KVM/arm64 updates for 5.9
Marc Zyngier
maintainerHi Paolo,
This is the bulk of the 5.9 patches for KVM/arm64. It is a pretty busy merge window for us this time, thanks to the ongoing Protected KVM work. We have changes all over the map, but the most important piece probably is the way we now build the EL2 code on non-VHE systems. On top of giving us better control over what gets pulled in there, it allowed us to enable instrumentation on VHE systems.
The rest is a mixed bag of new features (TTL TLB invalidation, Pointer Auth on non-VHE), preliminary patches for NV, some early MMU rework before the 5.10 onslaught, and tons of cleanups.
A few things to notice:
- We share a branch with the arm64 tree, which has gone in already.
- There are a number of known conflicts with Sean's MMU cache rework, as well as the late fixes that went in 5.8. The conflicts are pretty simple to resolve, and -next has the right resolutions already.
Please pull,
M.
The following changes since commit 9ebcfadb0610322ac537dd7aa5d9cbc2b2894c68:
Linux 5.8-rc3 (2020-06-28 15:00:24 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-5.9
for you to fetch changes up to 16314874b12b451bd5a1df86bcb69745eb487502:
Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next (2020-07-30 16:13:04 +0100)
----------------------------------------------------------------
KVM/arm64 updates for Linux 5.9:
- Split the VHE and nVHE hypervisor code bases, build the EL2 code separately, allowing for the VHE code to now be built with instrumentation
- Level-based TLB invalidation support
- Restructure of the vcpu register storage to accomodate the NV code
- Pointer Authentication available for guests on nVHE hosts
- Simplification of the system register table parsing
- MMU cleanups and fixes
- A number of post-32bit cleanups and other fixes
----------------------------------------------------------------
Alexander Graf (2):
KVM: arm64: vgic-its: Change default outer cacheability for {PEND, PROP}BASER
KVM: arm: Add trace name for ARM_NISV
Andrew Scull (3):
arm64: kvm: Remove kern_hyp_va from get_vcpu_ptr
KVM: arm64: Handle calls to prefixed hyp functions
KVM: arm64: Move hyp-init.S to nVHE
Christoffer Dall (1):
KVM: arm64: Factor out stage 2 page table data from struct kvm
David Brazdil (16):
KVM: arm64: Fix symbol dependency in __hyp_call_panic_nvhe
KVM: arm64: Move __smccc_workaround_1_smc to .rodata
KVM: arm64: Add build rules for separate VHE/nVHE object files
KVM: arm64: Use build-time defines in has_vhe()
KVM: arm64: Build hyp-entry.S separately for VHE/nVHE
KVM: arm64: Duplicate hyp/tlb.c for VHE/nVHE
KVM: arm64: Split hyp/switch.c to VHE/nVHE
KVM: arm64: Split hyp/debug-sr.c to VHE/nVHE
KVM: arm64: Split hyp/sysreg-sr.c to VHE/nVHE
KVM: arm64: Duplicate hyp/timer-sr.c for VHE/nVHE
KVM: arm64: Compile remaining hyp/ files for both VHE/nVHE
KVM: arm64: Remove __hyp_text macro, use build rules instead
KVM: arm64: Lift instrumentation restrictions on VHE
KVM: arm64: Make nVHE ASLR conditional on RANDOMIZE_BASE
KVM: arm64: Substitute RANDOMIZE_BASE for HARDEN_EL2_VECTORS
KVM: arm64: Ensure that all nVHE hyp code is in .hyp.text
Gavin Shan (1):
KVM: arm64: Rename HSR to ESR
James Morse (5):
KVM: arm64: Drop the target_table[] indirection
KVM: arm64: Tolerate an empty target_table list
KVM: arm64: Move ACTLR_EL1 emulation to the sys_reg_descs array
KVM: arm64: Remove target_table from exit handlers
KVM: arm64: Remove the target table
Marc Zyngier (31):
KVM: arm64: Enable Address Authentication at EL2 if available
KVM: arm64: Allow ARM64_PTR_AUTH when ARM64_VHE=n
KVM: arm64: Allow PtrAuth to be enabled from userspace on non-VHE systems
KVM: arm64: Check HCR_EL2 instead of shadow copy to swap PtrAuth registers
KVM: arm64: Simplify PtrAuth alternative patching
KVM: arm64: Allow in-atomic injection of SPIs
arm64: Detect the ARMv8.4 TTL feature
arm64: Document SW reserved PTE/PMD bits in Stage-2 descriptors
arm64: Add level-hinted TLB invalidation helper
Merge branch 'kvm-arm64/ttl-for-arm64' into HEAD
KVM: arm64: Use TTL hint in when invalidating stage-2 translations
KVM: arm64: Introduce accessor for ctxt->sys_reg
KVM: arm64: hyp: Use ctxt_sys_reg/__vcpu_sys_reg instead of raw sys_regs access
KVM: arm64: sve: Use __vcpu_sys_reg() instead of raw sys_regs access
KVM: arm64: pauth: Use ctxt_sys_reg() instead of raw sys_regs access
KVM: arm64: debug: Drop useless vpcu parameter
KVM: arm64: Make struct kvm_regs userspace-only
KVM: arm64: Move ELR_EL1 to the system register array
KVM: arm64: Move SP_EL1 to the system register array
KVM: arm64: Disintegrate SPSR array
KVM: arm64: Move SPSR_EL1 to the system register array
KVM: arm64: timers: Rename kvm_timer_sync_hwstate to kvm_timer_sync_user
KVM: arm64: timers: Move timer registers to the sys_regs file
KVM: arm64: Don't use has_vhe() for CHOOSE_HYP_SYM()
Merge branch 'kvm-arm64/el2-obj-v4.1' into kvmarm-master/next-WIP Merge branch 'kvm-arm64/pre-nv-5.9' into kvmarm-master/next-WIP Merge branch 'kvm-arm64/ptrauth-nvhe' into kvmarm-master/next-WIP Merge branch 'kvm-arm64/target-table-no-more' into kvmarm-master/next-WIP Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next-WIP Merge branch 'kvm-arm64/el2-obj-v4.1' into kvmarm-master/next Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next
Peng Hao (1):
KVM: arm64: Drop long gone function parameter documentation
Will Deacon (4):
KVM: arm64: Rename kvm_vcpu_dabt_isextabt()
KVM: arm64: Handle data and instruction external aborts the same way
KVM: arm64: Don't skip cache maintenance for read-only memslots
KVM: arm64: Move S1PTW S2 fault logic out of io_mem_abort()
arch/arm64/Kconfig | 20 +-
arch/arm64/include/asm/cpucaps.h | 3 +-
arch/arm64/include/asm/kvm_asm.h | 75 +-
arch/arm64/include/asm/kvm_coproc.h | 8 -
arch/arm64/include/asm/kvm_emulate.h | 75 +-
arch/arm64/include/asm/kvm_host.h | 94 ++-
arch/arm64/include/asm/kvm_hyp.h | 15 +-
arch/arm64/include/asm/kvm_mmu.h | 16 +-
arch/arm64/include/asm/kvm_ptrauth.h | 34 +-
arch/arm64/include/asm/mmu.h | 7 -
arch/arm64/include/asm/pgtable-hwdef.h | 2 +
arch/arm64/include/asm/stage2_pgtable.h | 9 +
arch/arm64/include/asm/sysreg.h | 1 +
arch/arm64/include/asm/tlbflush.h | 45 +
arch/arm64/include/asm/virt.h | 13 +-
arch/arm64/kernel/asm-offsets.c | 3 +-
arch/arm64/kernel/cpu_errata.c | 4 +-
arch/arm64/kernel/cpufeature.c | 11 +
arch/arm64/kernel/image-vars.h | 54 ++
arch/arm64/kvm/Kconfig | 2 +-
arch/arm64/kvm/Makefile | 4 +-
arch/arm64/kvm/arch_timer.c | 157 +++-
arch/arm64/kvm/arm.c | 57 +-
arch/arm64/kvm/fpsimd.c | 6 +-
arch/arm64/kvm/guest.c | 79 +-
arch/arm64/kvm/handle_exit.c | 32 +-
arch/arm64/kvm/hyp/Makefile | 24 +-
arch/arm64/kvm/hyp/aarch32.c | 8 +-
arch/arm64/kvm/hyp/entry.S | 4 +-
arch/arm64/kvm/hyp/fpsimd.S | 1 -
arch/arm64/kvm/hyp/hyp-entry.S | 21 +-
.../kvm/hyp/{debug-sr.c => include/hyp/debug-sr.h} | 88 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 511 +++++++++++
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 193 +++++
arch/arm64/kvm/hyp/nvhe/Makefile | 62 ++
arch/arm64/kvm/hyp/nvhe/debug-sr.c | 77 ++
arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S | 5 +
arch/arm64/kvm/hyp/nvhe/switch.c | 272 ++++++
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 46 +
arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c | 6 +-
arch/arm64/kvm/hyp/nvhe/tlb.c | 154 ++++
arch/arm64/kvm/hyp/smccc_wa.S | 32 +
arch/arm64/kvm/hyp/switch.c | 936 ---------------------
arch/arm64/kvm/hyp/sysreg-sr.c | 333 --------
arch/arm64/kvm/hyp/tlb.c | 242 ------
arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 +-
arch/arm64/kvm/hyp/vgic-v3-sr.c | 134 ++-
arch/arm64/kvm/hyp/vhe/Makefile | 11 +
arch/arm64/kvm/hyp/vhe/debug-sr.c | 26 +
arch/arm64/kvm/hyp/vhe/switch.c | 219 +++++
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 114 +++
arch/arm64/kvm/hyp/vhe/timer-sr.c | 12 +
arch/arm64/kvm/hyp/vhe/tlb.c | 162 ++++
arch/arm64/kvm/inject_fault.c | 2 +-
arch/arm64/kvm/mmio.c | 6 -
arch/arm64/kvm/mmu.c | 311 ++++---
arch/arm64/kvm/regmap.c | 37 +-
arch/arm64/kvm/reset.c | 23 +-
arch/arm64/kvm/sys_regs.c | 207 ++---
arch/arm64/kvm/sys_regs_generic_v8.c | 96 ---
arch/arm64/kvm/trace_arm.h | 8 +-
arch/arm64/kvm/va_layout.c | 2 +-
arch/arm64/kvm/vgic/vgic-irqfd.c | 24 +-
arch/arm64/kvm/vgic/vgic-its.c | 3 +-
arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +-
include/kvm/arm_arch_timer.h | 13 +-
include/trace/events/kvm.h | 2 +-
scripts/kallsyms.c | 1 +
68 files changed, 2883 insertions(+), 2377 deletions(-)
rename arch/arm64/kvm/hyp/{debug-sr.c => include/hyp/debug-sr.h} (66%)
create mode 100644 arch/arm64/kvm/hyp/include/hyp/switch.h
create mode 100644 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile
create mode 100644 arch/arm64/kvm/hyp/nvhe/debug-sr.c
rename arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S (95%)
create mode 100644 arch/arm64/kvm/hyp/nvhe/switch.c
create mode 100644 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c
rename arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c (84%)
create mode 100644 arch/arm64/kvm/hyp/nvhe/tlb.c
create mode 100644 arch/arm64/kvm/hyp/smccc_wa.S
delete mode 100644 arch/arm64/kvm/hyp/switch.c
delete mode 100644 arch/arm64/kvm/hyp/sysreg-sr.c
delete mode 100644 arch/arm64/kvm/hyp/tlb.c
create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
create mode 100644 arch/arm64/kvm/hyp/vhe/debug-sr.c
create mode 100644 arch/arm64/kvm/hyp/vhe/switch.c
create mode 100644 arch/arm64/kvm/hyp/vhe/sysreg-sr.c
create mode 100644 arch/arm64/kvm/hyp/vhe/timer-sr.c
create mode 100644 arch/arm64/kvm/hyp/vhe/tlb.c
delete mode 100644 arch/arm64/kvm/sys_regs_generic_v8.c_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 1/56 KVM: arm64: Enable Address Authentication at EL2 if available
While initializing EL2, enable Address Authentication if detected from EL1. We still use the EL1-provided keys though.
--- arch/arm64/kvm/hyp-init.S | 5 +++++ 1 file changed, 5 insertions(+)
arch/arm64/kvm/hyp-init.S +5 -0
@@ -104,6 +104,11 @@ alternative_else_nop_endif */ mov_q x4, (SCTLR_EL2_RES1 | (SCTLR_ELx_FLAGS & ~SCTLR_ELx_A)) CPU_BE( orr x4, x4, #SCTLR_ELx_EE) +alternative_if ARM64_HAS_ADDRESS_AUTH + mov_q x5, (SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | \ + SCTLR_ELx_ENDA | SCTLR_ELx_ENDB) + orr x4, x4, x5 +alternative_else_nop_endif msr sctlr_el2, x4 isb
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 2/56 KVM: arm64: Allow ARM64_PTR_AUTH when ARM64_VHE=n
In reply to Marc ZyngierWe currently prevent PtrAuth from even being built if KVM is selected, but VHE isn't. It is a bit of a pointless restriction, since we also check this at run time (rejecting the enabling of PtrAuth for the vcpu if we're not running with VHE).
Just drop this apparently useless restriction.
--- arch/arm64/Kconfig | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/arm64/Kconfig +1 -3
@@ -1516,7 +1516,6 @@ menu "ARMv8.3 architectural features" config ARM64_PTR_AUTH bool "Enable support for pointer authentication" default y - depends on !KVM || ARM64_VHE depends on (CC_HAS_SIGN_RETURN_ADDRESS || CC_HAS_BRANCH_PROT_PAC_RET) && AS_HAS_PAC # GCC 9.1 and later inserts a .note.gnu.property section note for PAC # which is only understood by binutils starting with version 2.33.1.
@@ -1543,8 +1542,7 @@ config ARM64_PTR_AUTH The feature is detected at runtime. If the feature is not present in hardware it will not be advertised to userspace/KVM guest nor will it - be enabled. However, KVM guest also require VHE mode and hence - CONFIG_ARM64_VHE=y option to use this feature. + be enabled. If the feature is present on the boot CPU but not on a late CPU, then the late CPU will be parked. Also, if the boot CPU does not have
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 3/56 KVM: arm64: Allow PtrAuth to be enabled from userspace on non-VHE systems
In reply to Marc ZyngierNow that the scene is set for enabling PtrAuth on non-VHE, drop the restrictions preventing userspace from enabling it.
--- arch/arm64/kvm/reset.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-)
arch/arm64/kvm/reset.c +10 -11
@@ -42,6 +42,11 @@ static u32 kvm_ipa_limit;#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ PSR_AA32_I_BIT | PSR_AA32_F_BIT) +static bool system_has_full_ptr_auth (void ) +{ +return system_supports_address_auth () &&system_supports_generic_auth (); +} +/** * kvm_arch_vm_ioctl_check_extension *
@@ -80,8 +85,7 @@ int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext)break ;case KVM_CAP_ARM_PTRAUTH_ADDRESS :case KVM_CAP_ARM_PTRAUTH_GENERIC : - r =has_vhe () &&system_supports_address_auth () && -system_supports_generic_auth (); + r =system_has_full_ptr_auth ();break ;default : r =0 ;
@@ -205,19 +209,14 @@ static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)static int kvm_vcpu_enable_ptrauth (struct kvm_vcpu *vcpu) { -/* Support ptrauth only if the system supports these capabilities. */ -if (!has_vhe ()) -return -EINVAL ; - -if (!system_supports_address_auth () || - !system_supports_generic_auth ()) -return -EINVAL ;/* * For now make sure that both address/generic pointer authentication -* features are requested by the userspace together. +* features are requested by the userspace together and the system +* supports these capabilities. */ if (!test_bit (KVM_ARM_VCPU_PTRAUTH_ADDRESS , vcpu->arch.features) || - !test_bit (KVM_ARM_VCPU_PTRAUTH_GENERIC , vcpu->arch.features)) + !test_bit (KVM_ARM_VCPU_PTRAUTH_GENERIC , vcpu->arch.features) || + !system_has_full_ptr_auth ())return -EINVAL ; vcpu->arch.flags |=KVM_ARM64_GUEST_HAS_PTRAUTH ;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 4/56 KVM: arm64: Check HCR_EL2 instead of shadow copy to swap PtrAuth registers
In reply to Marc ZyngierWhen save/restoring PtrAuth registers between host and guest, it is pretty useless to fetch the in-memory state, while we have the right state in the HCR_EL2 system register. Use that instead.
--- arch/arm64/include/asm/kvm_ptrauth.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_ptrauth.h +2 -2
@@ -74,7 +74,7 @@ alternative_if_not ARM64_HAS_ADDRESS_AUTH_IMP_DEF b 1001f alternative_else_nop_endif 1000: - ldr \reg1, [\g_ctxt, #(VCPU_HCR_EL2 - VCPU_CONTEXT)] + mrs \reg1, hcr_el2 and \reg1, \reg1, #(HCR_API | HCR_APK) cbz \reg1, 1001f add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1
@@ -90,7 +90,7 @@ alternative_if_not ARM64_HAS_ADDRESS_AUTH_IMP_DEF b 2001f alternative_else_nop_endif 2000: - ldr \reg1, [\g_ctxt, #(VCPU_HCR_EL2 - VCPU_CONTEXT)] + mrs \reg1, hcr_el2 and \reg1, \reg1, #(HCR_API | HCR_APK) cbz \reg1, 2001f add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 5/56 KVM: arm64: Simplify PtrAuth alternative patching
In reply to Marc ZyngierWe currently decide to execute the PtrAuth save/restore code based on a set of branches that evaluate as (ARM64_HAS_ADDRESS_AUTH_ARCH || ARM64_HAS_ADDRESS_AUTH_IMP_DEF). This can be easily replaced by a much simpler test as the ARM64_HAS_ADDRESS_AUTH capability is exactly this expression.
--- arch/arm64/include/asm/kvm_ptrauth.h | 30 ++++++++++------------------ 1 file changed, 11 insertions(+), 19 deletions(-)
arch/arm64/include/asm/kvm_ptrauth.h +11 -19
@@ -61,44 +61,36 @@/* * Both ptrauth_switch_to_guest and ptrauth_switch_to_host macros will -* check for the presence of one of the cpufeature flag -* ARM64_HAS_ADDRESS_AUTH_ARCH or ARM64_HAS_ADDRESS_AUTH_IMP_DEF and +* check for the presence ARM64_HAS_ADDRESS_AUTH, which is defined as +* (ARM64_HAS_ADDRESS_AUTH_ARCH || ARM64_HAS_ADDRESS_AUTH_IMP_DEF) and * then proceed ahead with the save/restore of Pointer Authentication -* key registers. +* key registers if enabled for the guest. */ .macro ptrauth_switch_to_guest g_ctxt, reg1, reg2, reg3 -alternative_if ARM64_HAS_ADDRESS_AUTH_ARCH - b 1000f +alternative_if_not ARM64_HAS_ADDRESS_AUTH + b .L__skip_switch\@ alternative_else_nop_endif -alternative_if_not ARM64_HAS_ADDRESS_AUTH_IMP_DEF - b 1001f -alternative_else_nop_endif -1000: mrs \reg1, hcr_el2 and \reg1, \reg1, #(HCR_API | HCR_APK) - cbz \reg1, 1001f + cbz \reg1, .L__skip_switch\@ add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1 ptrauth_restore_state \reg1, \reg2, \reg3 -1001: +.L__skip_switch\@: .endm .macro ptrauth_switch_to_host g_ctxt, h_ctxt, reg1, reg2, reg3 -alternative_if ARM64_HAS_ADDRESS_AUTH_ARCH - b 2000f -alternative_else_nop_endif -alternative_if_not ARM64_HAS_ADDRESS_AUTH_IMP_DEF - b 2001f +alternative_if_not ARM64_HAS_ADDRESS_AUTH + b .L__skip_switch\@ alternative_else_nop_endif -2000: mrs \reg1, hcr_el2 and \reg1, \reg1, #(HCR_API | HCR_APK) - cbz \reg1, 2001f + cbz \reg1, .L__skip_switch\@ add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1 ptrauth_save_state \reg1, \reg2, \reg3 add \reg1, \h_ctxt, #CPU_APIAKEYLO_EL1 ptrauth_restore_state \reg1, \reg2, \reg3 isb -2001: +.L__skip_switch\@: .endm #else /* !CONFIG_ARM64_PTR_AUTH */
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 6/56 KVM: arm64: Allow in-atomic injection of SPIs
In reply to Marc ZyngierOn a system that uses SPIs to implement MSIs (as it would be the case on a GICv2 system exposing a GICv2m to its guests), we deny the possibility of injecting SPIs on the in-atomic fast-path.
This results in a very large amount of context-switches (roughly equivalent to twice the interrupt rate) on the host, and suboptimal performance for the guest (as measured with a test workload involving a virtio interface backed by vhost-net). Given that GICv2 systems are usually on the low-end of the spectrum performance wise, they could do without the aggravation.
We solved this for GICv3+ITS by having a translation cache. But SPIs do not need any extra infrastructure, and can be immediately injected in the virtual distributor as the locking is already heavy enough that we don't need to worry about anything.
This halves the number of context switches for the same workload.
--- arch/arm64/kvm/vgic/vgic-irqfd.c | 24 +++++++++++++++++++----- arch/arm64/kvm/vgic/vgic-its.c | 3 +-- 2 files changed, 20 insertions(+), 7 deletions(-)
arch/arm64/kvm/vgic/vgic-irqfd.c +19 -5
@@ -100,19 +100,33 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,/** * kvm_arch_set_irq_inatomic: fast-path for irqfd injection -* -* Currently only direct MSI injection is supported. */ int kvm_arch_set_irq_inatomic (struct kvm_kernel_irq_routing_entry *e,struct kvm *kvm,int irq_source_id,int level,bool line_status) { -if (e->type ==KVM_IRQ_ROUTING_MSI &&vgic_has_its (kvm) && level) { +if (!level) +return -EWOULDBLOCK ; + +switch (e->type) { +case KVM_IRQ_ROUTING_MSI : {struct kvm_msi msi; +if (!vgic_has_its (kvm)) +break ; +kvm_populate_msi (e, &msi); -if (!vgic_its_inject_cached_translation (kvm, &msi)) -return 0 ; +return vgic_its_inject_cached_translation (kvm, &msi); + } + +case KVM_IRQ_ROUTING_IRQCHIP : +/* +* Injecting SPIs is always possible in atomic context +* as long as the damn vgic is initialized. +*/ +if (unlikely (!vgic_initialized (kvm))) +break ; +return vgic_irqfd_set_irq (e, kvm, irq_source_id,1 , line_status); }return -EWOULDBLOCK ;
arch/arm64/kvm/vgic/vgic-its.c +1 -2
@@ -757,9 +757,8 @@ int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi) db = (u64 )msi->address_hi <<32 | msi->address_lo; irq =vgic_its_check_cache (kvm, db, msi->devid, msi->data); -if (!irq) -return -1 ; +return -EWOULDBLOCK ;raw_spin_lock_irqsave (&irq->irq_lock, flags); irq->pending_latch = true;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 7/56 arm64: kvm: Remove kern_hyp_va from get_vcpu_ptr
In reply to Marc ZyngierFrom: Andrew Scull <[email protected]>
get_vcpu_ptr is an assembly accessor for the percpu value kvm_host_data->host_ctxt.__hyp_running_vcpu. kern_hyp_va only applies to nVHE however __hyp_running_vcpu is always assigned a pointer that has already had kern_hyp_va applied in __kvm_vcpu_run_nvhe.
kern_hyp_va is currently idempotent as it just masks and inserts the tag, but this could change in future and the second application is unnecessary.
--- arch/arm64/include/asm/kvm_asm.h | 1 - 1 file changed, 1 deletion(-)
arch/arm64/include/asm/kvm_asm.h +0 -1
@@ -143,7 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ]; .macro get_vcpu_ptr vcpu, ctxt get_host_ctxt \ctxt, \vcpu ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU] - kern_hyp_va \vcpu .endm #endif
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 8/56 KVM: arm64: Drop the target_table[] indirection
In reply to Marc ZyngierFrom: James Morse <[email protected]>
KVM for 32bit arm had a get/set target mechanism to allow for micro-architecture differences that are visible in system registers to be described.
KVM's user-space can query the supported targets for a CPU, and create vCPUs for that target. The target can override the handling of system registers to provide different reset or RES0 behaviour. On 32bit arm this was used to provide different ACTLR reset values for A7 and A15.
On 64bit arm, the first few CPUs out of the gate used this mechanism, before it was deemed redundant in commit bca556ac468a ("arm64/kvm:
Add generic v8 KVM target"). All future CPUs use the KVM_ARM_TARGET_GENERIC_V8 target.
The 64bit target_table[] stuff exists to preserve the ABI to user-space. As all targets registers genericv8_target_table, there is no reason to look the target up.
Until we can merge genericv8_target_table with the main sys_regs array, kvm_register_target_sys_reg_table() becomes kvm_check_target_sys_reg_table(), which uses BUG_ON() in keeping with the other callers in this file.
--- arch/arm64/include/asm/kvm_coproc.h | 3 +-- arch/arm64/kvm/sys_regs.c | 16 ++++------------ arch/arm64/kvm/sys_regs.h | 2 ++ arch/arm64/kvm/sys_regs_generic_v8.c | 15 ++------------- 4 files changed, 9 insertions(+), 27 deletions(-)
arch/arm64/include/asm/kvm_coproc.h +1 -2
@@ -24,8 +24,7 @@ struct kvm_sys_reg_target_table {struct kvm_sys_reg_table table32; }; -void kvm_register_target_sys_reg_table (unsigned int target, -struct kvm_sys_reg_target_table *table); +void kvm_check_target_sys_reg_table (struct kvm_sys_reg_target_table *table);int kvm_handle_cp14_load_store (struct kvm_vcpu *vcpu,struct kvm_run *run);int kvm_handle_cp14_32 (struct kvm_vcpu *vcpu,struct kvm_run *run);
arch/arm64/kvm/sys_regs.c +4 -12
@@ -2109,17 +2109,10 @@ static int check_sysreg_table(const struct sys_reg_desc *table, unsigned int n,return 0 ; } -/* Target specific emulation tables */ -static struct kvm_sys_reg_target_table *target_tables[KVM_ARM_NUM_TARGETS ]; - -void kvm_register_target_sys_reg_table (unsigned int target, -struct kvm_sys_reg_target_table *table) +void kvm_check_target_sys_reg_table (struct kvm_sys_reg_target_table *table) { -if (check_sysreg_table (table->table64.table, table->table64.num, false) || -check_sysreg_table (table->table32.table, table->table32.num, true)) -return ; - - target_tables[target] = table; +BUG_ON (check_sysreg_table (table->table64.table, table->table64.num, false)); +BUG_ON (check_sysreg_table (table->table32.table, table->table32.num, true)); }/* Get specific register table for this target. */
@@ -2127,9 +2120,8 @@ static const struct sys_reg_desc *get_target_table(unsigned target,bool mode_is_64, size_t *num) { - struct kvm_sys_reg_target_table *table; + struct kvm_sys_reg_target_table *table = &genericv8_target_table; - table = target_tables[target];if (mode_is_64) { *num = table->table64.num;return table->table64.table;
arch/arm64/kvm/sys_regs.h +2 -0
@@ -165,4 +165,6 @@ const struct sys_reg_desc *find_reg_by_id(u64 id, CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)), \ Op2(sys_reg_Op2(reg)) +extern struct kvm_sys_reg_target_table genericv8_target_table; +#endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
arch/arm64/kvm/sys_regs_generic_v8.c +2 -13
@@ -59,7 +59,7 @@ static const struct sys_reg_desc genericv8_cp15_regs[] = { access_actlr }, }; -static struct kvm_sys_reg_target_table genericv8_target_table = { +struct kvm_sys_reg_target_table genericv8_target_table = { .table64 = { .table = genericv8_sys_regs, .num =ARRAY_SIZE (genericv8_sys_regs),
@@ -78,18 +78,7 @@ static int __init sys_reg_genericv8_init(void)BUG_ON (cmp_sys_reg (&genericv8_sys_regs[i-1 ], &genericv8_sys_regs[i]) >=0 ); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_AEM_V8 , - &genericv8_target_table); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_FOUNDATION_V8 , - &genericv8_target_table); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_CORTEX_A53 , - &genericv8_target_table); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_CORTEX_A57 , - &genericv8_target_table); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_XGENE_POTENZA , - &genericv8_target_table); -kvm_register_target_sys_reg_table (KVM_ARM_TARGET_GENERIC_V8 , - &genericv8_target_table); +kvm_check_target_sys_reg_table (&genericv8_target_table);return 0 ; }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 9/56 KVM: arm64: Tolerate an empty target_table list
In reply to Marc ZyngierFrom: James Morse <[email protected]>
Before emptying the target_table lists, and then removing their infrastructure, add some tolerance to an empty list.
Instead of bugging-out on an empty list, pretend we already reached the end in the two-list-walk.
--- arch/arm64/kvm/sys_regs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/sys_regs.c +4 -1
@@ -2809,7 +2809,10 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind) i2 = sys_reg_descs; end2 = sys_reg_descs +ARRAY_SIZE (sys_reg_descs); -BUG_ON (i1 == end1 || i2 == end2); +if (i1 == end1) + i1 =NULL ; + +BUG_ON (i2 == end2);/* Walk carefully, as both tables may refer to the same register. */ while (i1 || i2) {
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 10/56 KVM: arm64: Move ACTLR_EL1 emulation to the sys_reg_descs array
In reply to Marc ZyngierFrom: James Morse <[email protected]>
The only entry in the genericv8_sys_regs arrays is for emulation of ACTLR_EL1. As all targets emulate this in the same way, move it to sys_reg_descs[].
--- arch/arm64/kvm/sys_regs.c | 28 ++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs_generic_v8.c | 30 ---------------------------- 2 files changed, 28 insertions(+), 30 deletions(-)
arch/arm64/kvm/sys_regs.c +28 -0
@@ -242,6 +242,25 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,return true; } +static bool access_actlr (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +return ignore_write (vcpu, p); + + p->regval =vcpu_read_sys_reg (vcpu,ACTLR_EL1 ); + +if (p->is_aarch32) { +if (r->Op2 &2 ) + p->regval =upper_32_bits (p->regval); +else + p->regval =lower_32_bits (p->regval); + } + +return true; +} +/* * Trap handler for the GICv3 SGI generation system register. * Forward the request to the VGIC emulation.
@@ -615,6 +634,12 @@ static void reset_amair_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)vcpu_write_sys_reg (vcpu, amair,AMAIR_EL1 ); } +static void reset_actlr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) +{ +u64 actlr =read_sysreg (actlr_el1); +vcpu_write_sys_reg (vcpu, actlr,ACTLR_EL1 ); +} +static void reset_mpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) {u64 mpidr;
@@ -1518,6 +1543,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {ID_UNALLOCATED (7 ,7 ), { SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, + { SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 }, { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, { SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0, .visibility = sve_visibility }, { SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 },
@@ -1957,6 +1983,8 @@ static const struct sys_reg_desc cp14_64_regs[] = {static const struct sys_reg_desc cp15_regs[] = { {Op1 (0 ),CRn (0 ),CRm (0 ),Op2 (1 ), access_ctr }, {Op1 (0 ),CRn (1 ),CRm (0 ),Op2 (0 ), access_vm_reg,NULL , c1_SCTLR }, + {Op1 (0 ),CRn (1 ),CRm (0 ),Op2 (1 ), access_actlr }, + {Op1 (0 ),CRn (1 ),CRm (0 ),Op2 (3 ), access_actlr }, {Op1 (0 ),CRn (2 ),CRm (0 ),Op2 (0 ), access_vm_reg,NULL , c2_TTBR0 }, {Op1 (0 ),CRn (2 ),CRm (0 ),Op2 (1 ), access_vm_reg,NULL , c2_TTBR1 }, {Op1 (0 ),CRn (2 ),CRm (0 ),Op2 (2 ), access_vm_reg,NULL , c2_TTBCR },
arch/arm64/kvm/sys_regs_generic_v8.c +0 -30
@@ -19,44 +19,14 @@#include "sys_regs.h" -static bool access_actlr (struct kvm_vcpu *vcpu, -struct sys_reg_params *p, -const struct sys_reg_desc *r) -{ -if (p->is_write) -return ignore_write (vcpu, p); - - p->regval =vcpu_read_sys_reg (vcpu,ACTLR_EL1 ); - -if (p->is_aarch32) { -if (r->Op2 &2 ) - p->regval =upper_32_bits (p->regval); -else - p->regval =lower_32_bits (p->regval); - } - -return true; -} - -static void reset_actlr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) -{ -__vcpu_sys_reg (vcpu,ACTLR_EL1 ) =read_sysreg (actlr_el1); -} -/* * Implementation specific sys-reg registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 */ static const struct sys_reg_desc genericv8_sys_regs[] = { - {SYS_DESC (SYS_ACTLR_EL1 ), access_actlr, reset_actlr,ACTLR_EL1 }, };static const struct sys_reg_desc genericv8_cp15_regs[] = { -/* ACTLR */ - {Op1 (0b000 ),CRn (0b0001 ),CRm (0b0000 ),Op2 (0b001 ), - access_actlr }, - {Op1 (0b000 ),CRn (0b0001 ),CRm (0b0000 ),Op2 (0b011 ), - access_actlr }, };struct kvm_sys_reg_target_table genericv8_target_table = {
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 11/56 KVM: arm64: Remove target_table from exit handlers
In reply to Marc ZyngierFrom: James Morse <[email protected]>
Whenever KVM searches for a register (e.g. due to a guest exit), it works with two tables, as the target table overrides the sys_regs array.
Now that everything is in the sys_regs array, and the target table is empty, stop doing that.
Remove the second table and its size from all the functions that take it.
--- arch/arm64/kvm/sys_regs.c | 87 +++++++-------------------------------- 1 file changed, 16 insertions(+), 71 deletions(-)
arch/arm64/kvm/sys_regs.c +16 -71
@@ -2269,9 +2269,7 @@ static void unhandled_cp_access(struct kvm_vcpu *vcpu,*/ static int kvm_handle_cp_64 (struct kvm_vcpu *vcpu,const struct sys_reg_desc *global, -size_t nr_global, -const struct sys_reg_desc *target_specific, -size_t nr_specific) +size_t nr_global) {struct sys_reg_params params;u32 hsr =kvm_vcpu_get_hsr (vcpu);
@@ -2298,14 +2296,11 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, }/* -* Try to emulate the coprocessor access using the target -* specific table first, and using the global table afterwards. -* If either of the tables contains a handler, handle the +* If the table contains a handler, handle the * potential register operation in the case of a read and return * with success. */ -if (!emulate_cp (vcpu, ¶ms, target_specific, nr_specific) || - !emulate_cp (vcpu, ¶ms, global, nr_global)) { +if (!emulate_cp (vcpu, ¶ms, global, nr_global)) {/* Split up the value between registers for the read side */ if (!params.is_write) {vcpu_set_reg (vcpu, Rt,lower_32_bits (params.regval));
@@ -2326,9 +2321,7 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu,*/ static int kvm_handle_cp_32 (struct kvm_vcpu *vcpu,const struct sys_reg_desc *global, -size_t nr_global, -const struct sys_reg_desc *target_specific, -size_t nr_specific) +size_t nr_global) {struct sys_reg_params params;u32 hsr =kvm_vcpu_get_hsr (vcpu);
@@ -2344,8 +2337,7 @@ static int kvm_handle_cp_32(struct kvm_vcpu *vcpu, params.Op1 = (hsr >>14 ) &0x7 ; params.Op2 = (hsr >>17 ) &0x7 ; -if (!emulate_cp (vcpu, ¶ms, target_specific, nr_specific) || - !emulate_cp (vcpu, ¶ms, global, nr_global)) { +if (!emulate_cp (vcpu, ¶ms, global, nr_global)) {if (!params.is_write)vcpu_set_reg (vcpu, Rt, params.regval);return 1 ;
@@ -2357,38 +2349,22 @@ static int kvm_handle_cp_32(struct kvm_vcpu *vcpu,int kvm_handle_cp15_64 (struct kvm_vcpu *vcpu,struct kvm_run *run) { -const struct sys_reg_desc *target_specific; -size_t num; - - target_specific =get_target_table (vcpu->arch.target, false, &num); -return kvm_handle_cp_64 (vcpu, - cp15_64_regs,ARRAY_SIZE (cp15_64_regs), - target_specific, num); +return kvm_handle_cp_64 (vcpu, cp15_64_regs,ARRAY_SIZE (cp15_64_regs)); }int kvm_handle_cp15_32 (struct kvm_vcpu *vcpu,struct kvm_run *run) { -const struct sys_reg_desc *target_specific; -size_t num; - - target_specific =get_target_table (vcpu->arch.target, false, &num); -return kvm_handle_cp_32 (vcpu, - cp15_regs,ARRAY_SIZE (cp15_regs), - target_specific, num); +return kvm_handle_cp_32 (vcpu, cp15_regs,ARRAY_SIZE (cp15_regs)); }int kvm_handle_cp14_64 (struct kvm_vcpu *vcpu,struct kvm_run *run) { -return kvm_handle_cp_64 (vcpu, - cp14_64_regs,ARRAY_SIZE (cp14_64_regs), -NULL ,0 ); +return kvm_handle_cp_64 (vcpu, cp14_64_regs,ARRAY_SIZE (cp14_64_regs)); }int kvm_handle_cp14_32 (struct kvm_vcpu *vcpu,struct kvm_run *run) { -return kvm_handle_cp_32 (vcpu, - cp14_regs,ARRAY_SIZE (cp14_regs), -NULL ,0 ); +return kvm_handle_cp_32 (vcpu, cp14_regs,ARRAY_SIZE (cp14_regs)); }static bool is_imp_def_sys_reg (struct sys_reg_params *params)
@@ -2400,15 +2376,9 @@ static bool is_imp_def_sys_reg(struct sys_reg_params *params)static int emulate_sys_reg (struct kvm_vcpu *vcpu,struct sys_reg_params *params) { -size_t num; -const struct sys_reg_desc *table, *r; - - table =get_target_table (vcpu->arch.target, true, &num); +const struct sys_reg_desc *r; -/* Search target-specific then generic table. */ - r =find_reg (params, table, num); -if (!r) - r =find_reg (params, sys_reg_descs,ARRAY_SIZE (sys_reg_descs)); + r =find_reg (params, sys_reg_descs,ARRAY_SIZE (sys_reg_descs));if (likely (r)) {perform_access (vcpu, params, r);
@@ -2512,8 +2482,7 @@ const struct sys_reg_desc *find_reg_by_id(u64 id,static const struct sys_reg_desc *index_to_sys_reg_desc (struct kvm_vcpu *vcpu,u64 id) { -size_t num; -const struct sys_reg_desc *table, *r; +const struct sys_reg_desc *r;struct sys_reg_params params;/* We only do sys_reg for now. */
@@ -2523,10 +2492,7 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,if (!index_to_params (id, ¶ms))return NULL ; - table =get_target_table (vcpu->arch.target, true, &num); - r =find_reg (¶ms, table, num); -if (!r) - r =find_reg (¶ms, sys_reg_descs,ARRAY_SIZE (sys_reg_descs)); + r =find_reg (¶ms, sys_reg_descs,ARRAY_SIZE (sys_reg_descs));/* Not saved in the sys_reg array and not otherwise accessible? */ if (r && !(r->reg || r->get_user))
@@ -2826,38 +2792,17 @@ static int walk_one_sys_reg(const struct kvm_vcpu *vcpu,/* Assumed ordered tables, see kvm_sys_reg_table_init. */ static int walk_sys_regs (struct kvm_vcpu *vcpu,u64 __user *uind) { -const struct sys_reg_desc *i1, *i2, *end1, *end2; +const struct sys_reg_desc *i2, *end2;unsigned int total =0 ; -size_t num;int err; -/* We check for duplicates here, to allow arch-specific overrides. */ - i1 =get_target_table (vcpu->arch.target, true, &num); - end1 = i1 + num; i2 = sys_reg_descs; end2 = sys_reg_descs +ARRAY_SIZE (sys_reg_descs); -if (i1 == end1) - i1 =NULL ; - -BUG_ON (i2 == end2); - -/* Walk carefully, as both tables may refer to the same register. */ -while (i1 || i2) { -int cmp =cmp_sys_reg (i1, i2); -/* target-specific overrides generic entry. */ -if (cmp <=0 ) - err =walk_one_sys_reg (vcpu, i1, &uind, &total); -else - err =walk_one_sys_reg (vcpu, i2, &uind, &total); - +while (i2 != end2) { + err =walk_one_sys_reg (vcpu, i2++, &uind, &total);if (err)return err; - -if (cmp <=0 && ++i1 == end1) - i1 =NULL ; -if (cmp >=0 && ++i2 == end2) - i2 =NULL ; }return total; }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 12/56 KVM: arm64: Remove the target table
In reply to Marc ZyngierFrom: James Morse <[email protected]>
Finally, remove the target table. Merge the code that checks the tables into kvm_reset_sys_regs() as there is now only one table.
--- arch/arm64/include/asm/kvm_coproc.h | 7 ---- arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/sys_regs.c | 57 ++++++---------------------- arch/arm64/kvm/sys_regs.h | 2 - arch/arm64/kvm/sys_regs_generic_v8.c | 55 --------------------------- 5 files changed, 12 insertions(+), 111 deletions(-) delete mode 100644 arch/arm64/kvm/sys_regs_generic_v8.c
arch/arm64/include/asm/kvm_coproc.h +0 -7
@@ -19,13 +19,6 @@ struct kvm_sys_reg_table {size_t num; }; -struct kvm_sys_reg_target_table { -struct kvm_sys_reg_table table64; -struct kvm_sys_reg_table table32; -}; - -void kvm_check_target_sys_reg_table (struct kvm_sys_reg_target_table *table); -int kvm_handle_cp14_load_store (struct kvm_vcpu *vcpu,struct kvm_run *run);int kvm_handle_cp14_32 (struct kvm_vcpu *vcpu,struct kvm_run *run);int kvm_handle_cp14_64 (struct kvm_vcpu *vcpu,struct kvm_run *run);
arch/arm64/kvm/Makefile +1 -1
@@ -14,7 +14,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ $(KVM)/vfio.o $(KVM)/irqchip.o \ arm.o mmu.o mmio.o psci.o perf.o hypercalls.o pvtime.o \ inject_fault.o regmap.o va_layout.o hyp.o hyp-init.o handle_exit.o \ - guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o \ + guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ aarch32.o arch_timer.o \ vgic/vgic.o vgic/vgic-init.o \
arch/arm64/kvm/sys_regs.c +11 -46
75 lines not on this page · raw message
arch/arm64/kvm/sys_regs.h +0 -2
@@ -165,6 +165,4 @@ const struct sys_reg_desc *find_reg_by_id(u64 id,CRn (sys_reg_CRn (reg)),CRm (sys_reg_CRm (reg)), \ Op2(sys_reg_Op2(reg)) -extern struct kvm_sys_reg_target_table genericv8_target_table; -#endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
arch/arm64/kvm/sys_regs_generic_v8.c deleted +0 -55
55 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 13/56 KVM: arm64: Fix symbol dependency in __hyp_call_panic_nvhe
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
__hyp_call_panic_nvhe contains inline assembly which did not declare its dependency on the __hyp_panic_string symbol.
The static-declared string has previously been kept alive because of a use in __hyp_call_panic_vhe. Fix this in preparation for separating the source files between VHE and nVHE when the two users land in two different compilation units. The static variable otherwise gets dropped when compiling the nVHE source file, causing an undefined symbol linker error later.
--- arch/arm64/kvm/hyp/switch.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/switch.c +1 -1
@@ -897,7 +897,7 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,* making sure it is a kernel address and not a PC-relative * reference. */ - asmvolatile ("ldr %0, =__hyp_panic_string" :"=r" (str_va)); + asmvolatile ("ldr %0, =%1" :"=r" (str_va) :"S" (__hyp_panic_string)); __hyp_do_panic(str_va, spsr, elr,
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 14/56 KVM: arm64: Move __smccc_workaround_1_smc to .rodata
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
This snippet of assembly is used by cpu_errata.c to overwrite parts of KVM hyp vector. Move it to its own source file and change its ELF section to .rodata.
--- arch/arm64/kvm/hyp/Makefile | 1 + arch/arm64/kvm/hyp/hyp-entry.S | 16 ---------------- arch/arm64/kvm/hyp/smccc_wa.S | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 33 insertions(+), 16 deletions(-) create mode 100644 arch/arm64/kvm/hyp/smccc_wa.S
arch/arm64/kvm/hyp/Makefile +1 -0
@@ -7,6 +7,7 @@ ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) obj-$(CONFIG_KVM) += hyp.o +obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ debug-sr.o entry.o switch.o fpsimd.o tlb.o hyp-entry.o
arch/arm64/kvm/hyp/hyp-entry.S +0 -16
@@ -318,20 +318,4 @@ SYM_CODE_START(__bp_harden_hyp_vecs) 1: .org __bp_harden_hyp_vecs + __BP_HARDEN_HYP_VECS_SZ .org 1b SYM_CODE_END(__bp_harden_hyp_vecs) - - .popsection - -SYM_CODE_START(__smccc_workaround_1_smc) - esb - sub sp, sp, #(8 * 4) - stp x2, x3, [sp, #(8 * 0)] - stp x0, x1, [sp, #(8 * 2)] - mov w0, #ARM_SMCCC_ARCH_WORKAROUND_1 - smc #0 - ldp x2, x3, [sp, #(8 * 0)] - ldp x0, x1, [sp, #(8 * 2)] - add sp, sp, #(8 * 4) -1: .org __smccc_workaround_1_smc + __SMCCC_WORKAROUND_1_SMC_SZ - .org 1b -SYM_CODE_END(__smccc_workaround_1_smc) #endif
arch/arm64/kvm/hyp/smccc_wa.S new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2015-2018 - ARM Ltd + * Author: Marc Zyngier <[email protected]> + */ + +#include <linux/arm-smccc.h> +#include <linux/linkage.h> + +#include <asm/kvm_asm.h> +#include <asm/kvm_mmu.h> + + /* + * This is not executed directly and is instead copied into the vectors + * by install_bp_hardening_cb(). + */ + .data + .pushsection .rodata + .global __smccc_workaround_1_smc +SYM_DATA_START(__smccc_workaround_1_smc) + esb + sub sp, sp, #(8 * 4) + stp x2, x3, [sp, #(8 * 0)] + stp x0, x1, [sp, #(8 * 2)] + mov w0, #ARM_SMCCC_ARCH_WORKAROUND_1 + smc #0 + ldp x2, x3, [sp, #(8 * 0)] + ldp x0, x1, [sp, #(8 * 2)] + add sp, sp, #(8 * 4) +1: .org __smccc_workaround_1_smc + __SMCCC_WORKAROUND_1_SMC_SZ + .org 1b +SYM_DATA_END(__smccc_workaround_1_smc)
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 15/56 KVM: arm64: Add build rules for separate VHE/nVHE object files
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for hyp-specific header files which will include code common to VHE/nVHE.
Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with -D__KVM_VHE_HYPERVISOR__.
Under nVHE compile each source file into a .hyp.tmp.o object first, then prefix all its symbols with "__kvm_nvhe_" using objcopy and produce a .hyp.o. Suffixes were chosen so that it would be possible for VHE and nVHE to share some source files, but compiled with different CFLAGS.
The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols will never appear in EL1 stack traces.
Due to symbol prefixing, add a section in image-vars.h for aliases of symbols that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.
--- arch/arm64/kernel/image-vars.h | 14 +++++++++++++ arch/arm64/kvm/hyp/Makefile | 10 +++++++--- arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ scripts/kallsyms.c | 1 + 5 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
arch/arm64/kernel/image-vars.h +14 -0
@@ -51,4 +51,18 @@ __efistub__ctype = _ctype;#endif +#ifdef CONFIG_KVM + +/* +* KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to +* separate it from the kernel proper. The following symbols are legally +* accessed by it, therefore provide aliases to make them linkable. +* Do not include symbols which may not be safely accessed under hypervisor +* memory mappings. +*/ + +#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; + +#endif /* CONFIG_KVM */ +#endif /* __ARM64_KERNEL_IMAGE_VARS_H */
arch/arm64/kvm/hyp/Makefile +7 -3
@@ -3,10 +3,14 @@ # Makefile for Kernel-based Virtual Machine module, HYP part # -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ - $(DISABLE_STACKLEAK_PLUGIN) +incdir := $(srctree)/$(src)/include +subdir-asflags-y := -I$(incdir) +subdir-ccflags-y := -I$(incdir) \ + -fno-stack-protector \ + -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += hyp.o +obj-$(CONFIG_KVM) += hyp.o nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \
arch/arm64/kvm/hyp/nvhe/Makefile new file +34 -0
@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_NVHE_HYPERVISOR__ +ccflags-y := -D__KVM_NVHE_HYPERVISOR__ + +obj-y := + +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y)) + +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE + $(call if_changed_rule,cc_o_c) +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE + $(call if_changed_rule,as_o_S) +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE + $(call if_changed,hypcopy) + +quiet_cmd_hypcopy = HYPCOPY $@ + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ + +# KVM nVHE code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +# Skip objtool checking for this directory because nVHE code is compiled with +# non-standard build rules. +OBJECT_FILES_NON_STANDARD := y
arch/arm64/kvm/hyp/vhe/Makefile new file +17 -0
@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_VHE_HYPERVISOR__ +ccflags-y := -D__KVM_VHE_HYPERVISOR__ + +obj-y := + +# KVM code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n
scripts/kallsyms.c +1 -0
@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ +"__kvm_nvhe_" ,/* arm64 non-VHE KVM namespace */ NULL };
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 15/56] KVM: arm64: Add build rules for separate VHE/nVHE object files
Hi David, Marc,
Quoted 146 lines · Marc Zyngier
From: David Brazdil <[email protected]>
Add new folders
arch/arm64/kvm/hyp/{vhe,nvhe}and Makefiles for building code that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for hyp-specific header files which will include code common to VHE/nVHE.Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with -D__KVM_VHE_HYPERVISOR__.
Under nVHE compile each source file into a
.hyp.tmp.oobject first, then prefix all its symbols with "__kvm_nvhe_" usingobjcopyand produce a.hyp.o. Suffixes were chosen so that it would be possible for VHE and nVHE to share some source files, but compiled with different CFLAGS.The nVHE ELF symbol prefix is added to
kallsyms.cas ignored. EL2-only symbols will never appear in EL1 stack traces.Due to symbol prefixing, add a section in
image-vars.hfor aliases of symbols that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.Signed-off-by David Brazdil <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/image-vars.h | 14 +++++++++++++ arch/arm64/kvm/hyp/Makefile | 10 +++++++--- arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ scripts/kallsyms.c | 1 + 5 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile create mode 100644 arch/arm64/kvm/hyp/vhe/Makefilearch/arm64/kernel/image-vars.h +14
-0@@ -51,4 +51,18 @@ __efistub__ctype = _ctype;#endif +#ifdef CONFIG_KVM + +/* +* KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to +* separate it from the kernel proper. The following symbols are legally +* accessed by it, therefore provide aliases to make them linkable. +* Do not include symbols which may not be safely accessed under hypervisor +* memory mappings. +*/ + +#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; + +#endif /* CONFIG_KVM */ +#endif /* __ARM64_KERNEL_IMAGE_VARS_H */ arch/arm64/kvm/hyp/Makefile +7
-3@@ -3,10 +3,14 @@ # Makefile for Kernel-based Virtual Machine module, HYP part # -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ - $(DISABLE_STACKLEAK_PLUGIN) +incdir := $(srctree)/$(src)/include +subdir-asflags-y := -I$(incdir) +subdir-ccflags-y := -I$(incdir) \ + -fno-stack-protector \ + -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += hyp.o +obj-$(CONFIG_KVM) += hyp.o nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \arch/arm64/kvm/hyp/nvhe/Makefile new file +34
-0@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_NVHE_HYPERVISOR__ +ccflags-y := -D__KVM_NVHE_HYPERVISOR__ + +obj-y := + +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y)) + +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE + $(call if_changed_rule,cc_o_c) +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE + $(call if_changed_rule,as_o_S) +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE + $(call if_changed,hypcopy) + +quiet_cmd_hypcopy = HYPCOPY $@ + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ + +# KVM nVHE code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +# Skip objtool checking for this directory because nVHE code is compiled with +# non-standard build rules. +OBJECT_FILES_NON_STANDARD := yarch/arm64/kvm/hyp/vhe/Makefile new file +17
-0@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_VHE_HYPERVISOR__ +ccflags-y := -D__KVM_VHE_HYPERVISOR__ + +obj-y := + +# KVM code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := nscripts/kallsyms.c +0
-0
scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 112
Quoted 4 lines
scripts/kallsyms.c +1
-0@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */
The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all __kvm_nvhe_ prefixed symbols, like
ERR : <addr> : __kvm_nvhe___invalid not on kallsyms ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms ERR : <addr> : __kvm_nvhe_reset not on kallsyms
../..
I understand we willingly hided those symbols from /proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly?
Thanks
Eric
scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 115
Quoted 4 lines
scripts/kallsyms.c +0
-0NULL };
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH 15/56] KVM: arm64: Add build rules for separate VHE/nVHE object files
Hi Eric,
Quoted 36 lines · Auger Eric
Hi David, Marc,
Quoted 32 lines · Marc Zyngier
From: David Brazdil <[email protected]>
Add new folders
arch/arm64/kvm/hyp/{vhe,nvhe}and Makefiles for building code that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for hyp-specific header files which will include code common to VHE/nVHE.Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with -D__KVM_VHE_HYPERVISOR__.
Under nVHE compile each source file into a
.hyp.tmp.oobject first, then prefix all its symbols with "__kvm_nvhe_" usingobjcopyand produce a.hyp.o. Suffixes were chosen so that it would be possible for VHE and nVHE to share some source files, but compiled with different CFLAGS.The nVHE ELF symbol prefix is added to
kallsyms.cas ignored. EL2-only symbols will never appear in EL1 stack traces.Due to symbol prefixing, add a section in
image-vars.hfor aliases of symbols that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.Signed-off-by David Brazdil <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/image-vars.h | 14 +++++++++++++ arch/arm64/kvm/hyp/Makefile | 10 +++++++--- arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ scripts/kallsyms.c | 1 + 5 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
[...]
Quoted 13 lines
scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 112
Quoted 9 lines
scripts/kallsyms.c +1
-0@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all
__kvm_nvhe_prefixed symbols, like
Quoted 8 lines
ERR : <addr> :
__kvm_nvhe___invalidnot on kallsyms ERR : <addr> :__kvm_nvhe___do_hyp_initnot on kallsyms ERR : <addr> :__kvm_nvhe___kvm_handle_stub_hvcnot on kallsyms ERR : <addr> :__kvm_nvhe_resetnot on kallsyms
../..I understand we willingly hided those symbols from
/proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly?
Hmmm. This test always fail here, no matter whether I have this line or not:
<quote>
maz@big-leg-emma:~$ sudo perf_5.10 test -v 1
1: vmlinux symtab matches kallsyms :
--- start ---
test child forked, pid 664
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
/proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
Looking at the vmlinux_path (8 entries long)
symsrc__init: cannot get elf header.
symsrc__init: cannot get elf header.
Couldn't find a vmlinux that matches the kernel running on this machine, skipping test
test child finished with -2
---- end ----
vmlinux symtab matches kallsyms: Skip
</quote>Rookie question: How do you provide a kernel to the test framework?
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 15/56] KVM: arm64: Add build rules for separate VHE/nVHE object files
Hi Marc,
Quoted 62 lines · Marc Zyngier
Hi Eric,
Quoted 36 lines · Auger Eric
Hi David, Marc,
Quoted 32 lines · Marc Zyngier
From: David Brazdil <[email protected]>
Add new folders
arch/arm64/kvm/hyp/{vhe,nvhe}and Makefiles for building code that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for hyp-specific header files which will include code common to VHE/nVHE.Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with -D__KVM_VHE_HYPERVISOR__.
Under nVHE compile each source file into a
.hyp.tmp.oobject first, then prefix all its symbols with "__kvm_nvhe_" usingobjcopyand produce a.hyp.o. Suffixes were chosen so that it would be possible for VHE and nVHE to share some source files, but compiled with different CFLAGS.The nVHE ELF symbol prefix is added to
kallsyms.cas ignored. EL2-only symbols will never appear in EL1 stack traces.Due to symbol prefixing, add a section in
image-vars.hfor aliases of symbols that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.Signed-off-by David Brazdil <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/image-vars.h | 14 +++++++++++++ arch/arm64/kvm/hyp/Makefile | 10 +++++++--- arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ scripts/kallsyms.c | 1 + 5 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile[...]
Quoted 19 lines
scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 112
Quoted 9 lines
scripts/kallsyms.c +1
-0@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all
__kvm_nvhe_prefixed symbols, likeERR : <addr> :
__kvm_nvhe___invalidnot on kallsyms ERR : <addr> :__kvm_nvhe___do_hyp_initnot on kallsyms ERR : <addr> :__kvm_nvhe___kvm_handle_stub_hvcnot on kallsyms ERR : <addr> :__kvm_nvhe_resetnot on kallsyms
../..
Quoted 5 lines
Quoted 2 lines
I understand we willingly hided those symbols from
/proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly?Hmmm. This test always fail here, no matter whether I have this line or not:
sorry Marc I missed your reply,
Quoted 25 lines
<quote>
maz@big-leg-emma:~$ sudo perf_5.10 test -v 1 1: vmlinux symtab matches kallsyms : --- start --- test child forked, pid 664 /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. symsrc__init: cannot get elf header. Couldn't find a vmlinux that matches the kernel running on this machine, skipping test test child finished with -2 ---- end ---- vmlinux symtab matches kallsyms: Skip </quote>
Quoted 1 line
Rookie question: How do you provide a kernel to the test framework?
On my end I compiled and installed a kernel and compiled perf from that same kernel. Should work. I don't know how to point to a specific vmlinux
Thanks
Eric
Quoted 5 lines
Thanks,
M.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 15/56] KVM: arm64: Add build rules for separate VHE/nVHE object files
In reply to Auger EricHi David, Marc,
Quoted 157 lines · Auger Eric
Hi David, Marc,
Quoted 146 lines · Marc Zyngier
From: David Brazdil <[email protected]>
Add new folders
arch/arm64/kvm/hyp/{vhe,nvhe}and Makefiles for building code that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for hyp-specific header files which will include code common to VHE/nVHE.Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with -D__KVM_VHE_HYPERVISOR__.
Under nVHE compile each source file into a
.hyp.tmp.oobject first, then prefix all its symbols with "__kvm_nvhe_" usingobjcopyand produce a.hyp.o. Suffixes were chosen so that it would be possible for VHE and nVHE to share some source files, but compiled with different CFLAGS.The nVHE ELF symbol prefix is added to
kallsyms.cas ignored. EL2-only symbols will never appear in EL1 stack traces.Due to symbol prefixing, add a section in
image-vars.hfor aliases of symbols that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.Signed-off-by David Brazdil <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/kernel/image-vars.h | 14 +++++++++++++ arch/arm64/kvm/hyp/Makefile | 10 +++++++--- arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ scripts/kallsyms.c | 1 + 5 files changed, 73 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile create mode 100644 arch/arm64/kvm/hyp/vhe/Makefilearch/arm64/kernel/image-vars.h +14
-0@@ -51,4 +51,18 @@ __efistub__ctype = _ctype;#endif +#ifdef CONFIG_KVM + +/* +* KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to +* separate it from the kernel proper. The following symbols are legally +* accessed by it, therefore provide aliases to make them linkable. +* Do not include symbols which may not be safely accessed under hypervisor +* memory mappings. +*/ + +#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; + +#endif /* CONFIG_KVM */ +#endif /* __ARM64_KERNEL_IMAGE_VARS_H */ arch/arm64/kvm/hyp/Makefile +7
-3@@ -3,10 +3,14 @@ # Makefile for Kernel-based Virtual Machine module, HYP part # -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ - $(DISABLE_STACKLEAK_PLUGIN) +incdir := $(srctree)/$(src)/include +subdir-asflags-y := -I$(incdir) +subdir-ccflags-y := -I$(incdir) \ + -fno-stack-protector \ + -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += hyp.o +obj-$(CONFIG_KVM) += hyp.o nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \arch/arm64/kvm/hyp/nvhe/Makefile new file +34
-0@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_NVHE_HYPERVISOR__ +ccflags-y := -D__KVM_NVHE_HYPERVISOR__ + +obj-y := + +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y)) + +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE + $(call if_changed_rule,cc_o_c) +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE + $(call if_changed_rule,as_o_S) +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE + $(call if_changed,hypcopy) + +quiet_cmd_hypcopy = HYPCOPY $@ + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ + +# KVM nVHE code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +# Skip objtool checking for this directory because nVHE code is compiled with +# non-standard build rules. +OBJECT_FILES_NON_STANDARD := yarch/arm64/kvm/hyp/vhe/Makefile new file +17
-0@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part +# + +asflags-y := -D__KVM_VHE_HYPERVISOR__ +ccflags-y := -D__KVM_VHE_HYPERVISOR__ + +obj-y := + +# KVM code is run at a different exception code with a different map, so +# compiler instrumentation that inserts callbacks or checks into the code may +# cause crashes. Just disable it. +GCOV_PROFILE := n +KASAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := nscripts/kallsyms.c +0
-0scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 112
Quoted 4 lines
scripts/kallsyms.c +1
-0@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all
__kvm_nvhe_prefixed symbols, like
Quoted 8 lines
ERR : <addr> :
__kvm_nvhe___invalidnot on kallsyms ERR : <addr> :__kvm_nvhe___do_hyp_initnot on kallsyms ERR : <addr> :__kvm_nvhe___kvm_handle_stub_hvcnot on kallsyms ERR : <addr> :__kvm_nvhe_resetnot on kallsyms
../..I understand we willingly hided those symbols from
/proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly?
Were you eventually able to reproduce?
Thanks
Eric
Quoted 10 lines
Thanks
Eric
Quoted 4 lines
scripts/kallsyms.c +0
-0NULL };
Marc Zyngier
maintainerRe: [PATCH 15/56] KVM: arm64: Add build rules for separate VHE/nVHE object files
Hi Eric,
Quoted 3 lines · Auger Eric
Hi David, Marc,
[...]
Quoted 17 lines
Quoted 17 lines
scripts/kallsyms.c static bool is_ignored_symbol(const char *name, char type) · line 112
Quoted 7 lines
scripts/kallsyms.c +1
-0@@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)".LASANPC" ,/* s390 kasan local symbols */ "__crc_" ,/* modversions */ "__efistub_" ,/* arm64 EFI stub namespace */ + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all
__kvm_nvhe_prefixed symbols, likeERR : <addr> :
__kvm_nvhe___invalidnot on kallsyms ERR : <addr> :__kvm_nvhe___do_hyp_initnot on kallsyms ERR : <addr> :__kvm_nvhe___kvm_handle_stub_hvcnot on kallsyms ERR : <addr> :__kvm_nvhe_resetnot on kallsyms
../..
Quoted 4 lines
Quoted 2 lines
I understand we willingly hided those symbols from
/proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly?Were you eventually able to reproduce?
Unfortunately not, I always end-up with the test failing even if I add the symbols back. I must be doing something wrong...
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Marc Zyngier
maintainerPATCH 16/56 KVM: arm64: Use build-time defines in has_vhe()
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
Build system compiles hyp code with macros specifying if the code belongs to VHE or nVHE. Use these macros to evaluate has_vhe() at compile time.
--- arch/arm64/include/asm/virt.h | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
arch/arm64/include/asm/virt.h +10 -3
@@ -85,10 +85,17 @@ static inline bool is_kernel_in_hyp_mode(void) static __always_inline bool has_vhe(void) { -if (cpus_have_final_cap (ARM64_HAS_VIRT_HOST_EXTN )) +/* +* The following macros are defined for code specic to VHE/nVHE. +* If has_vhe() is inlined into those compilation units, it can +* be determined statically. Otherwise fall back to caps. +*/ +if (__is_defined (__KVM_VHE_HYPERVISOR__))return true; - -return false; +else if (__is_defined (__KVM_NVHE_HYPERVISOR__)) +return false; +else +return cpus_have_final_cap (ARM64_HAS_VIRT_HOST_EXTN ); }#endif /* __ASSEMBLY__ */
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 17/56 KVM: arm64: Handle calls to prefixed hyp functions
In reply to Marc ZyngierFrom: Andrew Scull <[email protected]>
Once hyp functions are moved to a hyp object, they will have prefixed symbols. This change declares and gets the address of the prefixed version for calls to the hyp functions.
To aid migration, the hyp functions that have not yet moved have their prefixed versions aliased to their non-prefixed version. This begins with all the hyp functions being listed and will reduce to none of them once the migration is complete.
[David: Extracted kvm_call_hyp nVHE branches into own helper macros, added
comments around symbol aliases.]
--- arch/arm64/include/asm/kvm_asm.h | 19 +++++++++++++++++++ arch/arm64/include/asm/kvm_host.h | 17 ++++++++++++++--- arch/arm64/kernel/image-vars.h | 26 ++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_asm.h +19 -0
@@ -42,6 +42,24 @@#include <linux/mm.h> +/* +* Translate name of a symbol defined in nVHE hyp to the name seen +* by kernel proper. All nVHE symbols are prefixed by the build system +* to avoid clashes with the VHE variants. +*/ +#define kvm_nvhe_sym (sym) __kvm_nvhe_##sym + +#define DECLARE_KVM_VHE_SYM (sym) extern char sym[] +#define DECLARE_KVM_NVHE_SYM (sym) extern char kvm_nvhe_sym(sym)[] + +/* +* Define a pair of symbols sharing the same name but one defined in +* VHE and the other in nVHE hyp implementations. +*/ +#define DECLARE_KVM_HYP_SYM (sym) \ + DECLARE_KVM_VHE_SYM(sym); \ + DECLARE_KVM_NVHE_SYM(sym) +/* Translate a kernel address of @sym into its equivalent linear mapping */ #define kvm_ksym_ref (sym) \ ({ \
@@ -50,6 +68,7 @@ val =lm_alias (&sym); \ val; \ }) +#define kvm_ksym_ref_nvhe (sym) kvm_ksym_ref(kvm_nvhe_sym(sym))struct kvm ;struct kvm_vcpu ;
arch/arm64/include/asm/kvm_host.h +14 -3
@@ -448,6 +448,18 @@ void kvm_arm_resume_guest(struct kvm *kvm);u64 __kvm_call_hyp (void *hypfn, ...); +#define kvm_call_hyp_nvhe (f, ...) \ + do { \ + DECLARE_KVM_NVHE_SYM(f); \ + __kvm_call_hyp(kvm_ksym_ref_nvhe(f), ##__VA_ARGS__); \ + } while(0) + +#define kvm_call_hyp_nvhe_ret (f, ...) \ + ({ \ + DECLARE_KVM_NVHE_SYM(f); \ + __kvm_call_hyp(kvm_ksym_ref_nvhe(f), ##__VA_ARGS__); \ + }) +/* * The couple of isb() below are there to guarantee the same behaviour * on VHE as on !VHE, where the eret to EL1 acts as a context
@@ -459,7 +471,7 @@ u64 __kvm_call_hyp(void *hypfn, ...);f (__VA_ARGS__); \isb (); \ }else { \ - __kvm_call_hyp(kvm_ksym_ref(f), ##__VA_ARGS__); \ + kvm_call_hyp_nvhe(f, ##__VA_ARGS__); \ } \ }while (0 )
@@ -471,8 +483,7 @@ u64 __kvm_call_hyp(void *hypfn, ...); ret =f (__VA_ARGS__); \isb (); \ }else { \ - ret =__kvm_call_hyp (kvm_ksym_ref (f), \ - ##__VA_ARGS__); \ + ret = kvm_call_hyp_nvhe_ret(f, ##__VA_ARGS__); \ } \ \ ret; \
arch/arm64/kernel/image-vars.h +26 -0
@@ -63,6 +63,32 @@ __efistub__ctype = _ctype;#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; +/* Symbols defined in debug-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_get_mdcr_el2); + +/* Symbols defined in switch.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_vcpu_run_nvhe); + +/* Symbols defined in sysreg-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_enable_ssbs); + +/* Symbols defined in timer-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_timer_set_cntvoff); + +/* Symbols defined in tlb.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_flush_vm_context); +KVM_NVHE_ALIAS (__kvm_tlb_flush_local_vmid); +KVM_NVHE_ALIAS (__kvm_tlb_flush_vmid); +KVM_NVHE_ALIAS (__kvm_tlb_flush_vmid_ipa); + +/* Symbols defined in vgic-v3-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__vgic_v3_get_ich_vtr_el2); +KVM_NVHE_ALIAS (__vgic_v3_init_lrs); +KVM_NVHE_ALIAS (__vgic_v3_read_vmcr); +KVM_NVHE_ALIAS (__vgic_v3_restore_aprs); +KVM_NVHE_ALIAS (__vgic_v3_save_aprs); +KVM_NVHE_ALIAS (__vgic_v3_write_vmcr); +#endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 18/56 KVM: arm64: Build hyp-entry.S separately for VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
hyp-entry.S contains implementation of KVM hyp vectors. This code is mostly shared between VHE/nVHE, therefore compile it under both VHE and nVHE build rules. nVHE-specific host HVC handler is hidden behind __KVM_NVHE_HYPERVISOR__.
Adjust code which selects which KVM hyp vecs to install to choose the correct VHE/nVHE symbol.
--- arch/arm64/include/asm/kvm_asm.h | 22 +++++++++++++++++----- arch/arm64/include/asm/mmu.h | 7 ------- arch/arm64/kernel/image-vars.h | 24 ++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/hyp/Makefile | 4 ++-- arch/arm64/kvm/hyp/hyp-entry.S | 2 ++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/vhe/Makefile | 2 +- 8 files changed, 48 insertions(+), 17 deletions(-)
arch/arm64/include/asm/kvm_asm.h +17 -5
@@ -60,12 +60,17 @@DECLARE_KVM_VHE_SYM (sym); \ DECLARE_KVM_NVHE_SYM(sym) -/* Translate a kernel address of @sym into its equivalent linear mapping */ -#define kvm_ksym_ref (sym) \ +#define CHOOSE_VHE_SYM (sym) sym +#define CHOOSE_NVHE_SYM (sym) kvm_nvhe_sym(sym) +#define CHOOSE_HYP_SYM (sym) (has_vhe() ? CHOOSE_VHE_SYM(sym) \ + : CHOOSE_NVHE_SYM(sym)) + +/* Translate a kernel address @ptr into its equivalent linear mapping */ +#define kvm_ksym_ref (ptr) \ ({ \ - void *val = &sym; \ + void *val = (ptr); \ if (!is_kernel_in_hyp_mode()) \ - val = lm_alias(&sym); \ + val = lm_alias((ptr)); \ val; \ })#define kvm_ksym_ref_nvhe (sym) kvm_ksym_ref(kvm_nvhe_sym(sym))
@@ -76,7 +81,14 @@ struct kvm_vcpu;extern char __kvm_hyp_init[];extern char __kvm_hyp_init_end[]; -extern char __kvm_hyp_vector[]; +DECLARE_KVM_HYP_SYM (__kvm_hyp_vector); +#define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector) + +#ifdef CONFIG_KVM_INDIRECT_VECTORS +extern atomic_t arm64_el2_vector_last_slot; +DECLARE_KVM_HYP_SYM (__bp_harden_hyp_vecs); +#define __bp_harden_hyp_vecs CHOOSE_HYP_SYM(__bp_harden_hyp_vecs) +#endif extern void __kvm_flush_vm_context (void );extern void __kvm_tlb_flush_vmid_ipa (struct kvm *kvm,phys_addr_t ipa);
arch/arm64/include/asm/mmu.h +0 -7
@@ -45,13 +45,6 @@ struct bp_hardening_data {bp_hardening_cb_t fn; }; -#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR ) || \ - defined(CONFIG_HARDEN_EL2_VECTORS )) - -extern char __bp_harden_hyp_vecs[]; -extern atomic_t arm64_el2_vector_last_slot; -#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */ -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
arch/arm64/kernel/image-vars.h +24 -0
@@ -66,8 +66,17 @@ __efistub__ctype = _ctype;/* Symbols defined in debug-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_get_mdcr_el2); +/* Symbols defined in entry.S (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__guest_exit); +KVM_NVHE_ALIAS (abort_guest_exit_end); +KVM_NVHE_ALIAS (abort_guest_exit_start); + +/* Symbols defined in hyp-init.S (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__kvm_handle_stub_hvc); +/* Symbols defined in switch.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_vcpu_run_nvhe); +KVM_NVHE_ALIAS (hyp_panic);/* Symbols defined in sysreg-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_enable_ssbs);
@@ -89,6 +98,21 @@ KVM_NVHE_ALIAS(__vgic_v3_restore_aprs);KVM_NVHE_ALIAS (__vgic_v3_save_aprs);KVM_NVHE_ALIAS (__vgic_v3_write_vmcr); +/* Alternative callbacks for init-time patching of nVHE hyp code. */ +KVM_NVHE_ALIAS (arm64_enable_wa2_handling); +KVM_NVHE_ALIAS (kvm_patch_vector_branch); +KVM_NVHE_ALIAS (kvm_update_va_mask); + +/* Global kernel state accessed by nVHE hyp code. */ +KVM_NVHE_ALIAS (arm64_ssbd_callback_required); +KVM_NVHE_ALIAS (kvm_host_data); + +/* Kernel constant needed to compute idmap addresses. */ +KVM_NVHE_ALIAS (kimage_voffset); + +/* Kernel symbols used to call panic() from nVHE hyp code (via ERET). */ +KVM_NVHE_ALIAS (panic); +#endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
arch/arm64/kvm/arm.c +1 -1
@@ -1285,7 +1285,7 @@ static void cpu_init_hyp_mode(void)* so that we can use adr_l to access per-cpu variables in EL2. */ tpidr_el2 = ((unsigned long )this_cpu_ptr (&kvm_host_data) - - (unsigned long )kvm_ksym_ref (kvm_host_data)); + (unsigned long )kvm_ksym_ref (&kvm_host_data)); pgd_ptr =kvm_mmu_get_httbr (); hyp_stack_ptr =__this_cpu_read (kvm_arm_hyp_stack_page) +PAGE_SIZE ;
arch/arm64/kvm/hyp/Makefile +2 -2
@@ -10,11 +10,11 @@ subdir-ccflags-y := -I$(incdir) \ -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += hyp.o nvhe/ +obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ - debug-sr.o entry.o switch.o fpsimd.o tlb.o hyp-entry.o + debug-sr.o entry.o switch.o fpsimd.o tlb.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/hyp-entry.S +2 -0
@@ -40,6 +40,7 @@ el1_sync: // Guest trapped into EL2 ccmp x0, #ESR_ELx_EC_HVC32, #4, ne b.ne el1_trap +#ifdef __KVM_NVHE_HYPERVISOR__ mrs x1, vttbr_el2 // If vttbr is valid, the guest cbnz x1, el1_hvc_guest // called HVC
@@ -74,6 +75,7 @@ el1_sync: // Guest trapped into EL2 eret sb +#endif /* __KVM_NVHE_HYPERVISOR__ */ el1_hvc_guest: /*
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := +obj-y := ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := +obj-y := ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 19/56 KVM: arm64: Move hyp-init.S to nVHE
In reply to Marc ZyngierFrom: Andrew Scull <[email protected]>
hyp-init.S contains the identity mapped initialisation code for the non-VHE code that runs at EL2. It is only used for non-VHE.
Adjust code that calls into this to use the prefixed symbol name.
---
arch/arm64/include/asm/kvm_asm.h | 5 ++---
arch/arm64/kernel/image-vars.h | 9 ++++++---
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/hyp/nvhe/Makefile | 2 +-
arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S | 0
5 files changed, 10 insertions(+), 8 deletions(-)
rename arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S (100%)arch/arm64/include/asm/kvm_asm.h +2 -3
@@ -78,10 +78,9 @@struct kvm ;struct kvm_vcpu ; -extern char __kvm_hyp_init[]; -extern char __kvm_hyp_init_end[]; - +DECLARE_KVM_NVHE_SYM (__kvm_hyp_init);DECLARE_KVM_HYP_SYM (__kvm_hyp_vector); +#define __kvm_hyp_init CHOOSE_NVHE_SYM(__kvm_hyp_init)#define __kvm_hyp_vector CHOOSE_HYP_SYM(__kvm_hyp_vector)#ifdef CONFIG_KVM_INDIRECT_VECTORS
arch/arm64/kernel/image-vars.h +6 -3
@@ -71,9 +71,6 @@ KVM_NVHE_ALIAS(__guest_exit);KVM_NVHE_ALIAS (abort_guest_exit_end);KVM_NVHE_ALIAS (abort_guest_exit_start); -/* Symbols defined in hyp-init.S (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__kvm_handle_stub_hvc); -/* Symbols defined in switch.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_vcpu_run_nvhe);KVM_NVHE_ALIAS (hyp_panic);
@@ -113,6 +110,12 @@ KVM_NVHE_ALIAS(kimage_voffset);/* Kernel symbols used to call panic() from nVHE hyp code (via ERET). */ KVM_NVHE_ALIAS (panic); +/* Vectors installed by hyp-init on reset HVC. */ +KVM_NVHE_ALIAS (__hyp_stub_vectors); + +/* IDMAP TCR_EL1.T0SZ as computed by the EL1 init code */ +KVM_NVHE_ALIAS (idmap_t0sz); +#endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
arch/arm64/kvm/Makefile +1 -1
@@ -13,7 +13,7 @@ obj-$(CONFIG_KVM) += hyp/ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ $(KVM)/vfio.o $(KVM)/irqchip.o \ arm.o mmu.o mmio.o psci.o perf.o hypercalls.o pvtime.o \ - inject_fault.o regmap.o va_layout.o hyp.o hyp-init.o handle_exit.o \ + inject_fault.o regmap.o va_layout.o hyp.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ aarch32.o arch_timer.o \
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := ../hyp-entry.o +obj-y := hyp-init.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/hyp-init.S renamed from arch/arm64/kvm/hyp-init.S +0 -0
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 20/56 KVM: arm64: Duplicate hyp/tlb.c for VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
tlb.c contains code for flushing the TLB, with code shared between VHE/nVHE. Because common code is small, duplicate tlb.c and specialize each copy for VHE/nVHE.
---
arch/arm64/kernel/image-vars.h | 14 +--
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/nvhe/Makefile | 2 +-
arch/arm64/kvm/hyp/{ => nvhe}/tlb.c | 96 +----------------
arch/arm64/kvm/hyp/vhe/Makefile | 2 +-
arch/arm64/kvm/hyp/vhe/tlb.c | 161 ++++++++++++++++++++++++++++
6 files changed, 177 insertions(+), 100 deletions(-)
rename arch/arm64/kvm/hyp/{ => nvhe}/tlb.c (61%)
create mode 100644 arch/arm64/kvm/hyp/vhe/tlb.carch/arm64/kernel/image-vars.h +8 -6
@@ -81,12 +81,6 @@ KVM_NVHE_ALIAS(__kvm_enable_ssbs);/* Symbols defined in timer-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_timer_set_cntvoff); -/* Symbols defined in tlb.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__kvm_flush_vm_context); -KVM_NVHE_ALIAS (__kvm_tlb_flush_local_vmid); -KVM_NVHE_ALIAS (__kvm_tlb_flush_vmid); -KVM_NVHE_ALIAS (__kvm_tlb_flush_vmid_ipa); -/* Symbols defined in vgic-v3-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__vgic_v3_get_ich_vtr_el2);KVM_NVHE_ALIAS (__vgic_v3_init_lrs);
@@ -116,6 +110,14 @@ KVM_NVHE_ALIAS(__hyp_stub_vectors);/* IDMAP TCR_EL1.T0SZ as computed by the EL1 init code */ KVM_NVHE_ALIAS (idmap_t0sz); +/* Kernel symbol used by icache_is_vpipt(). */ +KVM_NVHE_ALIAS (__icache_flags); + +/* Kernel symbols needed for cpus_have_final/const_caps checks. */ +KVM_NVHE_ALIAS (arm64_const_caps_ready); +KVM_NVHE_ALIAS (cpu_hwcap_keys); +KVM_NVHE_ALIAS (cpu_hwcaps); +#endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
arch/arm64/kvm/hyp/Makefile +1 -1
@@ -14,7 +14,7 @@ obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ - debug-sr.o entry.o switch.o fpsimd.o tlb.o + debug-sr.o entry.o switch.o fpsimd.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := hyp-init.o ../hyp-entry.o +obj-y := tlb.o hyp-init.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/tlb.c renamed from arch/arm64/kvm/hyp/tlb.c +5 -91
128 lines not on this page · raw message
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := ../hyp-entry.o +obj-y := tlb.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/vhe/tlb.c new file +161 -0
161 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 21/56 KVM: arm64: Split hyp/switch.c to VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
switch.c implements context-switching for KVM, with large parts shared between VHE/nVHE. These common routines are moved to a header file, VHE-specific code is moved to vhe/switch.c and nVHE-specific code is moved to nvhe/switch.c.
Previously __kvm_vcpu_run needed a different symbol name for VHE/nVHE. This is cleaned up and the caller in arm.c simplified.
--- arch/arm64/include/asm/kvm_asm.h | 4 +- arch/arm64/include/asm/kvm_hyp.h | 6 + arch/arm64/kernel/image-vars.h | 37 +- arch/arm64/kvm/arm.c | 6 +- arch/arm64/kvm/hyp/Makefile | 2 +- arch/arm64/kvm/hyp/hyp-entry.S | 2 + arch/arm64/kvm/hyp/include/hyp/switch.h | 509 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 2 +- arch/arm64/kvm/hyp/nvhe/switch.c | 271 +++++++ arch/arm64/kvm/hyp/switch.c | 936 ------------------------ arch/arm64/kvm/hyp/sysreg-sr.c | 4 +- arch/arm64/kvm/hyp/vhe/Makefile | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 219 ++++++ 13 files changed, 1048 insertions(+), 952 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/hyp/switch.h create mode 100644 arch/arm64/kvm/hyp/nvhe/switch.c delete mode 100644 arch/arm64/kvm/hyp/switch.c create mode 100644 arch/arm64/kvm/hyp/vhe/switch.c
arch/arm64/include/asm/kvm_asm.h +1 -3
@@ -96,9 +96,7 @@ extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu);extern void __kvm_timer_set_cntvoff (u64 cntvoff); -extern int kvm_vcpu_run_vhe (struct kvm_vcpu *vcpu); - -extern int __kvm_vcpu_run_nvhe (struct kvm_vcpu *vcpu); +extern int __kvm_vcpu_run (struct kvm_vcpu *vcpu);extern void __kvm_enable_ssbs (void );
arch/arm64/include/asm/kvm_hyp.h +6 -0
@@ -81,11 +81,17 @@ void __debug_switch_to_host(struct kvm_vcpu *vcpu);void __fpsimd_save_state (struct user_fpsimd_state *fp_regs);void __fpsimd_restore_state (struct user_fpsimd_state *fp_regs); +#ifndef __KVM_NVHE_HYPERVISOR__void activate_traps_vhe_load (struct kvm_vcpu *vcpu);void deactivate_traps_vhe_put (void ); +#endif u64 __guest_enter (struct kvm_vcpu *vcpu,struct kvm_cpu_context *host_ctxt); + +void __noreturn hyp_panic (struct kvm_cpu_context *host_ctxt); +#ifdef __KVM_NVHE_HYPERVISOR__void __noreturn __hyp_do_panic (unsigned long , ...); +#endif #endif /* __ARM64_KVM_HYP_H__ */
arch/arm64/kernel/image-vars.h +34 -3
@@ -63,30 +63,50 @@ __efistub__ctype = _ctype;#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; +/* Symbols defined in aarch32.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (kvm_skip_instr32); +/* Symbols defined in debug-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__debug_switch_to_guest); +KVM_NVHE_ALIAS (__debug_switch_to_host);KVM_NVHE_ALIAS (__kvm_get_mdcr_el2);/* Symbols defined in entry.S (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__guest_enter);KVM_NVHE_ALIAS (__guest_exit);KVM_NVHE_ALIAS (abort_guest_exit_end);KVM_NVHE_ALIAS (abort_guest_exit_start); -/* Symbols defined in switch.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__kvm_vcpu_run_nvhe); -KVM_NVHE_ALIAS (hyp_panic); +/* Symbols defined in fpsimd.S (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__fpsimd_restore_state); +KVM_NVHE_ALIAS (__fpsimd_save_state);/* Symbols defined in sysreg-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_enable_ssbs); +KVM_NVHE_ALIAS (__sysreg32_restore_state); +KVM_NVHE_ALIAS (__sysreg32_save_state); +KVM_NVHE_ALIAS (__sysreg_restore_state_nvhe); +KVM_NVHE_ALIAS (__sysreg_save_state_nvhe);/* Symbols defined in timer-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_timer_set_cntvoff); +KVM_NVHE_ALIAS (__timer_disable_traps); +KVM_NVHE_ALIAS (__timer_enable_traps); + +/* Symbols defined in vgic-v2-cpuif-proxy.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__vgic_v2_perform_cpuif_access);/* Symbols defined in vgic-v3-sr.c (not yet compiled with nVHE build rules). */ +KVM_NVHE_ALIAS (__vgic_v3_activate_traps); +KVM_NVHE_ALIAS (__vgic_v3_deactivate_traps);KVM_NVHE_ALIAS (__vgic_v3_get_ich_vtr_el2);KVM_NVHE_ALIAS (__vgic_v3_init_lrs); +KVM_NVHE_ALIAS (__vgic_v3_perform_cpuif_access);KVM_NVHE_ALIAS (__vgic_v3_read_vmcr);KVM_NVHE_ALIAS (__vgic_v3_restore_aprs); +KVM_NVHE_ALIAS (__vgic_v3_restore_state);KVM_NVHE_ALIAS (__vgic_v3_save_aprs); +KVM_NVHE_ALIAS (__vgic_v3_save_state);KVM_NVHE_ALIAS (__vgic_v3_write_vmcr);/* Alternative callbacks for init-time patching of nVHE hyp code. */
@@ -97,11 +117,13 @@ KVM_NVHE_ALIAS(kvm_update_va_mask);/* Global kernel state accessed by nVHE hyp code. */ KVM_NVHE_ALIAS (arm64_ssbd_callback_required);KVM_NVHE_ALIAS (kvm_host_data); +KVM_NVHE_ALIAS (kvm_vgic_global_state);/* Kernel constant needed to compute idmap addresses. */ KVM_NVHE_ALIAS (kimage_voffset);/* Kernel symbols used to call panic() from nVHE hyp code (via ERET). */ +KVM_NVHE_ALIAS (__hyp_panic_string);KVM_NVHE_ALIAS (panic);/* Vectors installed by hyp-init on reset HVC. */
@@ -118,6 +140,15 @@ KVM_NVHE_ALIAS(arm64_const_caps_ready);KVM_NVHE_ALIAS (cpu_hwcap_keys);KVM_NVHE_ALIAS (cpu_hwcaps); +/* Static keys which are set if a vGIC trap should be handled in hyp. */ +KVM_NVHE_ALIAS (vgic_v2_cpuif_trap); +KVM_NVHE_ALIAS (vgic_v3_cpuif_trap); + +/* Static key checked in pmr_sync(). */ +#ifdef CONFIG_ARM64_PSEUDO_NMI +KVM_NVHE_ALIAS (gic_pmr_sync); +#endif +#endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
arch/arm64/kvm/arm.c +1 -5
@@ -748,11 +748,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)trace_kvm_entry (*vcpu_pc (vcpu));guest_enter_irqoff (); -if (has_vhe ()) { - ret =kvm_vcpu_run_vhe (vcpu); - }else { - ret =kvm_call_hyp_ret (__kvm_vcpu_run_nvhe, vcpu); - } + ret =kvm_call_hyp_ret (__kvm_vcpu_run, vcpu); vcpu->mode =OUTSIDE_GUEST_MODE ; vcpu->stat.exits++;
arch/arm64/kvm/hyp/Makefile +1 -1
@@ -14,7 +14,7 @@ obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ - debug-sr.o entry.o switch.o fpsimd.o + debug-sr.o entry.o fpsimd.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/hyp-entry.S +2 -0
@@ -182,6 +182,7 @@ el2_error: eret sb +#ifdef __KVM_NVHE_HYPERVISOR__ SYM_FUNC_START(__hyp_do_panic) mov lr, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT |\ PSR_MODE_EL1h)
@@ -191,6 +192,7 @@ SYM_FUNC_START(__hyp_do_panic) eret sb SYM_FUNC_END(__hyp_do_panic) +#endif SYM_CODE_START(__hyp_panic) get_host_ctxt x0, x1
arch/arm64/kvm/hyp/include/hyp/switch.h new file +509 -0
509 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := tlb.o hyp-init.o ../hyp-entry.o +obj-y := switch.o tlb.o hyp-init.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/switch.c new file +271 -0
271 lines not on this page · raw message
arch/arm64/kvm/hyp/switch.c deleted +0 -936
936 lines not on this page · raw message
arch/arm64/kvm/hyp/sysreg-sr.c +2 -2
@@ -114,7 +114,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)/* * Must only be done for guest registers, hence the context * test. We're coming from the host, so SCTLR.M is already -* set. Pairs with __activate_traps_nvhe(). +* set. Pairs with nVHE's __activate_traps(). */ write_sysreg_el1((ctxt->sys_regs[TCR_EL1] | TCR_EPD1_MASK | TCR_EPD0_MASK),
@@ -142,7 +142,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) ctxt->__hyp_running_vcpu) {/* * Must only be done for host registers, hence the context -* test. Pairs with __deactivate_traps_nvhe(). +* test. Pairs with nVHE's __deactivate_traps(). */ isb ();/*
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := tlb.o ../hyp-entry.o +obj-y := switch.o tlb.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/vhe/switch.c new file +219 -0
219 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 22/56 KVM: arm64: Split hyp/debug-sr.c to VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
debug-sr.c contains KVM's code for context-switching debug registers, with some code shared between VHE/nVHE. These common routines are moved to a header file, VHE-specific code is moved to vhe/debug-sr.c and nVHE-specific code to nvhe/debug-sr.c.
Functions are slightly refactored to move code hidden behind has_vhe() checks to the corresponding .c files.
---
arch/arm64/kernel/image-vars.h | 5 --
arch/arm64/kvm/hyp/Makefile | 2 +-
.../{debug-sr.c => include/hyp/debug-sr.h} | 78 +++----------------
arch/arm64/kvm/hyp/nvhe/Makefile | 2 +-
arch/arm64/kvm/hyp/nvhe/debug-sr.c | 77 ++++++++++++++++++
arch/arm64/kvm/hyp/vhe/Makefile | 2 +-
arch/arm64/kvm/hyp/vhe/debug-sr.c | 26 +++++++
7 files changed, 118 insertions(+), 74 deletions(-)
rename arch/arm64/kvm/hyp/{debug-sr.c => include/hyp/debug-sr.h} (71%)
create mode 100644 arch/arm64/kvm/hyp/nvhe/debug-sr.c
create mode 100644 arch/arm64/kvm/hyp/vhe/debug-sr.carch/arm64/kernel/image-vars.h +0 -5
@@ -66,11 +66,6 @@ __efistub__ctype = _ctype;/* Symbols defined in aarch32.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (kvm_skip_instr32); -/* Symbols defined in debug-sr.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__debug_switch_to_guest); -KVM_NVHE_ALIAS (__debug_switch_to_host); -KVM_NVHE_ALIAS (__kvm_get_mdcr_el2); -/* Symbols defined in entry.S (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__guest_enter);KVM_NVHE_ALIAS (__guest_exit);
arch/arm64/kvm/hyp/Makefile +1 -1
@@ -14,7 +14,7 @@ obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ - debug-sr.o entry.o fpsimd.o + entry.o fpsimd.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/include/hyp/debug-sr.h renamed from arch/arm64/kvm/hyp/debug-sr.c +12 -66
123 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := switch.o tlb.o hyp-init.o ../hyp-entry.o +obj-y := debug-sr.o switch.o tlb.o hyp-init.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/debug-sr.c new file +77 -0
77 lines not on this page · raw message
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := switch.o tlb.o ../hyp-entry.o +obj-y := debug-sr.o switch.o tlb.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/vhe/debug-sr.c new file +26 -0
@@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2015 - ARM Ltd +* Author: Marc Zyngier <[email protected]> +*/ + +#include <hyp/debug-sr.h> + +#include <linux/kvm_host.h> + +#include <asm/kvm_hyp.h> + +void __debug_switch_to_guest (struct kvm_vcpu *vcpu) +{ +__debug_switch_to_guest_common (vcpu); +} + +void __debug_switch_to_host (struct kvm_vcpu *vcpu) +{ +__debug_switch_to_host_common (vcpu); +} + +u32 __kvm_get_mdcr_el2 (void ) +{ +return read_sysreg (mdcr_el2); +}
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 23/56 KVM: arm64: Split hyp/sysreg-sr.c to VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
sysreg-sr.c contains KVM's code for saving/restoring system registers, with some code shared between VHE/nVHE. These common routines are moved to a header file, VHE-specific code is moved to vhe/sysreg-sr.c and nVHE-specific code to nvhe/sysreg-sr.c.
---
arch/arm64/include/asm/kvm_host.h | 6 +-
arch/arm64/include/asm/kvm_hyp.h | 5 +-
arch/arm64/kernel/image-vars.h | 7 -
arch/arm64/kvm/arm.c | 8 +-
arch/arm64/kvm/hyp/Makefile | 4 +-
.../{sysreg-sr.c => include/hyp/sysreg-sr.h} | 158 ++----------------
arch/arm64/kvm/hyp/nvhe/Makefile | 2 +-
arch/arm64/kvm/hyp/nvhe/switch.c | 1 +
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 46 +++++
arch/arm64/kvm/hyp/vhe/Makefile | 2 +-
arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 114 +++++++++++++
12 files changed, 191 insertions(+), 164 deletions(-)
rename arch/arm64/kvm/hyp/{sysreg-sr.c => include/hyp/sysreg-sr.h} (56%)
create mode 100644 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c
create mode 100644 arch/arm64/kvm/hyp/vhe/sysreg-sr.carch/arm64/include/asm/kvm_host.h +3 -3
@@ -338,7 +338,7 @@ struct kvm_vcpu_arch {struct vcpu_reset_state reset_state;/* True when deferrable sysregs are loaded on the physical CPU, -* see kvm_vcpu_load_sysregs and kvm_vcpu_put_sysregs. */ +* see kvm_vcpu_load_sysregs_vhe and kvm_vcpu_put_sysregs_vhe. */ bool sysregs_loaded_on_cpu;/* Guest PV state */
@@ -639,8 +639,8 @@ static inline int kvm_arm_have_ssbd(void) } } -void kvm_vcpu_load_sysregs (struct kvm_vcpu *vcpu); -void kvm_vcpu_put_sysregs (struct kvm_vcpu *vcpu); +void kvm_vcpu_load_sysregs_vhe (struct kvm_vcpu *vcpu); +void kvm_vcpu_put_sysregs_vhe (struct kvm_vcpu *vcpu);int kvm_set_ipa_limit (void );
arch/arm64/include/asm/kvm_hyp.h +3 -2
@@ -66,14 +66,15 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);void __timer_enable_traps (struct kvm_vcpu *vcpu);void __timer_disable_traps (struct kvm_vcpu *vcpu); +#ifdef __KVM_NVHE_HYPERVISOR__void __sysreg_save_state_nvhe (struct kvm_cpu_context *ctxt);void __sysreg_restore_state_nvhe (struct kvm_cpu_context *ctxt); +#else void sysreg_save_host_state_vhe (struct kvm_cpu_context *ctxt);void sysreg_restore_host_state_vhe (struct kvm_cpu_context *ctxt);void sysreg_save_guest_state_vhe (struct kvm_cpu_context *ctxt);void sysreg_restore_guest_state_vhe (struct kvm_cpu_context *ctxt); -void __sysreg32_save_state (struct kvm_vcpu *vcpu); -void __sysreg32_restore_state (struct kvm_vcpu *vcpu); +#endif void __debug_switch_to_guest (struct kvm_vcpu *vcpu);void __debug_switch_to_host (struct kvm_vcpu *vcpu);
arch/arm64/kernel/image-vars.h +0 -7
@@ -76,13 +76,6 @@ KVM_NVHE_ALIAS(abort_guest_exit_start);KVM_NVHE_ALIAS (__fpsimd_restore_state);KVM_NVHE_ALIAS (__fpsimd_save_state); -/* Symbols defined in sysreg-sr.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__kvm_enable_ssbs); -KVM_NVHE_ALIAS (__sysreg32_restore_state); -KVM_NVHE_ALIAS (__sysreg32_save_state); -KVM_NVHE_ALIAS (__sysreg_restore_state_nvhe); -KVM_NVHE_ALIAS (__sysreg_save_state_nvhe); -/* Symbols defined in timer-sr.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__kvm_timer_set_cntvoff);KVM_NVHE_ALIAS (__timer_disable_traps);
arch/arm64/kvm/arm.c +5 -3
@@ -351,7 +351,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_vgic_load (vcpu);kvm_timer_vcpu_load (vcpu); -kvm_vcpu_load_sysregs (vcpu); +if (has_vhe ()) +kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu);kvm_vcpu_pmu_restore_guest (vcpu);if (kvm_arm_is_pvtime_enabled (&vcpu->arch))
@@ -369,7 +370,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)void kvm_arch_vcpu_put (struct kvm_vcpu *vcpu) {kvm_arch_vcpu_put_fp (vcpu); -kvm_vcpu_put_sysregs (vcpu); +if (has_vhe ()) +kvm_vcpu_put_sysregs_vhe (vcpu);kvm_timer_vcpu_put (vcpu);kvm_vgic_put (vcpu);kvm_vcpu_pmu_restore_host (vcpu);
@@ -1302,7 +1304,7 @@ static void cpu_init_hyp_mode(void)*/ if (this_cpu_has_cap (ARM64_SSBS ) &&arm64_get_ssbd_state () ==ARM64_SSBD_FORCE_DISABLE ) { -kvm_call_hyp (__kvm_enable_ssbs); +kvm_call_hyp_nvhe (__kvm_enable_ssbs); } }
arch/arm64/kvm/hyp/Makefile +2 -2
@@ -13,8 +13,8 @@ subdir-ccflags-y := -I$(incdir) \ obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o -hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ - entry.o fpsimd.o +hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o entry.o \ + fpsimd.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h renamed from arch/arm64/kvm/hyp/sysreg-sr.c +14 -144
221 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := debug-sr.o switch.o tlb.o hyp-init.o ../hyp-entry.o +obj-y := sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/switch.c +1 -0
@@ -5,6 +5,7 @@*/ #include <hyp/switch.h> +#include <hyp/sysreg-sr.h> #include <linux/arm-smccc.h> #include <linux/kvm_host.h>
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c new file +46 -0
46 lines not on this page · raw message
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := debug-sr.o switch.o tlb.o ../hyp-entry.o +obj-y := sysreg-sr.o debug-sr.o switch.o tlb.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/vhe/switch.c +1 -1
@@ -120,7 +120,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)* HCR_EL2.TGE. * * We have already configured the guest's stage 1 translation in -* kvm_vcpu_load_sysregs above. We must now call __activate_vm +* kvm_vcpu_load_sysregs_vhe above. We must now call __activate_vm * before __activate_traps, because __activate_vm configures * stage 2 translation, and __activate_traps clear HCR_EL2.TGE * (among other things).
arch/arm64/kvm/hyp/vhe/sysreg-sr.c new file +114 -0
114 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 24/56 KVM: arm64: Duplicate hyp/timer-sr.c for VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
timer-sr.c contains a HVC handler for setting CNTVOFF_EL2 and two helper functions for controlling access to physical counter. The former is used by both VHE/nVHE and is duplicated, the latter are used only by nVHE and moved to nvhe/timer-sr.c.
---
arch/arm64/include/asm/kvm_hyp.h | 2 ++
arch/arm64/kernel/image-vars.h | 5 -----
arch/arm64/kvm/hyp/Makefile | 2 +-
arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++-
arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c | 2 +-
arch/arm64/kvm/hyp/vhe/Makefile | 2 +-
arch/arm64/kvm/hyp/vhe/timer-sr.c | 12 ++++++++++++
7 files changed, 19 insertions(+), 9 deletions(-)
rename arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c (95%)
create mode 100644 arch/arm64/kvm/hyp/vhe/timer-sr.carch/arm64/include/asm/kvm_hyp.h +2 -0
@@ -63,8 +63,10 @@ void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_restore_aprs (struct vgic_v3_cpu_if *cpu_if);int __vgic_v3_perform_cpuif_access (struct kvm_vcpu *vcpu); +#ifdef __KVM_NVHE_HYPERVISOR__void __timer_enable_traps (struct kvm_vcpu *vcpu);void __timer_disable_traps (struct kvm_vcpu *vcpu); +#endif #ifdef __KVM_NVHE_HYPERVISOR__void __sysreg_save_state_nvhe (struct kvm_cpu_context *ctxt);
arch/arm64/kernel/image-vars.h +0 -5
@@ -76,11 +76,6 @@ KVM_NVHE_ALIAS(abort_guest_exit_start);KVM_NVHE_ALIAS (__fpsimd_restore_state);KVM_NVHE_ALIAS (__fpsimd_save_state); -/* Symbols defined in timer-sr.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__kvm_timer_set_cntvoff); -KVM_NVHE_ALIAS (__timer_disable_traps); -KVM_NVHE_ALIAS (__timer_enable_traps); -/* Symbols defined in vgic-v2-cpuif-proxy.c (not yet compiled with nVHE build rules). */ KVM_NVHE_ALIAS (__vgic_v2_perform_cpuif_access);
arch/arm64/kvm/hyp/Makefile +1 -1
@@ -13,7 +13,7 @@ subdir-ccflags-y := -I$(incdir) \ obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o -hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o entry.o \ +hyp-y := vgic-v3-sr.o aarch32.o vgic-v2-cpuif-proxy.o entry.o \ fpsimd.o # KVM code is run at a different exception code with a different map, so
arch/arm64/kvm/hyp/nvhe/Makefile +2 -1
@@ -6,7 +6,8 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o ../hyp-entry.o +obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o \ + ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/nvhe/timer-sr.c renamed from arch/arm64/kvm/hyp/timer-sr.c +1 -1
@@ -10,7 +10,7 @@#include <asm/kvm_hyp.h> -void __hyp_text __kvm_timer_set_cntvoff(u64 cntvoff) +void __kvm_timer_set_cntvoff (u64 cntvoff) {write_sysreg (cntvoff, cntvoff_el2); }
arch/arm64/kvm/hyp/vhe/Makefile +1 -1
@@ -6,7 +6,7 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := sysreg-sr.o debug-sr.o switch.o tlb.o ../hyp-entry.o +obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/vhe/timer-sr.c new file +12 -0
@@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2012-2015 - ARM Ltd +* Author: Marc Zyngier <[email protected]> +*/ + +#include <asm/kvm_hyp.h> + +void __kvm_timer_set_cntvoff (u64 cntvoff) +{ +write_sysreg (cntvoff, cntvoff_el2); +}
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 25/56 KVM: arm64: Compile remaining hyp/ files for both VHE/nVHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
The following files in hyp/ contain only code shared by VHE/nVHE:
vgic-v3-sr.c, aarch32.c, vgic-v2-cpuif-proxy.c, entry.S, fpsimd.S
Compile them under both configurations. Deletions in image-vars.h reflect eliminated dependencies of nVHE code on the rest of the kernel.
--- arch/arm64/kernel/image-vars.h | 29 ----------------------------- arch/arm64/kvm/hyp/Makefile | 13 +------------ arch/arm64/kvm/hyp/nvhe/Makefile | 5 +++-- arch/arm64/kvm/hyp/vhe/Makefile | 4 +++- 4 files changed, 7 insertions(+), 44 deletions(-)
arch/arm64/kernel/image-vars.h +0 -29
@@ -63,35 +63,6 @@ __efistub__ctype = _ctype;#define KVM_NVHE_ALIAS (sym) __kvm_nvhe_##sym = sym; -/* Symbols defined in aarch32.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (kvm_skip_instr32); - -/* Symbols defined in entry.S (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__guest_enter); -KVM_NVHE_ALIAS (__guest_exit); -KVM_NVHE_ALIAS (abort_guest_exit_end); -KVM_NVHE_ALIAS (abort_guest_exit_start); - -/* Symbols defined in fpsimd.S (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__fpsimd_restore_state); -KVM_NVHE_ALIAS (__fpsimd_save_state); - -/* Symbols defined in vgic-v2-cpuif-proxy.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__vgic_v2_perform_cpuif_access); - -/* Symbols defined in vgic-v3-sr.c (not yet compiled with nVHE build rules). */ -KVM_NVHE_ALIAS (__vgic_v3_activate_traps); -KVM_NVHE_ALIAS (__vgic_v3_deactivate_traps); -KVM_NVHE_ALIAS (__vgic_v3_get_ich_vtr_el2); -KVM_NVHE_ALIAS (__vgic_v3_init_lrs); -KVM_NVHE_ALIAS (__vgic_v3_perform_cpuif_access); -KVM_NVHE_ALIAS (__vgic_v3_read_vmcr); -KVM_NVHE_ALIAS (__vgic_v3_restore_aprs); -KVM_NVHE_ALIAS (__vgic_v3_restore_state); -KVM_NVHE_ALIAS (__vgic_v3_save_aprs); -KVM_NVHE_ALIAS (__vgic_v3_save_state); -KVM_NVHE_ALIAS (__vgic_v3_write_vmcr); -/* Alternative callbacks for init-time patching of nVHE hyp code. */ KVM_NVHE_ALIAS (arm64_enable_wa2_handling);KVM_NVHE_ALIAS (kvm_patch_vector_branch);
arch/arm64/kvm/hyp/Makefile +1 -12
@@ -10,16 +10,5 @@ subdir-ccflags-y := -I$(incdir) \ -DDISABLE_BRANCH_PROFILING \ $(DISABLE_STACKLEAK_PLUGIN) -obj-$(CONFIG_KVM) += hyp.o vhe/ nvhe/ +obj-$(CONFIG_KVM) += vhe/ nvhe/ obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o - -hyp-y := vgic-v3-sr.o aarch32.o vgic-v2-cpuif-proxy.o entry.o \ - fpsimd.o - -# KVM code is run at a different exception code with a different map, so -# compiler instrumentation that inserts callbacks or checks into the code may -# cause crashes. Just disable it. -GCOV_PROFILE := n -KASAN_SANITIZE := n -UBSAN_SANITIZE := n -KCOV_INSTRUMENT := n
arch/arm64/kvm/hyp/nvhe/Makefile +3 -2
@@ -6,8 +6,9 @@ asflags-y := -D__KVM_NVHE_HYPERVISOR__ ccflags-y := -D__KVM_NVHE_HYPERVISOR__ -obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o \ - ../hyp-entry.o +obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o +obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ + ../fpsimd.o ../hyp-entry.o obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
arch/arm64/kvm/hyp/vhe/Makefile +3 -1
@@ -6,7 +6,9 @@ asflags-y := -D__KVM_VHE_HYPERVISOR__ ccflags-y := -D__KVM_VHE_HYPERVISOR__ -obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o ../hyp-entry.o +obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o +obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ + ../fpsimd.o ../hyp-entry.o # KVM code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 26/56 KVM: arm64: Remove __hyp_text macro, use build rules instead
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
With nVHE code now fully separated from the rest of the kernel, the effects of the __hyp_text macro (which had to be applied on all nVHE code) can be achieved with build rules instead. The macro used to:
(a) move code to .hyp.text ELF section, now done by renaming .text using objcopy, and
(b) notrace and __noscs would negate effects of CC_FLAGS_FTRACE and CC_FLAGS_SCS, respectivelly, now those flags are erased from KBUILD_CFLAGS (same way as in EFI stub).
Note that by removing __hyp_text from code shared with VHE, all VHE code is now compiled into .text and without notrace and __noscs.
Use of '.pushsection .hyp.text' removed from assembly files as this is now also covered by the build rules.
For MAINTAINERS: if needed to re-run, uses of macro were removed with the following command. Formatting was fixed up manually.
find arch/arm64/kvm/hyp -type f -name '*.c' -o -name '*.h' \
-exec sed -i 's/ __hyp_text//g' {} +--- arch/arm64/include/asm/kvm_emulate.h | 2 +- arch/arm64/include/asm/kvm_hyp.h | 2 - arch/arm64/kvm/hyp/aarch32.c | 6 +- arch/arm64/kvm/hyp/entry.S | 1 - arch/arm64/kvm/hyp/fpsimd.S | 1 - arch/arm64/kvm/hyp/hyp-entry.S | 1 - arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 16 +-- arch/arm64/kvm/hyp/include/hyp/switch.h | 36 +++--- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 20 ++-- arch/arm64/kvm/hyp/nvhe/Makefile | 8 +- arch/arm64/kvm/hyp/nvhe/debug-sr.c | 10 +- arch/arm64/kvm/hyp/nvhe/switch.c | 18 +-- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 6 +- arch/arm64/kvm/hyp/nvhe/timer-sr.c | 4 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 14 +-- arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 +- arch/arm64/kvm/hyp/vgic-v3-sr.c | 130 +++++++++------------ 17 files changed, 132 insertions(+), 147 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +1 -1
@@ -516,7 +516,7 @@ static __always_inline void kvm_skip_instr(struct kvm_vcpu *vcpu, bool is_wide_i* Skip an instruction which has been emulated at hyp while most guest sysregs * are live. */ -static __always_inline void __hyp_text __kvm_skip_instr(struct kvm_vcpu *vcpu) +static __always_inline void __kvm_skip_instr(struct kvm_vcpu *vcpu) { *vcpu_pc (vcpu) =read_sysreg_el2 (SYS_ELR ); vcpu->arch.ctxt.gp_regs.regs.pstate =read_sysreg_el2 (SYS_SPSR );
arch/arm64/include/asm/kvm_hyp.h +0 -2
@@ -12,8 +12,6 @@#include <asm/alternative.h> #include <asm/sysreg.h> -#define __hyp_text __section(.hyp.text) notrace __noscs -#define read_sysreg_elx (r,nvh,vh) \ ({ \ u64 reg; \
arch/arm64/kvm/hyp/aarch32.c +3 -3
@@ -44,7 +44,7 @@ static const unsigned short cc_map[16] = {/* * Check if a trapped instruction should have been executed or not. */ -bool __hyp_text kvm_condition_valid32(const struct kvm_vcpu *vcpu) +bool kvm_condition_valid32 (const struct kvm_vcpu *vcpu) {unsigned long cpsr;u32 cpsr_cond;
@@ -93,7 +93,7 @@ bool __hyp_text kvm_condition_valid32(const struct kvm_vcpu *vcpu)* * IT[7:0] -> CPSR[26:25],CPSR[15:10] */ -static void __hyp_text kvm_adjust_itstate(struct kvm_vcpu *vcpu) +static void kvm_adjust_itstate (struct kvm_vcpu *vcpu) {unsigned long itbits, cond;unsigned long cpsr = *vcpu_cpsr (vcpu);
@@ -123,7 +123,7 @@ static void __hyp_text kvm_adjust_itstate(struct kvm_vcpu *vcpu)* kvm_skip_instr - skip a trapped instruction and proceed to the next * @vcpu: The vcpu pointer */ -void __hyp_text kvm_skip_instr32(struct kvm_vcpu *vcpu, bool is_wide_instr) +void kvm_skip_instr32 (struct kvm_vcpu *vcpu,bool is_wide_instr) {u32 pc = *vcpu_pc (vcpu);bool is_thumb;
arch/arm64/kvm/hyp/entry.S +0 -1
@@ -21,7 +21,6 @@ #define CPU_SP_EL0_OFFSET (CPU_XREG_OFFSET(30) + 8) .text - .pushsection .hyp.text, "ax" /* * We treat x18 as callee-saved as the host may use it as a platform
arch/arm64/kvm/hyp/fpsimd.S +0 -1
@@ -9,7 +9,6 @@ #include <asm/fpsimdmacros.h> .text - .pushsection .hyp.text, "ax" SYM_FUNC_START(__fpsimd_save_state) fpsimd_save x0, 1
arch/arm64/kvm/hyp/hyp-entry.S +0 -1
@@ -16,7 +16,6 @@ #include <asm/mmu.h> .text - .pushsection .hyp.text, "ax" .macro do_el2_call /*
arch/arm64/kvm/hyp/include/hyp/debug-sr.h +8 -8
@@ -88,9 +88,9 @@default :write_debug (ptr[0 ], reg,0 ); \ } -static inline void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu, -struct kvm_guest_debug_arch *dbg, -struct kvm_cpu_context *ctxt) +static inline void __debug_save_state (struct kvm_vcpu *vcpu, +struct kvm_guest_debug_arch *dbg, +struct kvm_cpu_context *ctxt) {u64 aa64dfr0;int brps, wrps;
@@ -107,9 +107,9 @@ static inline void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu, ctxt->sys_regs[MDCCINT_EL1 ] =read_sysreg (mdccint_el1); } -static inline void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu, -struct kvm_guest_debug_arch *dbg, -struct kvm_cpu_context *ctxt) +static inline void __debug_restore_state (struct kvm_vcpu *vcpu, +struct kvm_guest_debug_arch *dbg, +struct kvm_cpu_context *ctxt) {u64 aa64dfr0;int brps, wrps;
@@ -127,7 +127,7 @@ static inline void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu,write_sysreg (ctxt->sys_regs[MDCCINT_EL1 ], mdccint_el1); } -static inline void __hyp_text __debug_switch_to_guest_common(struct kvm_vcpu *vcpu) +static inline void __debug_switch_to_guest_common (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;
@@ -146,7 +146,7 @@ static inline void __hyp_text __debug_switch_to_guest_common(struct kvm_vcpu *vc__debug_restore_state (vcpu, guest_dbg, guest_ctxt); } -static inline void __hyp_text __debug_switch_to_host_common(struct kvm_vcpu *vcpu) +static inline void __debug_switch_to_host_common (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;
arch/arm64/kvm/hyp/include/hyp/switch.h +18 -18
@@ -30,7 +30,7 @@extern const char __hyp_panic_string[];/* Check whether the FP regs were dirtied while in the host-side run loop: */ -static inline bool __hyp_text update_fp_enabled(struct kvm_vcpu *vcpu) +static inline bool update_fp_enabled (struct kvm_vcpu *vcpu) {/* * When the system doesn't support FP/SIMD, we cannot rely on
@@ -48,7 +48,7 @@ static inline bool __hyp_text update_fp_enabled(struct kvm_vcpu *vcpu) }/* Save the 32-bit only FPSIMD system register state */ -static inline void __hyp_text __fpsimd_save_fpexc32(struct kvm_vcpu *vcpu) +static inline void __fpsimd_save_fpexc32 (struct kvm_vcpu *vcpu) {if (!vcpu_el1_is_32bit (vcpu))return ;
@@ -56,7 +56,7 @@ static inline void __hyp_text __fpsimd_save_fpexc32(struct kvm_vcpu *vcpu) vcpu->arch.ctxt.sys_regs[FPEXC32_EL2 ] =read_sysreg (fpexc32_el2); } -static inline void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) +static inline void __activate_traps_fpsimd32 (struct kvm_vcpu *vcpu) {/* * We are about to set CPTR_EL2.TFP to trap all floating point
@@ -73,7 +73,7 @@ static inline void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) } } -static inline void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu) +static inline void __activate_traps_common (struct kvm_vcpu *vcpu) {/* Trap on AArch32 cp15 c15 (impdef sysregs) accesses (EL1 or EL0) */ write_sysreg (1 <<15 , hstr_el2);
@@ -89,13 +89,13 @@ static inline void __hyp_text __activate_traps_common(struct kvm_vcpu *vcpu)write_sysreg (vcpu->arch.mdcr_el2, mdcr_el2); } -static inline void __hyp_text __deactivate_traps_common(void) +static inline void __deactivate_traps_common (void ) {write_sysreg (0 , hstr_el2);write_sysreg (0 , pmuserenr_el0); } -static inline void __hyp_text ___activate_traps(struct kvm_vcpu *vcpu) +static inline void ___activate_traps (struct kvm_vcpu *vcpu) {u64 hcr = vcpu->arch.hcr_el2;
@@ -108,7 +108,7 @@ static inline void __hyp_text ___activate_traps(struct kvm_vcpu *vcpu)write_sysreg_s (vcpu->arch.vsesr_el2,SYS_VSESR_EL2 ); } -static inline void __hyp_text ___deactivate_traps(struct kvm_vcpu *vcpu) +static inline void ___deactivate_traps (struct kvm_vcpu *vcpu) {/* * If we pended a virtual abort, preserve it until it gets
@@ -122,12 +122,12 @@ static inline void __hyp_text ___deactivate_traps(struct kvm_vcpu *vcpu) } } -static inline void __hyp_text __activate_vm(struct kvm *kvm) +static inline void __activate_vm (struct kvm *kvm) {__load_guest_stage2 (kvm); } -static inline bool __hyp_text __translate_far_to_hpfar(u64 far, u64 *hpfar) +static inline bool __translate_far_to_hpfar (u64 far,u64 *hpfar) {u64 par, tmp;
@@ -156,7 +156,7 @@ static inline bool __hyp_text __translate_far_to_hpfar(u64 far, u64 *hpfar)return true; } -static inline bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu) +static inline bool __populate_fault_info (struct kvm_vcpu *vcpu) {u8 ec;u64 esr;
@@ -196,7 +196,7 @@ static inline bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu) }/* Check for an FPSIMD/SVE trap and handle as appropriate */ -static inline bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) +static inline bool __hyp_handle_fpsimd (struct kvm_vcpu *vcpu) {bool vhe, sve_guest, sve_host;u8 hsr_ec;
@@ -283,7 +283,7 @@ static inline bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu)return true; } -static inline bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) +static inline bool handle_tx2_tvm (struct kvm_vcpu *vcpu) {u32 sysreg =esr_sys64_to_sysreg (kvm_vcpu_get_hsr (vcpu));int rt =kvm_vcpu_sys_get_rt (vcpu);
@@ -338,7 +338,7 @@ static inline bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu)return true; } -static inline bool __hyp_text esr_is_ptrauth_trap(u32 esr) +static inline bool esr_is_ptrauth_trap (u32 esr) {u32 ec =ESR_ELx_EC (esr);
@@ -371,7 +371,7 @@ static inline bool __hyp_text esr_is_ptrauth_trap(u32 esr) regs[key ## KEYHI_EL1] = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ }) -static inline bool __hyp_text __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) +static inline bool __hyp_handle_ptrauth (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *ctxt;u64 val;
@@ -401,7 +401,7 @@ static inline bool __hyp_text __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)* the guest, false when we should restore the host state and return to the * main run loop. */ -static inline bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool fixup_guest_exit (struct kvm_vcpu *vcpu,u64 *exit_code) {if (ARM_EXCEPTION_CODE (*exit_code) !=ARM_EXCEPTION_IRQ ) vcpu->arch.fault.esr_el2 =read_sysreg_el2 (SYS_ESR );
@@ -473,7 +473,7 @@ static inline bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_return false; } -static inline bool __hyp_text __needs_ssbd_off(struct kvm_vcpu *vcpu) +static inline bool __needs_ssbd_off (struct kvm_vcpu *vcpu) {if (!cpus_have_final_cap (ARM64_SSBD ))return false;
@@ -481,7 +481,7 @@ static inline bool __hyp_text __needs_ssbd_off(struct kvm_vcpu *vcpu)return !(vcpu->arch.workaround_flags &VCPU_WORKAROUND_2_FLAG ); } -static inline void __hyp_text __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu) +static inline void __set_guest_arch_workaround_state (struct kvm_vcpu *vcpu) {#ifdef CONFIG_ARM64_SSBD /*
@@ -494,7 +494,7 @@ static inline void __hyp_text __set_guest_arch_workaround_state(struct kvm_vcpu#endif } -static inline void __hyp_text __set_host_arch_workaround_state(struct kvm_vcpu *vcpu) +static inline void __set_host_arch_workaround_state (struct kvm_vcpu *vcpu) {#ifdef CONFIG_ARM64_SSBD /*
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +10 -10
@@ -15,18 +15,18 @@#include <asm/kvm_emulate.h> #include <asm/kvm_hyp.h> -static inline void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_save_common_state (struct kvm_cpu_context *ctxt) { ctxt->sys_regs[MDSCR_EL1 ] =read_sysreg (mdscr_el1); } -static inline void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_save_user_state (struct kvm_cpu_context *ctxt) { ctxt->sys_regs[TPIDR_EL0 ] =read_sysreg (tpidr_el0); ctxt->sys_regs[TPIDRRO_EL0 ] =read_sysreg (tpidrro_el0); } -static inline void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_save_el1_state (struct kvm_cpu_context *ctxt) { ctxt->sys_regs[CSSELR_EL1 ] =read_sysreg (csselr_el1); ctxt->sys_regs[SCTLR_EL1 ] =read_sysreg_el1 (SYS_SCTLR );
@@ -51,7 +51,7 @@ static inline void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ct ctxt->gp_regs.spsr[KVM_SPSR_EL1 ]=read_sysreg_el1 (SYS_SPSR ); } -static inline void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt) { ctxt->gp_regs.regs.pc =read_sysreg_el2 (SYS_ELR ); ctxt->gp_regs.regs.pstate =read_sysreg_el2 (SYS_SPSR );
@@ -60,18 +60,18 @@ static inline void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_cont ctxt->sys_regs[DISR_EL1 ] =read_sysreg_s (SYS_VDISR_EL2 ); } -static inline void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_common_state (struct kvm_cpu_context *ctxt) {write_sysreg (ctxt->sys_regs[MDSCR_EL1 ], mdscr_el1); } -static inline void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_user_state (struct kvm_cpu_context *ctxt) {write_sysreg (ctxt->sys_regs[TPIDR_EL0 ], tpidr_el0);write_sysreg (ctxt->sys_regs[TPIDRRO_EL0 ], tpidrro_el0); } -static inline void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_el1_state (struct kvm_cpu_context *ctxt) {write_sysreg (ctxt->sys_regs[MPIDR_EL1 ], vmpidr_el2);write_sysreg (ctxt->sys_regs[CSSELR_EL1 ], csselr_el1);
@@ -130,7 +130,7 @@ static inline void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_contextwrite_sysreg_el1 (ctxt->gp_regs.spsr[KVM_SPSR_EL1 ],SYS_SPSR ); } -static inline void __hyp_text __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_el2_return_state (struct kvm_cpu_context *ctxt) {u64 pstate = ctxt->gp_regs.regs.pstate;u64 mode = pstate &PSR_AA32_MODE_MASK ;
@@ -156,7 +156,7 @@ static inline void __hyp_text __sysreg_restore_el2_return_state(struct kvm_cpu_cwrite_sysreg_s (ctxt->sys_regs[DISR_EL1 ],SYS_VDISR_EL2 ); } -static inline void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) +static inline void __sysreg32_save_state (struct kvm_vcpu *vcpu) {u64 *spsr, *sysreg;
@@ -178,7 +178,7 @@ static inline void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) sysreg[DBGVCR32_EL2 ] =read_sysreg (dbgvcr32_el2); } -static inline void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) +static inline void __sysreg32_restore_state (struct kvm_vcpu *vcpu) {u64 *spsr, *sysreg;
arch/arm64/kvm/hyp/nvhe/Makefile +7 -1
@@ -21,7 +21,13 @@ $(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE $(call if_changed,hypcopy) quiet_cmd_hypcopy = HYPCOPY $@ - cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \ + --rename-section=.text=.hyp.text \ + $< $@ + +# Remove ftrace and Shadow Call Stack CFLAGS. +# This is equivalent to the 'notrace' and '__noscs' annotations. +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # KVM nVHE code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
arch/arm64/kvm/hyp/nvhe/debug-sr.c +5 -5
@@ -14,7 +14,7 @@#include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> -static void __hyp_text __debug_save_spe(u64 *pmscr_el1) +static void __debug_save_spe (u64 *pmscr_el1) {u64 reg;
@@ -46,7 +46,7 @@ static void __hyp_text __debug_save_spe(u64 *pmscr_el1)dsb (nsh); } -static void __hyp_text __debug_restore_spe(u64 pmscr_el1) +static void __debug_restore_spe (u64 pmscr_el1) {if (!pmscr_el1)return ;
@@ -58,20 +58,20 @@ static void __hyp_text __debug_restore_spe(u64 pmscr_el1)write_sysreg_s (pmscr_el1,SYS_PMSCR_EL1 ); } -void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu) +void __debug_switch_to_guest (struct kvm_vcpu *vcpu) {/* Disable and flush SPE data generation */ __debug_save_spe (&vcpu->arch.host_debug_state.pmscr_el1);__debug_switch_to_guest_common (vcpu); } -void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) +void __debug_switch_to_host (struct kvm_vcpu *vcpu) {__debug_restore_spe (vcpu->arch.host_debug_state.pmscr_el1);__debug_switch_to_host_common (vcpu); } -u32 __hyp_text __kvm_get_mdcr_el2(void) +u32 __kvm_get_mdcr_el2 (void ) {return read_sysreg (mdcr_el2); }
arch/arm64/kvm/hyp/nvhe/switch.c +9 -9
@@ -27,7 +27,7 @@#include <asm/processor.h> #include <asm/thread_info.h> -static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) +static void __activate_traps (struct kvm_vcpu *vcpu) {u64 val;
@@ -58,7 +58,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) } } -static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) +static void __deactivate_traps (struct kvm_vcpu *vcpu) {u64 mdcr_el2;
@@ -93,13 +93,13 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)write_sysreg (CPTR_EL2_DEFAULT , cptr_el2); } -static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) +static void __deactivate_vm (struct kvm_vcpu *vcpu) {write_sysreg (0 , vttbr_el2); }/* Save VGICv3 state on non-VHE systems */ -static void __hyp_text __hyp_vgic_save_state(struct kvm_vcpu *vcpu) +static void __hyp_vgic_save_state (struct kvm_vcpu *vcpu) {if (static_branch_unlikely (&kvm_vgic_global_state.gicv3_cpuif)) {__vgic_v3_save_state (&vcpu->arch.vgic_cpu.vgic_v3);
@@ -108,7 +108,7 @@ static void __hyp_text __hyp_vgic_save_state(struct kvm_vcpu *vcpu) }/* Restore VGICv3 state on non_VEH systems */ -static void __hyp_text __hyp_vgic_restore_state(struct kvm_vcpu *vcpu) +static void __hyp_vgic_restore_state (struct kvm_vcpu *vcpu) {if (static_branch_unlikely (&kvm_vgic_global_state.gicv3_cpuif)) {__vgic_v3_activate_traps (&vcpu->arch.vgic_cpu.vgic_v3);
@@ -119,7 +119,7 @@ static void __hyp_text __hyp_vgic_restore_state(struct kvm_vcpu *vcpu)/** * Disable host events, enable guest events */ -static bool __hyp_text __pmu_switch_to_guest(struct kvm_cpu_context *host_ctxt) +static bool __pmu_switch_to_guest (struct kvm_cpu_context *host_ctxt) {struct kvm_host_data *host;struct kvm_pmu_events *pmu;
@@ -139,7 +139,7 @@ static bool __hyp_text __pmu_switch_to_guest(struct kvm_cpu_context *host_ctxt)/** * Disable guest events, enable host events */ -static void __hyp_text __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt) +static void __pmu_switch_to_host (struct kvm_cpu_context *host_ctxt) {struct kvm_host_data *host;struct kvm_pmu_events *pmu;
@@ -155,7 +155,7 @@ static void __hyp_text __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt) }/* Switch to the guest for legacy non-VHE systems */ -int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu) +int __kvm_vcpu_run (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *host_ctxt;struct kvm_cpu_context *guest_ctxt;
@@ -242,7 +242,7 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)return exit_code; } -void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt) +void __noreturn hyp_panic (struct kvm_cpu_context *host_ctxt) {u64 spsr =read_sysreg_el2 (SYS_SPSR );u64 elr =read_sysreg_el2 (SYS_ELR );
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +3 -3
@@ -18,7 +18,7 @@* Non-VHE: Both host and guest must save everything. */ -void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt) +void __sysreg_save_state_nvhe (struct kvm_cpu_context *ctxt) {__sysreg_save_el1_state (ctxt);__sysreg_save_common_state (ctxt);
@@ -26,7 +26,7 @@ void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)__sysreg_save_el2_return_state (ctxt); } -void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt) +void __sysreg_restore_state_nvhe (struct kvm_cpu_context *ctxt) {__sysreg_restore_el1_state (ctxt);__sysreg_restore_common_state (ctxt);
@@ -34,7 +34,7 @@ void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)__sysreg_restore_el2_return_state (ctxt); } -void __hyp_text __kvm_enable_ssbs(void) +void __kvm_enable_ssbs (void ) {u64 tmp;
arch/arm64/kvm/hyp/nvhe/timer-sr.c +2 -2
@@ -19,7 +19,7 @@ void __kvm_timer_set_cntvoff(u64 cntvoff)* Should only be called on non-VHE systems. * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe(). */ -void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu) +void __timer_disable_traps (struct kvm_vcpu *vcpu) {u64 val;
@@ -33,7 +33,7 @@ void __hyp_text __timer_disable_traps(struct kvm_vcpu *vcpu)* Should only be called on non-VHE systems. * VHE systems use EL2 timers and configure EL1 timers in kvm_timer_init_vhe(). */ -void __hyp_text __timer_enable_traps(struct kvm_vcpu *vcpu) +void __timer_enable_traps (struct kvm_vcpu *vcpu) {u64 val;
arch/arm64/kvm/hyp/nvhe/tlb.c +6 -8
@@ -12,8 +12,7 @@ struct tlb_inv_context {u64 tcr; }; -static void __hyp_text __tlb_switch_to_guest(struct kvm *kvm, -struct tlb_inv_context *cxt) +static void __tlb_switch_to_guest (struct kvm *kvm,struct tlb_inv_context *cxt) {if (cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT )) {u64 val;
@@ -36,8 +35,7 @@ static void __hyp_text __tlb_switch_to_guest(struct kvm *kvm, asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT)); } -static void __hyp_text __tlb_switch_to_host(struct kvm *kvm, -struct tlb_inv_context *cxt) +static void __tlb_switch_to_host (struct kvm *kvm,struct tlb_inv_context *cxt) {write_sysreg (0 , vttbr_el2);
@@ -49,7 +47,7 @@ static void __hyp_text __tlb_switch_to_host(struct kvm *kvm, } } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +void __kvm_tlb_flush_vmid_ipa (struct kvm *kvm,phys_addr_t ipa) {struct tlb_inv_context cxt;
@@ -103,7 +101,7 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa)__tlb_switch_to_host (kvm, &cxt); } -void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) +void __kvm_tlb_flush_vmid (struct kvm *kvm) {struct tlb_inv_context cxt;
@@ -120,7 +118,7 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm)__tlb_switch_to_host (kvm, &cxt); } -void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) +void __kvm_tlb_flush_local_vmid (struct kvm_vcpu *vcpu) {struct kvm *kvm =kern_hyp_va (kern_hyp_va (vcpu)->kvm);struct tlb_inv_context cxt;
@@ -135,7 +133,7 @@ void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu)__tlb_switch_to_host (kvm, &cxt); } -void __hyp_text __kvm_flush_vm_context(void) +void __kvm_flush_vm_context (void ) {dsb (ishst);__tlbi (alle1is);
arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c +2 -2
@@ -13,7 +13,7 @@#include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> -static bool __hyp_text __is_be(struct kvm_vcpu *vcpu) +static bool __is_be (struct kvm_vcpu *vcpu) {if (vcpu_mode_is_32bit (vcpu))return !!(read_sysreg_el2 (SYS_SPSR ) &PSR_AA32_E_BIT );
@@ -32,7 +32,7 @@ static bool __hyp_text __is_be(struct kvm_vcpu *vcpu)* 0: Not a GICV access * -1: Illegal GICV access successfully performed */ -int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu) +int __vgic_v2_perform_cpuif_access (struct kvm_vcpu *vcpu) {struct kvm *kvm =kern_hyp_va (vcpu->kvm);struct vgic_dist *vgic = &kvm->arch.vgic;
arch/arm64/kvm/hyp/vgic-v3-sr.c +58 -72
434 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 27/56 KVM: arm64: Lift instrumentation restrictions on VHE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
With VHE and nVHE executable code completely separated, remove build config that disabled GCOV/KASAN/UBSAN/KCOV instrumentation for VHE as these now execute under the same memory mappings as the rest of the kernel.
No violations are currently being reported by either KASAN or UBSAN.
--- arch/arm64/kvm/hyp/vhe/Makefile | 8 -------- 1 file changed, 8 deletions(-)
arch/arm64/kvm/hyp/vhe/Makefile +0 -8
@@ -9,11 +9,3 @@ ccflags-y := -D__KVM_VHE_HYPERVISOR__ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o - -# KVM code is run at a different exception code with a different map, so -# compiler instrumentation that inserts callbacks or checks into the code may -# cause crashes. Just disable it. -GCOV_PROFILE := n -KASAN_SANITIZE := n -UBSAN_SANITIZE := n -KCOV_INSTRUMENT := n
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 28/56 KVM: arm64: vgic-its: Change default outer cacheability for {PEND, PROP}BASER
In reply to Marc ZyngierFrom: Alexander Graf <[email protected]>
PENDBASER and PROPBASER define the outer caching mode for LPI tables. The memory backing them may not be outer sharable, so we mark them as nC by default. This however, breaks Windows on ARM which only accepts SameAsInner or RaWaWb as values for outer cachability.
We do today already allow the outer mode to be set to SameAsInner explicitly, so the easy fix is to default to that instead of nC for situations when an OS asks for a not fulfillable cachability request.
This fixes booting Windows in KVM with vgicv3 and ITS enabled for me.
--- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-mmio-v3.c +1 -1
@@ -389,7 +389,7 @@ u64 vgic_sanitise_outer_cacheability(u64 field)case GIC_BASER_CACHE_nC:return field;default : -return GIC_BASER_CACHE_nC; +return GIC_BASER_CACHE_SameAsInner; } }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 29/56 KVM: arm64: Drop long gone function parameter documentation
In reply to Marc ZyngierFrom: Peng Hao <[email protected]>
update_vmid() just has one parameter "vmid".The other parameter "kvm" is no longer used.
--- arch/arm64/kvm/arm.c | 1 - 1 file changed, 1 deletion(-)
arch/arm64/kvm/arm.c +0 -1
@@ -466,7 +466,6 @@ static bool need_new_vmid_gen(struct kvm_vmid *vmid)/** * update_vmid - Update the vmid with a valid VMID for the current generation -* @kvm: The guest that struct vmid belongs to * @vmid: The stage-2 VMID information struct */ static void update_vmid (struct kvm_vmid *vmid)
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 30/56 KVM: arm64: Rename HSR to ESR
In reply to Marc ZyngierFrom: Gavin Shan <[email protected]>
kvm/arm32 isn't supported since commit 541ad0150ca4 ("arm: Remove 32bit KVM host support"). So HSR isn't meaningful since then. This renames HSR to ESR accordingly. This shouldn't cause any functional changes:
* Rename kvm_vcpu_get_hsr() to kvm_vcpu_get_esr() to make the function names self-explanatory.
* Rename variables from @hsr to @esr to make them self-explanatory.
Note that the renaming on uapi and tracepoint will cause ABI changes, which we should avoid. Specificly, there are 4 related source files in this regard:
* arch/arm64/include/uapi/asm/kvm.h (struct kvm_debug_exit_arch::hsr) * arch/arm64/kvm/handle_exit.c (struct kvm_debug_exit_arch::hsr) * arch/arm64/kvm/trace_arm.h (tracepoints)
* arch/arm64/kvm/trace_handle_exit.h (tracepoints)
--- arch/arm64/include/asm/kvm_emulate.h | 34 ++++++++++++++-------------- arch/arm64/kvm/handle_exit.c | 32 +++++++++++++------------- arch/arm64/kvm/hyp/aarch32.c | 2 +- arch/arm64/kvm/hyp/switch.c | 14 ++++++------ arch/arm64/kvm/hyp/vgic-v3-sr.c | 4 ++-- arch/arm64/kvm/mmu.c | 6 ++--- arch/arm64/kvm/sys_regs.c | 28 +++++++++++------------ 7 files changed, 60 insertions(+), 60 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +17 -17
@@ -259,14 +259,14 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)return mode != PSR_MODE_EL0t; } -static __always_inline u32 kvm_vcpu_get_hsr(const struct kvm_vcpu *vcpu) +static __always_inline u32 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu) {return vcpu->arch.fault.esr_el2; } static __always_inline int kvm_vcpu_get_condition(const struct kvm_vcpu *vcpu) { -u32 esr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);if (esr & ESR_ELx_CV)return (esr & ESR_ELx_COND_MASK) >> ESR_ELx_COND_SHIFT;
@@ -291,64 +291,64 @@ static inline u64 kvm_vcpu_get_disr(const struct kvm_vcpu *vcpu)static inline u32 kvm_vcpu_hvc_get_imm (const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_hsr (vcpu) & ESR_ELx_xVC_IMM_MASK; +return kvm_vcpu_get_esr (vcpu) & ESR_ELx_xVC_IMM_MASK; } static __always_inline bool kvm_vcpu_dabt_isvalid(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_ISV); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_ISV); }static inline unsigned long kvm_vcpu_dabt_iss_nisv_sanitized (const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_hsr (vcpu) & (ESR_ELx_CM | ESR_ELx_WNR | ESR_ELx_FSC); +return kvm_vcpu_get_esr (vcpu) & (ESR_ELx_CM | ESR_ELx_WNR | ESR_ELx_FSC); }static inline bool kvm_vcpu_dabt_issext (const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_SSE); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_SSE); }static inline bool kvm_vcpu_dabt_issf (const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_SF); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_SF); } static __always_inline int kvm_vcpu_dabt_get_rd(const struct kvm_vcpu *vcpu) { -return (kvm_vcpu_get_hsr (vcpu) & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; +return (kvm_vcpu_get_esr (vcpu) & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; } static __always_inline bool kvm_vcpu_dabt_iss1tw(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_S1PTW); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_S1PTW); } static __always_inline bool kvm_vcpu_dabt_iswrite(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_WNR) || +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_WNR) ||kvm_vcpu_dabt_iss1tw (vcpu);/* AF/DBM update */ }static inline bool kvm_vcpu_dabt_is_cm (const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_CM); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_CM); } static __always_inline unsigned int kvm_vcpu_dabt_get_as(const struct kvm_vcpu *vcpu) { -return 1 << ((kvm_vcpu_get_hsr (vcpu) & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); +return 1 << ((kvm_vcpu_get_esr (vcpu) & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); }/* This one is not specific to Data Abort */ static __always_inline bool kvm_vcpu_trap_il_is32bit(const struct kvm_vcpu *vcpu) { -return !!(kvm_vcpu_get_hsr (vcpu) & ESR_ELx_IL); +return !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_IL); } static __always_inline u8 kvm_vcpu_trap_get_class(const struct kvm_vcpu *vcpu) { -return ESR_ELx_EC (kvm_vcpu_get_hsr (vcpu)); +return ESR_ELx_EC (kvm_vcpu_get_esr (vcpu)); }static inline bool kvm_vcpu_trap_is_iabt (const struct kvm_vcpu *vcpu)
@@ -358,12 +358,12 @@ static inline bool kvm_vcpu_trap_is_iabt(const struct kvm_vcpu *vcpu) static __always_inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_hsr (vcpu) & ESR_ELx_FSC; +return kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC; } static __always_inline u8 kvm_vcpu_trap_get_fault_type(const struct kvm_vcpu *vcpu) { -return kvm_vcpu_get_hsr (vcpu) & ESR_ELx_FSC_TYPE; +return kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC_TYPE; } static __always_inline bool kvm_vcpu_dabt_isextabt(const struct kvm_vcpu *vcpu)
@@ -387,7 +387,7 @@ static __always_inline bool kvm_vcpu_dabt_isextabt(const struct kvm_vcpu *vcpu) static __always_inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu) { -u32 esr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);return ESR_ELx_SYS64_ISS_RT (esr); }
arch/arm64/kvm/handle_exit.c +16 -16
@@ -89,7 +89,7 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu, struct kvm_run *run)*/ static int kvm_handle_wfx (struct kvm_vcpu *vcpu,struct kvm_run *run) { -if (kvm_vcpu_get_hsr (vcpu) & ESR_ELx_WFx_ISS_WFE) { +if (kvm_vcpu_get_esr (vcpu) & ESR_ELx_WFx_ISS_WFE) {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), true); vcpu->stat.wfe_exit_stat++;kvm_vcpu_on_spin (vcpu,vcpu_mode_priv (vcpu));
@@ -119,13 +119,13 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run)*/ static int kvm_handle_guest_debug (struct kvm_vcpu *vcpu,struct kvm_run *run) { -u32 hsr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);int ret =0 ; run->exit_reason =KVM_EXIT_DEBUG ; - run->debug.arch.hsr = hsr; + run->debug.arch.hsr = esr; -switch (ESR_ELx_EC (hsr)) { +switch (ESR_ELx_EC (esr)) {case ESR_ELx_EC_WATCHPT_LOW: run->debug.arch.far = vcpu->arch.fault.far_el2;/* fall through */
@@ -135,8 +135,8 @@ static int kvm_handle_guest_debug(struct kvm_vcpu *vcpu, struct kvm_run *run)case ESR_ELx_EC_BRK64:break ;default : -kvm_err ("%s: un-handled case hsr: %#08x\n" , - __func__, (unsigned int ) hsr); +kvm_err ("%s: un-handled case esr: %#08x\n" , + __func__, (unsigned int ) esr); ret =-1 ;break ; }
@@ -146,10 +146,10 @@ static int kvm_handle_guest_debug(struct kvm_vcpu *vcpu, struct kvm_run *run)static int kvm_handle_unknown_ec (struct kvm_vcpu *vcpu,struct kvm_run *run) { -u32 hsr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu); -kvm_pr_unimpl ("Unknown exception class: hsr: %#08x -- %s\n" , - hsr,esr_get_class_string (hsr)); +kvm_pr_unimpl ("Unknown exception class: esr: %#08x -- %s\n" , + esr,esr_get_class_string (esr));kvm_inject_undefined (vcpu);return 1 ;
@@ -200,10 +200,10 @@ static exit_handle_fn arm_exit_handlers[] = {static exit_handle_fn kvm_get_exit_handler (struct kvm_vcpu *vcpu) { -u32 hsr =kvm_vcpu_get_hsr (vcpu); -u8 hsr_ec =ESR_ELx_EC (hsr); +u32 esr =kvm_vcpu_get_esr (vcpu); +u8 esr_ec =ESR_ELx_EC (esr); -return arm_exit_handlers[hsr_ec]; +return arm_exit_handlers[esr_ec]; }/*
@@ -241,15 +241,15 @@ int handle_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, int exception_index) {if (ARM_SERROR_PENDING (exception_index)) { -u8 hsr_ec =ESR_ELx_EC (kvm_vcpu_get_hsr (vcpu)); +u8 esr_ec =ESR_ELx_EC (kvm_vcpu_get_esr (vcpu));/* * HVC/SMC already have an adjusted PC, which we need * to correct in order to return to after having * injected the SError. */ -if (hsr_ec == ESR_ELx_EC_HVC32 || hsr_ec == ESR_ELx_EC_HVC64 || - hsr_ec == ESR_ELx_EC_SMC32 || hsr_ec == ESR_ELx_EC_SMC64) { +if (esr_ec == ESR_ELx_EC_HVC32 || esr_ec == ESR_ELx_EC_HVC64 || + esr_ec == ESR_ELx_EC_SMC32 || esr_ec == ESR_ELx_EC_SMC64) {u32 adj =kvm_vcpu_trap_il_is32bit (vcpu) ?4 :2 ; *vcpu_pc (vcpu) -= adj; }
@@ -307,5 +307,5 @@ void handle_exit_early(struct kvm_vcpu *vcpu, struct kvm_run *run, exception_index =ARM_EXCEPTION_CODE (exception_index);if (exception_index ==ARM_EXCEPTION_EL1_SERROR ) -kvm_handle_guest_serror (vcpu,kvm_vcpu_get_hsr (vcpu)); +kvm_handle_guest_serror (vcpu,kvm_vcpu_get_esr (vcpu)); }
arch/arm64/kvm/hyp/aarch32.c +1 -1
@@ -51,7 +51,7 @@ bool __hyp_text kvm_condition_valid32(const struct kvm_vcpu *vcpu)int cond;/* Top two bits non-zero? Unconditional. */ -if (kvm_vcpu_get_hsr (vcpu) >>30 ) +if (kvm_vcpu_get_esr (vcpu) >>30 )return true;/* Is condition field valid? */
arch/arm64/kvm/hyp/switch.c +7 -7
@@ -356,7 +356,7 @@ static bool __hyp_text __populate_fault_info(struct kvm_vcpu *vcpu) static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) {bool vhe, sve_guest, sve_host; -u8 hsr_ec; +u8 esr_ec;if (!system_supports_fpsimd ())return false;
@@ -371,14 +371,14 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) vhe =has_vhe (); } - hsr_ec =kvm_vcpu_trap_get_class (vcpu); -if (hsr_ec != ESR_ELx_EC_FP_ASIMD && - hsr_ec != ESR_ELx_EC_SVE) + esr_ec =kvm_vcpu_trap_get_class (vcpu); +if (esr_ec != ESR_ELx_EC_FP_ASIMD && + esr_ec != ESR_ELx_EC_SVE)return false;/* Don't handle SVE traps for non-SVE vcpus here: */ if (!sve_guest) -if (hsr_ec != ESR_ELx_EC_FP_ASIMD) +if (esr_ec != ESR_ELx_EC_FP_ASIMD)return false;/* Valid trap. Switch the context: */
@@ -437,7 +437,7 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) { -u32 sysreg =esr_sys64_to_sysreg (kvm_vcpu_get_hsr (vcpu)); +u32 sysreg =esr_sys64_to_sysreg (kvm_vcpu_get_esr (vcpu));int rt =kvm_vcpu_sys_get_rt (vcpu);u64 val =vcpu_get_reg (vcpu, rt);
@@ -529,7 +529,7 @@ static bool __hyp_text __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)u64 val;if (!vcpu_has_ptrauth (vcpu) || - !esr_is_ptrauth_trap (kvm_vcpu_get_hsr (vcpu))) + !esr_is_ptrauth_trap (kvm_vcpu_get_esr (vcpu)))return false; ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt;
arch/arm64/kvm/hyp/vgic-v3-sr.c +2 -2
@@ -426,7 +426,7 @@ static int __hyp_text __vgic_v3_bpr_min(void) static int __hyp_text __vgic_v3_get_group(struct kvm_vcpu *vcpu) { -u32 esr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);u8 crm = (esr & ESR_ELx_SYS64_ISS_CRM_MASK) >> ESR_ELx_SYS64_ISS_CRM_SHIFT;return crm !=8 ;
@@ -992,7 +992,7 @@ int __hyp_text __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu)bool is_read;u32 sysreg; - esr =kvm_vcpu_get_hsr (vcpu); + esr =kvm_vcpu_get_esr (vcpu);if (vcpu_mode_is_32bit (vcpu)) {if (!kvm_condition_valid (vcpu)) {__kvm_skip_instr (vcpu);
arch/arm64/kvm/mmu.c +3 -3
@@ -2079,7 +2079,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)* For RAS the host kernel may handle this abort. * There is no need to pass the error into the guest. */ -if (!kvm_handle_guest_sea (fault_ipa,kvm_vcpu_get_hsr (vcpu))) +if (!kvm_handle_guest_sea (fault_ipa,kvm_vcpu_get_esr (vcpu)))return 1 ;if (unlikely (!is_iabt)) {
@@ -2088,7 +2088,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) } } -trace_kvm_guest_fault (*vcpu_pc (vcpu),kvm_vcpu_get_hsr (vcpu), +trace_kvm_guest_fault (*vcpu_pc (vcpu),kvm_vcpu_get_esr (vcpu),kvm_vcpu_get_hfar (vcpu), fault_ipa);/* Check the stage-2 fault is trans. fault or write fault */
@@ -2097,7 +2097,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)kvm_err ("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n" ,kvm_vcpu_trap_get_class (vcpu), (unsigned long )kvm_vcpu_trap_get_fault (vcpu), - (unsigned long )kvm_vcpu_get_hsr (vcpu)); + (unsigned long )kvm_vcpu_get_esr (vcpu));return -EFAULT ; }
arch/arm64/kvm/sys_regs.c +14 -14
@@ -2220,10 +2220,10 @@ static int emulate_cp(struct kvm_vcpu *vcpu,static void unhandled_cp_access (struct kvm_vcpu *vcpu,struct sys_reg_params *params) { -u8 hsr_ec =kvm_vcpu_trap_get_class (vcpu); +u8 esr_ec =kvm_vcpu_trap_get_class (vcpu);int cp =-1 ; -switch (hsr_ec) { +switch (esr_ec) {case ESR_ELx_EC_CP15_32:case ESR_ELx_EC_CP15_64: cp =15 ;
@@ -2254,17 +2254,17 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, size_t nr_specific) {struct sys_reg_params params; -u32 hsr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);int Rt =kvm_vcpu_sys_get_rt (vcpu); -int Rt2 = (hsr >>10 ) &0x1f ; +int Rt2 = (esr >>10 ) &0x1f ; params.is_aarch32 = true; params.is_32bit = false; - params.CRm = (hsr >>1 ) &0xf ; - params.is_write = ((hsr &1 ) ==0 ); + params.CRm = (esr >>1 ) &0xf ; + params.is_write = ((esr &1 ) ==0 ); params.Op0 =0 ; - params.Op1 = (hsr >>16 ) &0xf ; + params.Op1 = (esr >>16 ) &0xf ; params.Op2 =0 ; params.CRn =0 ;
@@ -2311,18 +2311,18 @@ static int kvm_handle_cp_32(struct kvm_vcpu *vcpu, size_t nr_specific) {struct sys_reg_params params; -u32 hsr =kvm_vcpu_get_hsr (vcpu); +u32 esr =kvm_vcpu_get_esr (vcpu);int Rt =kvm_vcpu_sys_get_rt (vcpu); params.is_aarch32 = true; params.is_32bit = true; - params.CRm = (hsr >>1 ) &0xf ; + params.CRm = (esr >>1 ) &0xf ; params.regval =vcpu_get_reg (vcpu, Rt); - params.is_write = ((hsr &1 ) ==0 ); - params.CRn = (hsr >>10 ) &0xf ; + params.is_write = ((esr &1 ) ==0 ); + params.CRn = (esr >>10 ) &0xf ; params.Op0 =0 ; - params.Op1 = (hsr >>14 ) &0x7 ; - params.Op2 = (hsr >>17 ) &0x7 ; + params.Op1 = (esr >>14 ) &0x7 ; + params.Op2 = (esr >>17 ) &0x7 ;if (!emulate_cp (vcpu, ¶ms, target_specific, nr_specific) || !emulate_cp (vcpu, ¶ms, global, nr_global)) {
@@ -2421,7 +2421,7 @@ static void reset_sys_reg_descs(struct kvm_vcpu *vcpu,int kvm_handle_sys_reg (struct kvm_vcpu *vcpu,struct kvm_run *run) {struct sys_reg_params params; -unsigned long esr =kvm_vcpu_get_hsr (vcpu); +unsigned long esr =kvm_vcpu_get_esr (vcpu);int Rt =kvm_vcpu_sys_get_rt (vcpu);int ret;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 31/56 arm64: Detect the ARMv8.4 TTL feature
In reply to Marc ZyngierIn order to reduce the cost of TLB invalidation, the ARMv8.4 TTL feature allows TLBs to be issued with a level allowing for quicker invalidation.
Let's detect the feature for now. Further patches will implement its actual usage.
--- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 3 files changed, 14 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpucaps.h +2 -1
@@ -62,7 +62,8 @@#define ARM64_HAS_GENERIC_AUTH 52#define ARM64_HAS_32BIT_EL1 53#define ARM64_BTI 54 +#define ARM64_HAS_ARMv8_4_TTL 55 -#define ARM64_NCAPS 55 +#define ARM64_NCAPS 56#endif /* __ASM_CPUCAPS_H */
arch/arm64/include/asm/sysreg.h +1 -0
@@ -746,6 +746,7 @@/* id_aa64mmfr2 */ #define ID_AA64MMFR2_E0PD_SHIFT 60 +#define ID_AA64MMFR2_TTL_SHIFT 48#define ID_AA64MMFR2_FWB_SHIFT 40#define ID_AA64MMFR2_AT_SHIFT 32#define ID_AA64MMFR2_LVA_SHIFT 16
arch/arm64/kernel/cpufeature.c +11 -0
@@ -323,6 +323,7 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr1[] = {static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = {ARM64_FTR_BITS (FTR_HIDDEN ,FTR_NONSTRICT ,FTR_LOWER_SAFE ,ID_AA64MMFR2_E0PD_SHIFT ,4 ,0 ), +ARM64_FTR_BITS (FTR_HIDDEN ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64MMFR2_TTL_SHIFT ,4 ,0 ),ARM64_FTR_BITS (FTR_HIDDEN ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64MMFR2_FWB_SHIFT ,4 ,0 ),ARM64_FTR_BITS (FTR_VISIBLE ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64MMFR2_AT_SHIFT ,4 ,0 ),ARM64_FTR_BITS (FTR_HIDDEN ,FTR_STRICT ,FTR_LOWER_SAFE ,ID_AA64MMFR2_LVA_SHIFT ,4 ,0 ),
@@ -1882,6 +1883,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .cpu_enable = cpu_has_fwb, }, + { + .desc = "ARMv8.4 Translation Table Level", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_ARMv8_4_TTL, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_TTL_SHIFT, + .min_field_value = 1, + .matches = has_cpuid_feature, + }, #ifdef CONFIG_ARM64_HW_AFDBM { /*
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 32/56 arm64: Document SW reserved PTE/PMD bits in Stage-2 descriptors
In reply to Marc ZyngierAdvertise bits [58:55] as reserved for SW in the S2 descriptors.
--- arch/arm64/include/asm/pgtable-hwdef.h | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/include/asm/pgtable-hwdef.h +2 -0
@@ -178,10 +178,12 @@#define PTE_S2_RDONLY (_AT(pteval_t, 1) << 6)/* HAP[2:1] */ #define PTE_S2_RDWR (_AT(pteval_t, 3) << 6)/* HAP[2:1] */ #define PTE_S2_XN (_AT(pteval_t, 2) << 53)/* XN[1:0] */ +#define PTE_S2_SW_RESVD (_AT(pteval_t, 15) << 55)/* Reserved for SW */ #define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6)/* HAP[2:1] */ #define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6)/* HAP[2:1] */ #define PMD_S2_XN (_AT(pmdval_t, 2) << 53)/* XN[1:0] */ +#define PMD_S2_SW_RESVD (_AT(pmdval_t, 15) << 55)/* Reserved for SW */ #define PUD_S2_RDONLY (_AT(pudval_t, 1) << 6)/* HAP[2:1] */ #define PUD_S2_RDWR (_AT(pudval_t, 3) << 6)/* HAP[2:1] */
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 33/56 arm64: Add level-hinted TLB invalidation helper
In reply to Marc ZyngierAdd a level-hinted TLB invalidation helper that only gets used if ARMv8.4-TTL gets detected.
--- arch/arm64/include/asm/stage2_pgtable.h | 9 +++++ arch/arm64/include/asm/tlbflush.h | 45 +++++++++++++++++++++++++ 2 files changed, 54 insertions(+)
arch/arm64/include/asm/stage2_pgtable.h +9 -0
@@ -256,4 +256,13 @@ stage2_pgd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) return (boundary - 1 < end - 1) ? boundary : end; } +/* +* Level values for the ARMv8.4-TTL extension, mapping PUD/PMD/PTE and +* the architectural page-table level. +*/ +#define S2_NO_LEVEL_HINT 0 +#define S2_PUD_LEVEL 1 +#define S2_PMD_LEVEL 2 +#define S2_PTE_LEVEL 3 +#endif /* __ARM64_S2_PGTABLE_H_ */
arch/arm64/include/asm/tlbflush.h +45 -0
57 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 34/56 KVM: arm64: Factor out stage 2 page table data from struct kvm
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
As we are about to reuse our stage 2 page table manipulation code for shadow stage 2 page tables in the context of nested virtualization, we are going to manage multiple stage 2 page tables for a single VM.
This requires some pretty invasive changes to our data structures, which moves the vmid and pgd pointers into a separate structure and change pretty much all of our mmu code to operate on this structure instead.
The new structure is called struct kvm_s2_mmu.
There is no intended functional change by this patch alone.
--- arch/arm64/include/asm/kvm_asm.h | 7 +- arch/arm64/include/asm/kvm_host.h | 32 ++- arch/arm64/include/asm/kvm_mmu.h | 16 +- arch/arm64/kvm/arm.c | 36 +-- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 33 ++- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/tlb.c | 26 +-- arch/arm64/kvm/mmu.c | 280 +++++++++++++----------- 10 files changed, 240 insertions(+), 198 deletions(-)
arch/arm64/include/asm/kvm_asm.h +4 -3
@@ -77,6 +77,7 @@struct kvm ;struct kvm_vcpu ; +struct kvm_s2_mmu ;DECLARE_KVM_NVHE_SYM (__kvm_hyp_init);DECLARE_KVM_HYP_SYM (__kvm_hyp_vector);
@@ -90,9 +91,9 @@ DECLARE_KVM_HYP_SYM(__bp_harden_hyp_vecs);#endif extern void __kvm_flush_vm_context (void ); -extern void __kvm_tlb_flush_vmid_ipa (struct kvm *kvm,phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid (struct kvm *kvm); -extern void __kvm_tlb_flush_local_vmid (struct kvm_vcpu *vcpu); +extern void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu);extern void __kvm_timer_set_cntvoff (u64 cntvoff);
arch/arm64/include/asm/kvm_host.h +25 -7
@@ -66,19 +66,34 @@ struct kvm_vmid {u32 vmid; }; -struct kvm_arch { +struct kvm_s2_mmu {struct kvm_vmid vmid; -/* stage2 entry level table */ -pgd_t *pgd; -phys_addr_t pgd_phys; - -/* VTCR_EL2 value for this VM */ -u64 vtcr; +/* +* stage2 entry level table +* +* Two kvm_s2_mmu structures in the same VM can point to the same +* pgd here. This happens when running a guest using a +* translation regime that isn't affected by its own stage-2 +* translation, such as a non-VHE hypervisor running at vEL2, or +* for vEL1/EL0 with vHCR_EL2.VM == 0. In that case, we use the +* canonical stage-2 page tables. +*/ +pgd_t *pgd; +phys_addr_t pgd_phys;/* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; +struct kvm *kvm; +}; + +struct kvm_arch { +struct kvm_s2_mmu mmu; + +/* VTCR_EL2 value for this VM */ +u64 vtcr; +/* The maximum number of vCPUs depends on the used GIC model */ int max_vcpus;
@@ -254,6 +269,9 @@ struct kvm_vcpu_arch {void *sve_state;unsigned int sve_max_vl; +/* Stage 2 paging state used by the hardware on next switch */ +struct kvm_s2_mmu *hw_mmu; +/* HYP configuration */ u64 hcr_el2;u32 mdcr_el2;
arch/arm64/include/asm/kvm_mmu.h +8 -8
@@ -134,8 +134,8 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,void free_hyp_pgds (void );void stage2_unmap_vm (struct kvm *kvm); -int kvm_alloc_stage2_pgd (struct kvm *kvm); -void kvm_free_stage2_pgd (struct kvm *kvm); +int kvm_init_stage2_mmu (struct kvm *kvm,struct kvm_s2_mmu *mmu); +void kvm_free_stage2_pgd (struct kvm_s2_mmu *mmu);int kvm_phys_addr_ioremap (struct kvm *kvm,phys_addr_t guest_ipa,phys_addr_t pa,unsigned long size,bool writable);
@@ -577,13 +577,13 @@ static inline u64 kvm_vttbr_baddr_mask(struct kvm *kvm)return vttbr_baddr_mask (kvm_phys_shift (kvm),kvm_stage2_levels (kvm)); } -static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) +static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) { -struct kvm_vmid *vmid = &kvm->arch.vmid; +struct kvm_vmid *vmid = &mmu->vmid;u64 vmid_field, baddr;u64 cnp =system_supports_cnp () ?VTTBR_CNP_BIT :0 ; - baddr = kvm->arch.pgd_phys; + baddr = mmu->pgd_phys; vmid_field = (u64 )vmid->vmid <<VTTBR_VMID_SHIFT ;return kvm_phys_to_vttbr (baddr) | vmid_field | cnp; }
@@ -592,10 +592,10 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm)* Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_guest_stage2(struct kvm *kvm) +static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) { -write_sysreg (kvm->arch.vtcr, vtcr_el2); -write_sysreg (kvm_get_vttbr (kvm), vttbr_el2); +write_sysreg (kern_hyp_va (mmu->kvm)->arch.vtcr, vtcr_el2); +write_sysreg (kvm_get_vttbr (mmu), vttbr_el2);/* * ARM errata 1165522 and 1530923 require the actual execution of the
arch/arm64/kvm/arm.c +12 -24
@@ -106,22 +106,15 @@ static int kvm_arm_default_max_vcpus(void)*/ int kvm_arch_init_vm (struct kvm *kvm,unsigned long type) { -int ret, cpu; +int ret; ret =kvm_arm_setup_stage2 (kvm, type);if (ret)return ret; - kvm->arch.last_vcpu_ran =alloc_percpu (typeof (*kvm->arch.last_vcpu_ran)); -if (!kvm->arch.last_vcpu_ran) -return -ENOMEM ; - -for_each_possible_cpu (cpu) - *per_cpu_ptr (kvm->arch.last_vcpu_ran, cpu) =-1 ; - - ret =kvm_alloc_stage2_pgd (kvm); + ret =kvm_init_stage2_mmu (kvm, &kvm->arch.mmu);if (ret) - goto out_fail_alloc; +return ret; ret =create_hyp_mappings (kvm, kvm +1 ,PAGE_HYP );if (ret)
@@ -129,18 +122,12 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)kvm_vgic_early_init (kvm); -/* Mark the initial VMID generation invalid */ - kvm->arch.vmid.vmid_gen =0 ; -/* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus =kvm_arm_default_max_vcpus ();return ret; out_free_stage2_pgd: -kvm_free_stage2_pgd (kvm); -out_fail_alloc: -free_percpu (kvm->arch.last_vcpu_ran); - kvm->arch.last_vcpu_ran =NULL ; +kvm_free_stage2_pgd (&kvm->arch.mmu);return ret; }
@@ -160,9 +147,6 @@ void kvm_arch_destroy_vm(struct kvm *kvm)kvm_vgic_destroy (kvm); -free_percpu (kvm->arch.last_vcpu_ran); - kvm->arch.last_vcpu_ran =NULL ; -for (i =0 ; i <KVM_MAX_VCPUS ; ++i) {if (kvm->vcpus[i]) {kvm_vcpu_destroy (kvm->vcpus[i]);
@@ -279,6 +263,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)kvm_arm_pvtime_vcpu_init (&vcpu->arch); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + err =kvm_vgic_vcpu_init (vcpu);if (err)return err;
@@ -334,16 +320,18 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu)void kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu) { +struct kvm_s2_mmu *mmu;int *last_ran; - last_ran =this_cpu_ptr (vcpu->kvm->arch.last_vcpu_ran); + mmu = vcpu->arch.hw_mmu; + last_ran =this_cpu_ptr (mmu->last_vcpu_ran);/* * We might get preempted before the vCPU actually runs, but * over-invalidation doesn't affect correctness. */ if (*last_ran != vcpu->vcpu_id) { -kvm_call_hyp (__kvm_tlb_flush_local_vmid, vcpu); +kvm_call_hyp (__kvm_tlb_flush_local_vmid, mmu); *last_ran = vcpu->vcpu_id; }
@@ -680,7 +668,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)*/ cond_resched (); -update_vmid (&vcpu->kvm->arch.vmid); +update_vmid (&vcpu->arch.hw_mmu->vmid);check_vcpu_requests (vcpu);
@@ -729,7 +717,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)*/ smp_store_mb (vcpu->mode,IN_GUEST_MODE ); -if (ret <=0 ||need_new_vmid_gen (&vcpu->kvm->arch.vmid) || +if (ret <=0 ||need_new_vmid_gen (&vcpu->arch.hw_mmu->vmid) ||kvm_request_pending (vcpu)) { vcpu->mode =OUTSIDE_GUEST_MODE ;isb ();/* Ensure work in x_flush_hwstate is committed */
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -2
@@ -122,9 +122,9 @@ static inline void ___deactivate_traps(struct kvm_vcpu *vcpu) } } -static inline void __activate_vm (struct kvm *kvm) +static inline void __activate_vm (struct kvm_s2_mmu *mmu) { -__load_guest_stage2 (kvm); +__load_guest_stage2 (mmu); }static inline bool __translate_far_to_hpfar (u64 far,u64 *hpfar)
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -194,7 +194,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)__sysreg32_restore_state (vcpu);__sysreg_restore_state_nvhe (guest_ctxt); -__activate_vm (kern_hyp_va (vcpu->kvm)); +__activate_vm (kern_hyp_va (vcpu->arch.hw_mmu));__activate_traps (vcpu);__hyp_vgic_restore_state (vcpu);
arch/arm64/kvm/hyp/nvhe/tlb.c +16 -17
@@ -12,7 +12,8 @@ struct tlb_inv_context {u64 tcr; }; -static void __tlb_switch_to_guest (struct kvm *kvm,struct tlb_inv_context *cxt) +static void __tlb_switch_to_guest (struct kvm_s2_mmu *mmu, +struct tlb_inv_context *cxt) {if (cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT )) {u64 val;
@@ -30,12 +31,10 @@ static void __tlb_switch_to_guest(struct kvm *kvm, struct tlb_inv_context *cxt)isb (); } -/* __load_guest_stage2() includes an ISB for the workaround. */ -__load_guest_stage2 (kvm); - asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT)); +__load_guest_stage2 (mmu); } -static void __tlb_switch_to_host (struct kvm *kvm,struct tlb_inv_context *cxt) +static void __tlb_switch_to_host (struct tlb_inv_context *cxt) {write_sysreg (0 , vttbr_el2);
@@ -47,15 +46,15 @@ static void __tlb_switch_to_host(struct kvm *kvm, struct tlb_inv_context *cxt) } } -void __kvm_tlb_flush_vmid_ipa (struct kvm *kvm,phys_addr_t ipa) +void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa) {struct tlb_inv_context cxt;dsb (ishst);/* Switch to requested VMID */ - kvm =kern_hyp_va (kvm); -__tlb_switch_to_guest (kvm, &cxt); + mmu =kern_hyp_va (mmu); +__tlb_switch_to_guest (mmu, &cxt);/* * We could do so much better if we had the VA as well.
@@ -98,39 +97,39 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa)if (icache_is_vpipt ())__flush_icache_all (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); } -void __kvm_tlb_flush_vmid (struct kvm *kvm) +void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu) {struct tlb_inv_context cxt;dsb (ishst);/* Switch to requested VMID */ - kvm =kern_hyp_va (kvm); -__tlb_switch_to_guest (kvm, &cxt); + mmu =kern_hyp_va (mmu); +__tlb_switch_to_guest (mmu, &cxt);__tlbi (vmalls12e1is);dsb (ish);isb (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); } -void __kvm_tlb_flush_local_vmid (struct kvm_vcpu *vcpu) +void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu) { -struct kvm *kvm =kern_hyp_va (kern_hyp_va (vcpu)->kvm);struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (kvm, &cxt); + mmu =kern_hyp_va (mmu); +__tlb_switch_to_guest (mmu, &cxt);__tlbi (vmalle1);dsb (nsh);isb (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); }void __kvm_flush_vm_context (void )
arch/arm64/kvm/hyp/vhe/switch.c +1 -1
@@ -125,7 +125,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)* stage 2 translation, and __activate_traps clear HCR_EL2.TGE * (among other things). */ -__activate_vm (vcpu->kvm); +__activate_vm (vcpu->arch.hw_mmu);__activate_traps (vcpu);sysreg_restore_guest_state_vhe (guest_ctxt);
arch/arm64/kvm/hyp/vhe/tlb.c +13 -13
@@ -16,7 +16,8 @@ struct tlb_inv_context {u64 sctlr; }; -static void __tlb_switch_to_guest (struct kvm *kvm,struct tlb_inv_context *cxt) +static void __tlb_switch_to_guest (struct kvm_s2_mmu *mmu, +struct tlb_inv_context *cxt) {u64 val;
@@ -52,14 +53,14 @@ static void __tlb_switch_to_guest(struct kvm *kvm, struct tlb_inv_context *cxt)* place before clearing TGE. __load_guest_stage2() already * has an ISB in order to deal with this. */ -__load_guest_stage2 (kvm); +__load_guest_stage2 (mmu); val =read_sysreg (hcr_el2); val &= ~HCR_TGE ;write_sysreg (val, hcr_el2);isb (); } -static void __tlb_switch_to_host (struct kvm *kvm,struct tlb_inv_context *cxt) +static void __tlb_switch_to_host (struct tlb_inv_context *cxt) {/* * We're done with the TLB operation, let's restore the host's
@@ -78,14 +79,14 @@ static void __tlb_switch_to_host(struct kvm *kvm, struct tlb_inv_context *cxt)local_irq_restore (cxt->flags); } -void __kvm_tlb_flush_vmid_ipa (struct kvm *kvm,phys_addr_t ipa) +void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa) {struct tlb_inv_context cxt;dsb (ishst);/* Switch to requested VMID */ -__tlb_switch_to_guest (kvm, &cxt); +__tlb_switch_to_guest (mmu, &cxt);/* * We could do so much better if we had the VA as well.
@@ -106,38 +107,37 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa)dsb (ish);isb (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); } -void __kvm_tlb_flush_vmid (struct kvm *kvm) +void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu) {struct tlb_inv_context cxt;dsb (ishst);/* Switch to requested VMID */ -__tlb_switch_to_guest (kvm, &cxt); +__tlb_switch_to_guest (mmu, &cxt);__tlbi (vmalls12e1is);dsb (ish);isb (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); } -void __kvm_tlb_flush_local_vmid (struct kvm_vcpu *vcpu) +void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu) { -struct kvm *kvm = vcpu->kvm;struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (kvm, &cxt); +__tlb_switch_to_guest (mmu, &cxt);__tlbi (vmalle1);dsb (nsh);isb (); -__tlb_switch_to_host (kvm, &cxt); +__tlb_switch_to_host (&cxt); }void __kvm_flush_vm_context (void )
arch/arm64/kvm/mmu.c +158 -122
854 lines not on this page · raw message
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 35/56 KVM: arm64: Use TTL hint in when invalidating stage-2 translations
In reply to Marc ZyngierSince we often have a precise idea of the level we're dealing with when invalidating TLBs, we can provide it to as a hint to our invalidation helper.
--- arch/arm64/include/asm/kvm_asm.h | 3 ++- arch/arm64/kvm/hyp/nvhe/tlb.c | 5 +++-- arch/arm64/kvm/hyp/vhe/tlb.c | 5 +++-- arch/arm64/kvm/mmu.c | 29 +++++++++++++++-------------- 4 files changed, 23 insertions(+), 19 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -1
@@ -91,7 +91,8 @@ DECLARE_KVM_HYP_SYM(__bp_harden_hyp_vecs);#endif extern void __kvm_flush_vm_context (void ); -extern void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa, +int level);extern void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu);extern void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu);
arch/arm64/kvm/hyp/nvhe/tlb.c +3 -2
@@ -46,7 +46,8 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt) } } -void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa) +void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu, +phys_addr_t ipa,int level) {struct tlb_inv_context cxt;
@@ -62,7 +63,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa)* whole of Stage-1. Weep... */ ipa >>=12 ; -__tlbi (ipas2e1is, ipa); +__tlbi_level (ipas2e1is, ipa, level);/* * We have to ensure completion of the invalidation at Stage-2,
arch/arm64/kvm/hyp/vhe/tlb.c +3 -2
@@ -79,7 +79,8 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt)local_irq_restore (cxt->flags); } -void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa) +void __kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu, +phys_addr_t ipa,int level) {struct tlb_inv_context cxt;
@@ -94,7 +95,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa)* whole of Stage-1. Weep... */ ipa >>=12 ; -__tlbi (ipas2e1is, ipa); +__tlbi_level (ipas2e1is, ipa, level);/* * We have to ensure completion of the invalidation at Stage-2,
arch/arm64/kvm/mmu.c +15 -14
@@ -58,9 +58,10 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)kvm_call_hyp (__kvm_tlb_flush_vmid, &kvm->arch.mmu); } -static void kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa) +static void kvm_tlb_flush_vmid_ipa (struct kvm_s2_mmu *mmu,phys_addr_t ipa, +int level) { -kvm_call_hyp (__kvm_tlb_flush_vmid_ipa, mmu, ipa); +kvm_call_hyp (__kvm_tlb_flush_vmid_ipa, mmu, ipa, level); }/*
@@ -102,7 +103,7 @@ static void stage2_dissolve_pmd(struct kvm_s2_mmu *mmu, phys_addr_t addr, pmd_treturn ;pmd_clear (pmd); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PMD_LEVEL );put_page (virt_to_page (pmd)); }
@@ -122,7 +123,7 @@ static void stage2_dissolve_pud(struct kvm_s2_mmu *mmu, phys_addr_t addr, pud_treturn ;stage2_pud_clear (kvm, pudp); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PUD_LEVEL );put_page (virt_to_page (pudp)); }
@@ -163,7 +164,7 @@ static void clear_stage2_pgd_entry(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addrstruct kvm *kvm = mmu->kvm;p4d_t *p4d_table__maybe_unused =stage2_p4d_offset (kvm, pgd,0UL );stage2_pgd_clear (kvm, pgd); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_NO_LEVEL_HINT );stage2_p4d_free (kvm, p4d_table);put_page (virt_to_page (pgd)); }
@@ -173,7 +174,7 @@ static void clear_stage2_p4d_entry(struct kvm_s2_mmu *mmu, p4d_t *p4d, phys_addrstruct kvm *kvm = mmu->kvm;pud_t *pud_table__maybe_unused =stage2_pud_offset (kvm, p4d,0 );stage2_p4d_clear (kvm, p4d); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_NO_LEVEL_HINT );stage2_pud_free (kvm, pud_table);put_page (virt_to_page (p4d)); }
@@ -185,7 +186,7 @@ static void clear_stage2_pud_entry(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addrVM_BUG_ON (stage2_pud_huge (kvm, *pud));stage2_pud_clear (kvm, pud); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_NO_LEVEL_HINT );stage2_pmd_free (kvm, pmd_table);put_page (virt_to_page (pud)); }
@@ -195,7 +196,7 @@ static void clear_stage2_pmd_entry(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addrpte_t *pte_table =pte_offset_kernel (pmd,0 );VM_BUG_ON (pmd_thp_or_huge (*pmd));pmd_clear (pmd); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_NO_LEVEL_HINT );free_page ((unsigned long )pte_table);put_page (virt_to_page (pmd)); }
@@ -273,7 +274,7 @@ static void unmap_stage2_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd,pte_t old_pte = *pte;kvm_set_pte (pte,__pte (0 )); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PTE_LEVEL );/* No need to invalidate the cache for device mappings */ if (!kvm_is_device_pfn (pte_pfn (old_pte)))
@@ -302,7 +303,7 @@ static void unmap_stage2_pmds(struct kvm_s2_mmu *mmu, pud_t *pud,pmd_t old_pmd = *pmd;pmd_clear (pmd); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PMD_LEVEL );kvm_flush_dcache_pmd (old_pmd);
@@ -332,7 +333,7 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, p4d_t *p4d,pud_t old_pud = *pud;stage2_pud_clear (kvm, pud); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PUD_LEVEL );kvm_flush_dcache_pud (old_pud);put_page (virt_to_page (pud)); }else {
@@ -1260,7 +1261,7 @@ static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu,*/ WARN_ON_ONCE (pmd_pfn (old_pmd) !=pmd_pfn (*new_pmd));pmd_clear (pmd); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PMD_LEVEL ); }else {get_page (virt_to_page (pmd)); }
@@ -1302,7 +1303,7 @@ static int stage2_set_pud_huge(struct kvm_s2_mmu *mmu,WARN_ON_ONCE (kvm_pud_pfn (old_pud) !=kvm_pud_pfn (*new_pudp));stage2_pud_clear (kvm, pudp); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PUD_LEVEL ); }else {get_page (virt_to_page (pudp)); }
@@ -1451,7 +1452,7 @@ static int stage2_set_pte(struct kvm_s2_mmu *mmu,return 0 ;kvm_set_pte (pte,__pte (0 )); -kvm_tlb_flush_vmid_ipa (mmu, addr); +kvm_tlb_flush_vmid_ipa (mmu, addr,S2_PTE_LEVEL ); }else {get_page (virt_to_page (pte)); }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 36/56 KVM: arm64: Introduce accessor for ctxt->sys_reg
In reply to Marc ZyngierIn order to allow the disintegration of the per-vcpu sysreg array, let's introduce a new helper (ctxt_sys_reg()) that returns the in-memory copy of a system register, picked from a given context.
__vcpu_sys_reg() is rewritten to use this helper.
--- arch/arm64/include/asm/kvm_host.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_host.h +10 -5
@@ -405,12 +405,17 @@ struct kvm_vcpu_arch {#define vcpu_gp_regs (v) (&(v)->arch.ctxt.gp_regs)/* -* Only use __vcpu_sys_reg if you know you want the memory backed version of a -* register, and not the one most recently accessed by a running VCPU. For -* example, for userspace access or for system registers that are never context -* switched, but only emulated. +* Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the +* memory backed version of a register, and not the one most recently +* accessed by a running VCPU. For example, for userspace access or +* for system registers that are never context switched, but only +* emulated. */ -#define __vcpu_sys_reg (v,r) ((v)->arch.ctxt.sys_regs[(r)]) +#define __ctxt_sys_reg (c,r) (&(c)->sys_regs[(r)]) + +#define ctxt_sys_reg (c,r) (*__ctxt_sys_reg(c,r)) + +#define __vcpu_sys_reg (v,r) (ctxt_sys_reg(&(v)->arch.ctxt, (r)))u64 vcpu_read_sys_reg (const struct kvm_vcpu *vcpu,int reg);void vcpu_write_sys_reg (struct kvm_vcpu *vcpu,u64 val,int reg);
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 37/56 KVM: arm64: hyp: Use ctxt_sys_reg/__vcpu_sys_reg instead of raw sys_regs access
In reply to Marc ZyngierSwitch the hypervisor code to using ctxt_sys_reg/__vcpu_sys_reg instead of raw sys_regs accesses. No intended functionnal change.
--- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 7 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 110 ++++++++++----------- arch/arm64/kvm/hyp/nvhe/switch.c | 4 +- 5 files changed, 62 insertions(+), 65 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -561,7 +561,7 @@ DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);static inline void kvm_init_host_cpu_context (struct kvm_cpu_context *cpu_ctxt) {/* The host's MPIDR is immutable, so let's set it up at boot time */ - cpu_ctxt->sys_regs[MPIDR_EL1 ] =read_cpuid_mpidr (); +ctxt_sys_reg (cpu_ctxt,MPIDR_EL1 ) =read_cpuid_mpidr (); }static inline bool kvm_arch_requires_vhe (void )
arch/arm64/kvm/hyp/include/hyp/debug-sr.h +2 -2
@@ -104,7 +104,7 @@ static inline void __debug_save_state(struct kvm_vcpu *vcpu,save_debug (dbg->dbg_wcr, dbgwcr, wrps);save_debug (dbg->dbg_wvr, dbgwvr, wrps); - ctxt->sys_regs[MDCCINT_EL1 ] =read_sysreg (mdccint_el1); +ctxt_sys_reg (ctxt,MDCCINT_EL1 ) =read_sysreg (mdccint_el1); } static inline void __debug_restore_state(struct kvm_vcpu *vcpu,
@@ -124,7 +124,7 @@ static inline void __debug_restore_state(struct kvm_vcpu *vcpu,restore_debug (dbg->dbg_wcr, dbgwcr, wrps);restore_debug (dbg->dbg_wvr, dbgwvr, wrps); -write_sysreg (ctxt->sys_regs[MDCCINT_EL1 ], mdccint_el1); +write_sysreg (ctxt_sys_reg (ctxt,MDCCINT_EL1 ), mdccint_el1); }static inline void __debug_switch_to_guest_common (struct kvm_vcpu *vcpu)
arch/arm64/kvm/hyp/include/hyp/switch.h +3 -4
@@ -53,7 +53,7 @@ static inline void __fpsimd_save_fpexc32(struct kvm_vcpu *vcpu)if (!vcpu_el1_is_32bit (vcpu))return ; - vcpu->arch.ctxt.sys_regs[FPEXC32_EL2 ] =read_sysreg (fpexc32_el2); +__vcpu_sys_reg (vcpu,FPEXC32_EL2 ) =read_sysreg (fpexc32_el2); }static inline void __activate_traps_fpsimd32 (struct kvm_vcpu *vcpu)
@@ -268,15 +268,14 @@ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu)sve_load_state (vcpu_sve_pffr (vcpu), &vcpu->arch.ctxt.gp_regs.fp_regs.fpsr,sve_vq_from_vl (vcpu->arch.sve_max_vl) -1 ); -write_sysreg_s (vcpu->arch.ctxt.sys_regs[ZCR_EL1 ],SYS_ZCR_EL12 ); +write_sysreg_s (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR_EL12 ); }else {__fpsimd_restore_state (&vcpu->arch.ctxt.gp_regs.fp_regs); }/* Skip restoring fpexc32 for AArch64 guests */ if (!(read_sysreg (hcr_el2) &HCR_RW )) -write_sysreg (vcpu->arch.ctxt.sys_regs[FPEXC32_EL2 ], - fpexc32_el2); +write_sysreg (__vcpu_sys_reg (vcpu,FPEXC32_EL2 ), fpexc32_el2); vcpu->arch.flags |=KVM_ARM64_FP_ENABLED ;
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +54 -56
203 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/switch.c +2 -2
@@ -52,9 +52,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu)* configured and enabled. We can now restore the guest's S1 * configuration: SCTLR, and only then TCR. */ -write_sysreg_el1 (ctxt->sys_regs[SCTLR_EL1 ],SYS_SCTLR ); +write_sysreg_el1 (ctxt_sys_reg (ctxt,SCTLR_EL1 ),SYS_SCTLR );isb (); -write_sysreg_el1 (ctxt->sys_regs[TCR_EL1 ],SYS_TCR ); +write_sysreg_el1 (ctxt_sys_reg (ctxt,TCR_EL1 ),SYS_TCR ); } }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 38/56 KVM: arm64: sve: Use __vcpu_sys_reg() instead of raw sys_regs access
In reply to Marc ZyngierNow that we have a wrapper for the sysreg accesses, let's use that consistently.
--- arch/arm64/kvm/fpsimd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/arm64/kvm/fpsimd.c +1 -3
@@ -109,12 +109,10 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)local_irq_save (flags);if (vcpu->arch.flags &KVM_ARM64_FP_ENABLED ) { -u64 *guest_zcr = &vcpu->arch.ctxt.sys_regs[ZCR_EL1 ]; -fpsimd_save_and_flush_cpu_state ();if (guest_has_sve) - *guest_zcr =read_sysreg_s (SYS_ZCR_EL12 ); +__vcpu_sys_reg (vcpu,ZCR_EL1 ) =read_sysreg_s (SYS_ZCR_EL12 ); }else if (host_has_sve) {/* * The FPSIMD/SVE state in the CPU has not been touched, and we
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 39/56 KVM: arm64: pauth: Use ctxt_sys_reg() instead of raw sys_regs access
In reply to Marc ZyngierNow that we have a wrapper for the sysreg accesses, let's use that consistently.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +13 -10
@@ -364,11 +364,14 @@ static inline bool esr_is_ptrauth_trap(u32 esr)return false; } -#define __ptrauth_save_key (regs, key) \ -({ \ - regs[key ## KEYLO_EL1] = read_sysreg_s(SYS_ ## key ## KEYLO_EL1); \ - regs[key ## KEYHI_EL1] = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ -}) +#define __ptrauth_save_key (ctxt, key) \ + do { \ + u64 __val; \ + __val = read_sysreg_s(SYS_ ## key ## KEYLO_EL1); \ + ctxt_sys_reg(ctxt, key ## KEYLO_EL1) = __val; \ + __val = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ + ctxt_sys_reg(ctxt, key ## KEYHI_EL1) = __val; \ +} while(0)static inline bool __hyp_handle_ptrauth (struct kvm_vcpu *vcpu) {
@@ -380,11 +383,11 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)return false; ctxt = &__hyp_this_cpu_ptr (kvm_host_data)->host_ctxt; -__ptrauth_save_key (ctxt->sys_regs,APIA ); -__ptrauth_save_key (ctxt->sys_regs,APIB ); -__ptrauth_save_key (ctxt->sys_regs,APDA ); -__ptrauth_save_key (ctxt->sys_regs,APDB ); -__ptrauth_save_key (ctxt->sys_regs,APGA ); +__ptrauth_save_key (ctxt,APIA ); +__ptrauth_save_key (ctxt,APIB ); +__ptrauth_save_key (ctxt,APDA ); +__ptrauth_save_key (ctxt,APDB ); +__ptrauth_save_key (ctxt,APGA );vcpu_ptrauth_enable (vcpu);
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 40/56 KVM: arm64: debug: Drop useless vpcu parameter
In reply to Marc ZyngierAs part of the ongoing spring cleanup, remove the now useless vcpu parameter that is passed around (host and guest contexts give us everything we need).
--- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)
arch/arm64/kvm/hyp/include/hyp/debug-sr.h +8 -10
@@ -88,9 +88,8 @@default :write_debug (ptr[0 ], reg,0 ); \ } -static inline void __debug_save_state (struct kvm_vcpu *vcpu, -struct kvm_guest_debug_arch *dbg, -struct kvm_cpu_context *ctxt) +static void __debug_save_state (struct kvm_guest_debug_arch *dbg, +struct kvm_cpu_context *ctxt) {u64 aa64dfr0;int brps, wrps;
@@ -107,9 +106,8 @@ static inline void __debug_save_state(struct kvm_vcpu *vcpu,ctxt_sys_reg (ctxt,MDCCINT_EL1 ) =read_sysreg (mdccint_el1); } -static inline void __debug_restore_state (struct kvm_vcpu *vcpu, -struct kvm_guest_debug_arch *dbg, -struct kvm_cpu_context *ctxt) +static void __debug_restore_state (struct kvm_guest_debug_arch *dbg, +struct kvm_cpu_context *ctxt) {u64 aa64dfr0;int brps, wrps;
@@ -142,8 +140,8 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu) host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr); -__debug_save_state (vcpu, host_dbg, host_ctxt); -__debug_restore_state (vcpu, guest_dbg, guest_ctxt); +__debug_save_state (host_dbg, host_ctxt); +__debug_restore_state (guest_dbg, guest_ctxt); }static inline void __debug_switch_to_host_common (struct kvm_vcpu *vcpu)
@@ -161,8 +159,8 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu) host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg =kern_hyp_va (vcpu->arch.debug_ptr); -__debug_save_state (vcpu, guest_dbg, guest_ctxt); -__debug_restore_state (vcpu, host_dbg, host_ctxt); +__debug_save_state (guest_dbg, guest_ctxt); +__debug_restore_state (host_dbg, host_ctxt); vcpu->arch.flags &= ~KVM_ARM64_DEBUG_DIRTY ; }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 41/56 KVM: arm64: Make struct kvm_regs userspace-only
In reply to Marc Zyngierstruct kvm_regs is used by userspace to indicate which register gets accessed by the {GET,SET}_ONE_REG API. But as we're about to refactor the layout of the in-kernel register structures, we need the kernel to move away from it.
Let's make kvm_regs userspace only, and let the kernel map it to its own internal representation.
--- arch/arm64/include/asm/kvm_emulate.h | 18 +++--- arch/arm64/include/asm/kvm_host.h | 12 +++- arch/arm64/kernel/asm-offsets.c | 3 +- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/guest.c | 70 +++++++++++++++++----- arch/arm64/kvm/hyp/entry.S | 3 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 24 ++++---- arch/arm64/kvm/regmap.c | 6 +- arch/arm64/kvm/reset.c | 2 +- 10 files changed, 96 insertions(+), 48 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +9 -9
@@ -124,12 +124,12 @@ static inline void vcpu_set_vsesr(struct kvm_vcpu *vcpu, u64 vsesr)static __always_inline unsigned long *vcpu_pc (const struct kvm_vcpu *vcpu) { -return (unsigned long *)&vcpu_gp_regs (vcpu)->regs.pc; +return (unsigned long *)&vcpu_gp_regs (vcpu)->pc; }static inline unsigned long *__vcpu_elr_el1 (const struct kvm_vcpu *vcpu) { -return (unsigned long *)&vcpu_gp_regs (vcpu)->elr_el1; +return (unsigned long *)&vcpu->arch.ctxt.elr_el1; }static inline unsigned long vcpu_read_elr_el1 (const struct kvm_vcpu *vcpu)
@@ -150,7 +150,7 @@ static inline void vcpu_write_elr_el1(const struct kvm_vcpu *vcpu, unsigned longstatic __always_inline unsigned long *vcpu_cpsr (const struct kvm_vcpu *vcpu) { -return (unsigned long *)&vcpu_gp_regs (vcpu)->regs.pstate; +return (unsigned long *)&vcpu_gp_regs (vcpu)->pstate; } static __always_inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu)
@@ -179,14 +179,14 @@ static inline void vcpu_set_thumb(struct kvm_vcpu *vcpu)static __always_inline unsigned long vcpu_get_reg (const struct kvm_vcpu *vcpu,u8 reg_num) { -return (reg_num ==31 ) ?0 :vcpu_gp_regs (vcpu)->regs.regs[reg_num]; +return (reg_num ==31 ) ?0 :vcpu_gp_regs (vcpu)->regs[reg_num]; } static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,unsigned long val) {if (reg_num !=31 ) -vcpu_gp_regs (vcpu)->regs.regs[reg_num] = val; +vcpu_gp_regs (vcpu)->regs[reg_num] = val; }static inline unsigned long vcpu_read_spsr (const struct kvm_vcpu *vcpu)
@@ -197,7 +197,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu)if (vcpu->arch.sysregs_loaded_on_cpu)return read_sysreg_el1 (SYS_SPSR );else -return vcpu_gp_regs (vcpu)->spsr[KVM_SPSR_EL1 ]; +return vcpu->arch.ctxt.spsr[KVM_SPSR_EL1 ]; }static inline void vcpu_write_spsr (struct kvm_vcpu *vcpu,unsigned long v)
@@ -210,7 +210,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v)if (vcpu->arch.sysregs_loaded_on_cpu)write_sysreg_el1 (v,SYS_SPSR );else -vcpu_gp_regs (vcpu)->spsr[KVM_SPSR_EL1 ] = v; + vcpu->arch.ctxt.spsr[KVM_SPSR_EL1 ] = v; }/*
@@ -519,11 +519,11 @@ static __always_inline void kvm_skip_instr(struct kvm_vcpu *vcpu, bool is_wide_i static __always_inline void __kvm_skip_instr(struct kvm_vcpu *vcpu) { *vcpu_pc (vcpu) =read_sysreg_el2 (SYS_ELR ); - vcpu->arch.ctxt.gp_regs.regs.pstate =read_sysreg_el2 (SYS_SPSR ); +vcpu_gp_regs (vcpu)->pstate =read_sysreg_el2 (SYS_SPSR );kvm_skip_instr (vcpu,kvm_vcpu_trap_il_is32bit (vcpu)); -write_sysreg_el2 (vcpu->arch.ctxt.gp_regs.regs.pstate,SYS_SPSR ); +write_sysreg_el2 (vcpu_gp_regs (vcpu)->pstate,SYS_SPSR );write_sysreg_el2 (*vcpu_pc (vcpu),SYS_ELR ); }
arch/arm64/include/asm/kvm_host.h +10 -2
@@ -236,7 +236,15 @@ enum vcpu_sysreg {#define NR_COPRO_REGS (NR_SYS_REGS * 2)struct kvm_cpu_context { -struct kvm_regs gp_regs; +struct user_pt_regs regs;/* sp = sp_el0 */ + +u64 sp_el1; +u64 elr_el1; + +u64 spsr[KVM_NR_SPSR ]; + +struct user_fpsimd_state fp_regs; +union {u64 sys_regs[NR_SYS_REGS ];u32 copro[NR_COPRO_REGS ];
@@ -402,7 +410,7 @@ struct kvm_vcpu_arch { system_supports_generic_auth()) && \ ((vcpu)->arch.flags & KVM_ARM64_GUEST_HAS_PTRAUTH)) -#define vcpu_gp_regs (v) (&(v)->arch.ctxt.gp_regs) +#define vcpu_gp_regs (v) (&(v)->arch.ctxt.regs)/* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the
arch/arm64/kernel/asm-offsets.c +1 -2
@@ -102,13 +102,12 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); -DEFINE (CPU_GP_REGS , offsetof(struct kvm_cpu_context , gp_regs)); +DEFINE (CPU_USER_PT_REGS , offsetof(struct kvm_cpu_context , regs)); DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1])); DEFINE(CPU_APIBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIBKEYLO_EL1])); DEFINE(CPU_APDAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDAKEYLO_EL1])); DEFINE(CPU_APDBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDBKEYLO_EL1])); DEFINE(CPU_APGAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APGAKEYLO_EL1])); -DEFINE (CPU_USER_PT_REGS , offsetof(struct kvm_regs , regs));DEFINE (HOST_CONTEXT_VCPU , offsetof(struct kvm_cpu_context , __hyp_running_vcpu));DEFINE (HOST_DATA_CONTEXT , offsetof(struct kvm_host_data , host_ctxt));#endif
arch/arm64/kvm/fpsimd.c +1 -1
@@ -85,7 +85,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)WARN_ON_ONCE (!irqs_disabled ());if (vcpu->arch.flags &KVM_ARM64_FP_ENABLED ) { -fpsimd_bind_state_to_cpu (&vcpu->arch.ctxt.gp_regs.fp_regs, +fpsimd_bind_state_to_cpu (&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, vcpu->arch.sve_max_vl);
arch/arm64/kvm/guest.c +56 -14
117 lines not on this page · raw message
arch/arm64/kvm/hyp/entry.S +1 -2
@@ -16,8 +16,7 @@ #include <asm/kvm_mmu.h> #include <asm/kvm_ptrauth.h> -#define CPU_GP_REG_OFFSET(x) (CPU_GP_REGS + x) -#define CPU_XREG_OFFSET(x) CPU_GP_REG_OFFSET(CPU_USER_PT_REGS + 8*x) +#define CPU_XREG_OFFSET(x) (CPU_USER_PT_REGS + 8*x) #define CPU_SP_EL0_OFFSET (CPU_XREG_OFFSET(30) + 8) .text
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -2
@@ -266,11 +266,11 @@ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu)if (sve_guest) {sve_load_state (vcpu_sve_pffr (vcpu), - &vcpu->arch.ctxt.gp_regs.fp_regs.fpsr, + &vcpu->arch.ctxt.fp_regs.fpsr,sve_vq_from_vl (vcpu->arch.sve_max_vl) -1 );write_sysreg_s (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR_EL12 ); }else { -__fpsimd_restore_state (&vcpu->arch.ctxt.gp_regs.fp_regs); +__fpsimd_restore_state (&vcpu->arch.ctxt.fp_regs); }/* Skip restoring fpexc32 for AArch64 guests */
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +12 -12
@@ -46,15 +46,15 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,PAR_EL1 ) =read_sysreg (par_el1);ctxt_sys_reg (ctxt,TPIDR_EL1 ) =read_sysreg (tpidr_el1); - ctxt->gp_regs.sp_el1 =read_sysreg (sp_el1); - ctxt->gp_regs.elr_el1 =read_sysreg_el1 (SYS_ELR ); - ctxt->gp_regs.spsr[KVM_SPSR_EL1 ]=read_sysreg_el1 (SYS_SPSR ); + ctxt->sp_el1 =read_sysreg (sp_el1); + ctxt->elr_el1 =read_sysreg_el1 (SYS_ELR ); + ctxt->spsr[KVM_SPSR_EL1 ] =read_sysreg_el1 (SYS_SPSR ); }static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt) { - ctxt->gp_regs.regs.pc =read_sysreg_el2 (SYS_ELR ); - ctxt->gp_regs.regs.pstate =read_sysreg_el2 (SYS_SPSR ); + ctxt->regs.pc =read_sysreg_el2 (SYS_ELR ); + ctxt->regs.pstate =read_sysreg_el2 (SYS_SPSR );if (cpus_have_final_cap (ARM64_HAS_RAS_EXTN ))ctxt_sys_reg (ctxt,DISR_EL1 ) =read_sysreg_s (SYS_VDISR_EL2 );
@@ -125,14 +125,14 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)write_sysreg_el1 (ctxt_sys_reg (ctxt,TCR_EL1 ),SYS_TCR ); } -write_sysreg (ctxt->gp_regs.sp_el1, sp_el1); -write_sysreg_el1 (ctxt->gp_regs.elr_el1,SYS_ELR ); -write_sysreg_el1 (ctxt->gp_regs.spsr[KVM_SPSR_EL1 ],SYS_SPSR ); +write_sysreg (ctxt->sp_el1, sp_el1); +write_sysreg_el1 (ctxt->elr_el1,SYS_ELR ); +write_sysreg_el1 (ctxt->spsr[KVM_SPSR_EL1 ],SYS_SPSR ); }static inline void __sysreg_restore_el2_return_state (struct kvm_cpu_context *ctxt) { -u64 pstate = ctxt->gp_regs.regs.pstate; +u64 pstate = ctxt->regs.pstate;u64 mode = pstate &PSR_AA32_MODE_MASK ;/*
@@ -149,7 +149,7 @@ static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxif (!(mode &PSR_MODE32_BIT ) && mode >= PSR_MODE_EL2t) pstate = PSR_MODE_EL2h |PSR_IL_BIT ; -write_sysreg_el2 (ctxt->gp_regs.regs.pc,SYS_ELR ); +write_sysreg_el2 (ctxt->regs.pc,SYS_ELR );write_sysreg_el2 (pstate,SYS_SPSR );if (cpus_have_final_cap (ARM64_HAS_RAS_EXTN ))
@@ -163,7 +163,7 @@ static inline void __sysreg32_save_state(struct kvm_vcpu *vcpu)if (!vcpu_el1_is_32bit (vcpu))return ; - spsr = vcpu->arch.ctxt.gp_regs.spsr; + spsr = vcpu->arch.ctxt.spsr; spsr[KVM_SPSR_ABT ] =read_sysreg (spsr_abt); spsr[KVM_SPSR_UND ] =read_sysreg (spsr_und);
@@ -184,7 +184,7 @@ static inline void __sysreg32_restore_state(struct kvm_vcpu *vcpu)if (!vcpu_el1_is_32bit (vcpu))return ; - spsr = vcpu->arch.ctxt.gp_regs.spsr; + spsr = vcpu->arch.ctxt.spsr;write_sysreg (spsr[KVM_SPSR_ABT ], spsr_abt);write_sysreg (spsr[KVM_SPSR_UND ], spsr_und);
arch/arm64/kvm/regmap.c +3 -3
@@ -100,7 +100,7 @@ static const unsigned long vcpu_reg_offsets[VCPU_NR_MODES][16] = {*/ unsigned long *vcpu_reg32 (const struct kvm_vcpu *vcpu,u8 reg_num) { -unsigned long *reg_array = (unsigned long *)&vcpu->arch.ctxt.gp_regs.regs; +unsigned long *reg_array = (unsigned long *)&vcpu->arch.ctxt.regs;unsigned long mode = *vcpu_cpsr (vcpu) &PSR_AA32_MODE_MASK ;switch (mode) {
@@ -148,7 +148,7 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu)int spsr_idx =vcpu_spsr32_mode (vcpu);if (!vcpu->arch.sysregs_loaded_on_cpu) -return vcpu_gp_regs (vcpu)->spsr[spsr_idx]; +return vcpu->arch.ctxt.spsr[spsr_idx];switch (spsr_idx) {case KVM_SPSR_SVC :
@@ -171,7 +171,7 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v)int spsr_idx =vcpu_spsr32_mode (vcpu);if (!vcpu->arch.sysregs_loaded_on_cpu) { -vcpu_gp_regs (vcpu)->spsr[spsr_idx] = v; + vcpu->arch.ctxt.spsr[spsr_idx] = v;return ; }
arch/arm64/kvm/reset.c +1 -1
@@ -288,7 +288,7 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)/* Reset core registers */ memset (vcpu_gp_regs (vcpu),0 ,sizeof (*vcpu_gp_regs (vcpu))); -vcpu_gp_regs (vcpu)->regs.pstate = pstate; +vcpu_gp_regs (vcpu)->pstate = pstate;/* Reset system registers */ kvm_reset_sys_regs (vcpu);
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 42/56 KVM: arm64: Move ELR_EL1 to the system register array
In reply to Marc ZyngierAs ELR-EL1 is a VNCR-capable register with ARMv8.4-NV, let's move it to the sys_regs array and repaint the accessors. While we're at it, let's kill the now useless accessors used only on the fault injection path.
--- arch/arm64/include/asm/kvm_emulate.h | 21 --------------------- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 4 ++-- arch/arm64/kvm/inject_fault.c | 2 +- arch/arm64/kvm/sys_regs.c | 2 ++ 6 files changed, 8 insertions(+), 26 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +0 -21
@@ -127,27 +127,6 @@ static __always_inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu)return (unsigned long *)&vcpu_gp_regs (vcpu)->pc; } -static inline unsigned long *__vcpu_elr_el1 (const struct kvm_vcpu *vcpu) -{ -return (unsigned long *)&vcpu->arch.ctxt.elr_el1; -} - -static inline unsigned long vcpu_read_elr_el1 (const struct kvm_vcpu *vcpu) -{ -if (vcpu->arch.sysregs_loaded_on_cpu) -return read_sysreg_el1 (SYS_ELR ); -else -return *__vcpu_elr_el1 (vcpu); -} - -static inline void vcpu_write_elr_el1 (const struct kvm_vcpu *vcpu,unsigned long v) -{ -if (vcpu->arch.sysregs_loaded_on_cpu) -write_sysreg_el1 (v,SYS_ELR ); -else - *__vcpu_elr_el1 (vcpu) = v; -} -static __always_inline unsigned long *vcpu_cpsr (const struct kvm_vcpu *vcpu) {return (unsigned long *)&vcpu_gp_regs (vcpu)->pstate;
arch/arm64/include/asm/kvm_host.h +2 -1
@@ -185,6 +185,8 @@ enum vcpu_sysreg {APGAKEYLO_EL1 ,APGAKEYHI_EL1 , +ELR_EL1 , +/* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */
@@ -239,7 +241,6 @@ struct kvm_cpu_context {struct user_pt_regs regs;/* sp = sp_el0 */ u64 sp_el1; -u64 elr_el1;u64 spsr[KVM_NR_SPSR ];
arch/arm64/kvm/guest.c +1 -1
@@ -132,7 +132,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return &vcpu->arch.ctxt.sp_el1;case KVM_REG_ARM_CORE_REG (elr_el1): -return &vcpu->arch.ctxt.elr_el1; +return __ctxt_sys_reg (&vcpu->arch.ctxt,ELR_EL1 ); case KVM_REG_ARM_CORE_REG(spsr[0]) ...KVM_REG_ARM_CORE_REG (spsr[KVM_NR_SPSR -1 ]):
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +2 -2
@@ -47,7 +47,7 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,TPIDR_EL1 ) =read_sysreg (tpidr_el1); ctxt->sp_el1 =read_sysreg (sp_el1); - ctxt->elr_el1 =read_sysreg_el1 (SYS_ELR ); +ctxt_sys_reg (ctxt,ELR_EL1 ) =read_sysreg_el1 (SYS_ELR ); ctxt->spsr[KVM_SPSR_EL1 ] =read_sysreg_el1 (SYS_SPSR ); }
@@ -126,7 +126,7 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) }write_sysreg (ctxt->sp_el1, sp_el1); -write_sysreg_el1 (ctxt->elr_el1,SYS_ELR ); +write_sysreg_el1 (ctxt_sys_reg (ctxt,ELR_EL1 ),SYS_ELR );write_sysreg_el1 (ctxt->spsr[KVM_SPSR_EL1 ],SYS_SPSR ); }
arch/arm64/kvm/inject_fault.c +1 -1
@@ -64,7 +64,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,case PSR_MODE_EL1h: vbar =vcpu_read_sys_reg (vcpu,VBAR_EL1 ); sctlr =vcpu_read_sys_reg (vcpu,SCTLR_EL1 ); -vcpu_write_elr_el1 (vcpu, *vcpu_pc (vcpu)); +vcpu_write_sys_reg (vcpu, *vcpu_pc (vcpu),ELR_EL1 );break ;default :/* Don't do that */
arch/arm64/kvm/sys_regs.c +2 -0
@@ -94,6 +94,7 @@ static bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val)case TPIDR_EL1 : *val =read_sysreg_s (SYS_TPIDR_EL1 );break ;case AMAIR_EL1 : *val =read_sysreg_s (SYS_AMAIR_EL12 );break ;case CNTKCTL_EL1 : *val =read_sysreg_s (SYS_CNTKCTL_EL12 );break ; +case ELR_EL1 : *val =read_sysreg_s (SYS_ELR_EL12 );break ;case PAR_EL1 : *val =read_sysreg_s (SYS_PAR_EL1 );break ;case DACR32_EL2 : *val =read_sysreg_s (SYS_DACR32_EL2 );break ;case IFSR32_EL2 : *val =read_sysreg_s (SYS_IFSR32_EL2 );break ;
@@ -133,6 +134,7 @@ static bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)case TPIDR_EL1 :write_sysreg_s (val,SYS_TPIDR_EL1 );break ;case AMAIR_EL1 :write_sysreg_s (val,SYS_AMAIR_EL12 );break ;case CNTKCTL_EL1 :write_sysreg_s (val,SYS_CNTKCTL_EL12 );break ; +case ELR_EL1 :write_sysreg_s (val,SYS_ELR_EL12 );break ;case PAR_EL1 :write_sysreg_s (val,SYS_PAR_EL1 );break ;case DACR32_EL2 :write_sysreg_s (val,SYS_DACR32_EL2 );break ;case IFSR32_EL2 :write_sysreg_s (val,SYS_IFSR32_EL2 );break ;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 43/56 KVM: arm64: Move SP_EL1 to the system register array
In reply to Marc ZyngierSP_EL1 being a VNCR-capable register with ARMv8.4-NV, move it to the system register array and update the accessors.
--- arch/arm64/include/asm/kvm_host.h | 3 +-- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 4 ++-- 3 files changed, 4 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -2
@@ -186,6 +186,7 @@ enum vcpu_sysreg {APGAKEYHI_EL1 ,ELR_EL1 , +SP_EL1 ,/* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2, /* Domain Access Control Register */
@@ -240,8 +241,6 @@ enum vcpu_sysreg {struct kvm_cpu_context {struct user_pt_regs regs;/* sp = sp_el0 */ -u64 sp_el1; -u64 spsr[KVM_NR_SPSR ];struct user_fpsimd_state fp_regs;
arch/arm64/kvm/guest.c +1 -1
@@ -129,7 +129,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return &vcpu->arch.ctxt.regs.pstate;case KVM_REG_ARM_CORE_REG (sp_el1): -return &vcpu->arch.ctxt.sp_el1; +return __ctxt_sys_reg (&vcpu->arch.ctxt,SP_EL1 );case KVM_REG_ARM_CORE_REG (elr_el1):return __ctxt_sys_reg (&vcpu->arch.ctxt,ELR_EL1 );
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +2 -2
@@ -46,7 +46,7 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,PAR_EL1 ) =read_sysreg (par_el1);ctxt_sys_reg (ctxt,TPIDR_EL1 ) =read_sysreg (tpidr_el1); - ctxt->sp_el1 =read_sysreg (sp_el1); +ctxt_sys_reg (ctxt,SP_EL1 ) =read_sysreg (sp_el1);ctxt_sys_reg (ctxt,ELR_EL1 ) =read_sysreg_el1 (SYS_ELR ); ctxt->spsr[KVM_SPSR_EL1 ] =read_sysreg_el1 (SYS_SPSR ); }
@@ -125,7 +125,7 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)write_sysreg_el1 (ctxt_sys_reg (ctxt,TCR_EL1 ),SYS_TCR ); } -write_sysreg (ctxt->sp_el1, sp_el1); +write_sysreg (ctxt_sys_reg (ctxt,SP_EL1 ), sp_el1);write_sysreg_el1 (ctxt_sys_reg (ctxt,ELR_EL1 ),SYS_ELR );write_sysreg_el1 (ctxt->spsr[KVM_SPSR_EL1 ],SYS_SPSR ); }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 44/56 KVM: arm64: Disintegrate SPSR array
In reply to Marc ZyngierAs we're about to move SPSR_EL1 into the VNCR page, we need to disassociate it from the rest of the 32bit cruft. Let's break the array into individual fields.
--- arch/arm64/include/asm/kvm_emulate.h | 4 +-- arch/arm64/include/asm/kvm_host.h | 6 +++- arch/arm64/kvm/guest.c | 19 ++++++++---- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +++++++---------- arch/arm64/kvm/regmap.c | 35 ++++++++++++++++++++-- 5 files changed, 63 insertions(+), 29 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +2 -2
@@ -176,7 +176,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu)if (vcpu->arch.sysregs_loaded_on_cpu)return read_sysreg_el1 (SYS_SPSR );else -return vcpu->arch.ctxt.spsr[KVM_SPSR_EL1 ]; +return vcpu->arch.ctxt.spsr_el1; }static inline void vcpu_write_spsr (struct kvm_vcpu *vcpu,unsigned long v)
@@ -189,7 +189,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v)if (vcpu->arch.sysregs_loaded_on_cpu)write_sysreg_el1 (v,SYS_SPSR );else - vcpu->arch.ctxt.spsr[KVM_SPSR_EL1 ] = v; + vcpu->arch.ctxt.spsr_el1 = v; }/*
arch/arm64/include/asm/kvm_host.h +5 -1
@@ -241,7 +241,11 @@ enum vcpu_sysreg {struct kvm_cpu_context {struct user_pt_regs regs;/* sp = sp_el0 */ -u64 spsr[KVM_NR_SPSR ]; +u64 spsr_el1;/* aka spsr_svc */ +u64 spsr_abt; +u64 spsr_und; +u64 spsr_irq; +u64 spsr_fiq;struct user_fpsimd_state fp_regs;
arch/arm64/kvm/guest.c +14 -5
@@ -134,11 +134,20 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)case KVM_REG_ARM_CORE_REG (elr_el1):return __ctxt_sys_reg (&vcpu->arch.ctxt,ELR_EL1 ); - case KVM_REG_ARM_CORE_REG(spsr[0]) ... -KVM_REG_ARM_CORE_REG (spsr[KVM_NR_SPSR -1 ]): - off -=KVM_REG_ARM_CORE_REG (spsr[0 ]); - off /=2 ; -return &vcpu->arch.ctxt.spsr[off]; +case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_EL1 ]): +return &vcpu->arch.ctxt.spsr_el1; + +case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_ABT ]): +return &vcpu->arch.ctxt.spsr_abt; + +case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_UND ]): +return &vcpu->arch.ctxt.spsr_und; + +case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_IRQ ]): +return &vcpu->arch.ctxt.spsr_irq; + +case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_FIQ ]): +return &vcpu->arch.ctxt.spsr_fiq; case KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]) ...KVM_REG_ARM_CORE_REG (fp_regs.vregs[31 ]):
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +10 -18
@@ -48,7 +48,7 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,SP_EL1 ) =read_sysreg (sp_el1);ctxt_sys_reg (ctxt,ELR_EL1 ) =read_sysreg_el1 (SYS_ELR ); - ctxt->spsr[KVM_SPSR_EL1 ] =read_sysreg_el1 (SYS_SPSR ); + ctxt->spsr_el1 =read_sysreg_el1 (SYS_SPSR ); }static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt)
@@ -127,7 +127,7 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)write_sysreg (ctxt_sys_reg (ctxt,SP_EL1 ), sp_el1);write_sysreg_el1 (ctxt_sys_reg (ctxt,ELR_EL1 ),SYS_ELR ); -write_sysreg_el1 (ctxt->spsr[KVM_SPSR_EL1 ],SYS_SPSR ); +write_sysreg_el1 (ctxt->spsr_el1,SYS_SPSR ); }static inline void __sysreg_restore_el2_return_state (struct kvm_cpu_context *ctxt)
@@ -158,17 +158,13 @@ static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxstatic inline void __sysreg32_save_state (struct kvm_vcpu *vcpu) { -u64 *spsr; -if (!vcpu_el1_is_32bit (vcpu))return ; - spsr = vcpu->arch.ctxt.spsr; - - spsr[KVM_SPSR_ABT ] =read_sysreg (spsr_abt); - spsr[KVM_SPSR_UND ] =read_sysreg (spsr_und); - spsr[KVM_SPSR_IRQ ] =read_sysreg (spsr_irq); - spsr[KVM_SPSR_FIQ ] =read_sysreg (spsr_fiq); + vcpu->arch.ctxt.spsr_abt =read_sysreg (spsr_abt); + vcpu->arch.ctxt.spsr_und =read_sysreg (spsr_und); + vcpu->arch.ctxt.spsr_irq =read_sysreg (spsr_irq); + vcpu->arch.ctxt.spsr_fiq =read_sysreg (spsr_fiq);__vcpu_sys_reg (vcpu,DACR32_EL2 ) =read_sysreg (dacr32_el2);__vcpu_sys_reg (vcpu,IFSR32_EL2 ) =read_sysreg (ifsr32_el2);
@@ -179,17 +175,13 @@ static inline void __sysreg32_save_state(struct kvm_vcpu *vcpu)static inline void __sysreg32_restore_state (struct kvm_vcpu *vcpu) { -u64 *spsr; -if (!vcpu_el1_is_32bit (vcpu))return ; - spsr = vcpu->arch.ctxt.spsr; - -write_sysreg (spsr[KVM_SPSR_ABT ], spsr_abt); -write_sysreg (spsr[KVM_SPSR_UND ], spsr_und); -write_sysreg (spsr[KVM_SPSR_IRQ ], spsr_irq); -write_sysreg (spsr[KVM_SPSR_FIQ ], spsr_fiq); +write_sysreg (vcpu->arch.ctxt.spsr_abt, spsr_abt); +write_sysreg (vcpu->arch.ctxt.spsr_und, spsr_und); +write_sysreg (vcpu->arch.ctxt.spsr_irq, spsr_irq); +write_sysreg (vcpu->arch.ctxt.spsr_fiq, spsr_fiq);write_sysreg (__vcpu_sys_reg (vcpu,DACR32_EL2 ), dacr32_el2);write_sysreg (__vcpu_sys_reg (vcpu,IFSR32_EL2 ), ifsr32_el2);
arch/arm64/kvm/regmap.c +32 -3
@@ -147,8 +147,20 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu) {int spsr_idx =vcpu_spsr32_mode (vcpu); -if (!vcpu->arch.sysregs_loaded_on_cpu) -return vcpu->arch.ctxt.spsr[spsr_idx]; +if (!vcpu->arch.sysregs_loaded_on_cpu) { +switch (spsr_idx) { +case KVM_SPSR_SVC : +return vcpu->arch.ctxt.spsr_el1; +case KVM_SPSR_ABT : +return vcpu->arch.ctxt.spsr_abt; +case KVM_SPSR_UND : +return vcpu->arch.ctxt.spsr_und; +case KVM_SPSR_IRQ : +return vcpu->arch.ctxt.spsr_irq; +case KVM_SPSR_FIQ : +return vcpu->arch.ctxt.spsr_fiq; + } + }switch (spsr_idx) {case KVM_SPSR_SVC :
@@ -171,7 +183,24 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v)int spsr_idx =vcpu_spsr32_mode (vcpu);if (!vcpu->arch.sysregs_loaded_on_cpu) { - vcpu->arch.ctxt.spsr[spsr_idx] = v; +switch (spsr_idx) { +case KVM_SPSR_SVC : + vcpu->arch.ctxt.spsr_el1 = v; +break ; +case KVM_SPSR_ABT : + vcpu->arch.ctxt.spsr_abt = v; +break ; +case KVM_SPSR_UND : + vcpu->arch.ctxt.spsr_und = v; +break ; +case KVM_SPSR_IRQ : + vcpu->arch.ctxt.spsr_irq = v; +break ; +case KVM_SPSR_FIQ : + vcpu->arch.ctxt.spsr_fiq = v; +break ; + } +return ; }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 45/56 KVM: arm64: Move SPSR_EL1 to the system register array
In reply to Marc ZyngierSPSR_EL1 being a VNCR-capable register with ARMv8.4-NV, move it to the sysregs array and update the accessors.
--- arch/arm64/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 4 ++-- arch/arm64/kvm/regmap.c | 4 ++-- 5 files changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +2 -2
@@ -176,7 +176,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu)if (vcpu->arch.sysregs_loaded_on_cpu)return read_sysreg_el1 (SYS_SPSR );else -return vcpu->arch.ctxt.spsr_el1; +return __vcpu_sys_reg (vcpu,SPSR_EL1 ); }static inline void vcpu_write_spsr (struct kvm_vcpu *vcpu,unsigned long v)
@@ -189,7 +189,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v)if (vcpu->arch.sysregs_loaded_on_cpu)write_sysreg_el1 (v,SYS_SPSR );else - vcpu->arch.ctxt.spsr_el1 = v; +__vcpu_sys_reg (vcpu,SPSR_EL1 ) = v; }/*
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -187,6 +187,7 @@ enum vcpu_sysreg {ELR_EL1 ,SP_EL1 , +SPSR_EL1 ,/* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2, /* Domain Access Control Register */
@@ -241,7 +242,6 @@ enum vcpu_sysreg {struct kvm_cpu_context {struct user_pt_regs regs;/* sp = sp_el0 */ -u64 spsr_el1;/* aka spsr_svc */ u64 spsr_abt;u64 spsr_und;u64 spsr_irq;
arch/arm64/kvm/guest.c +1 -1
@@ -135,7 +135,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)return __ctxt_sys_reg (&vcpu->arch.ctxt,ELR_EL1 );case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_EL1 ]): -return &vcpu->arch.ctxt.spsr_el1; +return __ctxt_sys_reg (&vcpu->arch.ctxt,SPSR_EL1 );case KVM_REG_ARM_CORE_REG (spsr[KVM_SPSR_ABT ]):return &vcpu->arch.ctxt.spsr_abt;
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +2 -2
@@ -48,7 +48,7 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,SP_EL1 ) =read_sysreg (sp_el1);ctxt_sys_reg (ctxt,ELR_EL1 ) =read_sysreg_el1 (SYS_ELR ); - ctxt->spsr_el1 =read_sysreg_el1 (SYS_SPSR ); +ctxt_sys_reg (ctxt,SPSR_EL1 ) =read_sysreg_el1 (SYS_SPSR ); }static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt)
@@ -127,7 +127,7 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)write_sysreg (ctxt_sys_reg (ctxt,SP_EL1 ), sp_el1);write_sysreg_el1 (ctxt_sys_reg (ctxt,ELR_EL1 ),SYS_ELR ); -write_sysreg_el1 (ctxt->spsr_el1,SYS_SPSR ); +write_sysreg_el1 (ctxt_sys_reg (ctxt,SPSR_EL1 ),SYS_SPSR ); }static inline void __sysreg_restore_el2_return_state (struct kvm_cpu_context *ctxt)
arch/arm64/kvm/regmap.c +2 -2
@@ -150,7 +150,7 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu)if (!vcpu->arch.sysregs_loaded_on_cpu) {switch (spsr_idx) {case KVM_SPSR_SVC : -return vcpu->arch.ctxt.spsr_el1; +return __vcpu_sys_reg (vcpu,SPSR_EL1 );case KVM_SPSR_ABT :return vcpu->arch.ctxt.spsr_abt;case KVM_SPSR_UND :
@@ -185,7 +185,7 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v)if (!vcpu->arch.sysregs_loaded_on_cpu) {switch (spsr_idx) {case KVM_SPSR_SVC : - vcpu->arch.ctxt.spsr_el1 = v; +__vcpu_sys_reg (vcpu,SPSR_EL1 ) = v;break ;case KVM_SPSR_ABT : vcpu->arch.ctxt.spsr_abt = v;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 46/56 KVM: arm64: timers: Rename kvm_timer_sync_hwstate to kvm_timer_sync_user
In reply to Marc Zyngierkvm_timer_sync_hwstate() has nothing to do with the timer HW state, but more to do with the state of a userspace interrupt controller. Change the suffix from _hwstate to_user, in keeping with the rest of the code.
--- arch/arm64/kvm/arch_timer.c | 2 +- arch/arm64/kvm/arm.c | 4 ++-- include/kvm/arm_arch_timer.h | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/arm64/kvm/arch_timer.c +1 -1
@@ -615,7 +615,7 @@ static void unmask_vtimer_irq_user(struct kvm_vcpu *vcpu) } } -void kvm_timer_sync_hwstate (struct kvm_vcpu *vcpu) +void kvm_timer_sync_user (struct kvm_vcpu *vcpu) {struct arch_timer_cpu *timer =vcpu_timer (vcpu);
arch/arm64/kvm/arm.c +2 -2
@@ -723,7 +723,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)isb ();/* Ensure work in x_flush_hwstate is committed */ kvm_pmu_sync_hwstate (vcpu);if (static_branch_unlikely (&userspace_irqchip_in_use)) -kvm_timer_sync_hwstate (vcpu); +kvm_timer_sync_user (vcpu);kvm_vgic_sync_hwstate (vcpu);local_irq_enable ();preempt_enable ();
@@ -768,7 +768,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)* timer virtual interrupt state. */ if (static_branch_unlikely (&userspace_irqchip_in_use)) -kvm_timer_sync_hwstate (vcpu); +kvm_timer_sync_user (vcpu);kvm_arch_vcpu_ctxsync_fp (vcpu);
include/kvm/arm_arch_timer.h +1 -1
@@ -71,7 +71,7 @@ int kvm_timer_hyp_init(bool);int kvm_timer_enable (struct kvm_vcpu *vcpu);int kvm_timer_vcpu_reset (struct kvm_vcpu *vcpu);void kvm_timer_vcpu_init (struct kvm_vcpu *vcpu); -void kvm_timer_sync_hwstate (struct kvm_vcpu *vcpu); +void kvm_timer_sync_user (struct kvm_vcpu *vcpu);bool kvm_timer_should_notify_user (struct kvm_vcpu *vcpu);void kvm_timer_update_run (struct kvm_vcpu *vcpu);void kvm_timer_vcpu_terminate (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 47/56 KVM: arm64: timers: Move timer registers to the sys_regs file
In reply to Marc ZyngierMove the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as arch_timer_ctx_index) can work correctly at an early stage.
--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)
arch/arm64/include/asm/kvm_host.h +6 -0
@@ -189,6 +189,12 @@ enum vcpu_sysreg {SP_EL1 ,SPSR_EL1 , +CNTVOFF_EL2 , +CNTV_CVAL_EL0 , +CNTV_CTL_EL0 , +CNTP_CVAL_EL0 , +CNTP_CTL_EL0 , +/* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */
arch/arm64/kvm/arch_timer.c +122 -33
272 lines not on this page · raw message
arch/arm64/kvm/trace_arm.h +4 -4
@@ -301,8 +301,8 @@ TRACE_EVENT(kvm_timer_save_state, ), TP_fast_assign( - __entry->ctl = ctx->cnt_ctl; - __entry->cval = ctx->cnt_cval; + __entry->ctl = timer_get_ctl(ctx); + __entry->cval =timer_get_cval (ctx); __entry->timer_idx =arch_timer_ctx_index (ctx); ),
@@ -323,8 +323,8 @@ TRACE_EVENT(kvm_timer_restore_state, ), TP_fast_assign( - __entry->ctl = ctx->cnt_ctl; - __entry->cval = ctx->cnt_cval; + __entry->ctl = timer_get_ctl(ctx); + __entry->cval =timer_get_cval (ctx); __entry->timer_idx =arch_timer_ctx_index (ctx); ),
include/kvm/arm_arch_timer.h +4 -7
@@ -26,16 +26,9 @@ enum kvm_arch_timer_regs {struct arch_timer_context {struct kvm_vcpu *vcpu; -/* Registers: control register, timer value */ -u32 cnt_ctl; -u64 cnt_cval; -/* Timer IRQ */ struct kvm_irq_level irq; -/* Virtual offset */ -u64 cntvoff; -/* Emulated Timer (may be unused) */ struct hrtimer hrtimer;
@@ -109,4 +102,8 @@ void kvm_arm_timer_write_sysreg(struct kvm_vcpu *vcpu,enum kvm_arch_timer_regs treg, u64 val); +/* Needed for tracing */ +u32 timer_get_ctl (struct arch_timer_context *ctxt); +u64 timer_get_cval (struct arch_timer_context *ctxt); +#endif
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Hi Marc,
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <[email protected]>
Cc: Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Move the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as arch_timer_ctx_index) can work correctly at an early stage.
--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)
+static u64 timer_get_offset(struct arch_timer_context *ctxt) {
+ struct kvm_vcpu *vcpu = ctxt->vcpu;
+
+ switch(arch_timer_ctx_index(ctxt)) {
+ case TIMER_VTIMER:
+ return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
+ default:
+ return 0;
+ }
+}
+
Can I export this helper? As in my ptp_kvm implementation I need get VCNT offset value separately not just give me a result of VCNT.
Thanks
Jianyong
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Quoted 41 lines · Jianyong Wu
Hi Marc,
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <[email protected]>
Cc: Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileMove the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as
arch_timer_ctx_index) can work correctly at an early stage.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)arch/arm64/kvm/arch_timer.c +11
-0+static u64 timer_get_offset (struct arch_timer_context *ctxt) { +struct kvm_vcpu *vcpu = ctxt->vcpu; + +switch (arch_timer_ctx_index (ctxt)) { +case TIMER_VTIMER : +return __vcpu_sys_reg (vcpu,CNTVOFF_EL2 ); +default : +return 0 ; + } +} +
Quoted 2 lines
Can I export this helper? As in my
ptp_kvmimplementation I need get VCNT offset value separately not just give me a result of VCNT.
Sorry, you need to give me a bit more context. What do you need the offset for exactly?
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
RE: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Quoted 45 lines
-----Original Message-----
From: Marc Zyngier <[email protected]>
Sent: Wednesday, August 19, 2020 6:00 PM
To: Jianyong Wu <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileQuoted 30 lines · Jianyong Wu
Hi Marc,
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <[email protected]>
Cc: Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileMove the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as
arch_timer_ctx_index) can work correctly at an early stage.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)
Quoted 16 lines
Quoted 11 lines
arch/arm64/kvm/arch_timer.c +11
-0+static u64 timer_get_offset (struct arch_timer_context *ctxt) { +struct kvm_vcpu *vcpu = ctxt->vcpu; + +switch (arch_timer_ctx_index (ctxt)) { +case TIMER_VTIMER : +return __vcpu_sys_reg (vcpu,CNTVOFF_EL2 ); +default : +return 0 ; + } +} +Quoted 2 lines
Can I export this helper? As in my
ptp_kvmimplementation I need get VCNT offset value separately not just give me a result of VCNT.Sorry, you need to give me a bit more context. What do you need the offset for exactly?
Yeah,
In my ptp_kvm implementation, I need acquire wall time and counter cycle in the same time in host. After get host counter cycle, I need subtract it by VCNT offset to obtain VCNT. See https://lkml.org/lkml/2020/6/19/441 https://lkml.org/lkml/2020/6/19/441
But now I can't get the VCNT offset easily like before using " vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like "kvm_arm_timer_read" as I need acquire the counter cycle in the same time with the host wall time.
Thanks
Jianyong
Quoted 4 lines
M.--
Jazz is not dead. It just smells funny...
Marc Zyngier
maintainerRe: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Quoted 60 lines · Jianyong Wu
Quoted 43 lines
-----Original Message-----
From: Marc Zyngier <[email protected]>
Sent: Wednesday, August 19, 2020 6:00 PM
To: Jianyong Wu <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileQuoted 30 lines · Jianyong Wu
Hi Marc,
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <[email protected]>
Cc: Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileMove the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as
arch_timer_ctx_index) can work correctly at an early stage.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)Quoted 16 lines
Quoted 11 lines
arch/arm64/kvm/arch_timer.c +11
-0+static u64 timer_get_offset (struct arch_timer_context *ctxt) { +struct kvm_vcpu *vcpu = ctxt->vcpu; + +switch (arch_timer_ctx_index (ctxt)) { +case TIMER_VTIMER : +return __vcpu_sys_reg (vcpu,CNTVOFF_EL2 ); +default : +return 0 ; + } +} +Quoted 2 lines
Can I export this helper? As in my
ptp_kvmimplementation I need get VCNT offset value separately not just give me a result of VCNT.Sorry, you need to give me a bit more context. What do you need the offset for exactly?
Quoted 10 lines
Yeah,
In myptp_kvmimplementation, I need acquire wall time and counter cycle in the same time in host. After get host counter cycle, I need subtract it by VCNT offset to obtain VCNT. See
https://lkml.org/lkml/2020/6/19/441
https://lkml.org/lkml/2020/6/19/441
But now I can't get the VCNT offset easily like before using " vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like "kvm_arm_timer_read" as I need acquire the counter cycle in the same time with the host wall time.
I must be missing something. CNTVOFF_EL2 is now implemented as a standard system register, and has the same visibility as any other vcpu sysreg.
Why doesn't vcpu_read_sys_reg(vcpu, CNTVOFF_EL2) work for you?
M.
Signature · 5 lines
Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
RE: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file
Quoted 86 lines
-----Original Message-----
From: Marc Zyngier <[email protected]>
Sent: Wednesday, August 19, 2020 6:39 PM
To: Jianyong Wu <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; linux- [email protected]
Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileQuoted 60 lines · Jianyong Wu
Quoted 43 lines
-----Original Message-----
From: Marc Zyngier <[email protected]>
Sent: Wednesday, August 19, 2020 6:00 PM
To: Jianyong Wu <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileQuoted 30 lines · Jianyong Wu
Hi Marc,
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <[email protected]>
Cc: Peng Hao <[email protected]>; [email protected]; [email protected]; Will Deacon <[email protected]>; Catalin Marinas <[email protected]>; Alexander Graf <[email protected]>; [email protected]; [email protected]
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to thesys_regsfileMove the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.
This requires moving the initialisation of the timer struct so that some of the helpers (such as
arch_timer_ctx_index) can work correctly at an early stage.Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-)Quoted 16 lines
Quoted 11 lines
arch/arm64/kvm/arch_timer.c +11
-0+static u64 timer_get_offset (struct arch_timer_context *ctxt) { +struct kvm_vcpu *vcpu = ctxt->vcpu; + +switch (arch_timer_ctx_index (ctxt)) { +case TIMER_VTIMER : +return __vcpu_sys_reg (vcpu,CNTVOFF_EL2 ); +default : +return 0 ; + } +} +Quoted 2 lines
Can I export this helper? As in my
ptp_kvmimplementation I need get VCNT offset value separately not just give me a result of VCNT.Sorry, you need to give me a bit more context. What do you need the offset for exactly?
Quoted 10 lines
Yeah,
In myptp_kvmimplementation, I need acquire wall time and counter cycle in the same time in host. After get host counter cycle, I need subtract it by VCNT offset to obtain VCNT. See
https://lkml.org/lkml/2020/6/19/441
https://lkml.org/lkml/2020/6/19/441
But now I can't get the VCNT offset easily like before using " vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like "kvm_arm_timer_read" as I need acquire the counter cycle in the same time with the host wall time.
Quoted 4 lines
I must be missing something.
CNTVOFF_EL2is now implemented as a standard system register, and has the same visibility as any other vcpu sysreg.Why doesn't
vcpu_read_sys_reg(vcpu, CNTVOFF_EL2)work for you?
Maybe it serves me, let me try it, thanks.
Thanks
Jianyong
Quoted 4 lines
M.--
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 48/56 KVM: arm64: Don't use has_vhe() for CHOOSE_HYP_SYM()
In reply to Marc ZyngierThe recently introduced CHOOSE_HYP_SYM() macro picks one symbol or another, depending on whether the kernel run as a VHE hypervisor or not. For that, it uses the has_vhe() helper, which is itself implemented as a final capability.
Unfortunately, __copy_hyp_vect_bpi now indirectly uses CHOOSE_HYP_SYM to get the __bp_harden_hyp_vecs symbol, using has_vhe() in the process. At this stage, the capability isn't final and things explode:
[ 0.000000] ACPI: SRAT not present [ 0.000000] percpu: Embedded 34 pages/cpu s101264 r8192 d29808 u139264 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] ------------[ cut here ]------------ [ 0.000000] kernel BUG at arch/arm64/include/asm/cpufeature.h:459! [ 0.000000] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc4-00080-gd630681366e5 #1388 [ 0.000000] pstate: 80000085 (Nzcv daIf -PAN -UAO BTYPE=--) [ 0.000000] pc : check_branch_predictor+0x3a4/0x408 [ 0.000000] lr : check_branch_predictor+0x2a4/0x408 [ 0.000000] sp : ffff800011693e90 [ 0.000000] x29: ffff800011693e90 x28: ffff8000116a1530 [ 0.000000] x27: ffff8000112c1008 x26: ffff800010ca6ff8 [ 0.000000] x25: ffff8000112c1000 x24: ffff8000116a1320 [ 0.000000] x23: 0000000000000000 x22: ffff8000112c1000 [ 0.000000] x21: ffff800010177120 x20: ffff8000116ae108 [ 0.000000] x19: 0000000000000000 x18: ffff800011965c90 [ 0.000000] x17: 0000000000022000 x16: 0000000000000003 [ 0.000000] x15: 00000000ffffffff x14: ffff8000118c3a38 [ 0.000000] x13: 0000000000000021 x12: 0000000000000022 [ 0.000000] x11: d37a6f4de9bd37a7 x10: 000000000000001d [ 0.000000] x9 : 0000000000000000 x8 : ffff800011f8dad8 [ 0.000000] x7 : ffff800011965ad0 x6 : 0000000000000003 [ 0.000000] x5 : 0000000000000000 x4 : 0000000000000000 [ 0.000000] x3 : 0000000000000100 x2 : 0000000000000004 [ 0.000000] x1 : ffff8000116ae148 x0 : 0000000000000000 [ 0.000000] Call trace: [ 0.000000] check_branch_predictor+0x3a4/0x408 [ 0.000000] update_cpu_capabilities+0x84/0x138 [ 0.000000] init_cpu_features+0x2c0/0x2d8 [ 0.000000] cpuinfo_store_boot_cpu+0x54/0x64 [ 0.000000] smp_prepare_boot_cpu+0x2c/0x60 [ 0.000000] start_kernel+0x16c/0x574 [ 0.000000] Code: 17ffffc7 91010281 14000198 17ffffca (d4210000)
This is addressed using a two-fold process:
- Replace has_vhe() with is_kernel_in_hyp_mode(), which tests whether we are running at EL2.
- Make CHOOSE_HYP_SYM() return an *undefined* symbol when compiled in the nVHE hypervisor, as we really should never use this helper in the nVHE-specific code.
With this in place, we're back to a bootable kernel again.
--- arch/arm64/include/asm/kvm_asm.h | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_asm.h +19 -1
@@ -62,8 +62,26 @@#define CHOOSE_VHE_SYM (sym) sym#define CHOOSE_NVHE_SYM (sym) kvm_nvhe_sym(sym) -#define CHOOSE_HYP_SYM (sym) (has_vhe() ? CHOOSE_VHE_SYM(sym) \ + +#ifndef __KVM_NVHE_HYPERVISOR__ +/* +* BIG FAT WARNINGS: +* +* - Don't be tempted to change the following is_kernel_in_hyp_mode() +* to has_vhe(). has_vhe() is implemented as a *final* capability, +* while this is used early at boot time, when the capabilities are +* not final yet.... +* +* - Don't let the nVHE hypervisor have access to this, as it will +* pick the *wrong* symbol (yes, it runs at EL2...). +*/ +#define CHOOSE_HYP_SYM (sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \ : CHOOSE_NVHE_SYM(sym)) +#else +/* The nVHE hypervisor shouldn't even try to access anything */ +extern void *__nvhe_undefined_symbol; +#define CHOOSE_HYP_SYM (sym) __nvhe_undefined_symbol +#endif /* Translate a kernel address @ptr into its equivalent linear mapping */ #define kvm_ksym_ref (ptr) \
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 49/56 KVM: arm64: Make nVHE ASLR conditional on RANDOMIZE_BASE
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
If there are spare bits in non-VHE hyp VA, KVM unconditionally replaces them with a random tag chosen at init. Disable this if the kernel is built without RANDOMIZE_BASE to align with kernel behavior.
--- arch/arm64/kvm/va_layout.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/va_layout.c +1 -1
@@ -48,7 +48,7 @@ __init void kvm_compute_layout(void) va_mask =GENMASK_ULL (tag_lsb -1 ,0 ); tag_val = hyp_va_msb; -if (tag_lsb != (vabits_actual -1 )) { +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) && tag_lsb != (vabits_actual -1 )) {/* We have some free bits to insert a random tag. */ tag_val |=get_random_long () &GENMASK_ULL (vabits_actual -2 , tag_lsb); }
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 50/56 KVM: arm64: Substitute RANDOMIZE_BASE for HARDEN_EL2_VECTORS
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
The HARDEN_EL2_VECTORS config maps vectors at a fixed location on cores which are susceptible to Spector variant 3a (A57, A72) to prevent defeating hyp layout randomization by leaking the value of VBAR_EL2.
Since this feature is only applicable when EL2 layout randomization is enabled, unify both behind the same RANDOMIZE_BASE Kconfig. Majority of code remains conditional on a capability selected for the affected cores.
--- arch/arm64/Kconfig | 16 ---------------- arch/arm64/include/asm/mmu.h | 6 ++---- arch/arm64/kernel/cpu_errata.c | 4 ++-- arch/arm64/kvm/Kconfig | 2 +- 4 files changed, 5 insertions(+), 23 deletions(-)
arch/arm64/Kconfig +0 -16
@@ -1182,22 +1182,6 @@ config HARDEN_BRANCH_PREDICTOR If unsure, say Y. -config HARDEN_EL2_VECTORS - bool "Harden EL2 vector mapping against system register leak" if EXPERT - default y - help - Speculation attacks against some high-performance processors can - be used to leak privileged information such as the vector base - register, resulting in a potential defeat of the EL2 layout - randomization. - - This config option will map the vectors to a fixed location, - independent of the EL2 code mapping, so that revealing VBAR_EL2 - to an attacker does not give away any extra information. This - only gets enabled on affected CPUs. - - If unsure, say Y. - config ARM64_SSBD bool "Speculative Store Bypass Disable" if EXPERT default y
arch/arm64/include/asm/mmu.h +2 -4
@@ -42,12 +42,10 @@ struct bp_hardening_data {bp_hardening_cb_t fn; }; -#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR ) || \ - defined(CONFIG_HARDEN_EL2_VECTORS )) - +#ifdef CONFIG_KVM_INDIRECT_VECTORS extern char __bp_harden_hyp_vecs[];extern atomic_t arm64_el2_vector_last_slot; -#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */ +#endif #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
arch/arm64/kernel/cpu_errata.c +2 -2
@@ -635,7 +635,7 @@ has_neoverse_n1_erratum_1542419(const struct arm64_cpu_capabilities *entry,return is_midr_in_range (midr, &range) && has_dic; } -#if defined(CONFIG_HARDEN_EL2_VECTORS ) +#ifdef CONFIG_RANDOMIZE_BASE static const struct midr_range ca57_a72[] = { MIDR_ALL_VERSIONS(MIDR_CORTEX_A57),
@@ -880,7 +880,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, .matches = check_branch_predictor, }, -#ifdef CONFIG_HARDEN_EL2_VECTORS +#ifdef CONFIG_RANDOMIZE_BASE { .desc = "EL2 vector hardening", .capability = ARM64_HARDEN_EL2_VECTORS,
arch/arm64/kvm/Kconfig +1 -1
@@ -58,7 +58,7 @@ config KVM_ARM_PMU virtual machines. config KVM_INDIRECT_VECTORS - def_bool HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS + def_bool HARDEN_BRANCH_PREDICTOR || RANDOMIZE_BASE endif # KVM
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 51/56 KVM: arm64: Ensure that all nVHE hyp code is in .hyp.text
In reply to Marc ZyngierFrom: David Brazdil <[email protected]>
Some compilers may put a subset of generated functions into '.text.*' ELF sections and the linker may leverage this division to optimize ELF layout. Unfortunately, the recently introduced HYPCOPY command assumes that all executable code (with the exception of specialized sections such as '.hyp.idmap.text') is in the '.text' section. If this assumption is broken, code in '.text.*' will be merged into kernel proper '.text' instead of the '.hyp.text' that is mapped in EL2.
To ensure that this cannot happen, insert an OBJDUMP assertion into HYPCOPY. The command dumps a list of ELF sections in the input object file and greps for '.text.'. If found, compilation fails. Tested with both binutils' and LLVM's objdump (the output format is different).
GCC offers '-fno-reorder-functions' to disable this behaviour. Select the flag if it is available. From inspection of GCC source (latest Git in July 2020), this flag does force all code into '.text'. By default, GCC uses profile data, heuristics and attributes to select a subsection.
LLVM/Clang currently does not have a similar optimization pass. It can place static constructors into '.text.startup' and it's optimizer can be provided with profile data to reorder hot/cold functions. Neither of these is applicable to nVHE hyp code. If this changes in the future, the OBJDUMP assertion should alert users to the problem.
--- arch/arm64/kvm/hyp/nvhe/Makefile | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/nvhe/Makefile +23 -3
@@ -20,10 +20,30 @@ $(obj)/%.hyp.tmp.o: $(src)/%.S FORCE $(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE $(call if_changed,hypcopy) +# Disable reordering functions by GCC (enabled at -O2). +# This pass puts functions into '.text.*' sections to aid the linker +# in optimizing ELF layout. See HYPCOPY comment below for more info. +ccflags-y += $(call cc-option,-fno-reorder-functions) + +# The HYPCOPY command uses `objcopy` to prefix all ELF symbol names +# and relevant ELF section names to avoid clashes with VHE code/data. +# +# Hyp code is assumed to be in the '.text' section of the input object +# files (with the exception of specialized sections such as +# '.hyp.idmap.text'). This assumption may be broken by a compiler that +# divides code into sections like '.text.unlikely' so as to optimize +# ELF layout. HYPCOPY checks that no such sections exist in the input +# using `objdump`, otherwise they would be linked together with other +# kernel code and not memory-mapped correctly at runtime. quiet_cmd_hypcopy = HYPCOPY $@ - cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \ - --rename-section=.text=.hyp.text \ - $< $@ + cmd_hypcopy = \ + if $(OBJDUMP) -h $< | grep -F '.text.'; then \ + echo "$@: function reordering not supported in nVHE hyp code" >&2; \ + /bin/false; \ + fi; \ + $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \ + --rename-section=.text=.hyp.text \ + $< $@ # Remove ftrace and Shadow Call Stack CFLAGS. # This is equivalent to the 'notrace' and '__noscs' annotations.
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 52/56 KVM: arm: Add trace name for ARM_NISV
In reply to Marc ZyngierFrom: Alexander Graf <[email protected]>
Commit c726200dd106d ("KVM: arm/arm64: Allow reporting non-ISV data aborts to userspace") introduced a mechanism to deflect MMIO traffic the kernel can not handle to user space. For that, it introduced a new exit reason.
However, it did not update the trace point array that gives human readable names to these exit reasons inside the trace log.
Let's fix that up after the fact, so that trace logs are pretty even when we get user space MMIO traps on ARM.
--- include/trace/events/kvm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/trace/events/kvm.h +1 -1
@@ -17,7 +17,7 @@ERSN (NMI ),ERSN (INTERNAL_ERROR ),ERSN (OSI ),ERSN (PAPR_HCALL ), \ERSN (S390_UCONTROL ),ERSN (WATCHDOG ),ERSN (S390_TSCH ),ERSN (EPR ),\ERSN (SYSTEM_EVENT ),ERSN (S390_STSI ),ERSN (IOAPIC_EOI ), \ - ERSN(HYPERV) + ERSN(HYPERV), ERSN(ARM_NISV) TRACE_EVENT(kvm_userspace_exit, TP_PROTO(__u32 reason, int errno),
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 53/56 KVM: arm64: Rename kvm_vcpu_dabt_isextabt()
In reply to Marc ZyngierFrom: Will Deacon <[email protected]>
kvm_vcpu_dabt_isextabt() is not specific to data aborts and, unlike kvm_vcpu_dabt_issext(), has nothing to do with sign extension.
Rename it to 'kvm_vcpu_abt_issea()'.
--- arch/arm64/include/asm/kvm_emulate.h | 2 +- arch/arm64/kvm/hyp/switch.c | 2 +- arch/arm64/kvm/mmu.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +1 -1
@@ -366,7 +366,7 @@ static __always_inline u8 kvm_vcpu_trap_get_fault_type(const struct kvm_vcpu *vcreturn kvm_vcpu_get_esr (vcpu) & ESR_ELx_FSC_TYPE; } -static __always_inline bool kvm_vcpu_dabt_isextabt(const struct kvm_vcpu *vcpu) +static __always_inline bool kvm_vcpu_abt_issea(const struct kvm_vcpu *vcpu) {switch (kvm_vcpu_trap_get_fault (vcpu)) {case FSC_SEA :
arch/arm64/kvm/hyp/switch.c +1 -1
@@ -594,7 +594,7 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) valid =kvm_vcpu_trap_get_class (vcpu) == ESR_ELx_EC_DABT_LOW &&kvm_vcpu_trap_get_fault_type (vcpu) ==FSC_FAULT &&kvm_vcpu_dabt_isvalid (vcpu) && - !kvm_vcpu_dabt_isextabt (vcpu) && + !kvm_vcpu_abt_issea (vcpu) && !kvm_vcpu_dabt_iss1tw (vcpu);if (valid) {
arch/arm64/kvm/mmu.c +1 -1
@@ -2074,7 +2074,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) is_iabt =kvm_vcpu_trap_is_iabt (vcpu);/* Synchronous External Abort? */ -if (kvm_vcpu_dabt_isextabt (vcpu)) { +if (kvm_vcpu_abt_issea (vcpu)) {/* * For RAS the host kernel may handle this abort. * There is no need to pass the error into the guest.
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 54/56 KVM: arm64: Handle data and instruction external aborts the same way
In reply to Marc ZyngierFrom: Will Deacon <[email protected]>
If the guest generates a synchronous external abort which is not handled by the host, we inject it back into the guest as a virtual SError, but only if the original fault was reported on the data side. Instruction faults are reported as "Unsupported FSC", causing the vCPU run loop to bail with -EFAULT.
Although synchronous external aborts from a guest are pretty unusual, treat them the same regardless of whether they are taken as data or instruction aborts by EL2.
--- arch/arm64/kvm/mmu.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
arch/arm64/kvm/mmu.c +3 -6
@@ -2079,13 +2079,10 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)* For RAS the host kernel may handle this abort. * There is no need to pass the error into the guest. */ -if (!kvm_handle_guest_sea (fault_ipa,kvm_vcpu_get_esr (vcpu))) -return 1 ; - -if (unlikely (!is_iabt)) { +if (kvm_handle_guest_sea (fault_ipa,kvm_vcpu_get_esr (vcpu)))kvm_inject_vabt (vcpu); -return 1 ; - } + +return 1 ; } trace_kvm_guest_fault(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 55/56 KVM: arm64: Don't skip cache maintenance for read-only memslots
In reply to Marc ZyngierFrom: Will Deacon <[email protected]>
If a guest performs cache maintenance on a read-only memslot, we should inform userspace rather than skip the instruction altogether.
--- arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/mmu.c +1 -1
@@ -2121,7 +2121,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)* So let's assume that the guest is just being * cautious, and skip the instruction. */ -if (kvm_vcpu_dabt_is_cm (vcpu)) { +if (kvm_is_error_hva (hva) &&kvm_vcpu_dabt_is_cm (vcpu)) {kvm_skip_instr (vcpu,kvm_vcpu_trap_il_is32bit (vcpu)); ret =1 ; goto out_unlock;
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH 56/56 KVM: arm64: Move S1PTW S2 fault logic out of io_mem_abort()
In reply to Marc ZyngierFrom: Will Deacon <[email protected]>
To allow for re-injection of stage-2 faults on stage-1 page-table walks due to either a missing or read-only memslot, move the triage logic out of io_mem_abort() and into kvm_handle_guest_abort(), where these aborts can be handled before anything else.
--- arch/arm64/kvm/mmio.c | 6 ------ arch/arm64/kvm/mmu.c | 13 ++++++++++++- 2 files changed, 12 insertions(+), 7 deletions(-)
arch/arm64/kvm/mmio.c +0 -6
@@ -145,12 +145,6 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run,return -ENOSYS ; } -/* Page table accesses IO mem: tell guest to fix its TTBR */ -if (kvm_vcpu_dabt_iss1tw (vcpu)) { -kvm_inject_dabt (vcpu,kvm_vcpu_get_hfar (vcpu)); -return 1 ; - } -/* * Prepare MMIO operation. First decode the syndrome data we get * from the CPU. Then try if some in-kernel emulation feels
arch/arm64/kvm/mmu.c +12 -1
@@ -2105,12 +2105,23 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) hva =gfn_to_hva_memslot_prot (memslot, gfn, &writable); write_fault =kvm_is_write_fault (vcpu);if (kvm_is_error_hva (hva) || (write_fault && !writable)) { +/* +* The guest has put either its instructions or its page-tables +* somewhere it shouldn't have. Userspace won't be able to do +* anything about this (there's no syndrome for a start), so +* re-inject the abort back into the guest. +*/ if (is_iabt) { -/* Prefetch Abort on I/O address */ ret = -ENOEXEC ; goto out; } +if (kvm_vcpu_dabt_iss1tw (vcpu)) { +kvm_inject_dabt (vcpu,kvm_vcpu_get_hfar (vcpu)); + ret =1 ; + goto out_unlock; + } +/* * Check for a cache maintenance operation. Since we * ended-up here, we know it is outside of any memory
Signature · 6 lines
2.27.0 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Paolo Bonzini
maintainerQuoted 13 lines · Marc Zyngier
Hi Paolo,
This is the bulk of the 5.9 patches for KVM/arm64. It is a pretty busy merge window for us this time, thanks to the ongoing Protected KVM work. We have changes all over the map, but the most important piece probably is the way we now build the EL2 code on non-VHE systems. On top of giving us better control over what gets pulled in there, it allowed us to enable instrumentation on VHE systems.
The rest is a mixed bag of new features (TTL TLB invalidation, Pointer Auth on non-VHE), preliminary patches for NV, some early MMU rework before the 5.10 onslaught, and tons of cleanups.
Quoted 7 lines
A few things to notice:
- We share a branch with the arm64 tree, which has gone in already.
- There are a number of known conflicts with Sean's MMU cache rework, as well as the late fixes that went in 5.8. The conflicts are pretty simple to resolve, and -next has the right resolutions already.
Ok, since I have already an x86 conflict I'll wait to pull this until next week.
Paolo
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Paolo Bonzini
maintainerQuoted 1 line · Marc Zyngier
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.gittags/kvmarm-5.9
Pulled, thanks.
Paolo
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm