KVM: arm64: Preamble for pKVM
Subsystems (4)
v3 cover KVM: arm64: Preamble for pKVM
Changes from V2 [0]:
- Dropped patches that are better suited for later on with pKVM [Marc]
- Squashed some patches together to make it easier for maintainers [Marc]
- Moved all fixes to the beginning of the series [patches 1 - 13]
- Misc fixes [Mark, Marc, Oliver]
We are getting closer to upstreaming the remaining part of pKVM [1]. To make the process easier for us and for our dear reviewers, we are sending out this patch series as a preamble to the upcoming patches.
This series is based on Linux 6.9-rc3 -- kvmarm/next (9ac5bab4deee). Most of the patches in this series are self-standing, and can be applied directly.
This series is a bit of a bombay-mix of patches we've been carrying. There's no one overarching theme, but they do improve the code by fixing existing bugs in pKVM, refactoring code to make it more readable and easier to re-use for pKVM, or adding functionality to the existing pKVM code upstream.
None of the patches in this series intentionally affect the functionality of non-protected modes. Patches 1 to 13 are fixes.
For a technical deep dive into pKVM, please refer to Quentin Perret's KVM Forum Presentation [2, 3]. The pKVM core series, which we plan on sending for review next, the code is here [1].
Cheers,
Fuad, Quentin, Will, and Marc
[0] https://lore.kernel.org/all/[email protected]/
[1] https://android-kvm.googlesource.com/linux/+/refs/heads/for-upstream/pkvm-core
[2] Protected KVM on arm64 (slides)
https://static.sched.com/hosted_files/kvmforum2022/88/KVM%20forum%202022%20-%20pKVM%20deep%20dive.pdf
[3] Protected KVM on arm64 (video)
https://www.youtube.com/watch?v=9npebeVFbFw
Fuad Tabba (15):
KVM: arm64: Initialize the kvm host data's fpsimd_state pointer in
pKVM
KVM: arm64: Move guest_owns_fp_regs() to increase its scope
KVM: arm64: Refactor checks for FP state ownership
KVM: arm64: Do not re-initialize the KVM lock
KVM: arm64: Do not map the host fpsimd state to hyp in pKVM
KVM: arm64: Fix comment for __pkvm_vcpu_init_traps()
KVM: arm64: Change kvm_handle_mmio_return() return polarity
KVM: arm64: Move setting the page as dirty out of the critical section
KVM: arm64: Introduce and use predicates that check for protected VMs
KVM: arm64: Move pstate reset value definitions to kvm_arm.h
KVM: arm64: Clarify rationale for ZCR_EL1 value restored on guest exit
KVM: arm64: Refactor calculating SVE state size to use helpers
KVM: arm64: Move some kvm_psci functions to a shared header
KVM: arm64: Refactor reset_mpidr() to extract its computation
KVM: arm64: Refactor kvm_vcpu_enable_ptrauth() for hyp use
Marc Zyngier (3):
KVM: arm64: Check for PTE validity when checking for
executable/cacheable
KVM: arm64: Simplify vgic-v3 hypercalls
KVM: arm64: Force injection of a data abort on NISV MMIO exit
Quentin Perret (4):
KVM: arm64: Issue CMOs when tearing down guest s2 pages
KVM: arm64: Avoid BUG-ing from the host abort path
KVM: arm64: Prevent kmemleak from accessing .hyp.data
KVM: arm64: Add is_pkvm_initialized() helper
Will Deacon (9):
KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE
KVM: arm64: Support TLB invalidation in guest context
KVM: arm64: Remove locking from EL2 allocation fast-paths
KVM: arm64: Introduce hyp_rwlock_t
KVM: arm64: Add atomics-based checking refcount implementation at EL2
KVM: arm64: Use atomic refcount helpers for 'struct
hyp_page::refcount'
KVM: arm64: Reformat/beautify PTP hypercall documentation
KVM: arm64: Rename firmware pseudo-register documentation file
KVM: arm64: Document the KVM/arm64-specific calls in hypercalls.rst
Documentation/virt/kvm/api.rst | 7 + .../virt/kvm/arm/fw-pseudo-registers.rst | 138 ++++++++++++++ Documentation/virt/kvm/arm/hypercalls.rst | 180 +++++------------- Documentation/virt/kvm/arm/index.rst | 1 + Documentation/virt/kvm/arm/ptp_kvm.rst | 38 ++-- arch/arm64/include/asm/kvm_arm.h | 12 ++ arch/arm64/include/asm/kvm_asm.h | 8 +- arch/arm64/include/asm/kvm_emulate.h | 11 +- arch/arm64/include/asm/kvm_host.h | 39 ++-- arch/arm64/include/asm/kvm_hyp.h | 4 +- arch/arm64/include/asm/virt.h | 12 +- arch/arm64/kvm/arm.c | 21 +- arch/arm64/kvm/fpsimd.c | 60 +++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 6 +- arch/arm64/kvm/hyp/include/nvhe/memory.h | 18 +- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 6 + arch/arm64/kvm/hyp/include/nvhe/refcount.h | 76 ++++++++ arch/arm64/kvm/hyp/include/nvhe/rwlock.h | 129 +++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 24 +-- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 21 +- arch/arm64/kvm/hyp/nvhe/pkvm.c | 14 +- arch/arm64/kvm/hyp/nvhe/setup.c | 1 + arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- arch/arm64/kvm/hyp/nvhe/tlb.c | 115 ++++++++--- arch/arm64/kvm/hyp/pgtable.c | 21 +- arch/arm64/kvm/hyp/vgic-v3-sr.c | 27 ++- arch/arm64/kvm/hyp/vhe/switch.c | 4 +- arch/arm64/kvm/mmio.c | 12 +- arch/arm64/kvm/mmu.c | 8 +- arch/arm64/kvm/pkvm.c | 2 +- arch/arm64/kvm/psci.c | 28 --- arch/arm64/kvm/reset.c | 20 +- arch/arm64/kvm/sys_regs.c | 14 +- arch/arm64/kvm/sys_regs.h | 19 ++ arch/arm64/kvm/vgic/vgic-v2.c | 9 +- arch/arm64/kvm/vgic/vgic-v3.c | 23 +-- arch/arm64/kvm/vgic/vgic.c | 11 -- arch/arm64/kvm/vgic/vgic.h | 2 - include/kvm/arm_psci.h | 29 +++ include/kvm/arm_vgic.h | 1 - 42 files changed, 775 insertions(+), 424 deletions(-) create mode 100644 Documentation/virt/kvm/arm/fw-pseudo-registers.rst create mode 100644 arch/arm64/kvm/hyp/include/nvhe/refcount.h create mode 100644 arch/arm64/kvm/hyp/include/nvhe/rwlock.h
base-commit: 9ac5bab4deeeeb99f36695250b99c2f9bfae2379
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 1/31 KVM: arm64: Initialize the kvm host data's fpsimd_state pointer in pKVM
Since the host_fpsimd_state has been removed from kvm_vcpu_arch, it isn't pointing to the hyp's version of the host fp_regs in protected mode.
Initialize the host_data fpsimd_state point to the host_data's context fp_regs on pKVM initialization.
--- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 1 + arch/arm64/kvm/hyp/nvhe/pkvm.c | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 1 + 3 files changed, 13 insertions(+)
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +1 -0
@@ -54,6 +54,7 @@ pkvm_hyp_vcpu_to_hyp_vm(struct pkvm_hyp_vcpu *hyp_vcpu) }void pkvm_hyp_vm_table_init (void *tbl); +void pkvm_host_fpsimd_state_init (void );int __pkvm_init_vm (struct kvm *host_kvm,unsigned long vm_hva,unsigned long pgd_hva);
arch/arm64/kvm/hyp/nvhe/pkvm.c +11 -0
@@ -247,6 +247,17 @@ void pkvm_hyp_vm_table_init(void *tbl) vm_table = tbl; } +void pkvm_host_fpsimd_state_init (void ) +{ +unsigned long i; + +for (i =0 ; i < hyp_nr_cpus; i++) { +struct kvm_host_data *host_data =per_cpu_ptr (&kvm_host_data, i); + + host_data->fpsimd_state = &host_data->host_ctxt.fp_regs; + } +} +/* * Return the hyp vm structure corresponding to the handle. */
arch/arm64/kvm/hyp/nvhe/setup.c +1 -0
@@ -300,6 +300,7 @@ void __noreturn __pkvm_init_finalise(void) goto out;pkvm_hyp_vm_table_init (vm_table_base); +pkvm_host_fpsimd_state_init (); out:/* * We tail-called to here from handle___pkvm_init() and will not return,
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 2/31 KVM: arm64: Move guest_owns_fp_regs() to increase its scope
In reply to Fuad Tabbaguest_owns_fp_regs() will be used to check fpsimd state ownership across kvm/arm64. Therefore, move it to kvm_host.h to widen its scope.
Moreover, the host state is not per-vcpu anymore, the vcpu parameter isn't used, so remove it as well.
No functional change intended.
--- arch/arm64/include/asm/kvm_host.h | 6 ++++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 6 ------ arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_host.h +6 -0
@@ -1207,6 +1207,12 @@ DECLARE_KVM_HYP_PER_CPU(struct kvm_host_data, kvm_host_data); &this_cpu_ptr_hyp_sym(kvm_host_data)->f)#endif +/* Check whether the FP regs are owned by the guest */ +static inline bool guest_owns_fp_regs (void ) +{ +return *host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ; +} +static inline void kvm_init_host_cpu_context (struct kvm_cpu_context *cpu_ctxt) {/* The host's MPIDR is immutable, so let's set it up at boot time */
arch/arm64/kvm/hyp/include/hyp/switch.h +0 -6
@@ -39,12 +39,6 @@ struct kvm_exception_table_entry {extern struct kvm_exception_table_entry __start___kvm_ex_table;extern struct kvm_exception_table_entry __stop___kvm_ex_table; -/* Check whether the FP regs are owned by the guest */ -static inline bool guest_owns_fp_regs (struct kvm_vcpu *vcpu) -{ -return *host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ; -} -/* Save the 32-bit only FPSIMD system register state */ static inline void __fpsimd_save_fpexc32 (struct kvm_vcpu *vcpu) {
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -53,7 +53,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) val |=CPTR_EL2_TSM ; } -if (!guest_owns_fp_regs (vcpu)) { +if (!guest_owns_fp_regs ()) {if (has_hvhe ()) val &= ~(CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN |CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN );
arch/arm64/kvm/hyp/vhe/switch.c +1 -1
@@ -75,7 +75,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) val |=CPTR_EL2_TAM ; -if (guest_owns_fp_regs (vcpu)) { +if (guest_owns_fp_regs ()) {if (vcpu_has_sve (vcpu)) val |=CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ; }else {
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 3/31 KVM: arm64: Refactor checks for FP state ownership
In reply to Fuad TabbaTo avoid direct comparison against the fp_owner enum, add a new function that performs the check, host_owns_fp_regs(), to complement the existing guest_owns_fp_regs().
To check for fpsimd state ownership, use the helpers instead of directly using the enums.
No functional change intended.
--- arch/arm64/include/asm/kvm_emulate.h | 6 ++---- arch/arm64/include/asm/kvm_host.h | 6 ++++++ arch/arm64/kvm/fpsimd.c | 5 ++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 2 +- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- 6 files changed, 13 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +2 -4
@@ -587,16 +587,14 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu) }else if (has_hvhe ()) { val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ); -if (!vcpu_has_sve (vcpu) || - (*host_data_ptr (fp_owner) !=FP_STATE_GUEST_OWNED )) +if (!vcpu_has_sve (vcpu) || !guest_owns_fp_regs ()) val |=CPACR_EL1_ZEN_EL1EN |CPACR_EL1_ZEN_EL0EN ;if (cpus_have_final_cap (ARM64_SME )) val |=CPACR_EL1_SMEN_EL1EN |CPACR_EL1_SMEN_EL0EN ; }else { val =CPTR_NVHE_EL2_RES1 ; -if (vcpu_has_sve (vcpu) && - (*host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED )) +if (vcpu_has_sve (vcpu) &&guest_owns_fp_regs ()) val |=CPTR_EL2_TZ ;if (cpus_have_final_cap (ARM64_SME )) val &= ~CPTR_EL2_TSM ;
arch/arm64/include/asm/kvm_host.h +6 -0
@@ -1213,6 +1213,12 @@ static inline bool guest_owns_fp_regs(void)return *host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ; } +/* Check whether the FP regs are owned by the host */ +static inline bool host_owns_fp_regs (void ) +{ +return *host_data_ptr (fp_owner) ==FP_STATE_HOST_OWNED ; +} +static inline void kvm_init_host_cpu_context (struct kvm_cpu_context *cpu_ctxt) {/* The host's MPIDR is immutable, so let's set it up at boot time */
arch/arm64/kvm/fpsimd.c +2 -3
@@ -141,8 +141,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)WARN_ON_ONCE (!irqs_disabled ()); -if (*host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ) { - +if (guest_owns_fp_regs ()) {/* * Currently we do not support SME guests so SVCR is * always 0 and we just need a variable to point to.
@@ -195,7 +194,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)isb (); } -if (*host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ) { +if (guest_owns_fp_regs ()) {if (vcpu_has_sve (vcpu)) {__vcpu_sys_reg (vcpu,ZCR_EL1 ) =read_sysreg_el1 (SYS_ZCR );
arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1
@@ -370,7 +370,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)isb ();/* Write out the host state if it's in the registers */ -if (*host_data_ptr (fp_owner) ==FP_STATE_HOST_OWNED ) +if (host_owns_fp_regs ())__fpsimd_save_state (*host_data_ptr (fpsimd_state));/* Restore the guest state */
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -337,7 +337,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)__sysreg_restore_state_nvhe (host_ctxt); -if (*host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ) +if (guest_owns_fp_regs ())__fpsimd_save_fpexc32 (vcpu);__debug_switch_to_host (vcpu);
arch/arm64/kvm/hyp/vhe/switch.c +1 -1
@@ -258,7 +258,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)sysreg_restore_host_state_vhe (host_ctxt); -if (*host_data_ptr (fp_owner) ==FP_STATE_GUEST_OWNED ) +if (guest_owns_fp_regs ())__fpsimd_save_fpexc32 (vcpu);__debug_switch_to_host (vcpu);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 4/31 KVM: arm64: Do not re-initialize the KVM lock
In reply to Fuad TabbaThe lock is already initialized in core KVM code at kvm_create_vm().
--- arch/arm64/kvm/pkvm.c | 1 - 1 file changed, 1 deletion(-)
arch/arm64/kvm/pkvm.c +0 -1
@@ -222,7 +222,6 @@ void pkvm_destroy_hyp_vm(struct kvm *host_kvm)int pkvm_init_host_vm (struct kvm *host_kvm) { -mutex_init (&host_kvm->lock);return 0 ; }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 5/31 KVM: arm64: Issue CMOs when tearing down guest s2 pages
In reply to Fuad TabbaFrom: Quentin Perret <[email protected]>
On the guest teardown path, pKVM will zero the pages used to back the guest data structures before returning them to the host as they may contain secrets (e.g. in the vCPU registers). However, the zeroing is done using a cacheable alias, and CMOs are missing, hence giving the host a potential opportunity to read the original content of the guest structs from memory.
Fix this by issuing CMOs after zeroing the pages.
--- arch/arm64/kvm/hyp/nvhe/pkvm.c | 1 + 1 file changed, 1 insertion(+)
arch/arm64/kvm/hyp/nvhe/pkvm.c +1 -0
@@ -441,6 +441,7 @@ static void *map_donated_memory(unsigned long host_va, size_t size)static void __unmap_donated_memory (void *va,size_t size) { +kvm_flush_dcache_to_poc (va, size);WARN_ON (__pkvm_hyp_donate_host (hyp_virt_to_pfn (va),PAGE_ALIGN (size) >>PAGE_SHIFT )); }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 6/31 KVM: arm64: Avoid BUG-ing from the host abort path
In reply to Fuad TabbaFrom: Quentin Perret <[email protected]>
Under certain circumstances __get_fault_info() may resolve the faulting address using the AT instruction. Given that this is being done outside of the host lock critical section, it is racy and the resolution via AT may fail. We currently BUG() in this situation, which is obviously less than ideal. Moving the address resolution to the critical section may have a performance impact, so let's keep it where it is, but bail out and return to the host to try a second time.
--- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/mem_protect.c +7 -1
@@ -533,7 +533,13 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)int ret =0 ; esr =read_sysreg_el2 (SYS_ESR ); -BUG_ON (!__get_fault_info (esr, &fault)); +if (!__get_fault_info (esr, &fault)) { +/* +* We've presumably raced with a page-table change which caused +* AT to fail, try again. +*/ +return ; + } addr = (fault.hpfar_el2 &HPFAR_MASK ) <<8 ; ret =host_stage2_idmap (addr);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 7/31 KVM: arm64: Check for PTE validity when checking for executable/cacheable
In reply to Fuad TabbaFrom: Marc Zyngier <[email protected]>
Don't just assume that the PTE is valid when checking whether it describes an executable or cacheable mapping.
This makes sure that we don't issue CMOs for invalid mappings.
--- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +3 -3
@@ -914,12 +914,12 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,static bool stage2_pte_cacheable (struct kvm_pgtable *pgt,kvm_pte_t pte) {u64 memattr = pte &KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR ; -return memattr ==KVM_S2_MEMATTR (pgt,NORMAL ); +return kvm_pte_valid (pte) && memattr ==KVM_S2_MEMATTR (pgt,NORMAL ); }static bool stage2_pte_executable (kvm_pte_t pte) { -return !(pte &KVM_PTE_LEAF_ATTR_HI_S2_XN ); +return kvm_pte_valid (pte) && !(pte &KVM_PTE_LEAF_ATTR_HI_S2_XN ); } static u64 stage2_map_walker_phys_addr(const struct kvm_pgtable_visit_ctx *ctx,
@@ -1370,7 +1370,7 @@ static int stage2_flush_walker(const struct kvm_pgtable_visit_ctx *ctx,struct kvm_pgtable *pgt = ctx->arg;struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops; -if (!kvm_pte_valid (ctx->old) || !stage2_pte_cacheable (pgt, ctx->old)) +if (!stage2_pte_cacheable (pgt, ctx->old))return 0 ;if (mm_ops->dcache_clean_inval_poc)
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 8/31 KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
Break-before-make (BBM) can be expensive, as transitioning via an invalid mapping (i.e. the "break" step) requires the completion of TLB invalidation and can also cause other agents to fault concurrently on the invalid mapping.
Since BBM is not required when changing only the software bits of a PTE, avoid the sequence in this case and just update the PTE directly.
--- arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/arm64/kvm/hyp/pgtable.c +15 -0
@@ -979,6 +979,21 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,if (!stage2_pte_needs_update (ctx->old, new))return -EAGAIN ; +/* If we're only changing software bits, then store them and go! */ +if (!kvm_pgtable_walk_shared (ctx) && + !((ctx->old ^ new) & ~KVM_PTE_LEAF_ATTR_HI_SW )) { +bool old_is_counted =stage2_pte_is_counted (ctx->old); + +if (old_is_counted !=stage2_pte_is_counted (new)) { +if (old_is_counted) + mm_ops->put_page (ctx->ptep); +else + mm_ops->get_page (ctx->ptep); + } +WARN_ON_ONCE (!stage2_try_set_pte (ctx, new)); +return 0 ; + } +if (!stage2_try_break_pte (ctx, data->mmu))return -EAGAIN ;
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 9/31 KVM: arm64: Support TLB invalidation in guest context
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
Typically, TLB invalidation of guest stage-2 mappings using nVHE is performed by a hypercall originating from the host. For the invalidation instruction to be effective, therefore, __tlb_switch_to_{guest,host}() swizzle the active stage-2 context around the TLBI instruction.
With guest-to-host memory sharing and unsharing hypercalls originating from the guest under pKVM, there is need to support both guest and host VMID invalidations issued from guest context.
Replace the __tlb_switch_to_{guest,host}() functions with a more general {enter,exit}_vmid_context() implementation which supports being invoked from guest context and acts as a no-op if the target context matches the running context.
--- arch/arm64/kvm/hyp/nvhe/tlb.c | 115 +++++++++++++++++++++++++++------- 1 file changed, 91 insertions(+), 24 deletions(-)
arch/arm64/kvm/hyp/nvhe/tlb.c +91 -24
@@ -11,13 +11,23 @@#include <nvhe/mem_protect.h> struct tlb_inv_context { -u64 tcr; +struct kvm_s2_mmu *mmu; +u64 tcr; +u64 sctlr; }; -static void __tlb_switch_to_guest (struct kvm_s2_mmu *mmu, -struct tlb_inv_context *cxt, -bool nsh) +static void enter_vmid_context (struct kvm_s2_mmu *mmu, +struct tlb_inv_context *cxt, +bool nsh) { +struct kvm_s2_mmu *host_s2_mmu = &host_mmu.arch.mmu; +struct kvm_cpu_context *host_ctxt; +struct kvm_vcpu *vcpu; + + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; + vcpu = host_ctxt->__hyp_running_vcpu; + cxt->mmu =NULL ; +/* * We have two requirements: *
@@ -40,20 +50,55 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,else dsb (ish); +/* +* If we're already in the desired context, then there's nothing to do. +*/ +if (vcpu) { +/* +* We're in guest context. However, for this to work, this needs +* to be called from within __kvm_vcpu_run(), which ensures that +* __hyp_running_vcpu is set to the current guest vcpu. +*/ +if (mmu == vcpu->arch.hw_mmu ||WARN_ON (mmu != host_s2_mmu)) +return ; + + cxt->mmu = vcpu->arch.hw_mmu; + }else { +/* We're in host context. */ +if (mmu == host_s2_mmu) +return ; + + cxt->mmu = host_s2_mmu; + } +if (cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT )) {u64 val;/* * For CPUs that are affected by ARM 1319367, we need to -* avoid a host Stage-1 walk while we have the guest's -* VMID set in the VTTBR in order to invalidate TLBs. -* We're guaranteed that the S1 MMU is enabled, so we can -* simply set the EPD bits to avoid any further TLB fill. +* avoid a Stage-1 walk with the old VMID while we have +* the new VMID set in the VTTBR in order to invalidate TLBs. +* We're guaranteed that the host S1 MMU is enabled, so +* we can simply set the EPD bits to avoid any further +* TLB fill. For guests, we ensure that the S1 MMU is +* temporarily enabled in the next context. */ val = cxt->tcr =read_sysreg_el1 (SYS_TCR ); val |=TCR_EPD1_MASK |TCR_EPD0_MASK ;write_sysreg_el1 (val,SYS_TCR );isb (); + +if (vcpu) { + val = cxt->sctlr =read_sysreg_el1 (SYS_SCTLR ); +if (!(val & SCTLR_ELx_M)) { + val |= SCTLR_ELx_M; +write_sysreg_el1 (val,SYS_SCTLR ); +isb (); + } + }else { +/* The host S1 MMU is always enabled. */ + cxt->sctlr = SCTLR_ELx_M; + } }/*
@@ -62,18 +107,40 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,* ensuring that we always have an ISB, but not two ISBs back * to back. */ -__load_stage2 (mmu,kern_hyp_va (mmu->arch)); +if (vcpu) +__load_host_stage2 (); +else +__load_stage2 (mmu,kern_hyp_va (mmu->arch)); + asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT)); } -static void __tlb_switch_to_host (struct tlb_inv_context *cxt) +static void exit_vmid_context (struct tlb_inv_context *cxt) { -__load_host_stage2 (); +struct kvm_s2_mmu *mmu = cxt->mmu; +struct kvm_cpu_context *host_ctxt; +struct kvm_vcpu *vcpu; + + host_ctxt = &this_cpu_ptr (&kvm_host_data)->host_ctxt; + vcpu = host_ctxt->__hyp_running_vcpu; + +if (!mmu) +return ; + +if (vcpu) +__load_stage2 (mmu,kern_hyp_va (mmu->arch)); +else +__load_host_stage2 ();if (cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT )) { -/* Ensure write of the host VMID */ +/* Ensure write of the old VMID */ isb (); -/* Restore the host's TCR_EL1 */ + +if (!(cxt->sctlr & SCTLR_ELx_M)) { +write_sysreg_el1 (cxt->sctlr,SYS_SCTLR ); +isb (); + } +write_sysreg_el1 (cxt->tcr,SYS_TCR ); } }
@@ -84,7 +151,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt, false); +enter_vmid_context (mmu, &cxt, false);/* * We could do so much better if we had the VA as well.
@@ -105,7 +172,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,dsb (ish);isb (); -__tlb_switch_to_host (&cxt); +exit_vmid_context (&cxt); } void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
@@ -114,7 +181,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt, true); +enter_vmid_context (mmu, &cxt, true);/* * We could do so much better if we had the VA as well.
@@ -135,7 +202,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,dsb (nsh);isb (); -__tlb_switch_to_host (&cxt); +exit_vmid_context (&cxt); } void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
@@ -152,7 +219,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, start =round_down (start, stride);/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt, false); +enter_vmid_context (mmu, &cxt, false);__flush_s2_tlb_range_op (ipas2e1is, start, pages, stride,TLBI_TTL_UNKNOWN );
@@ -162,7 +229,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,dsb (ish);isb (); -__tlb_switch_to_host (&cxt); +exit_vmid_context (&cxt); }void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu)
@@ -170,13 +237,13 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt, false); +enter_vmid_context (mmu, &cxt, false);__tlbi (vmalls12e1is);dsb (ish);isb (); -__tlb_switch_to_host (&cxt); +exit_vmid_context (&cxt); }void __kvm_flush_cpu_context (struct kvm_s2_mmu *mmu)
@@ -184,19 +251,19 @@ void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)struct tlb_inv_context cxt;/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt, false); +enter_vmid_context (mmu, &cxt, false);__tlbi (vmalle1); asmvolatile ("ic iallu" );dsb (nsh);isb (); -__tlb_switch_to_host (&cxt); +exit_vmid_context (&cxt); }void __kvm_flush_vm_context (void ) { -/* Same remark as in __tlb_switch_to_guest() */ +/* Same remark as in enter_vmid_context() */ dsb (ish);__tlbi (alle1is);dsb (ish);
Signature · 1 line
2.44.0.769.g3c40516874-goog
Re: [PATCH v3 09/31] KVM: arm64: Support TLB invalidation in guest context
Quoted 7 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Typically, TLB invalidation of guest stage-2 mappings using nVHE is performed by a hypercall originating from the host. For the invalidation instruction to be effective, therefore, __tlb_switch_to_{guest,host}() swizzle the active stage-2 context around the TLBI instruction.
Quoted 8 lines
With guest-to-host memory sharing and unsharing hypercalls originating from the guest under pKVM, there is need to support both guest and host VMID invalidations issued from guest context.
Replace the __tlb_switch_to_{guest,host}() functions with a more general {enter,exit}_vmid_context() implementation which supports being invoked from guest context and acts as a no-op if the target context matches the running context.
I'd rather not introduce unnecessary asymmetry between the nVHE and VHE TLB invalidations, the code duplication is annoying enough as is. Can you rename to {enter,exit}_vmid_context() on the VHE side as well?
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v3 09/31] KVM: arm64: Support TLB invalidation in guest context
Hi Oliver,
Quoted 13 lines · Oliver Upton
Quoted 11 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Typically, TLB invalidation of guest stage-2 mappings using nVHE is performed by a hypercall originating from the host. For the invalidation instruction to be effective, therefore, __tlb_switch_to_{guest,host}() swizzle the active stage-2 context around the TLBI instruction.
With guest-to-host memory sharing and unsharing hypercalls originating from the guest under pKVM, there is need to support both guest and host VMID invalidations issued from guest context.
Quoted 8 lines
Quoted 4 lines
Replace the __tlb_switch_to_{guest,host}() functions with a more general {enter,exit}_vmid_context() implementation which supports being invoked from guest context and acts as a no-op if the target context matches the running context.
I'd rather not introduce unnecessary asymmetry between the nVHE and VHE TLB invalidations, the code duplication is annoying enough as is. Can you rename to {enter,exit}_vmid_context() on the VHE side as well?
Will do.
Thanks for all the reviews!
/fuad
Quoted 3 lines
--
Thanks,
Oliver
PATCH v3 10/31 KVM: arm64: Do not map the host fpsimd state to hyp in pKVM
In reply to Fuad TabbapKVM maintains its own state at EL2 for tracking the host fpsimd state. Therefore, no need to map and share the host's view with it.
--- arch/arm64/include/asm/kvm_host.h | 3 --- arch/arm64/kvm/fpsimd.c | 31 ++++--------------------------- arch/arm64/kvm/reset.c | 1 - 3 files changed, 4 insertions(+), 31 deletions(-)
arch/arm64/include/asm/kvm_host.h +0 -3
@@ -663,8 +663,6 @@ struct kvm_vcpu_arch {struct kvm_guest_debug_arch vcpu_debug_state;struct kvm_guest_debug_arch external_debug_state; -struct task_struct *parent_task; -/* VGIC state */ struct vgic_cpu vgic_cpu;struct arch_timer_cpu timer_cpu;
@@ -1262,7 +1260,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu);void kvm_arch_vcpu_ctxflush_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_ctxsync_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_put_fp (struct kvm_vcpu *vcpu); -void kvm_vcpu_unshare_task_fp (struct kvm_vcpu *vcpu);static inline bool kvm_pmu_counter_deferred (struct perf_event_attr *attr) {
arch/arm64/kvm/fpsimd.c +4 -27
@@ -14,19 +14,6 @@#include <asm/kvm_mmu.h> #include <asm/sysreg.h> -void kvm_vcpu_unshare_task_fp (struct kvm_vcpu *vcpu) -{ -struct task_struct *p = vcpu->arch.parent_task; -struct user_fpsimd_state *fpsimd; - -if (!is_protected_kvm_enabled () || !p) -return ; - - fpsimd = &p->thread.uw.fpsimd_state; -kvm_unshare_hyp (fpsimd, fpsimd +1 ); -put_task_struct (p); -} -/* * Called on entry to KVM_RUN unless this vcpu previously ran at least * once and the most recent prior KVM_RUN for this vcpu was called from
@@ -38,28 +25,18 @@ void kvm_vcpu_unshare_task_fp(struct kvm_vcpu *vcpu)*/ int kvm_arch_vcpu_run_map_fp (struct kvm_vcpu *vcpu) { -int ret; -struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state; +int ret; -kvm_vcpu_unshare_task_fp (vcpu); +/* pKVM has its own tracking of the host fpsimd state. */ +if (is_protected_kvm_enabled ()) +return 0 ;/* Make sure the host task fpsimd state is visible to hyp: */ ret =kvm_share_hyp (fpsimd, fpsimd +1 );if (ret)return ret; -/* -* We need to keep current's task_struct pinned until its data has been -* unshared with the hypervisor to make sure it is not re-used by the -* kernel and donated to someone else while already shared -- see -* kvm_vcpu_unshare_task_fp() for the matching put_task_struct(). -*/ -if (is_protected_kvm_enabled ()) { -get_task_struct (current); - vcpu->arch.parent_task = current; - } -return 0 ; }
arch/arm64/kvm/reset.c +0 -1
@@ -151,7 +151,6 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) {void *sve_state = vcpu->arch.sve_state; -kvm_vcpu_unshare_task_fp (vcpu);kvm_unshare_hyp (vcpu, vcpu +1 );if (sve_state)kvm_unshare_hyp (sve_state, sve_state +vcpu_sve_state_size (vcpu));
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 11/31 KVM: arm64: Remove locking from EL2 allocation fast-paths
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
hyp_{get,put}_page() are called extensively from the page-table code to adjust reference counts on page-table pages. As a small step towards removing reader serialisation on these paths, drop the 'hyp_pool' lock in the case where the refcount remains positive, only taking the lock if the page is to be freed back to the allocator.
Remove a misleading comment at the same time, which implies that a page with a refcount of zero and which is not attached to a freelist is unsafe whereas in practice it's the other way around which can lead to problems.
--- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 6 +----- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 16 ++++------------ 2 files changed, 5 insertions(+), 17 deletions(-)
arch/arm64/kvm/hyp/include/nvhe/gfp.h +1 -5
@@ -10,11 +10,7 @@#define HYP_NO_ORDER USHRT_MAXstruct hyp_pool { -/* -* Spinlock protecting concurrent changes to the memory pool as well as -* the struct hyp_page of the pool's pages until we have a proper atomic -* API at EL2. -*/ +/* lock protecting concurrent changes to the memory pool. */ hyp_spinlock_t lock;struct list_head free_area[NR_PAGE_ORDERS ];phys_addr_t range_start;
arch/arm64/kvm/hyp/nvhe/page_alloc.c +4 -12
@@ -152,33 +152,25 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool,static void __hyp_put_page (struct hyp_pool *pool,struct hyp_page *p) { -if (hyp_page_ref_dec_and_test (p)) +if (hyp_page_ref_dec_and_test (p)) { +hyp_spin_lock (&pool->lock);__hyp_attach_page (pool, p); +hyp_spin_unlock (&pool->lock); + } } -/* -* Changes to the buddy tree and page refcounts must be done with the hyp_pool -* lock held. If a refcount change requires an update to the buddy tree (e.g. -* hyp_put_page()), both operations must be done within the same critical -* section to guarantee transient states (e.g. a page with null refcount but -* not yet attached to a free list) can't be observed by well-behaved readers. -*/ void hyp_put_page (struct hyp_pool *pool,void *addr) {struct hyp_page *p =hyp_virt_to_page (addr); -hyp_spin_lock (&pool->lock);__hyp_put_page (pool, p); -hyp_spin_unlock (&pool->lock); }void hyp_get_page (struct hyp_pool *pool,void *addr) {struct hyp_page *p =hyp_virt_to_page (addr); -hyp_spin_lock (&pool->lock);hyp_page_ref_inc (p); -hyp_spin_unlock (&pool->lock); }void hyp_split_page (struct hyp_page *p)
Signature · 1 line
2.44.0.769.g3c40516874-goog
Re: [PATCH v3 11/31] KVM: arm64: Remove locking from EL2 allocation fast-paths
Quoted 7 lines · Fuad Tabba
From: Will Deacon <[email protected]>
hyp_{get,put}_page() are called extensively from the page-table code to adjust reference counts on page-table pages. As a small step towards removing reader serialisation on these paths, drop the '
hyp_pool' lock in the case where the refcount remains positive, only taking the lock if the page is to be freed back to the allocator.
Surely the ordering of these patches is wrong... The refcount accessors are non-atomic until patch 27.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v3 11/31] KVM: arm64: Remove locking from EL2 allocation fast-paths
Hi Oliver,
Quoted 4 lines · Oliver Upton
Quoted 2 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Quoted 8 lines
Quoted 5 lines
hyp_{get,put}_page() are called extensively from the page-table code to adjust reference counts on page-table pages. As a small step towards removing reader serialisation on these paths, drop the '
hyp_pool' lock in the case where the refcount remains positive, only taking the lock if the page is to be freed back to the allocator.Surely the ordering of these patches is wrong... The refcount accessors are non-atomic until patch 27.
Yup. This is the patch sender's fault (i.e., mine), not the author's. I'll fix this on the respin.
Thanks!
/fuad
Quoted 3 lines
--
Thanks,
Oliver
PATCH v3 12/31 KVM: arm64: Prevent kmemleak from accessing .hyp.data
In reply to Fuad TabbaFrom: Quentin Perret <[email protected]>
We've added a .data section for the hypervisor, which kmemleak is eager to parse. This clearly doesn't go well, so add the section to kmemleak's block list.
--- arch/arm64/kvm/pkvm.c | 1 + 1 file changed, 1 insertion(+)
arch/arm64/kvm/pkvm.c +1 -0
@@ -258,6 +258,7 @@ static int __init finalize_pkvm(void)* at, which would end badly once inaccessible. */ kmemleak_free_part (__hyp_bss_start, __hyp_bss_end - __hyp_bss_start); +kmemleak_free_part (__hyp_rodata_start, __hyp_rodata_end - __hyp_rodata_start);kmemleak_free_part_phys (hyp_mem_base, hyp_mem_size); ret =pkvm_drop_host_privileges ();
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 13/31 KVM: arm64: Fix comment for __pkvm_vcpu_init_traps()
In reply to Fuad TabbaFix the comment to clarify that __pkvm_vcpu_init_traps() initializes traps for all VMs in protected mode, and not only for protected VMs.
--- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/pkvm.c +1 -1
@@ -200,7 +200,7 @@ static void pvm_init_trap_regs(struct kvm_vcpu *vcpu) }/* -* Initialize trap register values for protected VMs. +* Initialize trap register values in protected mode. */ void __pkvm_vcpu_init_traps (struct kvm_vcpu *vcpu) {
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 14/31 KVM: arm64: Change kvm_handle_mmio_return() return polarity
In reply to Fuad TabbaMost exit handlers return <= 0 to indicate that the host needs to handle the exit. Make kvm_handle_mmio_return() consistent with the exit handlers in handle_exit(). This makes the code easier to reason about, and makes it easier to add other handlers in future patches.
No functional change intended.
--- arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/mmio.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kvm/arm.c +1 -1
@@ -974,7 +974,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)if (run->exit_reason ==KVM_EXIT_MMIO ) { ret =kvm_handle_mmio_return (vcpu); -if (ret) +if (ret <=0 )return ret; }
arch/arm64/kvm/mmio.c +2 -2
@@ -86,7 +86,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)/* Detect an already handled MMIO return */ if (unlikely (!vcpu->mmio_needed)) -return 0 ; +return 1 ; vcpu->mmio_needed =0 ;
@@ -117,7 +117,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)*/ kvm_incr_pc (vcpu); -return 0 ; +return 1 ; }int io_mem_abort (struct kvm_vcpu *vcpu,phys_addr_t fault_ipa)
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 15/31 KVM: arm64: Move setting the page as dirty out of the critical section
In reply to Fuad TabbaMove the unlock earlier in user_mem_abort() to shorten the critical section. This also helps for future refactoring and reuse of similar code.
This moves out marking the page as dirty outside of the critical section. That code does not interact with the stage-2 page tables, which the read lock in the critical section protects.
--- arch/arm64/kvm/mmu.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/arm64/kvm/mmu.c +5 -3
@@ -1522,8 +1522,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,read_lock (&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; -if (mmu_invalidate_retry (kvm, mmu_seq)) +if (mmu_invalidate_retry (kvm, mmu_seq)) { + ret = -EAGAIN ; goto out_unlock; + }/* * If we are not forced to use page mapping, check if we are
@@ -1581,6 +1583,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, memcache,KVM_PGTABLE_WALK_HANDLE_FAULT | KVM_PGTABLE_WALK_SHARED); +out_unlock: +read_unlock (&kvm->mmu_lock);/* Mark the page dirty only if the fault is handled successfully */ if (writable && !ret) {
@@ -1588,8 +1592,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,mark_page_dirty_in_slot (kvm, memslot, gfn); } -out_unlock: -read_unlock (&kvm->mmu_lock);kvm_release_pfn_clean (pfn);return ret != -EAGAIN ? ret :0 ; }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 16/31 KVM: arm64: Simplify vgic-v3 hypercalls
In reply to Fuad TabbaFrom: Marc Zyngier <[email protected]>
Consolidate the GICv3 VMCR accessor hypercalls into the APR save/restore hypercalls so that all of the EL2 GICv3 state is covered by a single pair of hypercalls.
--- arch/arm64/include/asm/kvm_asm.h | 8 ++------ arch/arm64/include/asm/kvm_hyp.h | 4 ++-- arch/arm64/kvm/arm.c | 5 ++--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 24 ++++++------------------ arch/arm64/kvm/hyp/vgic-v3-sr.c | 27 +++++++++++++++++++++++---- arch/arm64/kvm/vgic/vgic-v2.c | 9 +-------- arch/arm64/kvm/vgic/vgic-v3.c | 23 ++--------------------- arch/arm64/kvm/vgic/vgic.c | 11 ----------- arch/arm64/kvm/vgic/vgic.h | 2 -- include/kvm/arm_vgic.h | 1 - 10 files changed, 38 insertions(+), 76 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -6
@@ -73,10 +73,8 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range, __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context, __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff, - __KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr, - __KVM_HOST_SMCCC_FUNC___vgic_v3_write_vmcr, - __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs, - __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs, + __KVM_HOST_SMCCC_FUNC___vgic_v3_save_vmcr_aprs, + __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs, __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_init_traps, __KVM_HOST_SMCCC_FUNC___pkvm_init_vm, __KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
@@ -241,8 +239,6 @@ extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);extern void __kvm_adjust_pc (struct kvm_vcpu *vcpu);extern u64 __vgic_v3_get_gic_config (void ); -extern u64 __vgic_v3_read_vmcr (void ); -extern void __vgic_v3_write_vmcr (u32 vmcr);extern void __vgic_v3_init_lrs (void );extern u64 __kvm_get_mdcr_el2 (void );
arch/arm64/include/asm/kvm_hyp.h +2 -2
@@ -80,8 +80,8 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_restore_state (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_activate_traps (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_deactivate_traps (struct vgic_v3_cpu_if *cpu_if); -void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if); -void __vgic_v3_restore_aprs (struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_save_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_restore_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if);int __vgic_v3_perform_cpuif_access (struct kvm_vcpu *vcpu);#ifdef __KVM_NVHE_HYPERVISOR__
arch/arm64/kvm/arm.c +2 -3
@@ -784,9 +784,8 @@ void kvm_vcpu_wfi(struct kvm_vcpu *vcpu)* doorbells to be signalled, should an interrupt become pending. */ preempt_disable (); -kvm_vgic_vmcr_sync (vcpu);vcpu_set_flag (vcpu,IN_WFI ); -vgic_v4_put (vcpu); +kvm_vgic_put (vcpu);preempt_enable ();kvm_vcpu_halt (vcpu);
@@ -794,7 +793,7 @@ void kvm_vcpu_wfi(struct kvm_vcpu *vcpu)preempt_disable ();vcpu_clear_flag (vcpu,IN_WFI ); -vgic_v4_load (vcpu); +kvm_vgic_load (vcpu);preempt_enable (); }
arch/arm64/kvm/hyp/nvhe/hyp-main.c +6 -18
@@ -175,16 +175,6 @@ static void handle___vgic_v3_get_gic_config(struct kvm_cpu_context *host_ctxt)cpu_reg (host_ctxt,1 ) =__vgic_v3_get_gic_config (); } -static void handle___vgic_v3_read_vmcr (struct kvm_cpu_context *host_ctxt) -{ -cpu_reg (host_ctxt,1 ) =__vgic_v3_read_vmcr (); -} - -static void handle___vgic_v3_write_vmcr (struct kvm_cpu_context *host_ctxt) -{ -__vgic_v3_write_vmcr (cpu_reg (host_ctxt,1 )); -} -static void handle___vgic_v3_init_lrs (struct kvm_cpu_context *host_ctxt) {__vgic_v3_init_lrs ();
@@ -195,18 +185,18 @@ static void handle___kvm_get_mdcr_el2(struct kvm_cpu_context *host_ctxt)cpu_reg (host_ctxt,1 ) =__kvm_get_mdcr_el2 (); } -static void handle___vgic_v3_save_aprs (struct kvm_cpu_context *host_ctxt) +static void handle___vgic_v3_save_vmcr_aprs (struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1); -__vgic_v3_save_aprs (kern_hyp_va (cpu_if)); +__vgic_v3_save_vmcr_aprs (kern_hyp_va (cpu_if)); } -static void handle___vgic_v3_restore_aprs (struct kvm_cpu_context *host_ctxt) +static void handle___vgic_v3_restore_vmcr_aprs (struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1); -__vgic_v3_restore_aprs (kern_hyp_va (cpu_if)); +__vgic_v3_restore_vmcr_aprs (kern_hyp_va (cpu_if)); }static void handle___pkvm_init (struct kvm_cpu_context *host_ctxt)
@@ -337,10 +327,8 @@ static const hcall_t host_hcall[] = {HANDLE_FUNC (__kvm_tlb_flush_vmid_range),HANDLE_FUNC (__kvm_flush_cpu_context),HANDLE_FUNC (__kvm_timer_set_cntvoff), -HANDLE_FUNC (__vgic_v3_read_vmcr), -HANDLE_FUNC (__vgic_v3_write_vmcr), -HANDLE_FUNC (__vgic_v3_save_aprs), -HANDLE_FUNC (__vgic_v3_restore_aprs), +HANDLE_FUNC (__vgic_v3_save_vmcr_aprs), +HANDLE_FUNC (__vgic_v3_restore_vmcr_aprs),HANDLE_FUNC (__pkvm_vcpu_init_traps),HANDLE_FUNC (__pkvm_init_vm), HANDLE_FUNC(__pkvm_init_vcpu),
arch/arm64/kvm/hyp/vgic-v3-sr.c +23 -4
@@ -330,7 +330,7 @@ void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if)write_gicreg (0 ,ICH_HCR_EL2 ); } -void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if) +static void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if) {u64 val;u32 nr_pre_bits;
@@ -363,7 +363,7 @@ void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if) } } -void __vgic_v3_restore_aprs (struct vgic_v3_cpu_if *cpu_if) +static void __vgic_v3_restore_aprs (struct vgic_v3_cpu_if *cpu_if) {u64 val;u32 nr_pre_bits;
@@ -455,16 +455,35 @@ u64 __vgic_v3_get_gic_config(void)return val; } -u64 __vgic_v3_read_vmcr (void ) +static u64 __vgic_v3_read_vmcr (void ) {return read_gicreg (ICH_VMCR_EL2 ); } -void __vgic_v3_write_vmcr (u32 vmcr) +static void __vgic_v3_write_vmcr (u32 vmcr) {write_gicreg (vmcr,ICH_VMCR_EL2 ); } +void __vgic_v3_save_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if) +{ +__vgic_v3_save_aprs (cpu_if); +if (cpu_if->vgic_sre) + cpu_if->vgic_vmcr =__vgic_v3_read_vmcr (); +} + +void __vgic_v3_restore_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if) +{ +/* +* If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen +* is dependent on ICC_SRE_EL1.SRE, and we have to perform the +* VMCR_EL2 save/restore in the world switch. +*/ +if (cpu_if->vgic_sre) +__vgic_v3_write_vmcr (cpu_if->vgic_vmcr); +__vgic_v3_restore_aprs (cpu_if); +} +static int __vgic_v3_bpr_min (void ) {/* See Pseudocode for VPriorityGroup */
arch/arm64/kvm/vgic/vgic-v2.c +1 -8
@@ -464,17 +464,10 @@ void vgic_v2_load(struct kvm_vcpu *vcpu) kvm_vgic_global_state.vctrl_base + GICH_APR); } -void vgic_v2_vmcr_sync (struct kvm_vcpu *vcpu) -{ -struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; - - cpu_if->vgic_vmcr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_VMCR ); -} -void vgic_v2_put (struct kvm_vcpu *vcpu) {struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; -vgic_v2_vmcr_sync (vcpu); + cpu_if->vgic_vmcr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_VMCR ); cpu_if->vgic_apr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_APR ); }
arch/arm64/kvm/vgic/vgic-v3.c +2 -21
@@ -722,15 +722,7 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; -/* -* If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen -* is dependent on ICC_SRE_EL1.SRE, and we have to perform the -* VMCR_EL2 save/restore in the world switch. -*/ -if (likely (cpu_if->vgic_sre)) -kvm_call_hyp (__vgic_v3_write_vmcr, cpu_if->vgic_vmcr); - -kvm_call_hyp (__vgic_v3_restore_aprs, cpu_if); +kvm_call_hyp (__vgic_v3_restore_vmcr_aprs, cpu_if);if (has_vhe ())__vgic_v3_activate_traps (cpu_if);
@@ -738,24 +730,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)WARN_ON (vgic_v4_load (vcpu)); } -void vgic_v3_vmcr_sync (struct kvm_vcpu *vcpu) -{ -struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - -if (likely (cpu_if->vgic_sre)) - cpu_if->vgic_vmcr =kvm_call_hyp_ret (__vgic_v3_read_vmcr); -} -void vgic_v3_put (struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; +kvm_call_hyp (__vgic_v3_save_vmcr_aprs, cpu_if);WARN_ON (vgic_v4_put (vcpu)); -vgic_v3_vmcr_sync (vcpu); - -kvm_call_hyp (__vgic_v3_save_aprs, cpu_if); -if (has_vhe ())__vgic_v3_deactivate_traps (cpu_if); }
arch/arm64/kvm/vgic/vgic.c +0 -11
@@ -939,17 +939,6 @@ void kvm_vgic_put(struct kvm_vcpu *vcpu)vgic_v3_put (vcpu); } -void kvm_vgic_vmcr_sync (struct kvm_vcpu *vcpu) -{ -if (unlikely (!irqchip_in_kernel (vcpu->kvm))) -return ; - -if (kvm_vgic_global_state.type ==VGIC_V2 ) -vgic_v2_vmcr_sync (vcpu); -else -vgic_v3_vmcr_sync (vcpu); -} -int kvm_vgic_vcpu_pending_irq (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
arch/arm64/kvm/vgic/vgic.h +0 -2
@@ -214,7 +214,6 @@ int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,void vgic_v2_init_lrs (void );void vgic_v2_load (struct kvm_vcpu *vcpu);void vgic_v2_put (struct kvm_vcpu *vcpu); -void vgic_v2_vmcr_sync (struct kvm_vcpu *vcpu);void vgic_v2_save_state (struct kvm_vcpu *vcpu);void vgic_v2_restore_state (struct kvm_vcpu *vcpu);
@@ -253,7 +252,6 @@ bool vgic_v3_check_base(struct kvm *kvm);void vgic_v3_load (struct kvm_vcpu *vcpu);void vgic_v3_put (struct kvm_vcpu *vcpu); -void vgic_v3_vmcr_sync (struct kvm_vcpu *vcpu);bool vgic_has_its (struct kvm *kvm);int kvm_vgic_register_its_device (void );
include/kvm/arm_vgic.h +0 -1
@@ -388,7 +388,6 @@ int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu);void kvm_vgic_load (struct kvm_vcpu *vcpu);void kvm_vgic_put (struct kvm_vcpu *vcpu); -void kvm_vgic_vmcr_sync (struct kvm_vcpu *vcpu);#define irqchip_in_kernel (k) (!!((k)->arch.vgic.in_kernel))#define vgic_initialized (k) ((k)->arch.vgic.initialized)
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 17/31 KVM: arm64: Add is_pkvm_initialized() helper
In reply to Fuad TabbaFrom: Quentin Perret <[email protected]>
Add a helper allowing to check when the pkvm static key is enabled to ease the introduction of pkvm hooks in other parts of the code.
--- arch/arm64/include/asm/virt.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/arm64/include/asm/virt.h +8 -4
@@ -82,6 +82,12 @@ bool is_kvm_arm_initialised(void);DECLARE_STATIC_KEY_FALSE (kvm_protected_mode_initialized); +static inline bool is_pkvm_initialized (void ) +{ +return IS_ENABLED (CONFIG_KVM ) && +static_branch_likely (&kvm_protected_mode_initialized); +} +/* Reports the availability of HYP mode */ static inline bool is_hyp_mode_available (void ) {
@@ -89,8 +95,7 @@ static inline bool is_hyp_mode_available(void)* If KVM protected mode is initialized, all CPUs must have been booted * in EL2. Avoid checking __boot_cpu_mode as CPUs now come up in EL1. */ -if (IS_ENABLED (CONFIG_KVM ) && -static_branch_likely (&kvm_protected_mode_initialized)) +if (is_pkvm_initialized ())return true; return (__boot_cpu_mode[0] == BOOT_CPU_MODE_EL2 &&
@@ -104,8 +109,7 @@ static inline bool is_hyp_mode_mismatched(void)* If KVM protected mode is initialized, all CPUs must have been booted * in EL2. Avoid checking __boot_cpu_mode as CPUs now come up in EL1. */ -if (IS_ENABLED (CONFIG_KVM ) && -static_branch_likely (&kvm_protected_mode_initialized)) +if (is_pkvm_initialized ())return false;return __boot_cpu_mode[0 ] != __boot_cpu_mode[1 ];
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 18/31 KVM: arm64: Introduce and use predicates that check for protected VMs
In reply to Fuad TabbaIn order to determine whether or not a VM or vcpu are protected, introduce helpers to query this state. While at it, use the vcpu helper to check vcpus protected state instead of the kvm one.
--- arch/arm64/include/asm/kvm_host.h | 8 ++++---- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 5 +++++ arch/arm64/kvm/hyp/nvhe/switch.c | 6 ++---- 3 files changed, 11 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_host.h +4 -4
@@ -211,6 +211,7 @@ typedef unsigned int pkvm_handle_t;struct kvm_protected_vm {pkvm_handle_t handle;struct kvm_hyp_memcache teardown_mc; +bool enabled; };struct kvm_mpidr_data {
@@ -1295,10 +1296,9 @@ struct kvm *kvm_arch_alloc_vm(void);#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE -static inline bool kvm_vm_is_protected (struct kvm *kvm) -{ -return false; -} +#define kvm_vm_is_protected (kvm) (is_protected_kvm_enabled() && (kvm)->arch.pkvm.enabled) + +#define vcpu_is_protected (vcpu) kvm_vm_is_protected((vcpu)->kvm)int kvm_arm_vcpu_finalize (struct kvm_vcpu *vcpu,int feature);bool kvm_arm_vcpu_is_finalized (struct kvm_vcpu *vcpu);
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +5 -0
@@ -53,6 +53,11 @@ pkvm_hyp_vcpu_to_hyp_vm(struct pkvm_hyp_vcpu *hyp_vcpu) return container_of(hyp_vcpu->vcpu.kvm, struct pkvm_hyp_vm, kvm); } +static inline bool pkvm_hyp_vcpu_is_protected (struct pkvm_hyp_vcpu *hyp_vcpu) +{ +return vcpu_is_protected (&hyp_vcpu->vcpu); +} +void pkvm_hyp_vm_table_init (void *tbl);void pkvm_host_fpsimd_state_init (void );
arch/arm64/kvm/hyp/nvhe/switch.c +2 -4
@@ -209,7 +209,7 @@ static const exit_handler_fn pvm_exit_handlers[] = {static const exit_handler_fn *kvm_get_exit_handler_array (struct kvm_vcpu *vcpu) { -if (unlikely (kvm_vm_is_protected (kern_hyp_va (vcpu->kvm)))) +if (unlikely (vcpu_is_protected (vcpu)))return pvm_exit_handlers;return hyp_exit_handlers;
@@ -228,9 +228,7 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)*/ static void early_exit_filter (struct kvm_vcpu *vcpu,u64 *exit_code) { -struct kvm *kvm =kern_hyp_va (vcpu->kvm); - -if (kvm_vm_is_protected (kvm) &&vcpu_mode_is_32bit (vcpu)) { +if (unlikely (vcpu_is_protected (vcpu) &&vcpu_mode_is_32bit (vcpu))) {/* * As we have caught the guest red-handed, decide that it isn't * fit for purpose anymore by making the vcpu invalid. The VMM
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 19/31 KVM: arm64: Move pstate reset value definitions to kvm_arm.h
In reply to Fuad TabbaMove the macro defines of the pstate reset values to a shared header to be used by hyp in future patches.
No functional change intended.
--- arch/arm64/include/asm/kvm_arm.h | 12 ++++++++++++ arch/arm64/kvm/reset.c | 12 ------------ 2 files changed, 12 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_arm.h +12 -0
@@ -432,4 +432,16 @@ { PSR_AA32_MODE_UND, "32-bit UND" }, \ { PSR_AA32_MODE_SYS, "32-bit SYS" } +/* +* ARMv8 Reset Values +*/ +#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + +#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ + PSR_AA32_I_BIT | PSR_AA32_F_BIT) +#endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/kvm/reset.c +0 -12
@@ -33,18 +33,6 @@/* Maximum phys_shift supported for any VM on this host */ static u32 __ro_after_init kvm_ipa_limit; -/* -* ARMv8 Reset Values -*/ -#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ - PSR_F_BIT | PSR_D_BIT) - -#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ - PSR_F_BIT | PSR_D_BIT) - -#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ - PSR_AA32_I_BIT | PSR_AA32_F_BIT) -unsigned int __ro_after_init kvm_sve_max_vl;int __init kvm_arm_init_sve (void )
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 20/31 KVM: arm64: Clarify rationale for ZCR_EL1 value restored on guest exit
In reply to Fuad TabbaExpand comment clarifying why the host value representing SVE vector length being restored for ZCR_EL1 on guest exit isn't the same as it was on guest entry.
--- arch/arm64/kvm/fpsimd.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
arch/arm64/kvm/fpsimd.c +23 -1
@@ -175,12 +175,34 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)if (vcpu_has_sve (vcpu)) {__vcpu_sys_reg (vcpu,ZCR_EL1 ) =read_sysreg_el1 (SYS_ZCR ); -/* Restore the VL that was saved when bound to the CPU */ +/* +* Restore the VL that was saved when bound to the CPU, +* which is the maximum VL for the guest. Because the +* layout of the data when saving the sve state depends +* on the VL, we need to use a consistent (i.e., the +* maximum) VL. +* Note that this means that at guest exit ZCR_EL1 is +* not necessarily the same as on guest entry. +* +* Restoring the VL isn't needed in VHE mode since +* ZCR_EL2 (accessed via ZCR_EL1) would fulfill the same +* role when doing the save from EL2. +*/ if (!has_vhe ())sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL1 ); } +/* +* Flush (save and invalidate) the fpsimd/sve state so that if +* the host tries to use fpsimd/sve, it's not using stale data +* from the guest. +* +* Flushing the state sets the TIF_FOREIGN_FPSTATE bit for the +* context unconditionally, in both nVHE and VHE. This allows +* the kernel to restore the fpsimd/sve state, including ZCR_EL1 +* when needed. +*/ fpsimd_save_and_flush_cpu_state (); }else if (has_vhe () &&system_supports_sve ()) {/*
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 21/31 KVM: arm64: Refactor calculating SVE state size to use helpers
In reply to Fuad TabbaThe main factor for determining the SVE state size is the vector length, and future patches will need to calculate it without necessarily having a vcpu as a reference.
No functional change intended.
--- arch/arm64/include/asm/kvm_host.h | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kvm_host.h +9 -7
@@ -854,22 +854,24 @@ struct kvm_vcpu_arch {#define vcpu_sve_pffr (vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ sve_ffr_offset((vcpu)->arch.sve_max_vl)) -#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) - -#define vcpu_sve_state_size (vcpu) ({ \ +#define sve_state_size (sve_max_vl) ({ \ size_t __size_ret; \ - unsigned int __vcpu_vq; \ + unsigned int __vq; \ \ - if (WARN_ON(!sve_vl_valid((vcpu)->arch.sve_max_vl))) { \ + if (WARN_ON(!sve_vl_valid(sve_max_vl))) { \ __size_ret = 0; \ } else { \ - __vcpu_vq = vcpu_sve_max_vq(vcpu); \ - __size_ret = SVE_SIG_REGS_SIZE(__vcpu_vq); \ + __vq = sve_vq_from_vl(sve_max_vl); \ + __size_ret = SVE_SIG_REGS_SIZE(__vq); \ } \ \ __size_ret; \ }) +#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) + +#define vcpu_sve_state_size (vcpu) sve_state_size((vcpu)->arch.sve_max_vl) +#define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \ KVM_GUESTDBG_USE_HW | \
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 22/31 KVM: arm64: Move some kvm_psci functions to a shared header
In reply to Fuad TabbaMove some PSCI functions and macros to a shared header to be used by hyp in protected mode.
No functional change intended.
--- arch/arm64/kvm/psci.c | 28 ---------------------------- include/kvm/arm_psci.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 28 deletions(-)
arch/arm64/kvm/psci.c +0 -28
@@ -21,16 +21,6 @@* as described in ARM document number ARM DEN 0022A. */ -#define AFFINITY_MASK (level) ~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1) - -static unsigned long psci_affinity_mask (unsigned long affinity_level) -{ -if (affinity_level <=3 ) -return MPIDR_HWID_BITMASK &AFFINITY_MASK (affinity_level); - -return 0 ; -} -static unsigned long kvm_psci_vcpu_suspend (struct kvm_vcpu *vcpu) {/*
@@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)return PSCI_RET_SUCCESS ; } -static inline bool kvm_psci_valid_affinity (struct kvm_vcpu *vcpu, -unsigned long affinity) -{ -return !(affinity & ~MPIDR_HWID_BITMASK ); -} -static unsigned long kvm_psci_vcpu_on (struct kvm_vcpu *source_vcpu) {struct vcpu_reset_state *reset_state;
@@ -214,18 +198,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu) run->exit_reason =KVM_EXIT_SYSTEM_EVENT ; } -static void kvm_psci_narrow_to_32bit (struct kvm_vcpu *vcpu) -{ -int i; - -/* -* Zero the input registers' upper 32 bits. They will be fully -* zeroed on exit, so we're fine changing them in place. -*/ -for (i =1 ; i <4 ; i++) -vcpu_set_reg (vcpu, i,lower_32_bits (vcpu_get_reg (vcpu, i))); -} -static unsigned long kvm_psci_check_allowed_function (struct kvm_vcpu *vcpu,u32 fn) {/*
include/kvm/arm_psci.h +29 -0
@@ -36,6 +36,35 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu)return KVM_ARM_PSCI_0_1 ; } +/* Narrow the PSCI register arguments (r1 to r3) to 32 bits. */ +static inline void kvm_psci_narrow_to_32bit (struct kvm_vcpu *vcpu) +{ +int i; + +/* +* Zero the input registers' upper 32 bits. They will be fully +* zeroed on exit, so we're fine changing them in place. +*/ +for (i =1 ; i <4 ; i++) +vcpu_set_reg (vcpu, i,lower_32_bits (vcpu_get_reg (vcpu, i))); +} + +static inline bool kvm_psci_valid_affinity (struct kvm_vcpu *vcpu, +unsigned long affinity) +{ +return !(affinity & ~MPIDR_HWID_BITMASK ); +} + + +#define AFFINITY_MASK (level) ~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1) + +static inline unsigned long psci_affinity_mask (unsigned long affinity_level) +{ +if (affinity_level <=3 ) +return MPIDR_HWID_BITMASK &AFFINITY_MASK (affinity_level); + +return 0 ; +}int kvm_psci_call (struct kvm_vcpu *vcpu);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 23/31 KVM: arm64: Refactor reset_mpidr() to extract its computation
In reply to Fuad TabbaMove the computation of the mpidr to its own function in a shared header, as the computation will be used by hyp in protected mode.
No functional change intended.
--- arch/arm64/kvm/sys_regs.c | 14 +------------- arch/arm64/kvm/sys_regs.h | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 13 deletions(-)
arch/arm64/kvm/sys_regs.c +1 -13
@@ -816,21 +816,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)static u64 reset_mpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) { -u64 mpidr; +u64 mpidr =calculate_mpidr (vcpu); -/* -* Map the vcpu_id into the first three affinity level fields of -* the MPIDR. We limit the number of VCPUs in level 0 due to a -* limitation to 16 CPUs in that level in the ICC_SGIxR registers -* of the GICv3 to be able to address each CPU directly when -* sending IPIs. -*/ - mpidr = (vcpu->vcpu_id &0x0f ) <<MPIDR_LEVEL_SHIFT (0 ); - mpidr |= ((vcpu->vcpu_id >>4 ) &0xff ) <<MPIDR_LEVEL_SHIFT (1 ); - mpidr |= ((vcpu->vcpu_id >>12 ) &0xff ) <<MPIDR_LEVEL_SHIFT (2 ); - mpidr |= (1ULL <<31 );vcpu_write_sys_reg (vcpu, mpidr,MPIDR_EL1 ); -return mpidr; }
arch/arm64/kvm/sys_regs.h +19 -0
@@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],return __inline_bsearch ((void *)pval, table, num,sizeof (table[0 ]), match_sys_reg); } +static inline u64 calculate_mpidr (const struct kvm_vcpu *vcpu) +{ +u64 mpidr; + +/* +* Map the vcpu_id into the first three affinity level fields of +* the MPIDR. We limit the number of VCPUs in level 0 due to a +* limitation to 16 CPUs in that level in the ICC_SGIxR registers +* of the GICv3 to be able to address each CPU directly when +* sending IPIs. +*/ + mpidr = (vcpu->vcpu_id &0x0f ) <<MPIDR_LEVEL_SHIFT (0 ); + mpidr |= ((vcpu->vcpu_id >>4 ) &0xff ) <<MPIDR_LEVEL_SHIFT (1 ); + mpidr |= ((vcpu->vcpu_id >>12 ) &0xff ) <<MPIDR_LEVEL_SHIFT (2 ); + mpidr |= (1ULL <<31 ); + +return mpidr; +} +const struct sys_reg_desc *get_reg_by_id (u64 id,const struct sys_reg_desc table[],unsigned int num);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 24/31 KVM: arm64: Refactor kvm_vcpu_enable_ptrauth() for hyp use
In reply to Fuad TabbaMove kvm_vcpu_enable_ptrauth() to a shared header to be used by hyp in protected mode.
No functional change intended.
--- arch/arm64/include/asm/kvm_emulate.h | 5 +++++ arch/arm64/kvm/reset.c | 7 +------ 2 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +5 -0
@@ -609,4 +609,9 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu) kvm_write_cptr_el2(val); } + +static inline void kvm_vcpu_enable_ptrauth (struct kvm_vcpu *vcpu) +{ +vcpu_set_flag (vcpu,GUEST_HAS_PTRAUTH ); +}#endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/kvm/reset.c +1 -6
@@ -105,7 +105,7 @@ static int kvm_vcpu_finalize_sve(struct kvm_vcpu *vcpu)kfree (buf);return ret; } - + vcpu->arch.sve_state = buf;vcpu_set_flag (vcpu,VCPU_SVE_FINALIZED );return 0 ;
@@ -152,11 +152,6 @@ static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)memset (vcpu->arch.sve_state,0 ,vcpu_sve_state_size (vcpu)); } -static void kvm_vcpu_enable_ptrauth (struct kvm_vcpu *vcpu) -{ -vcpu_set_flag (vcpu,GUEST_HAS_PTRAUTH ); -} -/** * kvm_reset_vcpu - sets core registers and sys_regs to reset value * @vcpu: The VCPU pointer
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 25/31 KVM: arm64: Introduce hyp_rwlock_t
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
Introduce a simple counter-based rwlock for EL2 which can reduce locking contention on read-mostly data structures when compared to a spinlock.
--- arch/arm64/kvm/hyp/include/nvhe/rwlock.h | 129 +++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/rwlock.h
arch/arm64/kvm/hyp/include/nvhe/rwlock.h new file +129 -0
@@ -0,0 +1,129 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* A stand-alone rwlock implementation for use by the non-VHE KVM +* hypervisor code running at EL2. This is *not* a fair lock and is +* likely to scale very badly under contention. +* +* Copyright (C) 2022 Google LLC +* Author: Will Deacon <[email protected]> +* +* Heavily based on the implementation removed by 087133ac9076 which was: +* Copyright (C) 2012 ARM Ltd. +*/ + +#ifndef __ARM64_KVM_NVHE_RWLOCK_H__ +#define __ARM64_KVM_NVHE_RWLOCK_H__ + +#include <linux/bits.h> + +typedef struct { + u32 __val; +} hyp_rwlock_t; + +#define __HYP_RWLOCK_INITIALIZER \ + { .__val = 0 } + +#define __HYP_RWLOCK_UNLOCKED \ + ((hyp_rwlock_t) __HYP_RWLOCK_INITIALIZER) + +#define DEFINE_HYP_RWLOCK(x) hyp_rwlock_t x = __HYP_RWLOCK_UNLOCKED + +#define hyp_rwlock_init(l) \ +do { \ + *(l) = __HYP_RWLOCK_UNLOCKED; \ +} while (0) + +#define __HYP_RWLOCK_WRITER_BIT 31 + +static inline void hyp_write_lock(hyp_rwlock_t *lock) +{ + u32 tmp; + + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " sevl\n" + "1: wfe\n" + "2: ldaxr %w0, %1\n" + " cbnz %w0, 1b\n" + " stxr %w0, %w2, %1\n" + " cbnz %w0, 2b\n" + __nops(1), + /* LSE atomics */ + "1: mov %w0, wzr\n" + "2: casa %w0, %w2, %1\n" + " cbz %w0, 3f\n" + " ldxr %w0, %1\n" + " cbz %w0, 2b\n" + " wfe\n" + " b 1b\n" + "3:") + : "=&r" (tmp), "+Q" (lock->__val) + : "r" (BIT(__HYP_RWLOCK_WRITER_BIT)) + : "memory"); +} + +static inline void hyp_write_unlock(hyp_rwlock_t *lock) +{ + asm volatile(ARM64_LSE_ATOMIC_INSN( + " stlr wzr, %0", + " swpl wzr, wzr, %0") + : "=Q" (lock->__val) :: "memory"); +} + +static inline void hyp_read_lock(hyp_rwlock_t *lock) +{ + u32 tmp, tmp2; + + asm volatile( + " sevl\n" + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + "1: wfe\n" + "2: ldaxr %w0, %2\n" + " add %w0, %w0, #1\n" + " tbnz %w0, %3, 1b\n" + " stxr %w1, %w0, %2\n" + " cbnz %w1, 2b\n" + __nops(1), + /* LSE atomics */ + "1: wfe\n" + "2: ldxr %w0, %2\n" + " adds %w1, %w0, #1\n" + " tbnz %w1, %3, 1b\n" + " casa %w0, %w1, %2\n" + " sbc %w0, %w1, %w0\n" + " cbnz %w0, 2b") + : "=&r" (tmp), "=&r" (tmp2), "+Q" (lock->__val) + : "i" (__HYP_RWLOCK_WRITER_BIT) + : "cc", "memory"); +} + +static inline void hyp_read_unlock(hyp_rwlock_t *lock) +{ + u32 tmp, tmp2; + + asm volatile(ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + "1: ldxr %w0, %2\n" + " sub %w0, %w0, #1\n" + " stlxr %w1, %w0, %2\n" + " cbnz %w1, 1b", + /* LSE atomics */ + " movn %w0, #0\n" + " staddl %w0, %2\n" + __nops(2)) + : "=&r" (tmp), "=&r" (tmp2), "+Q" (lock->__val) + : + : "memory"); +} + +#ifdef CONFIG_NVHE_EL2_DEBUG +static inline void hyp_assert_write_lock_held(hyp_rwlock_t *lock) +{ + BUG_ON(!(READ_ONCE(lock->__val) & BIT(__HYP_RWLOCK_WRITER_BIT))); +} +#else +static inline void hyp_assert_write_lock_held(hyp_rwlock_t *lock) { } +#endif + +#endif /* __ARM64_KVM_NVHE_RWLOCK_H__ */
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 26/31 KVM: arm64: Add atomics-based checking refcount implementation at EL2
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
The current nVHE refcount implementation at EL2 uses a simple spinlock to serialise accesses. Although this works, it forces serialisation in places where it is not necessary, so introduce a simple atomics-based refcount implementation instead.
--- arch/arm64/kvm/hyp/include/nvhe/refcount.h | 76 ++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/refcount.h
arch/arm64/kvm/hyp/include/nvhe/refcount.h new file +76 -0
@@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Atomics-based checking refcount implementation. +* Copyright (C) 2023 Google LLC +* Author: Will Deacon <[email protected]> +*/ +#ifndef __ARM64_KVM_NVHE_REFCOUNT_H__ +#define __ARM64_KVM_NVHE_REFCOUNT_H__ + +#include <asm/lse.h> + +static inline s16 __ll_sc_refcount_fetch_add_16 (u16 *refcount,s16 addend) +{ +u16 new; +u32 flag; + + asmvolatile ( +" prfm pstl1strm, %[refcount]\n" +"1: ldxrh %w[new], %[refcount]\n" +" add %w[new], %w[new], %w[addend]\n" +" stxrh %w[flag], %w[new], %[refcount]\n" +" cbnz %w[flag], 1b" + : [refcount]"+Q" (*refcount), + [new]"=&r" (new), + [flag]"=&r" (flag) + : [addend]"Ir" (addend)); + +return new; +} + +#ifdef CONFIG_ARM64_LSE_ATOMICS + +static inline s16 __lse_refcount_fetch_add_16 (u16 *refcount,s16 addend) +{ +s16 old; + + asmvolatile (__LSE_PREAMBLE +" ldaddh %w[addend], %w[old], %[refcount]" + : [refcount]"+Q" (*refcount), + [old]"=r" (old) + : [addend]"r" (addend)); + +return old + addend; +} + +#endif /* CONFIG_ARM64_LSE_ATOMICS */ + +static inline u64 __hyp_refcount_fetch_add (void *refcount,const size_t size, +const s64 addend) +{ +s64 new; + +switch (size) { +case 2 : + new =__lse_ll_sc_body (refcount_fetch_add_16, refcount, addend); +break ; +default : +BUILD_BUG_ON_MSG (1 ,"Unsupported refcount size" ); +unreachable (); + } + +BUG_ON (new <0 ); +return new; +} + + +#define hyp_refcount_inc (r) __hyp_refcount_fetch_add(&(r), sizeof(r), 1) +#define hyp_refcount_dec (r) __hyp_refcount_fetch_add(&(r), sizeof(r), -1) +#define hyp_refcount_get (r) READ_ONCE(r) +#define hyp_refcount_set (r, v) do { \ + typeof(r) *__rp = &(r); \ + WARN_ON(hyp_refcount_get(*__rp)); \ + WRITE_ONCE(*__rp, v); \ +} while (0) + +#endif /* __ARM64_KVM_NVHE_REFCOUNT_H__ */
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 27/31 KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
Convert the 'struct hyp_page' refcount manipulation functions over to the new atomic refcount helpers. For now, this will make absolutely no difference because the 'struct hyp_pool' locking is still serialising everything. One step at a time...
--- arch/arm64/kvm/hyp/include/nvhe/memory.h | 18 +++++++----------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 5 ++++- 3 files changed, 12 insertions(+), 13 deletions(-)
arch/arm64/kvm/hyp/include/nvhe/memory.h +7 -11
@@ -6,6 +6,7 @@#include <asm/page.h> #include <linux/types.h> +#include <nvhe/refcount.h> struct hyp_page {unsigned short refcount;
@@ -39,37 +40,32 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr)#define hyp_page_to_pool (page) (((struct hyp_page *)page)->pool)/* -* Refcounting for 'struct hyp_page'. -* hyp_pool::lock must be held if atomic access to the refcount is required. +* Refcounting wrappers for 'struct hyp_page'. */ static inline int hyp_page_count (void *addr) {struct hyp_page *p =hyp_virt_to_page (addr); -return p->refcount; +return hyp_refcount_get (p->refcount); }static inline void hyp_page_ref_inc (struct hyp_page *p) { -BUG_ON (p->refcount ==USHRT_MAX ); - p->refcount++; +hyp_refcount_inc (p->refcount); }static inline void hyp_page_ref_dec (struct hyp_page *p) { -BUG_ON (!p->refcount); - p->refcount--; +hyp_refcount_dec (p->refcount); }static inline int hyp_page_ref_dec_and_test (struct hyp_page *p) { -hyp_page_ref_dec (p); -return (p->refcount ==0 ); +return hyp_refcount_dec (p->refcount) ==0 ; }static inline void hyp_set_page_refcounted (struct hyp_page *p) { -BUG_ON (p->refcount); - p->refcount =1 ; +hyp_refcount_set (p->refcount,1 ); }#endif /* __KVM_HYP_MEMORY_H */
arch/arm64/kvm/hyp/nvhe/mem_protect.c +1 -1
@@ -202,7 +202,7 @@ static void *guest_s2_zalloc_page(void *mc)memset (addr,0 ,PAGE_SIZE ); p =hyp_virt_to_page (addr);memset (p,0 ,sizeof (*p)); - p->refcount =1 ; +hyp_set_page_refcounted (p);return addr; }
arch/arm64/kvm/hyp/nvhe/page_alloc.c +4 -1
@@ -55,7 +55,10 @@ static struct hyp_page *__find_buddy_avail(struct hyp_pool *pool, {struct hyp_page *buddy =__find_buddy_nocheck (pool, p, order); -if (!buddy || buddy->order != order || buddy->refcount) +if (!buddy) +return NULL ; + +if (buddy->order != order ||hyp_refcount_get (buddy->refcount))return NULL ;return buddy;
Signature · 1 line
2.44.0.769.g3c40516874-goog
Re: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
Quoted 38 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Convert the 'struct
hyp_page' refcount manipulation functions over to the new atomic refcount helpers. For now, this will make absolutely no difference because the 'structhyp_pool' locking is still serialising everything. One step at a time...Signed-off-by Will Deacon <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/kvm/hyp/include/nvhe/memory.h | 18 +++++++----------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 5 ++++- 3 files changed, 12 insertions(+), 13 deletions(-)arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-2@@ -6,6 +6,7 @@#include <asm/page.h> #include <linux/types.h> +#include <nvhe/refcount.h> struct hyp_page {unsigned short refcount;@@ -39,37 +40,32 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr)#define hyp_page_to_pool (page) (((struct hyp_page *)page)->pool)/* -* Refcounting for 'struct hyp_page'. -* hyp_pool::lock must be held if atomic access to the refcount is required. +* Refcounting wrappers for 'struct hyp_page'. */ static inline int hyp_page_count (void *addr) {struct hyp_page *p =hyp_virt_to_page (addr);
arch/arm64/kvm/hyp/include/nvhe/memory.h static inline phys_addr_t hyp_virt_to_phys(void *addr) · line 54
Quoted 10 lines
arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-3-return p->refcount; +return hyp_refcount_get (p->refcount); }static inline void hyp_page_ref_inc (struct hyp_page *p) { -BUG_ON (p->refcount ==USHRT_MAX ); - p->refcount++; +hyp_refcount_inc (p->refcount);
Adding a BUG_ON() for taking a reference on a non-refcounted page (i.e. p->refcount was 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.
Seems like otherwise we'd get actually hit the BUG_ON() in an unrelated allocation path.
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
Quoted 56 lines · Oliver Upton
Quoted 38 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Convert the 'struct
hyp_page' refcount manipulation functions over to the new atomic refcount helpers. For now, this will make absolutely no difference because the 'structhyp_pool' locking is still serialising everything. One step at a time...Signed-off-by Will Deacon <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/kvm/hyp/include/nvhe/memory.h | 18 +++++++----------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 5 ++++- 3 files changed, 12 insertions(+), 13 deletions(-)arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-2@@ -6,6 +6,7 @@#include <asm/page.h> #include <linux/types.h> +#include <nvhe/refcount.h> struct hyp_page {unsigned short refcount;@@ -39,37 +40,32 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr)#define hyp_page_to_pool (page) (((struct hyp_page *)page)->pool)/* -* Refcounting for 'struct hyp_page'. -* hyp_pool::lock must be held if atomic access to the refcount is required. +* Refcounting wrappers for 'struct hyp_page'. */ static inline int hyp_page_count (void *addr) {struct hyp_page *p =hyp_virt_to_page (addr);arch/arm64/kvm/hyp/include/nvhe/memory.h static inline phys_addr_t hyp_virt_to_phys(void *addr) · line 54
Quoted 10 lines
arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-3-return p->refcount; +return hyp_refcount_get (p->refcount); }static inline void hyp_page_ref_inc (struct hyp_page *p) { -BUG_ON (p->refcount ==USHRT_MAX ); - p->refcount++; +hyp_refcount_inc (p->refcount);Adding a
BUG_ON()for taking a reference on a non-refcounted page (i.e.p->refcountwas 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.
Quoted 2 lines
Seems like otherwise we'd get actually hit the
BUG_ON()in an unrelated allocation path.
Will do.
Thanks,
/fuad
Quoted 4 lines
--
Thanks,
Oliver
Re: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
Hi Oliver,
Quoted 58 lines · Fuad Tabba
Quoted 56 lines · Oliver Upton
Quoted 38 lines · Fuad Tabba
From: Will Deacon <[email protected]>
Convert the 'struct
hyp_page' refcount manipulation functions over to the new atomic refcount helpers. For now, this will make absolutely no difference because the 'structhyp_pool' locking is still serialising everything. One step at a time...Signed-off-by Will Deacon <[email protected]>Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/kvm/hyp/include/nvhe/memory.h | 18 +++++++----------- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 2 +- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 5 ++++- 3 files changed, 12 insertions(+), 13 deletions(-)arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-2@@ -6,6 +6,7 @@#include <asm/page.h> #include <linux/types.h> +#include <nvhe/refcount.h> struct hyp_page {unsigned short refcount;@@ -39,37 +40,32 @@ static inline phys_addr_t hyp_virt_to_phys(void *addr)#define hyp_page_to_pool (page) (((struct hyp_page *)page)->pool)/* -* Refcounting for 'struct hyp_page'. -* hyp_pool::lock must be held if atomic access to the refcount is required. +* Refcounting wrappers for 'struct hyp_page'. */ static inline int hyp_page_count (void *addr) {struct hyp_page *p =hyp_virt_to_page (addr);arch/arm64/kvm/hyp/include/nvhe/memory.h static inline phys_addr_t hyp_virt_to_phys(void *addr) · line 54
Quoted 10 lines
arch/arm64/kvm/hyp/include/nvhe/memory.h +2
-3-return p->refcount; +return hyp_refcount_get (p->refcount); }static inline void hyp_page_ref_inc (struct hyp_page *p) { -BUG_ON (p->refcount ==USHRT_MAX ); - p->refcount++; +hyp_refcount_inc (p->refcount);Adding a
BUG_ON()for taking a reference on a non-refcounted page (i.e.p->refcountwas 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.
Quoted 2 lines
Quoted 2 lines
Seems like otherwise we'd get actually hit the
BUG_ON()in an unrelated allocation path.
Actually, the refcount can be 0 without it being an error. For example, when hyp pins memory shared with it by the host (mem_protect.c:hyp_pin_shared_mem()).
Cheers,
/fuad
Quoted 8 lines
Will do.
Thanks,
/fuadQuoted 4 lines
--
Thanks,
Oliver
Re: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
On Mon, Apr 22, 2024 at 02:08:17PM +0100, Fuad Tabba wrote:
[...]
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Adding a
BUG_ON()for taking a reference on a non-refcounted page (i.e.p->refcountwas 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.Seems like otherwise we'd get actually hit the
BUG_ON()in an unrelated allocation path.Actually, the refcount can be 0 without it being an error. For example, when hyp pins memory shared with it by the host (mem_protect.c:hyp_pin_shared_mem()).
Are those not by their very definition non-refcounted pages? I can't imagine we'd want pages in a shared state with the host to ever get returned to the hyp allocator. Seems an erroneous hyp_put_page() would get you there, though.
Signature · 2 lines
Thanks, Oliver
Will Deacon
maintainerRe: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
Hi Oliver,
Quoted 9 lines · Oliver Upton
On Mon, Apr 22, 2024 at 02:08:17PM +0100, Fuad Tabba wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Adding a
BUG_ON()for taking a reference on a non-refcounted page (i.e.p->refcountwas 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Seems like otherwise we'd get actually hit the
BUG_ON()in an unrelated allocation path.Actually, the refcount can be 0 without it being an error. For example, when hyp pins memory shared with it by the host (mem_protect.c:hyp_pin_shared_mem()).
Are those not by their very definition non-refcounted pages?
Right, we're using the refcount for two things here: (1) so that the allocator knows when to return the page to the pool and (2) so that the hypervisor can transiently prevent a page which has been shared by the host from being unshared. That second part is needed to e.g. prevent a page holding a host vCPU structure being donated to a guest as normal memory and then having the hypervisor write to it as a result of a host hypercall. We use the refcount for this because the same page can be shared with the hypervisor multiple times and we need to know when the last host sharer has dropped its pin.
Quoted 3 lines
I can't imagine we'd want pages in a shared state with the host to ever get returned to the hyp allocator. Seems an erroneous
hyp_put_page()would get you there, though.
Given the dual-use above, I don't think a BUG_ON() on the refcount is the right fix. Instead, we'd probably want a (cheap) mechanism to differentiate pages in states (1) and (2). This could be a new flag in 'struct hyp_page' or perhaps we could be creative and set the 'order' to HYP_NO_ORDER for pinned pages and then have a BUG() to check 'p->order' against 'pool->max_order' in hyp_put_page().
What do you think?
Will
Re: [PATCH v3 27/31] KVM: arm64: Use atomic refcount helpers for 'struct hyp_page::refcount'
Quoted 21 lines · Will Deacon
Hi Oliver,
Quoted 9 lines · Oliver Upton
On Mon, Apr 22, 2024 at 02:08:17PM +0100, Fuad Tabba wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Adding a
BUG_ON()for taking a reference on a non-refcounted page (i.e.p->refcountwas 0) would be nice, especially since we're past the point of serializing everything and you can theoretically have a zero count page outside of the free list.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Seems like otherwise we'd get actually hit the
BUG_ON()in an unrelated allocation path.Actually, the refcount can be 0 without it being an error. For example, when hyp pins memory shared with it by the host (mem_protect.c:hyp_pin_shared_mem()).
Are those not by their very definition non-refcounted pages?
Quoted 9 lines
Right, we're using the refcount for two things here: (1) so that the allocator knows when to return the page to the pool and (2) so that the hypervisor can transiently prevent a page which has been shared by the host from being unshared. That second part is needed to e.g. prevent a page holding a host vCPU structure being donated to a guest as normal memory and then having the hypervisor write to it as a result of a host hypercall. We use the refcount for this because the same page can be shared with the hypervisor multiple times and we need to know when the last host sharer has dropped its pin.
Ah, right. I don't have the muscle memory for the pKVM bits upstream (even though I should), I missed the very obvious refcount test in hyp_ack_unshare().
So it is a refcount on the page state, be it allocated or shared. And the 0 -> 1 transition on a shared page happens through an increment rather than an initializer like pages from the hyp pool.
Quoted 4 lines
Quoted 3 lines
I can't imagine we'd want pages in a shared state with the host to ever get returned to the hyp allocator. Seems an erroneous
hyp_put_page()would get you there, though.
Quoted 8 lines
Given the dual-use above, I don't think a
BUG_ON()on the refcount is the right fix. Instead, we'd probably want a (cheap) mechanism to differentiate pages in states (1) and (2). This could be a new flag in 'structhyp_page' or perhaps we could be creative and set the 'order' toHYP_NO_ORDERfor pinned pages and then have aBUG()to check 'p->order' against 'pool->max_order' inhyp_put_page().What do you think?
Excellent idea. Having a way to disambiguate page states would be great, all the better if it can (ab)use an existing field.
Signature · 2 lines
Thanks, Oliver
PATCH v3 28/31 KVM: arm64: Reformat/beautify PTP hypercall documentation
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
The PTP hypercall documentation doesn't produce the best-looking table when formatting in HTML as all of the return value definitions end up on the same line.
Reformat the PTP hypercall documentation to follow the formatting used by hypercalls.rst.
--- Documentation/virt/kvm/arm/ptp_kvm.rst | 38 ++++++++++++++++---------- 1 file changed, 24 insertions(+), 14 deletions(-)
Documentation/virt/kvm/arm/ptp_kvm.rst +24 -14
@@ -7,19 +7,29 @@ PTP_KVM is used for high precision time sync between host and guests. It relies on transferring the wall clock and counter value from the host to the guest using a KVM-specific hypercall. -* ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID: 0x86000001 +``ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID`` +---------------------------------------- -This hypercall uses the SMC32/HVC32 calling convention: +Retrieve current time information for the specific counter. There are no +endianness restrictions. -ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID - ============== ======== ===================================== - Function ID: (uint32) 0x86000001 - Arguments: (uint32) KVM_PTP_VIRT_COUNTER(0) - KVM_PTP_PHYS_COUNTER(1) - Return Values: (int32) NOT_SUPPORTED(-1) on error, or - (uint32) Upper 32 bits of wall clock time (r0) - (uint32) Lower 32 bits of wall clock time (r1) - (uint32) Upper 32 bits of counter (r2) - (uint32) Lower 32 bits of counter (r3) - Endianness: No Restrictions. - ============== ======== ===================================== ++---------------------+-------------------------------------------------------+ +| Presence: | Optional | ++---------------------+-------------------------------------------------------+ +| Calling convention: | HVC32 | ++---------------------+----------+--------------------------------------------+ +| Function ID: | (uint32) | 0x86000001 | ++---------------------+----------+----+---------------------------------------+ +| Arguments: | (uint32) | R1 | ``KVM_PTP_VIRT_COUNTER (0)`` | +| | | +---------------------------------------+ +| | | | ``KVM_PTP_PHYS_COUNTER (1)`` | ++---------------------+----------+----+---------------------------------------+ +| Return Values: | (int32) | R0 | ``NOT_SUPPORTED (-1)`` on error, else | +| | | | upper 32 bits of wall clock time | +| +----------+----+---------------------------------------+ +| | (uint32) | R1 | Lower 32 bits of wall clock time | +| +----------+----+---------------------------------------+ +| | (uint32) | R2 | Upper 32 bits of counter | +| +----------+----+---------------------------------------+ +| | (uint32) | R3 | Lower 32 bits of counter | ++---------------------+----------+----+---------------------------------------+
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 29/31 KVM: arm64: Rename firmware pseudo-register documentation file
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
In preparation for describing the guest view of KVM/arm64 hypercalls in hypercalls.rst, move the existing contents of the file concerning the firmware pseudo-registers elsewhere.
---
.../kvm/arm/{hypercalls.rst => fw-pseudo-registers.rst} | 6 +++---
Documentation/virt/kvm/arm/index.rst | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
rename Documentation/virt/kvm/arm/{hypercalls.rst => fw-pseudo-registers.rst} (97%)Documentation/virt/kvm/arm/fw-pseudo-registers.rst renamed from Documentation/virt/kvm/arm/hypercalls.rst +3 -3
@@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 -======================= -ARM Hypercall Interface -======================= +======================================= +ARM firmware pseudo-registers interface +======================================= KVM handles the hypercall services as requested by the guests. New hypercall services are regularly made available by the ARM specification or by KVM (as
Documentation/virt/kvm/arm/index.rst +1 -1
@@ -7,8 +7,8 @@ ARM .. toctree:: :maxdepth: 2 + fw-pseudo-registers hyp-abi - hypercalls pvtime ptp_kvm vcpu-features
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 30/31 KVM: arm64: Document the KVM/arm64-specific calls in hypercalls.rst
In reply to Fuad TabbaFrom: Will Deacon <[email protected]>
KVM/arm64 makes use of the SMCCC "Vendor Specific Hypervisor Service Call Range" to expose KVM-specific hypercalls to guests in a discoverable and extensible fashion.
Document the existence of this interface and the discovery hypercall.
--- Documentation/virt/kvm/arm/hypercalls.rst | 46 +++++++++++++++++++++++ Documentation/virt/kvm/arm/index.rst | 1 + 2 files changed, 47 insertions(+) create mode 100644 Documentation/virt/kvm/arm/hypercalls.rst
Documentation/virt/kvm/arm/hypercalls.rst new file +46 -0
@@ -0,0 +1,46 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============================================== +KVM/arm64-specific hypercalls exposed to guests +=============================================== + +This file documents the KVM/arm64-specific hypercalls which may be +exposed by KVM/arm64 to guest operating systems. These hypercalls are +issued using the HVC instruction according to version 1.1 of the Arm SMC +Calling Convention (DEN0028/C): + +https://developer.arm.com/docs/den0028/c + +All KVM/arm64-specific hypercalls are allocated within the "Vendor +Specific Hypervisor Service Call" range with a UID of +``28b46fb6-2ec5-11e9-a9ca-4b564d003a74``. This UID should be queried by the +guest using the standard "Call UID" function for the service range in +order to determine that the KVM/arm64-specific hypercalls are available. + +``ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID`` +--------------------------------------------- + +Provides a discovery mechanism for other KVM/arm64 hypercalls. + ++---------------------+-------------------------------------------------------------+ +| Presence: | Mandatory for the KVM/arm64 UID | ++---------------------+-------------------------------------------------------------+ +| Calling convention: | HVC32 | ++---------------------+----------+--------------------------------------------------+ +| Function ID: | (uint32) | 0x86000000 | ++---------------------+----------+--------------------------------------------------+ +| Arguments: | None | ++---------------------+----------+----+---------------------------------------------+ +| Return Values: | (uint32) | R0 | Bitmap of available function numbers 0-31 | +| +----------+----+---------------------------------------------+ +| | (uint32) | R1 | Bitmap of available function numbers 32-63 | +| +----------+----+---------------------------------------------+ +| | (uint32) | R2 | Bitmap of available function numbers 64-95 | +| +----------+----+---------------------------------------------+ +| | (uint32) | R3 | Bitmap of available function numbers 96-127 | ++---------------------+----------+----+---------------------------------------------+ + +``ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID`` +---------------------------------------- + +See ptp_kvm.rst
Documentation/virt/kvm/arm/index.rst +1 -0
@@ -9,6 +9,7 @@ ARM fw-pseudo-registers hyp-abi + hypercalls pvtime ptp_kvm vcpu-features
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 31/31 KVM: arm64: Force injection of a data abort on NISV MMIO exit
In reply to Fuad TabbaFrom: Marc Zyngier <[email protected]>
If a vcpu exits for a data abort with an invalid syndrome, the expectations are that userspace has a chance to save the day if it has requested to see such exits.
However, this is completely futile in the case of a protected VM, as none of the state is available. In this particular case, inject a data abort directly into the vcpu, consistent with what userspace could do.
This also helps with pKVM, which discards all syndrome information when forwarding data aborts that are not known to be MMIO.
Finally, hide the RETURN_NISV_IO_ABORT_TO_USER cap from userspace on protected VMs, and document this tweak to the API.
--- Documentation/virt/kvm/api.rst | 7 +++++++ arch/arm64/kvm/arm.c | 14 ++++++++++---- arch/arm64/kvm/mmio.c | 8 ++++++++ 3 files changed, 25 insertions(+), 4 deletions(-)
Documentation/virt/kvm/api.rst +7 -0
@@ -6894,6 +6894,13 @@ Note that KVM does not skip the faulting instruction as it does for KVM_EXIT_MMIO, but userspace has to emulate any change to the processing state if it decides to decode and emulate the instruction. +This feature isn't available to protected VMs, as userspace does not +have access to the state that is required to perform the emulation. +Instead, a data abort exception is directly injected in the guest. +Note that although KVM_CAP_ARM_NISV_TO_USER will be reported if +queried outside of a protected VM context, the feature will not be +exposed if queried on a protected VM file descriptor. + :: /* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */
arch/arm64/kvm/arm.c +10 -4
@@ -80,9 +80,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,switch (cap->cap) {case KVM_CAP_ARM_NISV_TO_USER : - r =0 ; -set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , - &kvm->arch.flags); +if (kvm_vm_is_protected (kvm)) { + r = -EINVAL ; + }else { + r =0 ; +set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , + &kvm->arch.flags); + }break ;case KVM_CAP_ARM_MTE :mutex_lock (&kvm->lock);
@@ -237,7 +241,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_IMMEDIATE_EXIT :case KVM_CAP_VCPU_EVENTS :case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 : -case KVM_CAP_ARM_NISV_TO_USER :case KVM_CAP_ARM_INJECT_EXT_DABT :case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES :
@@ -247,6 +250,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET : r =1 ;break ; +case KVM_CAP_ARM_NISV_TO_USER : + r = !kvm || !kvm_vm_is_protected (kvm); +break ;case KVM_CAP_SET_GUEST_DEBUG2 :return KVM_GUESTDBG_VALID_MASK ;case KVM_CAP_ARM_SET_DEVICE_ADDR :
arch/arm64/kvm/mmio.c +8 -0
@@ -133,11 +133,19 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)/* * No valid syndrome? Ask userspace for help if it has * volunteered to do so, and bail out otherwise. +* +* In the protected VM case, there isn't much userspace can do +* though, so directly deliver an exception to the guest. */ if (!kvm_vcpu_dabt_isvalid (vcpu)) {trace_kvm_mmio_nisv (*vcpu_pc (vcpu),kvm_vcpu_get_esr (vcpu),kvm_vcpu_get_hfar (vcpu), fault_ipa); +if (is_protected_kvm_enabled () &&vcpu_is_protected (vcpu)) { +kvm_inject_dabt (vcpu,kvm_vcpu_get_hfar (vcpu)); +return 1 ; + } +if (test_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , &vcpu->kvm->arch.flags)) { run->exit_reason =KVM_EXIT_ARM_NISV ;
Signature · 1 line
2.44.0.769.g3c40516874-goog
Re: [PATCH v3 31/31] KVM: arm64: Force injection of a data abort on NISV MMIO exit
Quoted 11 lines · Fuad Tabba
From: Marc Zyngier <[email protected]>
If a vcpu exits for a data abort with an invalid syndrome, the expectations are that userspace has a chance to save the day if it has requested to see such exits.
However, this is completely futile in the case of a protected VM, as none of the state is available. In this particular case, inject a data abort directly into the vcpu, consistent with what userspace could do.
Quoted 5 lines
This also helps with pKVM, which discards all syndrome information when forwarding data aborts that are not known to be MMIO.
Finally, hide the
RETURN_NISV_IO_ABORT_TO_USERcap from userspace on protected VMs, and document this tweak to the API.
Are there going to be more KVM caps that we hide from protected VMs in the future? If so, it might be better to have a common helper that enforces a denylist of capabilities not supported by pKVM.
That way we can avoid adding kvm_vm_is_protected() checks all over the shop.
Quoted 8 lines
arch/arm64/kvm/arm.c +0
-0@@ -80,9 +80,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,switch (cap->cap) {case KVM_CAP_ARM_NISV_TO_USER :
arch/arm64/kvm/arm.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 89
Quoted 10 lines
arch/arm64/kvm/arm.c +7
-3- r =0 ; -set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , - &kvm->arch.flags); +if (kvm_vm_is_protected (kvm)) { + r = -EINVAL ; + }else { + r =0 ; +set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , + &kvm->arch.flags); + }
nitpick: initialize r = -EINVAL and get rid of all the error-path initializations in kvm_vm_ioctl_enable_cap()
Quoted 28 lines
arch/arm64/kvm/arm.c +3
-1break ;case KVM_CAP_ARM_MTE :mutex_lock (&kvm->lock);@@ -237,7 +241,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_IMMEDIATE_EXIT :case KVM_CAP_VCPU_EVENTS :case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 : -case KVM_CAP_ARM_NISV_TO_USER :case KVM_CAP_ARM_INJECT_EXT_DABT :case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES :@@ -247,6 +250,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET : r =1 ;break ; +case KVM_CAP_ARM_NISV_TO_USER : + r = !kvm || !kvm_vm_is_protected (kvm); +break ;case KVM_CAP_SET_GUEST_DEBUG2 :return KVM_GUESTDBG_VALID_MASK ;case KVM_CAP_ARM_SET_DEVICE_ADDR :arch/arm64/kvm/mmio.c +0
-0@@ -133,11 +133,19 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)/* * No valid syndrome? Ask userspace for help if it has
arch/arm64/kvm/mmio.c int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) · line 144
Quoted 10 lines
arch/arm64/kvm/mmio.c +4
-0* volunteered to do so, and bail out otherwise. +* +* In the protected VM case, there isn't much userspace can do +* though, so directly deliver an exception to the guest. */ if (!kvm_vcpu_dabt_isvalid (vcpu)) {trace_kvm_mmio_nisv (*vcpu_pc (vcpu),kvm_vcpu_get_esr (vcpu),kvm_vcpu_get_hfar (vcpu), fault_ipa); +if (is_protected_kvm_enabled () &&vcpu_is_protected (vcpu)) {
You can drop the is_protected_kvm_enabled() now that it got folded in to kvm_vm_is_protected().
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v3 31/31] KVM: arm64: Force injection of a data abort on NISV MMIO exit
Hi Oliver,
Quoted 19 lines · Oliver Upton
Quoted 11 lines · Fuad Tabba
From: Marc Zyngier <[email protected]>
If a vcpu exits for a data abort with an invalid syndrome, the expectations are that userspace has a chance to save the day if it has requested to see such exits.
However, this is completely futile in the case of a protected VM, as none of the state is available. In this particular case, inject a data abort directly into the vcpu, consistent with what userspace could do.
Quoted 5 lines
This also helps with pKVM, which discards all syndrome information when forwarding data aborts that are not known to be MMIO.
Finally, hide the
RETURN_NISV_IO_ABORT_TO_USERcap from userspace on protected VMs, and document this tweak to the API.
Quoted 6 lines
Are there going to be more KVM caps that we hide from protected VMs in the future? If so, it might be better to have a common helper that enforces a denylist of capabilities not supported by pKVM.
That way we can avoid adding
kvm_vm_is_protected()checks all over the shop.
Yes there will be. I'll add a patch to do that and rework this one.
Quoted 6 lines
Quoted 6 lines
arch/arm64/kvm/arm.c +0
-0@@ -80,9 +80,13 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
Quoted 15 lines
arch/arm64/kvm/arm.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 89
Quoted 12 lines
arch/arm64/kvm/arm.c +7
-3switch (cap->cap) {case KVM_CAP_ARM_NISV_TO_USER : - r =0 ; -set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , - &kvm->arch.flags); +if (kvm_vm_is_protected (kvm)) { + r = -EINVAL ; + }else { + r =0 ; +set_bit (KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER , + &kvm->arch.flags); + }nitpick: initialize r =
-EINVALand get rid of all the error-path initializations inkvm_vm_ioctl_enable_cap()
Ack.
Quoted 37 lines
Quoted 37 lines
arch/arm64/kvm/arm.c +3
-1break ;case KVM_CAP_ARM_MTE :mutex_lock (&kvm->lock);@@ -237,7 +241,6 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_IMMEDIATE_EXIT :case KVM_CAP_VCPU_EVENTS :case KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 : -case KVM_CAP_ARM_NISV_TO_USER :case KVM_CAP_ARM_INJECT_EXT_DABT :case KVM_CAP_SET_GUEST_DEBUG :case KVM_CAP_VCPU_ATTRIBUTES :@@ -247,6 +250,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_COUNTER_OFFSET : r =1 ;break ; +case KVM_CAP_ARM_NISV_TO_USER : + r = !kvm || !kvm_vm_is_protected (kvm); +break ;case KVM_CAP_SET_GUEST_DEBUG2 :return KVM_GUESTDBG_VALID_MASK ;case KVM_CAP_ARM_SET_DEVICE_ADDR :arch/arm64/kvm/mmio.c +3
-0@@ -133,11 +133,19 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)/* * No valid syndrome? Ask userspace for help if it has * volunteered to do so, and bail out otherwise. +* +* In the protected VM case, there isn't much userspace can do +* though, so directly deliver an exception to the guest. */ if (!kvm_vcpu_dabt_isvalid (vcpu)) {trace_kvm_mmio_nisv (*vcpu_pc (vcpu),kvm_vcpu_get_esr (vcpu),kvm_vcpu_get_hfar (vcpu), fault_ipa);
Quoted 4 lines
arch/arm64/kvm/mmio.c int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) · line 144
Quoted 1 line
arch/arm64/kvm/mmio.c +1
-0+if (is_protected_kvm_enabled () &&vcpu_is_protected (vcpu)) {You can drop the
is_protected_kvm_enabled()now that it got folded in tokvm_vm_is_protected().
Ack.
Thank you!
/fuad
Quoted 3 lines
--
Thanks,
Oliver