MTE support for KVM guest
Subsystems (4)
v15 cover MTE support for KVM guest
This series adds support for using the Arm Memory Tagging Extensions (MTE) in a KVM guest.
I realise there are still open questions[1] around the performance of this series (the 'big lock', tag_sync_lock, introduced in the first patch). But there should be no impact on non-MTE workloads and until we get real MTE-enabled hardware it's hard to know whether there is a need for something more sophisticated or not. Peter Collingbourne's patch[3] to clear the tags at page allocation time should hide more of the impact for non-VM cases. So the remaining concern is around VM startup which could be effectively serialised through the lock.
Changes since v14[2]:
* Dropped "Handle MTE tags zeroing" patch in favour of Peter's similar patch[3] (now in arm64 tree).
* Improved documentation following Catalin's review.
[1]: https://lore.kernel.org/r/874ke7z3ng.wl-maz%40kernel.org
[2]: https://lore.kernel.org/r/[email protected]/
[3]: https://lore.kernel.org/r/[email protected]/
Steven Price (7):
arm64: mte: Handle race when synchronising tags
arm64: mte: Sync tags for pages where PTE is untagged
KVM: arm64: Introduce MTE VM feature
KVM: arm64: Save/restore MTE registers
KVM: arm64: Expose KVM_ARM_CAP_MTE
KVM: arm64: ioctl to fetch/store tags in a guest
KVM: arm64: Document MTE capability and ioctl
Documentation/virt/kvm/api.rst | 57 +++++++++++++++ arch/arm64/include/asm/kvm_arm.h | 3 +- arch/arm64/include/asm/kvm_emulate.h | 3 + arch/arm64/include/asm/kvm_host.h | 12 ++++ arch/arm64/include/asm/kvm_mte.h | 66 +++++++++++++++++ arch/arm64/include/asm/mte-def.h | 1 + arch/arm64/include/asm/mte.h | 8 ++- arch/arm64/include/asm/pgtable.h | 22 +++++- arch/arm64/include/asm/sysreg.h | 3 +- arch/arm64/include/uapi/asm/kvm.h | 11 +++ arch/arm64/kernel/asm-offsets.c | 2 + arch/arm64/kernel/mte.c | 54 ++++++++++++-- arch/arm64/kvm/arm.c | 16 +++++ arch/arm64/kvm/guest.c | 82 ++++++++++++++++++++++ arch/arm64/kvm/hyp/entry.S | 7 ++ arch/arm64/kvm/hyp/exception.c | 3 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 21 ++++++ arch/arm64/kvm/mmu.c | 42 ++++++++++- arch/arm64/kvm/reset.c | 3 +- arch/arm64/kvm/sys_regs.c | 32 +++++++-- include/uapi/linux/kvm.h | 2 + 21 files changed, 429 insertions(+), 21 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_mte.h
Signature · 1 line
2.20.1
PATCH v15 1/7 arm64: mte: Handle race when synchronising tags
mte_sync_tags() used test_and_set_bit() to set the PG_mte_tagged flag before restoring/zeroing the MTE tags. However if another thread were to race and attempt to sync the tags on the same page before the first thread had completed restoring/zeroing then it would see the flag is already set and continue without waiting. This would potentially expose the previous contents of the tags to user space, and cause any updates that user space makes before the restoring/zeroing has completed to potentially be lost.
Since this code is run from atomic contexts we can't just lock the page during the process. Instead implement a new (global) spinlock to protect the mte_sync_page_tags() function.
--- arch/arm64/kernel/mte.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
arch/arm64/kernel/mte.c +17 -3
@@ -25,6 +25,7 @@u64 gcr_kernel_excl__ro_after_init ;static bool report_fault_once = true; +static DEFINE_SPINLOCK (tag_sync_lock);#ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */
@@ -34,13 +35,22 @@ EXPORT_SYMBOL_GPL(mte_async_mode);static void mte_sync_page_tags (struct page *page,pte_t *ptep,bool check_swap) { +unsigned long flags;pte_t old_pte =READ_ONCE (*ptep); +spin_lock_irqsave (&tag_sync_lock, flags); + +/* Recheck with the lock held */ +if (test_bit (PG_mte_tagged, &page->flags)) + goto out; +if (check_swap &&is_swap_pte (old_pte)) {swp_entry_t entry =pte_to_swp_entry (old_pte); -if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) -return ; +if (!non_swap_entry (entry) &&mte_restore_tags (entry, page)) { +set_bit (PG_mte_tagged, &page->flags); + goto out; + } }page_kasan_tag_reset (page);
@@ -53,6 +63,10 @@ static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)*/ smp_wmb ();mte_clear_page_tags (page_address (page)); +set_bit (PG_mte_tagged, &page->flags); + +out: +spin_unlock_irqrestore (&tag_sync_lock, flags); }void mte_sync_tags (pte_t *ptep,pte_t pte)
@@ -63,7 +77,7 @@ void mte_sync_tags(pte_t *ptep, pte_t pte)/* if PG_mte_tagged is set, tags have already been initialised */ for (i =0 ; i < nr_pages; i++, page++) { -if (!test_and_set_bit (PG_mte_tagged, &page->flags)) +if (!test_bit (PG_mte_tagged, &page->flags))mte_sync_page_tags (page, ptep, check_swap); } }
Signature · 1 line
2.20.1
PATCH v15 2/7 arm64: mte: Sync tags for pages where PTE is untagged
In reply to Steven PriceA KVM guest could store tags in a page even if the VMM hasn't mapped the page with PROT_MTE. So when restoring pages from swap we will need to check to see if there are any saved tags even if !pte_tagged().
However don't check pages for which pte_access_permitted() returns false as these will not have been swapped out.
--- arch/arm64/include/asm/mte.h | 4 ++-- arch/arm64/include/asm/pgtable.h | 22 +++++++++++++++++++--- arch/arm64/kernel/mte.c | 17 +++++++++++++---- 3 files changed, 34 insertions(+), 9 deletions(-)
arch/arm64/include/asm/mte.h +2 -2
@@ -37,7 +37,7 @@ void mte_free_tag_storage(char *storage);/* track which pages have valid allocation tags */ #define PG_mte_tagged PG_arch_2 -void mte_sync_tags (pte_t *ptep,pte_t pte); +void mte_sync_tags (pte_t old_pte,pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void mte_thread_init_user (void );void mte_thread_switch (struct task_struct *next);
@@ -53,7 +53,7 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request,/* unused if !CONFIG_ARM64_MTE, silence the compiler */ #define PG_mte_tagged 0 -static inline void mte_sync_tags (pte_t *ptep,pte_t pte) +static inline void mte_sync_tags (pte_t old_pte,pte_t pte) { }static inline void mte_copy_page_tags (void *kto,const void *kfrom)
arch/arm64/include/asm/pgtable.h +19 -3
@@ -314,9 +314,25 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,if (pte_present (pte) &&pte_user_exec (pte) && !pte_special (pte))__sync_icache_dcache (pte); -if (system_supports_mte () && -pte_present (pte) &&pte_tagged (pte) && !pte_special (pte)) -mte_sync_tags (ptep, pte); +/* +* If the PTE would provide user space access to the tags associated +* with it then ensure that the MTE tags are synchronised. Although +* pte_access_permitted() returns false for exec only mappings, they +* don't expose tags (instruction fetches don't check tags). +*/ +if (system_supports_mte () &&pte_access_permitted (pte, false) && + !pte_special (pte)) { +pte_t old_pte =READ_ONCE (*ptep); +/* +* We only need to synchronise if the new PTE has tags enabled +* or if swapping in (in which case another mapping may have +* set tags in the past even if this PTE isn't tagged). +* (!pte_none() && !pte_present()) is an open coded version of +* is_swap_pte() +*/ +if (pte_tagged (pte) || (!pte_none (old_pte) && !pte_present (old_pte))) +mte_sync_tags (old_pte, pte); + }__check_racy_pte_update (mm, ptep, pte);
arch/arm64/kernel/mte.c +13 -4
@@ -33,10 +33,10 @@ DEFINE_STATIC_KEY_FALSE(mte_async_mode);EXPORT_SYMBOL_GPL (mte_async_mode);#endif -static void mte_sync_page_tags (struct page *page,pte_t *ptep,bool check_swap) +static void mte_sync_page_tags (struct page *page,pte_t old_pte, +bool check_swap,bool pte_is_tagged) {unsigned long flags; -pte_t old_pte =READ_ONCE (*ptep);spin_lock_irqsave (&tag_sync_lock, flags);
@@ -53,6 +53,9 @@ static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap) } } +if (!pte_is_tagged) + goto out; +page_kasan_tag_reset (page);/* * We need smp_wmb() in between setting the flags and clearing the
@@ -69,16 +72,22 @@ static void mte_sync_page_tags(struct page *page, pte_t *ptep, bool check_swap)spin_unlock_irqrestore (&tag_sync_lock, flags); } -void mte_sync_tags (pte_t *ptep,pte_t pte) +void mte_sync_tags (pte_t old_pte,pte_t pte) {struct page *page =pte_page (pte);long i, nr_pages =compound_nr (page);bool check_swap = nr_pages ==1 ; +bool pte_is_tagged =pte_tagged (pte); + +/* Early out if there's nothing to do */ +if (!check_swap && !pte_is_tagged) +return ;/* if PG_mte_tagged is set, tags have already been initialised */ for (i =0 ; i < nr_pages; i++, page++) {if (!test_bit (PG_mte_tagged, &page->flags)) -mte_sync_page_tags (page, ptep, check_swap); +mte_sync_page_tags (page, old_pte, check_swap, + pte_is_tagged); } }
Signature · 6 lines
2.20.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v15 3/7 KVM: arm64: Introduce MTE VM feature
In reply to Steven PriceAdd a new VM feature 'KVM_ARM_CAP_MTE' which enables memory tagging for a VM. This will expose the feature to the guest and automatically tag memory pages touched by the VM as PG_mte_tagged (and clear the tag storage) to ensure that the guest cannot see stale tags, and so that the tags are correctly saved/restored across swap.
Actually exposing the new capability to user space happens in a later patch.
--- arch/arm64/include/asm/kvm_emulate.h | 3 ++ arch/arm64/include/asm/kvm_host.h | 3 ++ arch/arm64/include/asm/mte.h | 4 +++ arch/arm64/kernel/mte.c | 17 +++++++++++ arch/arm64/kvm/hyp/exception.c | 3 +- arch/arm64/kvm/mmu.c | 42 +++++++++++++++++++++++++++- arch/arm64/kvm/sys_regs.c | 7 +++++ include/uapi/linux/kvm.h | 1 + 8 files changed, 78 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +3 -0
@@ -84,6 +84,9 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)if (cpus_have_const_cap (ARM64_MISMATCHED_CACHE_TYPE ) ||vcpu_el1_is_32bit (vcpu)) vcpu->arch.hcr_el2 |=HCR_TID2 ; + +if (kvm_has_mte (vcpu->kvm)) + vcpu->arch.hcr_el2 |=HCR_ATA ; }static inline unsigned long *vcpu_hcr (struct kvm_vcpu *vcpu)
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -132,6 +132,8 @@ struct kvm_arch {u8 pfr0_csv2;u8 pfr0_csv3; +/* Memory Tagging Extension enabled for the guest */ +bool mte_enabled; };struct kvm_vcpu_fault_info {
@@ -769,6 +771,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);#define kvm_arm_vcpu_sve_finalized (vcpu) \ ((vcpu)->arch.flags & KVM_ARM64_VCPU_SVE_FINALIZED) +#define kvm_has_mte (kvm) (system_supports_mte() && (kvm)->arch.mte_enabled)#define kvm_vcpu_has_pmu (vcpu) \ (test_bit(KVM_ARM_VCPU_PMU_V3, (vcpu)->arch.features))
arch/arm64/include/asm/mte.h +4 -0
@@ -37,6 +37,7 @@ void mte_free_tag_storage(char *storage);/* track which pages have valid allocation tags */ #define PG_mte_tagged PG_arch_2 +void mte_prepare_page_tags (struct page *page);void mte_sync_tags (pte_t old_pte,pte_t pte);void mte_copy_page_tags (void *kto,const void *kfrom);void mte_thread_init_user (void );
@@ -53,6 +54,9 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request,/* unused if !CONFIG_ARM64_MTE, silence the compiler */ #define PG_mte_tagged 0 +static inline void mte_prepare_page_tags (struct page *page) +{ +}static inline void mte_sync_tags (pte_t old_pte,pte_t pte) { }
arch/arm64/kernel/mte.c +17 -0
@@ -72,6 +72,23 @@ static void mte_sync_page_tags(struct page *page, pte_t old_pte,spin_unlock_irqrestore (&tag_sync_lock, flags); } +void mte_prepare_page_tags (struct page *page) +{ +unsigned long flags; + +spin_lock_irqsave (&tag_sync_lock, flags); + +/* Recheck with the lock held */ +if (test_bit (PG_mte_tagged, &page->flags)) + goto out; + +mte_clear_page_tags (page_address (page)); +set_bit (PG_mte_tagged, &page->flags); + +out: +spin_unlock_irqrestore (&tag_sync_lock, flags); +} +void mte_sync_tags (pte_t old_pte,pte_t pte) {struct page *page =pte_page (pte);
arch/arm64/kvm/hyp/exception.c +2 -1
@@ -112,7 +112,8 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= (old &PSR_C_BIT ); new |= (old &PSR_V_BIT ); -// TODO: TCO (if/when ARMv8.5-MemTag is exposed to guests) +if (kvm_has_mte (vcpu->kvm)) + new |=PSR_TCO_BIT ; new |= (old &PSR_DIT_BIT );
arch/arm64/kvm/mmu.c +41 -1
67 lines not on this page · raw message
arch/arm64/kvm/sys_regs.c +7 -0
@@ -1047,6 +1047,13 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu,break ;case SYS_ID_AA64PFR1_EL1 : val &= ~FEATURE (ID_AA64PFR1_MTE ); +if (kvm_has_mte (vcpu->kvm)) { +u64 pfr, mte; + + pfr =read_sanitised_ftr_reg (SYS_ID_AA64PFR1_EL1 ); + mte =cpuid_feature_extract_unsigned_field (pfr,ID_AA64PFR1_MTE_SHIFT ); + val |=FIELD_PREP (FEATURE (ID_AA64PFR1_MTE ), mte); + }break ;case SYS_ID_AA64ISAR1_EL1 :if (!vcpu_has_ptrauth (vcpu))
include/uapi/linux/kvm.h +1 -0
@@ -1082,6 +1082,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_SGX_ATTRIBUTE 196#define KVM_CAP_VM_COPY_ENC_CONTEXT_FROM 197#define KVM_CAP_PTP_KVM 198 +#define KVM_CAP_ARM_MTE 199#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.20.1
PATCH v15 4/7 KVM: arm64: Save/restore MTE registers
In reply to Steven PriceDefine the new system registers that MTE introduces and context switch them. The MTE feature is still hidden from the ID register as it isn't supported in a VM yet.
--- arch/arm64/include/asm/kvm_arm.h | 3 +- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/include/asm/kvm_mte.h | 66 ++++++++++++++++++++++ arch/arm64/include/asm/sysreg.h | 3 +- arch/arm64/kernel/asm-offsets.c | 2 + arch/arm64/kvm/hyp/entry.S | 7 +++ arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 21 +++++++ arch/arm64/kvm/sys_regs.c | 22 ++++++-- 8 files changed, 124 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_mte.h
arch/arm64/include/asm/kvm_arm.h +2 -1
@@ -12,7 +12,8 @@#include <asm/types.h> /* Hyp Configuration Register (HCR) bits */ -#define HCR_ATA (UL(1) << 56) +#define HCR_ATA_SHIFT 56 +#define HCR_ATA (UL(1) << HCR_ATA_SHIFT)#define HCR_FWB (UL(1) << 46)#define HCR_API (UL(1) << 41)#define HCR_APK (UL(1) << 40)
arch/arm64/include/asm/kvm_host.h +6 -0
@@ -208,6 +208,12 @@ enum vcpu_sysreg {CNTP_CVAL_EL0 ,CNTP_CTL_EL0 , +/* Memory Tagging Extension registers */ +RGSR_EL1 ,/* Random Allocation Tag Seed Register */ +GCR_EL1 ,/* Tag Control Register */ +TFSR_EL1 ,/* Tag Fault Status Register (EL1) */ +TFSRE0_EL1 ,/* Tag Fault Status Register (EL0) */ +/* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */
arch/arm64/include/asm/kvm_mte.h new file +66 -0
66 lines not on this page · raw message
arch/arm64/include/asm/sysreg.h +2 -1
@@ -651,7 +651,8 @@#define INIT_SCTLR_EL2_MMU_ON \ (SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA | SCTLR_ELx_I | \ - SCTLR_ELx_IESB | SCTLR_ELx_WXN | ENDIAN_SET_EL2 | SCTLR_EL2_RES1) + SCTLR_ELx_IESB | SCTLR_ELx_WXN | ENDIAN_SET_EL2 | \ + SCTLR_ELx_ITFSB | SCTLR_EL2_RES1)#define INIT_SCTLR_EL2_MMU_OFF \ (SCTLR_EL2_RES1 | ENDIAN_SET_EL2)
arch/arm64/kernel/asm-offsets.c +2 -0
@@ -111,6 +111,8 @@ int main(void) DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2));DEFINE (CPU_USER_PT_REGS , offsetof(struct kvm_cpu_context , regs)); + DEFINE(CPU_RGSR_EL1, offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1])); + DEFINE(CPU_GCR_EL1, offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1])); DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1])); DEFINE(CPU_APIBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIBKEYLO_EL1])); DEFINE(CPU_APDAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDAKEYLO_EL1]));
arch/arm64/kvm/hyp/entry.S +7 -0
@@ -13,6 +13,7 @@ #include <asm/kvm_arm.h> #include <asm/kvm_asm.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_mte.h> #include <asm/kvm_ptrauth.h> .text
@@ -51,6 +52,9 @@ alternative_else_nop_endif add x29, x0, #VCPU_CONTEXT + // mte_switch_to_guest(g_ctxt, h_ctxt, tmp1) + mte_switch_to_guest x29, x1, x2 + // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code
@@ -142,6 +146,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // when this feature is enabled for kernel code. ptrauth_switch_to_hyp x1, x2, x3, x4, x5 + // mte_switch_to_hyp(g_ctxt, h_ctxt, reg1) + mte_switch_to_hyp x1, x2, x3 + // Restore hyp's sp_el0 restore_sp_el0 x2, x3
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +21 -0
@@ -14,6 +14,7 @@#include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> #include <asm/kvm_hyp.h> +#include <asm/kvm_mmu.h> static inline void __sysreg_save_common_state (struct kvm_cpu_context *ctxt) {
@@ -26,6 +27,16 @@ static inline void __sysreg_save_user_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,TPIDRRO_EL0 ) =read_sysreg (tpidrro_el0); } +static inline bool ctxt_has_mte (struct kvm_cpu_context *ctxt) +{ +struct kvm_vcpu *vcpu = ctxt->__hyp_running_vcpu; + +if (!vcpu) + vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + +return kvm_has_mte (kern_hyp_va (vcpu->kvm)); +} +static inline void __sysreg_save_el1_state (struct kvm_cpu_context *ctxt) {ctxt_sys_reg (ctxt,CSSELR_EL1 ) =read_sysreg (csselr_el1);
@@ -46,6 +57,11 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,PAR_EL1 ) =read_sysreg_par ();ctxt_sys_reg (ctxt,TPIDR_EL1 ) =read_sysreg (tpidr_el1); +if (ctxt_has_mte (ctxt)) { +ctxt_sys_reg (ctxt,TFSR_EL1 ) =read_sysreg_el1 (SYS_TFSR ); +ctxt_sys_reg (ctxt,TFSRE0_EL1 ) =read_sysreg_s (SYS_TFSRE0_EL1 ); + } +ctxt_sys_reg (ctxt,SP_EL1 ) =read_sysreg (sp_el1);ctxt_sys_reg (ctxt,ELR_EL1 ) =read_sysreg_el1 (SYS_ELR );ctxt_sys_reg (ctxt,SPSR_EL1 ) =read_sysreg_el1 (SYS_SPSR );
@@ -107,6 +123,11 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)write_sysreg (ctxt_sys_reg (ctxt,PAR_EL1 ), par_el1);write_sysreg (ctxt_sys_reg (ctxt,TPIDR_EL1 ), tpidr_el1); +if (ctxt_has_mte (ctxt)) { +write_sysreg_el1 (ctxt_sys_reg (ctxt,TFSR_EL1 ),SYS_TFSR ); +write_sysreg_s (ctxt_sys_reg (ctxt,TFSRE0_EL1 ),SYS_TFSRE0_EL1 ); + } +if (!has_vhe () &&cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT ) && ctxt->__hyp_running_vcpu) {
arch/arm64/kvm/sys_regs.c +18 -4
@@ -1309,6 +1309,20 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,return true; } +static unsigned int mte_visibility (const struct kvm_vcpu *vcpu, +const struct sys_reg_desc *rd) +{ +return REG_HIDDEN ; +} + +#define MTE_REG (name) { \ + SYS_DESC(SYS_##name), \ + .access = undef_access, \ + .reset = reset_unknown, \ + .reg = name, \ + .visibility = mte_visibility, \ +} +/* sys_reg_desc initialiser for known cpufeature ID registers */ #define ID_SANITISED (name) { \ SYS_DESC(SYS_##name), \
@@ -1477,8 +1491,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 }, { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, - { SYS_DESC(SYS_RGSR_EL1), undef_access }, - { SYS_DESC(SYS_GCR_EL1), undef_access }, +MTE_REG (RGSR_EL1 ), +MTE_REG (GCR_EL1 ), { SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0, .visibility = sve_visibility }, { SYS_DESC(SYS_TRFCR_EL1), undef_access },
@@ -1505,8 +1519,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ERXMISC0_EL1), trap_raz_wi }, { SYS_DESC(SYS_ERXMISC1_EL1), trap_raz_wi }, - { SYS_DESC(SYS_TFSR_EL1), undef_access }, - { SYS_DESC(SYS_TFSRE0_EL1), undef_access }, +MTE_REG (TFSR_EL1 ), +MTE_REG (TFSRE0_EL1 ), { SYS_DESC(SYS_FAR_EL1), access_vm_reg, reset_unknown, FAR_EL1 }, { SYS_DESC(SYS_PAR_EL1), NULL, reset_unknown, PAR_EL1 },
Signature · 1 line
2.20.1
PATCH v15 5/7 KVM: arm64: Expose KVM_ARM_CAP_MTE
In reply to Steven PriceIt's now safe for the VMM to enable MTE in a guest, so expose the capability to user space.
--- arch/arm64/kvm/arm.c | 9 +++++++++ arch/arm64/kvm/reset.c | 3 ++- arch/arm64/kvm/sys_regs.c | 3 +++ 3 files changed, 14 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +9 -0
@@ -93,6 +93,12 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, r =0 ; kvm->arch.return_nisv_io_abort_to_user = true;break ; +case KVM_CAP_ARM_MTE : +if (!system_supports_mte () || kvm->created_vcpus) +return -EINVAL ; + r =0 ; + kvm->arch.mte_enabled = true; +break ;default : r = -EINVAL ;break ;
@@ -237,6 +243,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)*/ r =1 ;break ; +case KVM_CAP_ARM_MTE : + r =system_supports_mte (); +break ;case KVM_CAP_STEAL_TIME : r =kvm_arm_pvtime_supported ();break ;
arch/arm64/kvm/reset.c +2 -1
@@ -220,7 +220,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)switch (vcpu->arch.target) {default :if (test_bit (KVM_ARM_VCPU_EL1_32BIT , vcpu->arch.features)) { -if (!cpus_have_const_cap (ARM64_HAS_32BIT_EL1 )) { +if (!cpus_have_const_cap (ARM64_HAS_32BIT_EL1 ) || + vcpu->kvm->arch.mte_enabled) { ret = -EINVAL ; goto out; }
arch/arm64/kvm/sys_regs.c +3 -0
@@ -1312,6 +1312,9 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p,static unsigned int mte_visibility (const struct kvm_vcpu *vcpu,const struct sys_reg_desc *rd) { +if (kvm_has_mte (vcpu->kvm)) +return 0 ; +return REG_HIDDEN ; }
Signature · 1 line
2.20.1
PATCH v15 6/7 KVM: arm64: ioctl to fetch/store tags in a guest
In reply to Steven PriceThe VMM may not wish to have it's own mapping of guest memory mapped with PROT_MTE because this causes problems if the VMM has tag checking enabled (the guest controls the tags in physical RAM and it's unlikely the tags are correct for the VMM).
Instead add a new ioctl which allows the VMM to easily read/write the tags from guest memory, allowing the VMM's mapping to be non-PROT_MTE while the VMM can still read/write the tags for the purpose of migration.
--- arch/arm64/include/asm/kvm_host.h | 3 ++ arch/arm64/include/asm/mte-def.h | 1 + arch/arm64/include/uapi/asm/kvm.h | 11 +++++ arch/arm64/kvm/arm.c | 7 +++ arch/arm64/kvm/guest.c | 82 +++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 1 + 6 files changed, 105 insertions(+)
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -729,6 +729,9 @@ int kvm_arm_vcpu_arch_get_attr(struct kvm_vcpu *vcpu,int kvm_arm_vcpu_arch_has_attr (struct kvm_vcpu *vcpu,struct kvm_device_attr *attr); +long kvm_vm_ioctl_mte_copy_tags (struct kvm *kvm, +struct kvm_arm_copy_mte_tags *copy_tags); +/* Guest/host FPSIMD coordination helpers */ int kvm_arch_vcpu_run_map_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu);
arch/arm64/include/asm/mte-def.h +1 -0
@@ -7,6 +7,7 @@#define MTE_GRANULE_SIZE UL(16)#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_GRANULES_PER_PAGE (PAGE_SIZE / MTE_GRANULE_SIZE)#define MTE_TAG_SHIFT 56#define MTE_TAG_SIZE 4#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT)
arch/arm64/include/uapi/asm/kvm.h +11 -0
@@ -184,6 +184,17 @@ struct kvm_vcpu_events {__u32 reserved[12 ]; }; +struct kvm_arm_copy_mte_tags { +__u64 guest_ipa; +__u64 length; +void __user *addr; +__u64 flags; +__u64 reserved[2 ]; +}; + +#define KVM_ARM_TAGS_TO_GUEST 0 +#define KVM_ARM_TAGS_FROM_GUEST 1 +/* If you need to interpret the index values, here is the key: */ #define KVM_REG_ARM_COPROC_MASK 0x000000000FFF0000#define KVM_REG_ARM_COPROC_SHIFT 16
arch/arm64/kvm/arm.c +7 -0
@@ -1345,6 +1345,13 @@ long kvm_arch_vm_ioctl(struct file *filp,return 0 ; } +case KVM_ARM_MTE_COPY_TAGS : { +struct kvm_arm_copy_mte_tags copy_tags; + +if (copy_from_user (©_tags, argp,sizeof (copy_tags))) +return -EFAULT ; +return kvm_vm_ioctl_mte_copy_tags (kvm, ©_tags); + }default :return -EINVAL ; }
arch/arm64/kvm/guest.c +82 -0
85 lines not on this page · raw message
include/uapi/linux/kvm.h +1 -0
@@ -1428,6 +1428,7 @@ struct kvm_s390_ucas_mapping {/* Available with KVM_CAP_PMU_EVENT_FILTER */ #define KVM_SET_PMU_EVENT_FILTER _IOW(KVMIO, 0xb2, struct kvm_pmu_event_filter)#define KVM_PPC_SVM_OFF _IO(KVMIO, 0xb3) +#define KVM_ARM_MTE_COPY_TAGS _IOR(KVMIO, 0xb4, struct kvm_arm_copy_mte_tags)/* ioctl for vm fd */ #define KVM_CREATE_DEVICE _IOWR(KVMIO, 0xe0, struct kvm_create_device)
Signature · 1 line
2.20.1
PATCH v15 7/7 KVM: arm64: Document MTE capability and ioctl
In reply to Steven PriceA new capability (KVM_CAP_ARM_MTE) identifies that the kernel supports granting a guest access to the tags, and provides a mechanism for the VMM to enable it.
A new ioctl (KVM_ARM_MTE_COPY_TAGS) provides a simple way for a VMM to access the tags of a guest without having to maintain a PROT_MTE mapping in userspace. The above capability gates access to the ioctl.
--- Documentation/virt/kvm/api.rst | 57 ++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
Documentation/virt/kvm/api.rst +57 -0
@@ -5034,6 +5034,43 @@ see KVM_XEN_VCPU_SET_ATTR above. The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used with the KVM_XEN_VCPU_GET_ATTR ioctl. +4.130 KVM_ARM_MTE_COPY_TAGS +--------------------------- + +:Capability: KVM_CAP_ARM_MTE +:Architectures: arm64 +:Type: vm ioctl +:Parameters: struct kvm_arm_copy_mte_tags +:Returns: number of bytes copied, < 0 on error (-EINVAL for incorrect + arguments, -EFAULT if memory cannot be accessed). + +:: + + struct kvm_arm_copy_mte_tags { + __u64 guest_ipa; + __u64 length; + void __user *addr; + __u64 flags; + __u64 reserved[2]; + }; + +Copies Memory Tagging Extension (MTE) tags to/from guest tag memory. The +``guest_ipa`` and ``length`` fields must be ``PAGE_SIZE`` aligned. The ``addr`` +field must point to a buffer which the tags will be copied to or from. + +``flags`` specifies the direction of copy, either ``KVM_ARM_TAGS_TO_GUEST`` or +``KVM_ARM_TAGS_FROM_GUEST``. + +The size of the buffer to store the tags is ``(length / 16)`` bytes +(granules in MTE are 16 bytes long). Each byte contains a single tag +value. This matches the format of ``PTRACE_PEEKMTETAGS`` and +``PTRACE_POKEMTETAGS``. + +If an error occurs before any data is copied then a negative error code is +returned. If some tags have been copied before an error occurs then the number +of bytes successfully copied is returned. If the call completes successfully +then ``length`` is returned. + 5. The kvm_run structure ========================
@@ -6362,6 +6399,26 @@ default. See Documentation/x86/sgx/2.Kernel-internals.rst for more details. +7.26 KVM_CAP_ARM_MTE +-------------------- + +:Architectures: arm64 +:Parameters: none + +This capability indicates that KVM (and the hardware) supports exposing the +Memory Tagging Extensions (MTE) to the guest. It must also be enabled by the +VMM before creating any VCPUs to allow the guest access. Note that MTE is only +available to a guest running in AArch64 mode and enabling this capability will +cause attempts to create AArch32 VCPUs to fail. + +When enabled the guest is able to access tags associated with any memory given +to the guest. KVM will ensure that the tags are maintained during swap or +hibernation of the host; however the VMM needs to manually save/restore the +tags as appropriate if the VM is migrated. + +When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to +perform a bulk copy of tags to/from the guest. + 8. Other capabilities. ======================
Signature · 1 line
2.20.1
Catalin Marinas
maintainerQuoted 8 lines · Steven Price
I realise there are still open questions[1] around the performance of this series (the 'big lock',
tag_sync_lock, introduced in the first patch). But there should be no impact on non-MTE workloads and until we get real MTE-enabled hardware it's hard to know whether there is a need for something more sophisticated or not. Peter Collingbourne's patch[3] to clear the tags at page allocation time should hide more of the impact for non-VM cases. So the remaining concern is around VM startup which could be effectively serialised through the lock.
[...]
Quoted 1 line
[1]: https://lore.kernel.org/r/874ke7z3ng.wl-maz%40kernel.org
Start-up, VM resume, migration could be affected by this lock, basically any time you fault a page into the guest. As you said, for now it should be fine as long as the hardware doesn't support MTE or qemu doesn't enable MTE in guests. But the problem won't go away.
We have a partial solution with an array of locks to mitigate against this but there's still the question of whether we should actually bother for something that's unlikely to happen in practice: MAP_SHARED memory in guests (ignoring the stage 1 case for now).
If MAP_SHARED in guests is not a realistic use-case, we have the vma in user_mem_abort() and if the VM_SHARED flag is set together with MTE enabled for guests, we can reject the mapping.
We can discuss the stage 1 case separately from this series.
Signature · 1 line
Catalin
Marc Zyngier
maintainerQuoted 10 lines · Catalin Marinas
Quoted 8 lines · Steven Price
I realise there are still open questions[1] around the performance of this series (the 'big lock',
tag_sync_lock, introduced in the first patch). But there should be no impact on non-MTE workloads and until we get real MTE-enabled hardware it's hard to know whether there is a need for something more sophisticated or not. Peter Collingbourne's patch[3] to clear the tags at page allocation time should hide more of the impact for non-VM cases. So the remaining concern is around VM startup which could be effectively serialised through the lock.
Quoted 7 lines
[...]
Quoted 1 line
[1]: https://lore.kernel.org/r/874ke7z3ng.wl-maz%40kernel.org
Start-up, VM resume, migration could be affected by this lock, basically any time you fault a page into the guest. As you said, for now it should be fine as long as the hardware doesn't support MTE or qemu doesn't enable MTE in guests. But the problem won't go away.
Indeed. And I find it odd to say "it's not a problem, we don't have any HW available". By this token, why should we merge this work the first place, or any of the MTE work that has gone into the kernel over the past years?
Quoted 8 lines
We have a partial solution with an array of locks to mitigate against this but there's still the question of whether we should actually bother for something that's unlikely to happen in practice:
MAP_SHAREDmemory in guests (ignoring the stage 1 case for now).If
MAP_SHAREDin guests is not a realistic use-case, we have the vma inuser_mem_abort()and if theVM_SHAREDflag is set together with MTE enabled for guests, we can reject the mapping.
That's a reasonable approach. I wonder whether we could do that right at the point where the memslot is associated with the VM, like this:
arch/arm64/kvm/mmu.c +3 -0
@@ -1376,6 +1376,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (!vma)break ; +if (kvm_has_mte (kvm) && vma->vm_flags &VM_SHARED ) +return -EINVAL ; +/* * Take the intersection of this VMA with the memory region */
which takes the problem out of the fault path altogether? We document the restriction and move on. With that, we can use a non-locking version of mte_sync_page_tags().
Quoted 1 line
We can discuss the stage 1 case separately from this series.
Works for me.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 52 lines · Marc Zyngier
Quoted 10 lines · Catalin Marinas
Quoted 8 lines · Steven Price
I realise there are still open questions[1] around the performance of this series (the 'big lock',
tag_sync_lock, introduced in the first patch). But there should be no impact on non-MTE workloads and until we get real MTE-enabled hardware it's hard to know whether there is a need for something more sophisticated or not. Peter Collingbourne's patch[3] to clear the tags at page allocation time should hide more of the impact for non-VM cases. So the remaining concern is around VM startup which could be effectively serialised through the lock.Quoted 7 lines
[...]
Quoted 1 line
[1]: https://lore.kernel.org/r/874ke7z3ng.wl-maz%40kernel.org
Start-up, VM resume, migration could be affected by this lock, basically any time you fault a page into the guest. As you said, for now it should be fine as long as the hardware doesn't support MTE or qemu doesn't enable MTE in guests. But the problem won't go away.
Indeed. And I find it odd to say "it's not a problem, we don't have any HW available". By this token, why should we merge this work the first place, or any of the MTE work that has gone into the kernel over the past years?
Quoted 8 lines
We have a partial solution with an array of locks to mitigate against this but there's still the question of whether we should actually bother for something that's unlikely to happen in practice:
MAP_SHAREDmemory in guests (ignoring the stage 1 case for now).If
MAP_SHAREDin guests is not a realistic use-case, we have the vma inuser_mem_abort()and if theVM_SHAREDflag is set together with MTE enabled for guests, we can reject the mapping.That's a reasonable approach. I wonder whether we could do that right at the point where the memslot is associated with the VM, like this:
arch/arm64/kvm/mmu.c +3
-0@@ -1376,6 +1376,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (!vma)break ; +if (kvm_has_mte (kvm) && vma->vm_flags &VM_SHARED ) +return -EINVAL ; +/* * Take the intersection of this VMA with the memory region */
Quoted 3 lines
which takes the problem out of the fault path altogether? We document the restriction and move on. With that, we can use a non-locking version of
mte_sync_page_tags().
Does this deal with the case where the VMAs are changed after the memslot is created? While we can do the check here to give the VMM a heads-up if it gets it wrong, I think we also need it in user_mem_abort() to deal with a VMM which mmap()s over the VA of the memslot. Or am I missing something?
But if everyone is happy with the restriction (just for KVM) of not allowing MTE+VM_SHARED then that sounds like a good way forward.
Thanks,
Steve
Quoted 8 lines
Quoted 1 line
We can discuss the stage 1 case separately from this series.
Works for me.
Thanks,
M.
Marc Zyngier
maintainerQuoted 54 lines · Steven Price
Quoted 52 lines · Marc Zyngier
Quoted 10 lines · Catalin Marinas
Quoted 8 lines · Steven Price
I realise there are still open questions[1] around the performance of this series (the 'big lock',
tag_sync_lock, introduced in the first patch). But there should be no impact on non-MTE workloads and until we get real MTE-enabled hardware it's hard to know whether there is a need for something more sophisticated or not. Peter Collingbourne's patch[3] to clear the tags at page allocation time should hide more of the impact for non-VM cases. So the remaining concern is around VM startup which could be effectively serialised through the lock.Quoted 7 lines
[...]
Quoted 1 line
[1]: https://lore.kernel.org/r/874ke7z3ng.wl-maz%40kernel.org
Start-up, VM resume, migration could be affected by this lock, basically any time you fault a page into the guest. As you said, for now it should be fine as long as the hardware doesn't support MTE or qemu doesn't enable MTE in guests. But the problem won't go away.
Indeed. And I find it odd to say "it's not a problem, we don't have any HW available". By this token, why should we merge this work the first place, or any of the MTE work that has gone into the kernel over the past years?
Quoted 8 lines
We have a partial solution with an array of locks to mitigate against this but there's still the question of whether we should actually bother for something that's unlikely to happen in practice:
MAP_SHAREDmemory in guests (ignoring the stage 1 case for now).If
MAP_SHAREDin guests is not a realistic use-case, we have the vma inuser_mem_abort()and if theVM_SHAREDflag is set together with MTE enabled for guests, we can reject the mapping.That's a reasonable approach. I wonder whether we could do that right at the point where the memslot is associated with the VM, like this:
arch/arm64/kvm/mmu.c +3
-0@@ -1376,6 +1376,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,if (!vma)break ; +if (kvm_has_mte (kvm) && vma->vm_flags &VM_SHARED ) +return -EINVAL ; +/* * Take the intersection of this VMA with the memory region */
Quoted 9 lines
Quoted 3 lines
which takes the problem out of the fault path altogether? We document the restriction and move on. With that, we can use a non-locking version of
mte_sync_page_tags().Does this deal with the case where the VMAs are changed after the memslot is created? While we can do the check here to give the VMM a heads-up if it gets it wrong, I think we also need it in
user_mem_abort()to deal with a VMM which mmap()s over the VA of the memslot. Or am I missing something?
No, you're right. I wish the memslot API wasn't so lax... Anyway, even a VMA flag check in user_mem_abort() will be cheaper than this new BKL.
Quoted 2 lines
But if everyone is happy with the restriction (just for KVM) of not allowing MTE+VM_SHARED then that sounds like a good way forward.
Definitely works for me.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.