KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings
cover KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings
hVHE currently disables TTBR1_EL2 and places both the identity mapping and the hyp runtime mappings in TTBR0_EL2. As a result, it continues to use the legacy nVHE virtual-address layout even though hVHE uses the EL2&0 translation regime and can retain canonical kernel-image addresses.
This series moves the hVHE runtime mappings to TTBR1_EL2 and reserves TTBR0_EL2 for the identity mapping used during MMU and page-table transitions. Conventional VHE is unaffected, and nVHE continues to use the existing TTBR0-only layout, including when protected mode is enabled.
The existing code does not consistently distinguish kernel-image symbols from linear-map addresses because both currently use the same EL2 address transformation. The first four patches make this distinction explicit by:
- adding helpers for kernel-image symbol addresses;
- splitting the hyp mapping APIs by address type;
- providing separate symbol and linear VA-to-PA conversions; and
- clarifying the private hyp VA allocation terminology.
The final patch implements the TTBR1 layout. Hyp symbols retain their linked kernel-image VAs, while pools, per-CPU regions, SVE state and shared memory continue to use linear-map addresses. A separate TTBR0 page table contains the idmap, and both roots are installed during initial EL2 setup and pKVM finalization.
The resulting layouts are:
nVHE:
TTBR0_EL2: idmap and runtime mappings
TTBR1_EL2: unused
hVHE:
TTBR0_EL2: idmap
TTBR1_EL2: runtime mappings
Aneesh Kumar K.V (Arm) (5):
KVM: arm64: Make hyp symbol address conversion explicit
KVM: arm64: Split hyp mapping APIs by address type
KVM: arm64: Split hyp VA-to-PA conversion by address type
KVM: arm64: Rename the hyp private VA allocation base
KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings
arch/arm64/include/asm/kvm_asm.h | 13 +-- arch/arm64/include/asm/kvm_host.h | 2 + arch/arm64/include/asm/kvm_hyp.h | 4 +- arch/arm64/include/asm/kvm_mmu.h | 89 ++++++++++++++-- arch/arm64/include/asm/kvm_pgtable.h | 27 +++++ arch/arm64/include/asm/kvm_pkvm.h | 3 + arch/arm64/kernel/asm-offsets.c | 5 +- arch/arm64/kvm/arm.c | 92 +++++++++------- arch/arm64/kvm/hyp/include/nvhe/memory.h | 13 +-- arch/arm64/kvm/hyp/include/nvhe/mm.h | 9 +- arch/arm64/kvm/hyp/nvhe/early_alloc.c | 1 + arch/arm64/kvm/hyp/nvhe/events.c | 2 +- arch/arm64/kvm/hyp/nvhe/host.S | 4 +- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 29 ++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 30 +++--- arch/arm64/kvm/hyp/nvhe/mm.c | 110 +++++++++++++------ arch/arm64/kvm/hyp/nvhe/psci-relay.c | 12 +-- arch/arm64/kvm/hyp/nvhe/setup.c | 55 +++++++--- arch/arm64/kvm/hyp/pgtable.c | 62 ++++++++++- arch/arm64/kvm/hyp_trace.c | 2 +- arch/arm64/kvm/mmu.c | 128 ++++++++++++++++------- arch/arm64/kvm/pkvm.c | 3 + arch/arm64/kvm/va_layout.c | 32 +++++- 23 files changed, 546 insertions(+), 181 deletions(-)
base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa
Signature · 1 line
2.43.0
RFC PATCH 1/5 KVM: arm64: Make hyp symbol address conversion explicit
kvm_ksym_ref() and kvm_ksym_ref_nvhe() have confusing names which do not clearly describe the address conversion they perform.
Replace them with kvm_hyp_kimg_kaddr() for selecting the kernel address of a hyp symbol and kern_sym_hyp_va() for obtaining its hyp VA. Use lm_alias() directly where a linear-map address is required.
This is a naming cleanup with no functional change.
--- arch/arm64/include/asm/kvm_asm.h | 10 --------- arch/arm64/include/asm/kvm_mmu.h | 33 +++++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 38 ++++++++++++++++---------------- arch/arm64/kvm/va_layout.c | 2 +- 4 files changed, 53 insertions(+), 30 deletions(-)
arch/arm64/include/asm/kvm_asm.h +0 -10
@@ -233,16 +233,6 @@ struct kvm_nvhe_stacktrace_info {unsigned long pc; }; -/* Translate a kernel address @ptr into its equivalent linear mapping */ -#define kvm_ksym_ref (ptr) \ - ({ \ - void *val = (ptr); \ - if (!is_kernel_in_hyp_mode()) \ - val = lm_alias((ptr)); \ - val; \ - }) -#define kvm_ksym_ref_nvhe (sym) kvm_ksym_ref(kvm_nvhe_sym(sym)) -struct kvm ;struct kvm_vcpu ;struct kvm_s2_mmu ;
arch/arm64/include/asm/kvm_mmu.h +33 -0
@@ -107,6 +107,28 @@ u32 kvm_hyp_va_bits(void);void kvm_apply_hyp_relocations (void );#define __hyp_pa (x) (((phys_addr_t)(x)) + hyp_physvirt_offset) +/** +* kvm_hyp_kimg_kaddr - Select the kernel address used for an EL2 mapping +* @ptr: Address within the kernel image +* +* Return @ptr when EL2 uses the kernel-image address directly, as with +* VHE. Otherwise return the corresponding linear-map alias used as input +* to the legacy nVHE VA conversion. +* +* Return: A kernel address suitable as the basis for an EL2 virtual address. +*/ +static __always_inline void *kvm_hyp_kimg_kaddr(void *ptr) +{ +/* +* With VHE, the host runs at EL2 using the kernel mapping, so +* kernel-image VAs need no translation. +* Only legacy nVHE requires a linear-map alias. +*/ +if (is_kernel_in_hyp_mode ()) +return ptr; + +return lm_alias (ptr); +}/* * Convert a kernel VA into a HYP VA.
@@ -140,6 +162,17 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)#define kern_hyp_va (v) ((typeof(v))(__kern_hyp_va((unsigned long)(v)))) +/** +* kern_sym_hyp_va - Convert a kernel image symbol address to its runtime hyp VA +* @ptr: Address of a symbol in the kernel image +* +* Return: The runtime hyp virtual address corresponding to @ptr. +*/ +static __always_inline void *kern_sym_hyp_va(void *ptr) +{ +return kern_hyp_va (kvm_hyp_kimg_kaddr (ptr)); +} +extern u32 __hyp_va_bits;/*
arch/arm64/kvm/arm.c +19 -19
@@ -2105,10 +2105,10 @@ static int kvm_init_vector_slots(void)int err;void *base; - base =kern_hyp_va (kvm_ksym_ref (__kvm_hyp_vector)); + base =kern_sym_hyp_va (__kvm_hyp_vector);kvm_init_vector_slot (base,HYP_VECTOR_DIRECT ); - base =kern_hyp_va (kvm_ksym_ref (__bp_harden_hyp_vecs)); + base =kern_sym_hyp_va (__bp_harden_hyp_vecs);kvm_init_vector_slot (base,HYP_VECTOR_SPECTRE_DIRECT ); if (kvm_system_needs_idmapped_vectors() &&
@@ -2136,7 +2136,7 @@ static void __init cpu_prepare_hyp_mode(int cpu, u32 hyp_va_bits)* Also drop the KASAN tag which gets in the way... */ params->tpidr_el2 = (unsigned long )kasan_reset_tag (per_cpu_ptr_nvhe_sym (__per_cpu_start, cpu)) - - (unsigned long )kvm_ksym_ref (CHOOSE_NVHE_SYM (__per_cpu_start)); + (unsigned long )kvm_hyp_kimg_kaddr (CHOOSE_NVHE_SYM (__per_cpu_start)); params->mair_el2 =read_sysreg (mair_el1);
@@ -2532,13 +2532,13 @@ static void __init teardown_hyp_mode(void)static int __init do_pkvm_init (u32 hyp_va_bits) { -void *per_cpu_base =kvm_ksym_ref (kvm_nvhe_sym (kvm_arm_hyp_percpu_base)); +void *per_cpu_base =kvm_nvhe_sym (kvm_arm_hyp_percpu_base);int ret;preempt_disable ();cpu_hyp_init_context (); ret =kvm_call_hyp_nvhe (__pkvm_init, hyp_mem_base, hyp_mem_size, -kern_hyp_va (per_cpu_base), +kern_sym_hyp_va (per_cpu_base), hyp_va_bits);cpu_hyp_init_features ();
@@ -2611,8 +2611,8 @@ static void kvm_hyp_init_symbols(void)* Flush entire BSS since part of its data containing init symbols is read * while the MMU is off. */ -kvm_flush_dcache_to_poc (kvm_ksym_ref (__hyp_bss_start), -kvm_ksym_ref (__hyp_bss_end) -kvm_ksym_ref (__hyp_bss_start)); +kvm_flush_dcache_to_poc (lm_alias (__hyp_bss_start), +lm_alias (__hyp_bss_end) -lm_alias (__hyp_bss_start)); }static int __init kvm_hyp_init_protection (u32 hyp_va_bits)
@@ -2772,29 +2772,29 @@ static int __init init_hyp_mode(void)/* * Map the Hyp-code called directly from the host */ - err =create_hyp_mappings (kvm_ksym_ref (__hyp_text_start), -kvm_ksym_ref (__hyp_text_end),PAGE_HYP_EXEC ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_text_start), +kvm_hyp_kimg_kaddr (__hyp_text_end),PAGE_HYP_EXEC );if (err) {kvm_err ("Cannot map world-switch code\n" ); goto out_err; } - err =create_hyp_mappings (kvm_ksym_ref (__hyp_data_start), -kvm_ksym_ref (__hyp_data_end),PAGE_HYP ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_data_start), +kvm_hyp_kimg_kaddr (__hyp_data_end),PAGE_HYP );if (err) {kvm_err ("Cannot map .hyp.data section\n" ); goto out_err; } - err =create_hyp_mappings (kvm_ksym_ref (__hyp_rodata_start), -kvm_ksym_ref (__hyp_rodata_end),PAGE_HYP_RO ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_rodata_start), +kvm_hyp_kimg_kaddr (__hyp_rodata_end),PAGE_HYP_RO );if (err) {kvm_err ("Cannot map .hyp.rodata section\n" ); goto out_err; } - err =create_hyp_mappings (kvm_ksym_ref (__start_rodata), -kvm_ksym_ref (__end_rodata),PAGE_HYP_RO ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__start_rodata), +kvm_hyp_kimg_kaddr (__end_rodata),PAGE_HYP_RO );if (err) {kvm_err ("Cannot map rodata section\n" ); goto out_err;
@@ -2805,15 +2805,15 @@ static int __init init_hyp_mode(void)* section thanks to an assertion in the linker script. Map it RW and * the rest of .bss RO. */ - err =create_hyp_mappings (kvm_ksym_ref (__hyp_bss_start), -kvm_ksym_ref (__hyp_bss_end),PAGE_HYP ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_bss_start), +kvm_hyp_kimg_kaddr (__hyp_bss_end),PAGE_HYP );if (err) {kvm_err ("Cannot map hyp bss section: %d\n" , err); goto out_err; } - err =create_hyp_mappings (kvm_ksym_ref (__hyp_bss_end), -kvm_ksym_ref (__bss_stop),PAGE_HYP_RO ); + err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_bss_end), +kvm_hyp_kimg_kaddr (__bss_stop),PAGE_HYP_RO );if (err) {kvm_err ("Cannot map bss section\n" ); goto out_err;
arch/arm64/kvm/va_layout.c +1 -1
@@ -223,7 +223,7 @@ void kvm_patch_vector_branch(struct alt_instr *alt,/* * Compute HYP VA by using the same computation as kern_hyp_va() */ - addr =__early_kern_hyp_va ((u64 )kvm_ksym_ref (__kvm_hyp_vector)); + addr =__early_kern_hyp_va ((u64 )kvm_hyp_kimg_kaddr (__kvm_hyp_vector));/* Use PC[10:7] to branch to the same vector in KVM */ addr |= ((u64 )origptr &GENMASK_ULL (10 ,7 ));
Signature · 1 line
2.43.0
RFC PATCH 2/5 KVM: arm64: Split hyp mapping APIs by address type
In reply to Aneesh Kumar K.V (Arm)Kernel-image symbols and linear-map addresses may refer to the same physical memory, but they are not interchangeable when constructing the EL2 address space. A kernel-image symbol requires __pa_symbol() for its physical address and kern_sym_hyp_va() for its runtime hyp VA, whereas a linear-map address requires __pa() and kern_hyp_va().
The existing create_hyp_mappings() and pkvm_create_mappings() interfaces do not express this distinction. Host callers prepare symbol addresses before invoking create_hyp_mappings(), while the common implementation attempts to infer the physical address from the resulting pointer. The pKVM implementation similarly assumes that every input can be translated using the linear hyp VA-to-PA offset.
This ambiguity works while hyp symbols and linear allocations share the same fixed-offset translation. A later patch introduces an hVHE TTBR1 layout which retains kernel-image VAs for hyp symbols. That layout requires symbol addresses to be distinguished from linear-map addresses.
Replace create_hyp_mappings() with explicit create_hyp_symbol_mappings() and create_hyp_linear_mappings() interfaces. Resolve the appropriate hyp VA and PA in each wrapper, then pass those values to a common mapping helper. This also removes the need for the generic kernel-address-to-physical-address classifier.
Apply the same separation to the pKVM mapping code. Introduce symbol and linear-map wrappers around an internal helper which receives an already resolved hyp VA range and starting PA. Retain a locked linear variant for callers which already hold the hyp page-table lock.
Update all callers to use the interface matching the address type they provide.
--- arch/arm64/include/asm/kvm_mmu.h | 5 +- arch/arm64/kvm/arm.c | 29 +++++------ arch/arm64/kvm/hyp/include/nvhe/mm.h | 8 ++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 16 ++++-- arch/arm64/kvm/hyp/nvhe/mm.c | 46 ++++++++++++----- arch/arm64/kvm/hyp/nvhe/setup.c | 18 ++++--- arch/arm64/kvm/hyp_trace.c | 2 +- arch/arm64/kvm/mmu.c | 71 +++++++++++++++------------ 8 files changed, 122 insertions(+), 73 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +4 -1
@@ -190,7 +190,10 @@ extern u32 __hyp_va_bits;int kvm_share_hyp (void *from,void *to);void kvm_unshare_hyp (void *from,void *to); -int create_hyp_mappings (void *from,void *to,enum kvm_pgtable_prot prot); +int create_hyp_linear_mappings (void *from,void *to, +enum kvm_pgtable_prot prot); +int create_hyp_symbol_mappings (void *from,void *to, +enum kvm_pgtable_prot prot);int __create_hyp_mappings (unsigned long start,unsigned long size,unsigned long phys,enum kvm_pgtable_prot prot);int hyp_alloc_private_va_range (size_t size,unsigned long *haddr);
arch/arm64/kvm/arm.c +15 -14
@@ -2620,7 +2620,7 @@ static int __init kvm_hyp_init_protection(u32 hyp_va_bits)void *addr =phys_to_virt (hyp_mem_base);int ret; - ret =create_hyp_mappings (addr, addr + hyp_mem_size,PAGE_HYP ); + ret =create_hyp_linear_mappings (addr, addr + hyp_mem_size,PAGE_HYP );if (ret)return ret;
@@ -2772,29 +2772,29 @@ static int __init init_hyp_mode(void)/* * Map the Hyp-code called directly from the host */ - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_text_start), -kvm_hyp_kimg_kaddr (__hyp_text_end),PAGE_HYP_EXEC ); + err =create_hyp_symbol_mappings (__hyp_text_start, __hyp_text_end, +PAGE_HYP_EXEC );if (err) {kvm_err ("Cannot map world-switch code\n" ); goto out_err; } - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_data_start), -kvm_hyp_kimg_kaddr (__hyp_data_end),PAGE_HYP ); + err =create_hyp_symbol_mappings (__hyp_data_start, __hyp_data_end, +PAGE_HYP );if (err) {kvm_err ("Cannot map .hyp.data section\n" ); goto out_err; } - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_rodata_start), -kvm_hyp_kimg_kaddr (__hyp_rodata_end),PAGE_HYP_RO ); + err =create_hyp_symbol_mappings (__hyp_rodata_start, __hyp_rodata_end, +PAGE_HYP_RO );if (err) {kvm_err ("Cannot map .hyp.rodata section\n" ); goto out_err; } - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__start_rodata), -kvm_hyp_kimg_kaddr (__end_rodata),PAGE_HYP_RO ); + err =create_hyp_symbol_mappings (__start_rodata, __end_rodata, +PAGE_HYP_RO );if (err) {kvm_err ("Cannot map rodata section\n" ); goto out_err;
@@ -2805,15 +2805,15 @@ static int __init init_hyp_mode(void)* section thanks to an assertion in the linker script. Map it RW and * the rest of .bss RO. */ - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_bss_start), -kvm_hyp_kimg_kaddr (__hyp_bss_end),PAGE_HYP ); + err =create_hyp_symbol_mappings (__hyp_bss_start, __hyp_bss_end, +PAGE_HYP );if (err) {kvm_err ("Cannot map hyp bss section: %d\n" , err); goto out_err; } - err =create_hyp_mappings (kvm_hyp_kimg_kaddr (__hyp_bss_end), -kvm_hyp_kimg_kaddr (__bss_stop),PAGE_HYP_RO ); + err =create_hyp_symbol_mappings (__hyp_bss_end, __bss_stop, +PAGE_HYP_RO );if (err) {kvm_err ("Cannot map bss section\n" ); goto out_err;
@@ -2846,7 +2846,8 @@ static int __init init_hyp_mode(void)char *percpu_end = percpu_begin +nvhe_percpu_size ();/* Map Hyp percpu pages */ - err =create_hyp_mappings (percpu_begin, percpu_end,PAGE_HYP ); + err =create_hyp_linear_mappings (percpu_begin, percpu_end, +PAGE_HYP );if (err) {kvm_err ("Cannot map hyp percpu region\n" ); goto out_err;
arch/arm64/kvm/hyp/include/nvhe/mm.h +6 -2
@@ -23,8 +23,12 @@ int hyp_create_idmap(u32 hyp_va_bits);int hyp_map_vectors (void );int hyp_back_vmemmap (phys_addr_t back);int pkvm_cpu_set_vector (enum arm64_hyp_spectre_vector slot); -int pkvm_create_mappings (void *from,void *to,enum kvm_pgtable_prot prot); -int pkvm_create_mappings_locked (void *from,void *to,enum kvm_pgtable_prot prot); +int pkvm_create_linear_mappings (void *from,void *to, +enum kvm_pgtable_prot prot); +int pkvm_create_symbol_mappings (void *from,void *to, +enum kvm_pgtable_prot prot); +int pkvm_create_linear_mappings_locked (void *from,void *to, +enum kvm_pgtable_prot prot);int __pkvm_create_private_mapping (phys_addr_t phys,size_t size,enum kvm_pgtable_prot prot,unsigned long *haddr);
arch/arm64/kvm/hyp/nvhe/mem_protect.c +11 -5
@@ -1109,7 +1109,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) goto unlock;__hyp_set_page_state_range (phys, size,PKVM_PAGE_OWNED ); -WARN_ON (pkvm_create_mappings_locked (virt, virt + size,PAGE_HYP )); +WARN_ON (pkvm_create_linear_mappings_locked (virt, virt + size,PAGE_HYP ));WARN_ON (host_stage2_set_owner_locked (phys, size,PKVM_ID_HYP )); unlock:
@@ -1175,12 +1175,18 @@ int hyp_pin_shared_mem(void *from, void *to) goto unlock;for (cur = start; cur < end; cur +=PAGE_SIZE ) { +void *addr = (void *)cur; + p =hyp_virt_to_page (cur);hyp_page_ref_inc (p); -if (p->refcount ==1 ) -WARN_ON (pkvm_create_mappings_locked ((void *)cur, - (void *)cur +PAGE_SIZE , -PAGE_HYP )); +if (p->refcount ==1 ) { +int err; + + err =pkvm_create_linear_mappings_locked (addr, + addr +PAGE_SIZE , +PAGE_HYP ); +WARN_ON (err); + } } unlock:
arch/arm64/kvm/hyp/nvhe/mm.c +35 -11
70 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/setup.c +11 -7
@@ -86,7 +86,7 @@ static int pkvm_create_host_sve_mappings(void) start =kern_hyp_va (sve_regs); end = start +PAGE_ALIGN (pkvm_host_sve_state_size ()); - ret =pkvm_create_mappings (start, end,PAGE_HYP ); + ret =pkvm_create_linear_mappings (start, end,PAGE_HYP );if (ret)return ret; }
@@ -121,23 +121,27 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size,if (ret)return ret; - ret =pkvm_create_mappings (__hyp_text_start, __hyp_text_end,PAGE_HYP_EXEC ); + ret =pkvm_create_symbol_mappings (__hyp_text_start, __hyp_text_end, +PAGE_HYP_EXEC );if (ret)return ret; - ret =pkvm_create_mappings (__hyp_data_start, __hyp_data_end,PAGE_HYP ); + ret =pkvm_create_symbol_mappings (__hyp_data_start, __hyp_data_end, +PAGE_HYP );if (ret)return ret; - ret =pkvm_create_mappings (__hyp_rodata_start, __hyp_rodata_end,PAGE_HYP_RO ); + ret =pkvm_create_symbol_mappings (__hyp_rodata_start, __hyp_rodata_end, +PAGE_HYP_RO );if (ret)return ret; - ret =pkvm_create_mappings (__hyp_bss_start, __hyp_bss_end,PAGE_HYP ); + ret =pkvm_create_symbol_mappings (__hyp_bss_start, __hyp_bss_end, +PAGE_HYP );if (ret)return ret; - ret =pkvm_create_mappings (virt, virt + size,PAGE_HYP ); + ret =pkvm_create_linear_mappings (virt, virt + size,PAGE_HYP );if (ret)return ret;
@@ -146,7 +150,7 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size, start = (void *)kern_hyp_va (per_cpu_base[i]); end = start +PAGE_ALIGN (hyp_percpu_size); - ret =pkvm_create_mappings (start, end,PAGE_HYP ); + ret =pkvm_create_linear_mappings (start, end,PAGE_HYP );if (ret)return ret;
arch/arm64/kvm/hyp_trace.c +1 -1
@@ -142,7 +142,7 @@ static int __map_hyp(void *start, size_t size)if (is_protected_kvm_enabled ())return 0 ; -return create_hyp_mappings (start, start + size,PAGE_HYP ); +return create_hyp_linear_mappings (start, start + size,PAGE_HYP ); }static int __share_page (unsigned long va)
arch/arm64/kvm/mmu.c +39 -32
105 lines not on this page · raw message
Signature · 1 line
2.43.0
RFC PATCH 3/5 KVM: arm64: Split hyp VA-to-PA conversion by address type
In reply to Aneesh Kumar K.V (Arm)The existing hyp VA-to-PA conversion uses one physvirt offset for both kernel-image symbols and linear-map addresses. This only works while both address types use the same EL2 VA transformation.
With hVHE TTBR1, hyp symbols retain their kernel-image VAs while pools, per-CPU data and shared memory remain linear-map-derived. Introduce a separate symbol physvirt offset and explicit symbol and linear conversion helpers.
Update every caller according to its address type and remove the ambiguous hyp_pa and __hyp_pa helpers. The offsets remain same as hyp_physvirt_offset, making this preparatory for the TTBR1 layout.
--- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/include/asm/kvm_mmu.h | 17 ++++++++++++----- arch/arm64/kvm/arm.c | 2 +- arch/arm64/kvm/hyp/include/nvhe/memory.h | 13 +++++++------ arch/arm64/kvm/hyp/nvhe/early_alloc.c | 1 + arch/arm64/kvm/hyp/nvhe/events.c | 2 +- arch/arm64/kvm/hyp/nvhe/host.S | 4 ++-- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 14 +++++++------- arch/arm64/kvm/hyp/nvhe/mm.c | 11 ++++++----- arch/arm64/kvm/hyp/nvhe/psci-relay.c | 12 ++++++------ arch/arm64/kvm/hyp/nvhe/setup.c | 4 ++-- arch/arm64/kvm/va_layout.c | 2 ++ 12 files changed, 49 insertions(+), 35 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -827,6 +827,8 @@ extern struct kvm_host_psci_config kvm_nvhe_sym(kvm_host_psci_config);extern s64 kvm_nvhe_sym (hyp_physvirt_offset );#define hyp_physvirt_offset CHOOSE_NVHE_SYM(hyp_physvirt_offset) +extern s64 kvm_nvhe_sym (hyp_symbol_physvirt_offset ); +#define hyp_symbol_physvirt_offset CHOOSE_NVHE_SYM(hyp_symbol_physvirt_offset)extern u64 kvm_nvhe_sym (hyp_cpu_logical_map )[NR_CPUS ];#define hyp_cpu_logical_map CHOOSE_NVHE_SYM(hyp_cpu_logical_map)
arch/arm64/include/asm/kvm_mmu.h +12 -5
@@ -54,15 +54,21 @@#include <asm/alternative.h> /* -* Convert a hypervisor VA to a PA +* Convert an address in the hypervisor linear map to a PA * reg: hypervisor address to be converted in place * tmp: temporary register */ -.macro hyp_pa reg, tmp +.macro hyp_linear_pa reg, tmp ldr_l \tmp, hyp_physvirt_offset add \reg, \reg, \tmp .endm +/* Convert a hypervisor kernel-image VA to a PA. */ +.macro hyp_symbol_pa reg, tmp + ldr_l \tmp, hyp_symbol_physvirt_offset + add \reg, \reg, \tmp +.endm +/* * Convert a hypervisor VA to a kernel image address * reg: hypervisor address to be converted in place
@@ -74,8 +80,8 @@* specific registers encoded in the instructions). */ .macro hyp_kimg_va reg, tmp -/* Convert hyp VA -> PA. */ - hyp_pa \reg, \tmp +/* Convert hyp kernel-image VA -> PA. */ + hyp_symbol_pa \reg, \tmp /* Load kimage_voffset. */ alternative_cb ARM64_ALWAYS_SYSTEM, kvm_get_kimage_voffset
@@ -106,7 +112,6 @@ void kvm_compute_layout(void);u32 kvm_hyp_va_bits (void );void kvm_apply_hyp_relocations (void ); -#define __hyp_pa (x) (((phys_addr_t)(x)) + hyp_physvirt_offset)/** * kvm_hyp_kimg_kaddr - Select the kernel address used for an EL2 mapping * @ptr: Address within the kernel image
@@ -130,6 +135,8 @@ static __always_inline void *kvm_hyp_kimg_kaddr(void *ptr)return lm_alias (ptr); } +#define __hyp_linear_pa (x) (((phys_addr_t)(x)) + hyp_physvirt_offset) +#define __hyp_symbol_pa (x) (((phys_addr_t)(x)) + hyp_symbol_physvirt_offset)/* * Convert a kernel VA into a HYP VA. *
arch/arm64/kvm/arm.c +1 -1
@@ -2835,7 +2835,7 @@ static int __init init_hyp_mode(void)/* * Save the stack PA in nvhe_init_params. This will be needed * to recreate the stack mapping in protected nVHE mode. -* __hyp_pa() won't do the right thing there, since the stack +* __hyp_linear_pa() won't do the right thing there, since the stack * has been mapped in the flexible private VA space. */ params->stack_pa =__pa (stack_base);
arch/arm64/kvm/hyp/include/nvhe/memory.h +7 -6
@@ -73,16 +73,17 @@ struct hyp_page {extern u64 __hyp_vmemmap;#define hyp_vmemmap ((struct hyp_page *)__hyp_vmemmap) -#define __hyp_va (phys) ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset)) +#define __hyp_linear_va (phys) \ + ((void *)((phys_addr_t)(phys) - hyp_physvirt_offset))static inline void *hyp_phys_to_virt (phys_addr_t phys) { -return __hyp_va (phys); +return __hyp_linear_va (phys); }static inline phys_addr_t hyp_virt_to_phys (void *addr) { -return __hyp_pa (addr); +return __hyp_linear_pa (addr); }#define hyp_phys_to_pfn (phys) ((phys) >> PAGE_SHIFT)
@@ -94,12 +95,12 @@ static inline struct hyp_page *hyp_phys_to_page(phys_addr_t phys)return &hyp_vmemmap[hyp_phys_to_pfn (phys)]; } -#define hyp_virt_to_page (virt) hyp_phys_to_page(__hyp_pa(virt)) -#define hyp_virt_to_pfn (virt) hyp_phys_to_pfn(__hyp_pa(virt)) +#define hyp_virt_to_page (virt) hyp_phys_to_page(__hyp_linear_pa(virt)) +#define hyp_virt_to_pfn (virt) hyp_phys_to_pfn(__hyp_linear_pa(virt))#define hyp_page_to_pfn (page) ((struct hyp_page *)(page) - hyp_vmemmap)#define hyp_page_to_phys (page) hyp_pfn_to_phys((hyp_page_to_pfn(page))) -#define hyp_page_to_virt (page) __hyp_va(hyp_page_to_phys(page)) +#define hyp_page_to_virt (page) __hyp_linear_va(hyp_page_to_phys(page))#define hyp_page_to_pool (page) (((struct hyp_page *)page)->pool)static inline enum pkvm_page_state get_host_state (struct hyp_page *p)
arch/arm64/kvm/hyp/nvhe/early_alloc.c +1 -0
@@ -11,6 +11,7 @@struct kvm_pgtable_mm_ops hyp_early_alloc_mm_ops;s64 __ro_after_init hyp_physvirt_offset; +s64 __ro_after_init hyp_symbol_physvirt_offset;static unsigned long base;static unsigned long end;
arch/arm64/kvm/hyp/nvhe/events.c +1 -1
@@ -17,7 +17,7 @@ int __tracing_enable_event(unsigned short id, bool enable)if (event_id >= __hyp_event_ids_end)return -EINVAL ; - enabled =hyp_fixmap_map (__hyp_pa (&event_id->enabled)); + enabled =hyp_fixmap_map (__hyp_symbol_pa (&event_id->enabled));atomic_set (enabled, enable);hyp_fixmap_unmap ();
arch/arm64/kvm/hyp/nvhe/host.S +2 -2
@@ -133,7 +133,7 @@ SYM_FUNC_START(__hyp_do_panic) mrs x0, esr_el2 mov x4, x3 mov x3, x2 - hyp_pa x3, x6 + hyp_symbol_pa x3, x6 get_vcpu_ptr x5, x6 mrs x6, far_el2 mrs x7, hpfar_el2
@@ -163,7 +163,7 @@ alternative_else_nop_endif * Preserve x0-x4, which may contain stub parameters. */ adr_l x5, __kvm_handle_stub_hvc - hyp_pa x5, x6 + hyp_symbol_pa x5, x6 br x5 SYM_FUNC_END(__host_hvc)
arch/arm64/kvm/hyp/nvhe/mem_protect.c +7 -7
@@ -175,7 +175,7 @@ int kvm_host_prepare_stage2(void *pgt_pool_base)if (ret)return ret; - mmu->pgd_phys =__hyp_pa (host_mmu.pgt.pgd); + mmu->pgd_phys =__hyp_linear_pa (host_mmu.pgt.pgd); mmu->pgt = &host_mmu.pgt;atomic64_set (&mmu->vmid.id,0 );
@@ -243,9 +243,9 @@ static void __apply_guest_page(void *va, size_t size,void *map;if (IS_ALIGNED ((unsigned long )va,PMD_SIZE ) && size >=PMD_SIZE ) - map =hyp_fixblock_map (__hyp_pa (va), &map_size); + map =hyp_fixblock_map (__hyp_linear_pa (va), &map_size);else - map =hyp_fixmap_map (__hyp_pa (va)); + map =hyp_fixmap_map (__hyp_linear_pa (va));func (map, map_size);
@@ -300,7 +300,7 @@ int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd)if (ret)return ret; - vm->kvm.arch.mmu.pgd_phys =__hyp_pa (vm->pgt.pgd); + vm->kvm.arch.mmu.pgd_phys =__hyp_linear_pa (vm->pgt.pgd);return 0 ; }
@@ -1092,7 +1092,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) {u64 phys =hyp_pfn_to_phys (pfn);u64 size =PAGE_SIZE * nr_pages; -void *virt =__hyp_va (phys); +void *virt =__hyp_linear_va (phys);int ret;if (!pfn_range_is_valid (pfn, nr_pages))
@@ -1123,7 +1123,7 @@ int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) {u64 phys =hyp_pfn_to_phys (pfn);u64 size =PAGE_SIZE * nr_pages; -u64 virt = (u64 )__hyp_va (phys); +u64 virt = (u64 )__hyp_linear_va (phys);int ret;if (!pfn_range_is_valid (pfn, nr_pages))
@@ -1158,7 +1158,7 @@ int hyp_pin_shared_mem(void *from, void *to) {u64 cur, start =ALIGN_DOWN ((u64 )from,PAGE_SIZE );u64 end =PAGE_ALIGN ((u64 )to); -u64 phys =__hyp_pa (start); +u64 phys =__hyp_linear_pa (start);u64 size = end - start;struct hyp_page *p;int ret;
arch/arm64/kvm/hyp/nvhe/mm.c +6 -5
@@ -159,7 +159,7 @@ int pkvm_create_symbol_mappings(void *from, void *to, enum kvm_pgtable_prot prot) {unsigned long start = (unsigned long )from &PAGE_MASK ; -phys_addr_t phys =__hyp_pa (start); +phys_addr_t phys =__hyp_symbol_pa (start);int ret;hyp_spin_lock (&pkvm_pgd_lock);
@@ -242,7 +242,7 @@ int hyp_map_vectors(void)return 0 ; } - phys =__hyp_pa (__bp_harden_hyp_vecs); + phys =__hyp_symbol_pa (__bp_harden_hyp_vecs); ret =__pkvm_create_private_mapping (phys, __BP_HARDEN_HYP_VECS_SZ,PAGE_HYP_EXEC , &bp_base);if (ret)
@@ -418,7 +418,8 @@ int hyp_create_fixmap(void)return ret; ret =kvm_pgtable_hyp_map (&pkvm_pgtable, addr,PAGE_SIZE , -__hyp_pa (__hyp_bss_start),PAGE_HYP ); +__hyp_symbol_pa (__hyp_bss_start), +PAGE_HYP );if (ret)return ret;
@@ -434,10 +435,10 @@ int hyp_create_idmap(u32 hyp_va_bits) {unsigned long start, end; - start =hyp_virt_to_phys ((void *)__hyp_idmap_text_start); + start =__hyp_symbol_pa (__hyp_idmap_text_start); start =ALIGN_DOWN (start,PAGE_SIZE ); - end =hyp_virt_to_phys ((void *)__hyp_idmap_text_end); + end =__hyp_symbol_pa (__hyp_idmap_text_end); end =ALIGN (end,PAGE_SIZE );/*
arch/arm64/kvm/hyp/nvhe/psci-relay.c +6 -6
@@ -139,8 +139,8 @@ static int psci_cpu_on(u64 func_id, struct kvm_cpu_context *host_ctxt)wmb (); ret =psci_call (func_id, mpidr, -__hyp_pa (&kvm_hyp_cpu_entry), -__hyp_pa (init_params)); +__hyp_symbol_pa (&kvm_hyp_cpu_entry), +__hyp_linear_pa (init_params));/* If successful, the lock will be released by the target CPU. */ if (ret !=PSCI_RET_SUCCESS )
@@ -173,8 +173,8 @@ static int psci_cpu_suspend(u64 func_id, struct kvm_cpu_context *host_ctxt)* point if it is a deep sleep state. */ return psci_call (func_id, power_state, -__hyp_pa (&kvm_hyp_cpu_resume), -__hyp_pa (init_params)); +__hyp_symbol_pa (&kvm_hyp_cpu_resume), +__hyp_linear_pa (init_params)); }static int psci_system_suspend (u64 func_id,struct kvm_cpu_context *host_ctxt)
@@ -197,8 +197,8 @@ static int psci_system_suspend(u64 func_id, struct kvm_cpu_context *host_ctxt)/* Will only return on error. */ return psci_call (func_id, -__hyp_pa (&kvm_hyp_cpu_resume), -__hyp_pa (init_params),0 ); +__hyp_symbol_pa (&kvm_hyp_cpu_resume), +__hyp_linear_pa (init_params),0 ); }static void __noreturn __kvm_host_psci_cpu_entry (unsigned long pc,unsigned long r0)
arch/arm64/kvm/hyp/nvhe/setup.c +2 -2
@@ -169,7 +169,7 @@ static void update_nvhe_init_params(void)for (i =0 ; i < hyp_nr_cpus; i++) { params =per_cpu_ptr (&kvm_init_params, i); - params->pgd_pa =__hyp_pa (pkvm_pgtable.pgd); + params->pgd_pa =__hyp_linear_pa (pkvm_pgtable.pgd);dcache_clean_inval_poc ((unsigned long )params, (unsigned long )params +sizeof (*params)); }
@@ -371,7 +371,7 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas/* Jump in the idmap page to switch to the new page-tables */ params =this_cpu_ptr (&kvm_init_params); - fn = (typeof (fn ))__hyp_pa (__pkvm_init_switch_pgd); + fn = (typeof (fn ))__hyp_symbol_pa (__pkvm_init_switch_pgd);fn (params->pgd_pa, params->stack_hyp_va, __pkvm_init_finalise);unreachable ();
arch/arm64/kvm/va_layout.c +2 -0
@@ -44,6 +44,8 @@ static void init_hyp_physvirt_offset(void) kern_va = (u64 )lm_alias (__hyp_text_start); hyp_va =__early_kern_hyp_va (kern_va); hyp_physvirt_offset = (s64 )__pa (kern_va) - (s64 )hyp_va; + + hyp_symbol_physvirt_offset = hyp_physvirt_offset; }/*
Signature · 1 line
2.43.0
RFC PATCH 4/5 KVM: arm64: Rename the hyp private VA allocation base
In reply to Aneesh Kumar K.V (Arm)io_map_base is not used for I/O mappings. It is the allocation base for private hyp VAs used by stacks, fixmaps and others.
Rename it to __hyp_private_va_base in the hyp code
No functional change in this patch.
--- arch/arm64/kvm/hyp/nvhe/mm.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
arch/arm64/kvm/hyp/nvhe/mm.c +16 -15
@@ -24,7 +24,7 @@ hyp_spinlock_t pkvm_pgd_lock;struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS ];unsigned int hyp_memblock_nr; -static u64 __io_map_base; +static u64 __hyp_private_va_base;struct hyp_fixmap_slot {u64 addr;
@@ -50,7 +50,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)hyp_assert_lock_held (&pkvm_pgd_lock); -if (!start || start < __io_map_base) +if (!start || start < __hyp_private_va_base)return -EINVAL ;/* The allocated size is always a multiple of PAGE_SIZE */
@@ -60,7 +60,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)if (cur > __hyp_vmemmap)return -ENOMEM ; - __io_map_base = cur; + __hyp_private_va_base = cur;return 0 ; }
@@ -70,8 +70,8 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)* @size: The size of the VA range to reserve. * @haddr: The hypervisor virtual start address of the allocation. * -* The private virtual address (VA) range is allocated above __io_map_base -* and aligned based on the order of @size. +* The private virtual address (VA) range is allocated from +* __hyp_private_va_base and aligned based on the order of @size. * * Return: 0 on success or negative error code on failure. */
@@ -81,7 +81,7 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr)int ret;hyp_spin_lock (&pkvm_pgd_lock); - addr = __io_map_base; + addr = __hyp_private_va_base; ret =__pkvm_alloc_private_va_range (addr, size);hyp_spin_unlock (&pkvm_pgd_lock);
@@ -365,7 +365,7 @@ static int create_fixblock(void)return -EINVAL ;hyp_spin_lock (&pkvm_pgd_lock); - addr =ALIGN (__io_map_base,PMD_SIZE ); + addr =ALIGN (__hyp_private_va_base,PMD_SIZE ); ret =__pkvm_alloc_private_va_range (addr,PMD_SIZE );if (ret) goto unlock;
@@ -446,12 +446,13 @@ int hyp_create_idmap(u32 hyp_va_bits)* memory -- see va_layout.c for more details. The other half of the VA * space contains the trampoline page, and needs some care. Split that * second half in two and find the quarter of VA space not conflicting -* with the idmap to place the IOs and the vmemmap. IOs use the lower -* half of the quarter and the vmemmap the upper half. +* with the idmap to place private mappings and the vmemmap. Private +* mappings use the lower half of the quarter and the vmemmap the upper +* half. */ - __io_map_base = start &BIT (hyp_va_bits -2 ); - __io_map_base ^=BIT (hyp_va_bits -2 ); - __hyp_vmemmap = __io_map_base |BIT (hyp_va_bits -3 ); + __hyp_private_va_base = start &BIT (hyp_va_bits -2 ); + __hyp_private_va_base ^=BIT (hyp_va_bits -2 ); + __hyp_vmemmap = __hyp_private_va_base |BIT (hyp_va_bits -3 );return __pkvm_create_mappings (start, end - start, start,PAGE_HYP_EXEC ); }
@@ -464,13 +465,13 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)hyp_spin_lock (&pkvm_pgd_lock); - prev_base = __io_map_base; + prev_base = __hyp_private_va_base;/* * Efficient stack verification using the NVHE_STACK_SHIFT bit implies * an alignment of our allocation on the order of the size. */ size =NVHE_STACK_SIZE *2 ; - addr =ALIGN (__io_map_base, size); + addr =ALIGN (__hyp_private_va_base, size); ret =__pkvm_alloc_private_va_range (addr, size);if (!ret) {
@@ -486,7 +487,7 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr) ret =kvm_pgtable_hyp_map (&pkvm_pgtable, addr +NVHE_STACK_SIZE ,NVHE_STACK_SIZE , phys,PAGE_HYP );if (ret) - __io_map_base = prev_base; + __hyp_private_va_base = prev_base; }hyp_spin_unlock (&pkvm_pgd_lock);
Signature · 1 line
2.43.0
RFC PATCH 5/5 KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings
In reply to Aneesh Kumar K.V (Arm)hVHE currently disables TTBR1_EL2 and places both the identity mapping and runtime hyp mappings in TTBR0_EL2. This unnecessarily retains the legacy nVHE VA layout even though hVHE can use canonical kernel-image addresses through the EL2&0 translation regime.
Place hVHE runtime mappings in TTBR1_EL2 and reserve TTBR0_EL2 for the identity mapping used during MMU and page-table transitions. Select this layout based on hVHE support independently of protected mode, while retaining the existing TTBR0-only layout for nVHE and protected nVHE.
Keep hyp symbols at their linked kernel-image VAs, disable the legacy VA relocations for hVHE, and configure TCR_EL2 for both translation ranges. Allocate separate runtime and idmap page tables during initial setup and pKVM finalization, pass both roots through the per-CPU initialization parameters, and install both during page-table transitions.
Add support for walking canonical TTBR1 hyp page tables, including handling the wrap at the top of the address space. Use this when fixing page-table reference counts and destroying hyp page tables.
--- arch/arm64/include/asm/kvm_asm.h | 3 +- arch/arm64/include/asm/kvm_hyp.h | 4 +- arch/arm64/include/asm/kvm_mmu.h | 46 ++++++++++++++++++--- arch/arm64/include/asm/kvm_pgtable.h | 27 ++++++++++++ arch/arm64/include/asm/kvm_pkvm.h | 3 ++ arch/arm64/kernel/asm-offsets.c | 5 ++- arch/arm64/kvm/arm.c | 47 +++++++++++++-------- arch/arm64/kvm/hyp/include/nvhe/mm.h | 1 + arch/arm64/kvm/hyp/nvhe/hyp-init.S | 29 ++++++++++--- arch/arm64/kvm/hyp/nvhe/mm.c | 24 +++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 35 ++++++++++++++-- arch/arm64/kvm/hyp/pgtable.c | 62 +++++++++++++++++++++++++--- arch/arm64/kvm/mmu.c | 57 +++++++++++++++++++++++-- arch/arm64/kvm/pkvm.c | 3 ++ arch/arm64/kvm/va_layout.c | 32 +++++++++++--- 15 files changed, 328 insertions(+), 50 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -1
@@ -210,7 +210,8 @@ struct kvm_nvhe_init_params {unsigned long tpidr_el2;unsigned long stack_hyp_va;unsigned long stack_pa; -phys_addr_t pgd_pa; +phys_addr_t ttbr0_pgd_pa; +phys_addr_t ttbr1_pgd_pa;unsigned long hcr_el2;unsigned long vttbr;unsigned long vtcr;
arch/arm64/include/asm/kvm_hyp.h +2 -2
@@ -131,8 +131,8 @@ void __noreturn __hyp_do_panic(struct kvm_cpu_context *host_ctxt, u64 spsr,#endif #ifdef __KVM_NVHE_HYPERVISOR__ -void __pkvm_init_switch_pgd (phys_addr_t pgd,unsigned long sp, -void (*fn)(void )); +void __pkvm_init_switch_pgd (phys_addr_t ttbr0,phys_addr_t ttbr1, +unsigned long sp,void (*fn)(void ));int __pkvm_init (phys_addr_t phys,unsigned long size,unsigned long *per_cpu_base,u32 hyp_va_bits);void __noreturn __host_enter (struct kvm_cpu_context *host_ctxt);#endif
arch/arm64/include/asm/kvm_mmu.h +40 -6
81 lines not on this page · raw message
arch/arm64/include/asm/kvm_pgtable.h +27 -0
@@ -255,6 +255,14 @@ enum kvm_pgtable_stage2_flags { KVM_PGTABLE_S2_AS_S1 = BIT(1), }; +/** +* enum kvm_pgtable_hyp_flags - Hypervisor page-table flags. +* @KVM_PGTABLE_HYP_TTBR1: Walk TTBR1 canonical virtual addresses. +*/ +enum kvm_pgtable_hyp_flags { +KVM_PGTABLE_HYP_TTBR1 =BIT (0 ), +}; +/** * enum kvm_pgtable_prot - Page-table permissions and attributes. * @KVM_PGTABLE_PROT_UX: Unprivileged execute permission.
@@ -443,6 +451,7 @@ static inline bool kvm_pgtable_walk_lock_held(void)* @start_level: Level at which the page-table walk starts. * @pgd: Pointer to the first top-level entry of the page-table. * @mm_ops: Memory management callbacks. +* @hyp_flags: Hypervisor page-table flags. * @mmu: Stage-2 KVM MMU struct. Unused for stage-1 page-tables. * @flags: Stage-2 page-table flags. * @force_pte_cb: Function that returns true if page level mappings must
@@ -457,6 +466,9 @@ struct kvm_pgtable {kvm_pteref_t pgd;struct kvm_pgtable_mm_ops *mm_ops; +/* Hyp only */ +enum kvm_pgtable_hyp_flags hyp_flags; +/* Stage-2 only */ enum kvm_pgtable_stage2_flags flags;kvm_pgtable_force_pte_cb_t force_pte_cb;
@@ -465,6 +477,11 @@ struct kvm_pgtable {struct kvm_s2_mmu *mmu; }; +static inline void kvm_pgtable_hyp_enable_ttbr1 (struct kvm_pgtable *pgt) +{ + pgt->hyp_flags |=KVM_PGTABLE_HYP_TTBR1 ; +} +/** * kvm_pgtable_hyp_init() - Initialise a hypervisor stage-1 page-table. * @pgt: Uninitialised page-table structure to initialise.
@@ -851,6 +868,16 @@ int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,int kvm_pgtable_walk (struct kvm_pgtable *pgt,u64 addr,u64 size,struct kvm_pgtable_walker *walker); +/** +* kvm_pgtable_hyp_walk() - Walk the entire usable VA range of a hyp pgtable. +* @pgt: Page-table structure initialised by kvm_pgtable_hyp_init(). +* @walker: Walker callback description. +* +* Return: 0 on success, negative error code on failure. +*/ +int kvm_pgtable_hyp_walk (struct kvm_pgtable *pgt, +struct kvm_pgtable_walker *walker); +/** * kvm_pgtable_get_leaf() - Walk a page-table and retrieve the leaf entry * with its level.
arch/arm64/include/asm/kvm_pkvm.h +3 -0
@@ -139,6 +139,9 @@ static inline unsigned long hyp_s1_pgtable_pages(void)/* Allow 1 GiB for private mappings */ res +=__hyp_pgtable_max_pages (SZ_1G >>PAGE_SHIFT ); +/* Allow a separate TTBR0 idmap page-table. */ + res +=__hyp_pgtable_max_pages (1 ); +return res; }
arch/arm64/kernel/asm-offsets.c +4 -1
@@ -120,7 +120,10 @@ int main(void)DEFINE (NVHE_INIT_TCR_EL2 , offsetof(struct kvm_nvhe_init_params , tcr_el2));DEFINE (NVHE_INIT_TPIDR_EL2 , offsetof(struct kvm_nvhe_init_params , tpidr_el2));DEFINE (NVHE_INIT_STACK_HYP_VA , offsetof(struct kvm_nvhe_init_params , stack_hyp_va)); -DEFINE (NVHE_INIT_PGD_PA , offsetof(struct kvm_nvhe_init_params , pgd_pa)); +DEFINE (NVHE_INIT_TTBR0_PGD_PA , + offsetof(struct kvm_nvhe_init_params , ttbr0_pgd_pa)); +DEFINE (NVHE_INIT_TTBR1_PGD_PA , + offsetof(struct kvm_nvhe_init_params , ttbr1_pgd_pa));DEFINE (NVHE_INIT_HCR_EL2 , offsetof(struct kvm_nvhe_init_params , hcr_el2));DEFINE (NVHE_INIT_VTTBR , offsetof(struct kvm_nvhe_init_params , vttbr));DEFINE (NVHE_INIT_VTCR , offsetof(struct kvm_nvhe_init_params , vtcr));
arch/arm64/kvm/arm.c +30 -17
68 lines not on this page · raw message
arch/arm64/kvm/hyp/include/nvhe/mm.h +1 -0
@@ -11,6 +11,7 @@#include <nvhe/spinlock.h> extern struct kvm_pgtable pkvm_pgtable; +extern struct kvm_pgtable pkvm_idmap_pgtable;extern hyp_spinlock_t pkvm_pgd_lock;int hyp_create_fixmap (void );
arch/arm64/kvm/hyp/nvhe/hyp-init.S +24 -5
@@ -127,13 +127,22 @@ SYM_CODE_START_LOCAL(___kvm_hyp_init) ldr x1, [x0, #NVHE_INIT_VTCR] msr vtcr_el2, x1 - ldr x1, [x0, #NVHE_INIT_PGD_PA] + ldr x1, [x0, #NVHE_INIT_TTBR0_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP orr x2, x2, #TTBRx_EL1_CnP alternative_else_nop_endif msr ttbr0_el2, x2 + ldr x1, [x0, #NVHE_INIT_TTBR1_PGD_PA] + cbz x1, 2f + phys_to_ttbr x2, x1 +alternative_if ARM64_HAS_CNP + orr x2, x2, #TTBRx_EL1_CnP +alternative_else_nop_endif + msr_s SYS_TTBR1_EL2, x2 +2: + ldr x0, [x0, #NVHE_INIT_TCR_EL2] msr tcr_el2, x0
@@ -261,13 +270,15 @@ reset: SYM_CODE_END(__kvm_handle_stub_hvc) /* - * void __pkvm_init_switch_pgd(phys_addr_t pgd, unsigned long sp, - * void (*fn)(void)); + * void __pkvm_init_switch_pgd(phys_addr_t ttbr0, phys_addr_t ttbr1, + * unsigned long sp, void (*fn)(void)); * * SYM_TYPED_FUNC_START() allows C to call this ID-mapped function indirectly * using a physical pointer without triggering a kCFI failure. */ SYM_TYPED_FUNC_START(__pkvm_init_switch_pgd) + mov x6, x3 + /* Turn the MMU off */ pre_disable_mmu_workaround mrs x3, sctlr_el2
@@ -284,14 +295,22 @@ alternative_if ARM64_HAS_CNP alternative_else_nop_endif msr ttbr0_el2, x5 + cbz x1, 1f + phys_to_ttbr x5, x1 +alternative_if ARM64_HAS_CNP + orr x5, x5, #TTBRx_EL1_CnP +alternative_else_nop_endif + msr_s SYS_TTBR1_EL2, x5 +1: + /* Set the new stack pointer */ - mov sp, x1 + mov sp, x2 /* And turn the MMU back on! */ dsb nsh isb set_sctlr_el2 x3 - ret x2 + ret x6 SYM_FUNC_END(__pkvm_init_switch_pgd) .popsection
arch/arm64/kvm/hyp/nvhe/mm.c +24 -0
@@ -19,6 +19,7 @@#include <nvhe/spinlock.h> struct kvm_pgtable pkvm_pgtable; +struct kvm_pgtable pkvm_idmap_pgtable;hyp_spinlock_t pkvm_pgd_lock;struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS ];
@@ -431,6 +432,22 @@ int hyp_create_fixmap(void)return create_fixblock (); } +static int pkvm_map_idmap_text (unsigned long start,unsigned long end) +{ +unsigned long idmap_start, idmap_end; + + idmap_start =__hyp_symbol_pa (__hyp_idmap_text_start); + idmap_start =ALIGN_DOWN (idmap_start,PAGE_SIZE ); + idmap_end =__hyp_symbol_pa (__hyp_idmap_text_end); + idmap_end =ALIGN (idmap_end,PAGE_SIZE ); + +if (WARN_ON (start != idmap_start || end != idmap_end)) +return -EINVAL ; + +return kvm_pgtable_hyp_map (&pkvm_idmap_pgtable, start, + end - start, start,PAGE_HYP_EXEC ); +} +int hyp_create_idmap (u32 hyp_va_bits) {unsigned long start, end;
@@ -441,6 +458,13 @@ int hyp_create_idmap(u32 hyp_va_bits) end =__hyp_symbol_pa (__hyp_idmap_text_end); end =ALIGN (end,PAGE_SIZE ); +if (kvm_hyp_uses_ttbr1 ()) { + __hyp_private_va_base =kvm_hyp_ttbr1_private_start (); + __hyp_vmemmap =kvm_hyp_ttbr1_vmemmap_base (); + +return pkvm_map_idmap_text (start, end); + } +/* * One half of the VA space is reserved to linearly map portions of * memory -- see va_layout.c for more details. The other half of the VA
arch/arm64/kvm/hyp/nvhe/setup.c +31 -4
@@ -109,6 +109,15 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size,if (ret)return ret; +if (kvm_hyp_uses_ttbr1 ()) { +kvm_pgtable_hyp_enable_ttbr1 (&pkvm_pgtable); + + ret =kvm_pgtable_hyp_init (&pkvm_idmap_pgtable, hyp_va_bits, + &hyp_early_alloc_mm_ops); +if (ret) +return ret; + } + ret =hyp_create_idmap (hyp_va_bits);if (ret)return ret;
@@ -169,12 +178,27 @@ static void update_nvhe_init_params(void)for (i =0 ; i < hyp_nr_cpus; i++) { params =per_cpu_ptr (&kvm_init_params, i); - params->pgd_pa =__hyp_linear_pa (pkvm_pgtable.pgd); + params->ttbr0_pgd_pa =kvm_hyp_uses_ttbr1 () ? +__hyp_linear_pa (pkvm_idmap_pgtable.pgd) : +__hyp_linear_pa (pkvm_pgtable.pgd); + params->ttbr1_pgd_pa =kvm_hyp_uses_ttbr1 () ? +__hyp_linear_pa (pkvm_pgtable.pgd) :0 ;dcache_clean_inval_poc ((unsigned long )params, (unsigned long )params +sizeof (*params)); } } +static bool pkvm_init_ttbrs_valid (struct kvm_nvhe_init_params *params) +{ +if (!params->ttbr0_pgd_pa) +return false; + +if (kvm_hyp_uses_ttbr1 ()) +return params->ttbr1_pgd_pa; + +return !params->ttbr1_pgd_pa; +} +static void *hyp_zalloc_hyp_page (void *arg) {return hyp_alloc_pages (&hpool,0 );
@@ -284,8 +308,7 @@ static int fix_hyp_pgtable_refcnt(void) .arg = pkvm_pgtable.mm_ops, }; -return kvm_pgtable_walk (&pkvm_pgtable,0 ,BIT (pkvm_pgtable.ia_bits), - &walker); +return kvm_pgtable_hyp_walk (&pkvm_pgtable, &walker); }void __noreturn __pkvm_init_finalise (void )
@@ -371,8 +394,12 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas/* Jump in the idmap page to switch to the new page-tables */ params =this_cpu_ptr (&kvm_init_params); +if (WARN_ON (!pkvm_init_ttbrs_valid (params))) +return -EINVAL ; + fn = (typeof (fn ))__hyp_symbol_pa (__pkvm_init_switch_pgd); -fn (params->pgd_pa, params->stack_hyp_va, __pkvm_init_finalise); +fn (params->ttbr0_pgd_pa, params->ttbr1_pgd_pa, + params->stack_hyp_va, __pkvm_init_finalise);unreachable (); }
arch/arm64/kvm/hyp/pgtable.c +57 -5
111 lines not on this page · raw message
arch/arm64/kvm/mmu.c +53 -4
99 lines not on this page · raw message
arch/arm64/kvm/pkvm.c +3 -0
@@ -292,6 +292,9 @@ static int __init finalize_pkvm(void) ret =pkvm_drop_host_privileges ();if (ret)pr_err ("Failed to finalize Hyp protection: %d\n" , ret); +else +kvm_info ("pKVM finalized in %s mode\n" , +kvm_hyp_uses_ttbr1 () ?"hVHE" :"nVHE" );return ret; }
arch/arm64/kvm/va_layout.c +27 -5
@@ -33,19 +33,29 @@ static u64 __early_kern_hyp_va(u64 addr)return addr; } +static u64 __early_kern_sym_hyp_va (void *ptr) +{ +if (is_kernel_in_hyp_mode () ||kvm_hyp_init_uses_ttbr1 ()) +return (u64 )ptr; + +return __early_kern_hyp_va ((u64 )lm_alias (ptr)); +} +/* * Store a hyp VA <-> PA offset into a EL2-owned variable. */ static void init_hyp_physvirt_offset (void ) { -u64 kern_va, hyp_va; +u64 kern_va, hyp_va, sym_va;/* Compute the offset from the hyp VA and PA of a random symbol. */ kern_va = (u64 )lm_alias (__hyp_text_start); hyp_va =__early_kern_hyp_va (kern_va); hyp_physvirt_offset = (s64 )__pa (kern_va) - (s64 )hyp_va; - hyp_symbol_physvirt_offset = hyp_physvirt_offset; + sym_va =__early_kern_sym_hyp_va (__hyp_text_start); + hyp_symbol_physvirt_offset = (s64 )__pa_symbol (__hyp_text_start) - + (s64 )sym_va; }/*
@@ -87,6 +97,14 @@ __init void kvm_compute_layout(void)u64 hyp_va_msb;u32 hyp_va_bits =kvm_hyp_va_bits (); +if (kvm_hyp_init_uses_ttbr1 ()) { + tag_lsb =0 ; + tag_val =0 ; + va_mask = ~0ULL ; +init_hyp_physvirt_offset (); +return ; + } +/* Where is my RAM region? */ hyp_va_msb = idmap_addr &BIT (hyp_va_bits -1 ); hyp_va_msb ^=BIT (hyp_va_bits -1 );
@@ -117,6 +135,9 @@ __init void kvm_apply_hyp_relocations(void)int32_t *begin = (int32_t *)__hyp_reloc_begin;int32_t *end = (int32_t *)__hyp_reloc_end; +if (kvm_hyp_init_uses_ttbr1 ()) +return ; +for (rel = begin; rel < end; ++rel) {uintptr_t *ptr, kimg_va;
@@ -187,14 +208,15 @@ void __init kvm_update_va_mask(struct alt_instr *alt,u32 rd, rn, insn, oinsn;/* -* VHE doesn't need any address translation, let's NOP +* VHE/hVHE doesn't need any address translation, let's NOP * everything. * * Alternatively, if the tag is zero (because the layout * dictates it and we don't have any spare bits in the * address), NOP everything after masking the kernel VA. */ -if (cpus_have_cap (ARM64_HAS_VIRT_HOST_EXTN ) || (!tag_val && i >0 )) { +if (cpus_have_cap (ARM64_HAS_VIRT_HOST_EXTN ) || +kvm_hyp_init_uses_ttbr1 () || (!tag_val && i >0 )) { updptr[i] =cpu_to_le32 (aarch64_insn_gen_nop ());continue ; }
@@ -225,7 +247,7 @@ void kvm_patch_vector_branch(struct alt_instr *alt,/* * Compute HYP VA by using the same computation as kern_hyp_va() */ - addr =__early_kern_hyp_va ((u64 )kvm_hyp_kimg_kaddr (__kvm_hyp_vector)); + addr =__early_kern_sym_hyp_va (__kvm_hyp_vector);/* Use PC[10:7] to branch to the same vector in KVM */ addr |= ((u64 )origptr &GENMASK_ULL (10 ,7 ));
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [RFC PATCH 2/5] KVM: arm64: Split hyp mapping APIs by address type
In reply to Aneesh Kumar K.V (Arm)Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but it remains present: Treating EL2 page table allocation failure as fatal by wrapping pkvm_create_linear_mappings_locked() in WARN_ON(). This converts a recoverable -ENOMEM condition into a fatal hypervisor panic. --
--- Patch [2]: [RFC PATCH 2/5] KVM: arm64: Split hyp mapping APIs by address type --- commit 7eb3ee309a427de1a370d3d0ea72c38646079ff1
Author: Aneesh Kumar K.V (Arm) <[email protected]>
KVM: arm64: Split hyp mapping APIs by address type
This commit replaces create_hyp_mappings() and pkvm_create_mappings() with explicit symbol and linear-map interfaces to distinguish between kernel-image symbols and linear-map addresses. It updates all callers to use the appropriate interface for the address type they provide.
arch/arm64/kvm/hyp/nvhe/mem_protect.c int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) · line 1112
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/mem_protect.c +1
-1@@ -1109,7 +1109,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) goto unlock;__hyp_set_page_state_range (phys, size,PKVM_PAGE_OWNED ); -WARN_ON (pkvm_create_mappings_locked (virt, virt + size,PAGE_HYP )); +WARN_ON (pkvm_create_linear_mappings_locked (virt, virt + size,PAGE_HYP ));
[Severity: High]
This is a pre-existing issue, but wrapping pkvm_create_linear_mappings_locked() in a WARN_ON() in __pkvm_host_donate_hyp() converts a legitimate EL2 page table allocation failure into a fatal hypervisor panic. At EL2, WARN_ON() expands to BRK, taking down the host. Because userspace processes can exhaust the EL2 memory pool by triggering operations like host memory donation, could this panic be used for a host denial-of-service attack?
[ ... ]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/mem_protect.c +2
-1@@ -1175,12 +1175,18 @@ int hyp_pin_shared_mem(void *from, void *to) goto unlock;for (cur = start; cur < end; cur +=PAGE_SIZE ) { +void *addr = (void *)cur; + p =hyp_virt_to_page (cur);hyp_page_ref_inc (p); -if (p->refcount ==1 )
arch/arm64/kvm/hyp/nvhe/mem_protect.c int hyp_pin_shared_mem(void *from, void *to) · line 1188
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/mem_protect.c +7
-3-WARN_ON (pkvm_create_mappings_locked ((void *)cur, - (void *)cur +PAGE_SIZE , -PAGE_HYP )); +if (p->refcount ==1 ) { +int err; + + err =pkvm_create_linear_mappings_locked (addr, + addr +PAGE_SIZE , +PAGE_HYP ); +WARN_ON (err);
[Severity: High]
This isn't a bug introduced by this patch, but similar to the above, wrapping pkvm_create_linear_mappings_locked() in WARN_ON() in hyp_pin_shared_mem() treats a legitimate -ENOMEM condition as fatal. If the EL2 buddy allocator fails when allocating new page table pages, should this path instead return the error and gracefully fail the hypercall rather than crashing the hypervisor?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [High] Spectre V3A vector hardening branch synthesis truncates TTBR1 canonical addresses, leading to a…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] Spectre V3A vector hardening branch synthesis truncates TTBR1 canonical addresses, leading to a fatal EL2 Instruction Abort.
- [High] Writes to control-plane system registers (SYS_TTBR1_EL2) are not immediately followed by an isb() instruction.
- [Medium] The error-handling branches following WARN_ON at EL2 are dead code, as WARN_ON expands to a fatal BRK instruction.
- [Medium] Unvalidated software feature override leads to fatal hVHE layout mismatch on unsupported hardware. --
--- Patch [5]: [RFC PATCH 5/5] KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings --- commit c5a5f8a98b4a3d3fc85945a12a0935f8530f85d7
Author: Aneesh Kumar K.V (Arm) <[email protected]>
KVM: arm64: Use TTBR1_EL2 for hVHE runtime mappings
This patch updates hVHE to place runtime mappings in TTBR1_EL2 and reserves TTBR0_EL2 for the identity mapping used during MMU transitions. It introduces support for walking canonical TTBR1 hyp page tables while retaining the existing TTBR0-only layout for nVHE.
Quoted 5 lines
arch/arm64/include/asm/kvm_mmu.h +0
-0
arch/arm64/include/asm/kvm_mmu.h void kvm_compute_layout(void); · line 121
Quoted 9 lines
arch/arm64/include/asm/kvm_mmu.h +6
-0@@ -112,13 +113,28 @@ void kvm_compute_layout(void);u32 kvm_hyp_va_bits (void );void kvm_apply_hyp_relocations (void ); +static __always_inline bool kvm_hyp_init_uses_ttbr1(void) +{ +BUILD_BUG_ON (__is_defined (__KVM_NVHE_HYPERVISOR__)); + +return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_HVHE ); +}
[Severity: Medium]
Will this lead to a fatal hVHE layout mismatch on hardware that does not support VHE?
If a user specifies the arm64.sw_feature_override=hvhe command-line parameter on an unsupported CPU, early initialization will use the TTBR1 layout based on this override. However, the final CPU capabilities check in kvm_hyp_uses_ttbr1() evaluates to false, causing a desynchronization between early-boot and runtime page table layouts.
[ ... ]
Quoted 17 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +2
-1@@ -127,13 +127,22 @@ SYM_CODE_START_LOCAL(___kvm_hyp_init) ldr x1, [x0, #NVHE_INIT_VTCR] msr vtcr_el2, x1 - ldr x1, [x0, #NVHE_INIT_PGD_PA] + ldr x1, [x0, #NVHE_INIT_TTBR0_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP orr x2, x2, #TTBRx_EL1_CnP alternative_else_nop_endif msr ttbr0_el2, x2 + ldr x1, [x0, #NVHE_INIT_TTBR1_PGD_PA]
arch/arm64/kvm/hyp/nvhe/hyp-init.S SYM_CODE_START_LOCAL(___kvm_hyp_init) · line 147
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +8
-0+ cbz x1, 2f + phys_to_ttbr x2, x1 +alternative_if ARM64_HAS_CNP + orr x2, x2, #TTBRx_EL1_CnP +alternative_else_nop_endif + msr_s SYS_TTBR1_EL2, x2 +2: + ldr x0, [x0, #NVHE_INIT_TCR_EL2] msr tcr_el2, x0
[Severity: High]
Does the write to SYS_TTBR1_EL2 require an immediate isb?
Subsystem architectural invariants require every write to a control-plane system register to be followed by an isb as the very next instruction. Since it is delayed here behind register loads and another msr, intervening instructions might execute against an undefined pipeline state.
[ ... ]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +5
-0@@ -284,14 +295,22 @@ alternative_if ARM64_HAS_CNP alternative_else_nop_endif msr ttbr0_el2, x5 + cbz x1, 1f + phys_to_ttbr x5, x1 +alternative_if ARM64_HAS_CNP + orr x5, x5, #TTBRx_EL1_CnP +alternative_else_nop_endif
arch/arm64/kvm/hyp/nvhe/hyp-init.S alternative_if ARM64_HAS_CNP · line 311
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +4
-1+ msr_s SYS_TTBR1_EL2, x5 +1: + /* Set the new stack pointer */ - mov sp, x1 + mov sp, x2 /* And turn the MMU back on! */ dsb nsh isb
[Severity: High]
Could the lack of an immediate isb after msr_s SYS_TTBR1_EL2 cause unpredictable behavior here as well?
The isb is placed after the stack pointer assignment and dsb, which leaves a window where instructions execute against an undefined pipeline state.
[ ... ]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/mm.c +1
-0@@ -431,6 +432,22 @@ int hyp_create_fixmap(void)return create_fixblock (); } +static int pkvm_map_idmap_text (unsigned long start,unsigned long end)
arch/arm64/kvm/hyp/nvhe/mm.c int hyp_create_fixmap(void) · line 445
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/mm.c +10
-0+{ +unsigned long idmap_start, idmap_end; + + idmap_start =__hyp_symbol_pa (__hyp_idmap_text_start); + idmap_start =ALIGN_DOWN (idmap_start,PAGE_SIZE ); + idmap_end =__hyp_symbol_pa (__hyp_idmap_text_end); + idmap_end =ALIGN (idmap_end,PAGE_SIZE ); + +if (WARN_ON (start != idmap_start || end != idmap_end)) +return -EINVAL ;
[Severity: Medium]
Is the error-handling branch following this WARN_ON dead code?
At EL2 in nVHE/pKVM, WARN_ON expands to a fatal BRK instruction which is caught by the hypervisor panic handler. The subsystem does not support warn-and-return semantics, meaning the return -EINVAL will never execute.
[ ... ]
arch/arm64/kvm/hyp/nvhe/setup.c int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas · line 399
Quoted 11 lines
arch/arm64/kvm/hyp/nvhe/setup.c +3
-0@@ -371,8 +394,12 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas/* Jump in the idmap page to switch to the new page-tables */ params =this_cpu_ptr (&kvm_init_params); +if (WARN_ON (!pkvm_init_ttbrs_valid (params))) +return -EINVAL ; +
[Severity: Medium]
Similarly here, is this return -EINVAL unreachable?
The WARN_ON will trigger a hypervisor panic, so the recovery path cannot be executed.
[ ... ]
arch/arm64/kvm/va_layout.c void kvm_patch_vector_branch(struct alt_instr *alt, · line 253
Quoted 13 lines
arch/arm64/kvm/va_layout.c +1
-1@@ -225,7 +247,7 @@ void kvm_patch_vector_branch(struct alt_instr *alt,/* * Compute HYP VA by using the same computation as kern_hyp_va() */ - addr =__early_kern_hyp_va ((u64 )kvm_hyp_kimg_kaddr (__kvm_hyp_vector)); + addr =__early_kern_sym_hyp_va (__kvm_hyp_vector);/* Use PC[10:7] to branch to the same vector in KVM */ addr |= ((u64 )origptr &GENMASK_ULL (10 ,7 ));
[Severity: High]
Does the Spectre V3A vector hardening branch synthesis truncate TTBR1 canonical addresses?
The AARCH64_INSN_MOVEWIDE_ZERO instruction generated below clears bits 16-63 to zero:
insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
(u16)addr,
0,
AARCH64_INSN_VARIANT_64BIT,
AARCH64_INSN_MOVEWIDE_ZERO);
*updptr++ = cpu_to_le32(insn);This branch synthesis relies on a TTBR0 address format. Since a TTBR1 address requires bits 48-63 to be set to 1, the address becomes corrupted. When the hypervisor exits to the host, this truncated address could lead to a fatal EL2 Instruction Abort.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Quoted 37 lines · [email protected] writes:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] Spectre V3A vector hardening branch synthesis truncates TTBR1 canonical addresses, leading to a fatal EL2 Instruction Abort.
- [High] Writes to control-plane system registers (SYS_TTBR1_EL2) are not immediately followed by anisb()instruction.
- [Medium] The error-handling branches followingWARN_ONat EL2 are dead code, asWARN_ONexpands to a fatalBRKinstruction.
- [Medium] Unvalidated software feature override leads to fatal hVHE layout mismatch on unsupported hardware. ----- Patch [5]: [RFC PATCH 5/5] KVM: arm64: Use
TTBR1_EL2for hVHE runtime mappings --- commitc5a5f8a98b4a3d3fc85945a12a0935f8530f85d7
Author: Aneesh Kumar K.V (Arm) <[email protected]>KVM: arm64: Use
TTBR1_EL2for hVHE runtime mappingsThis patch updates hVHE to place runtime mappings in
TTBR1_EL2and reservesTTBR0_EL2for the identity mapping used during MMU transitions. It introduces support for walking canonical TTBR1 hyp page tables while retaining the existing TTBR0-only layout for nVHE.Quoted 5 lines
arch/arm64/include/asm/kvm_mmu.h +0
-0arch/arm64/include/asm/kvm_mmu.h void kvm_compute_layout(void); · line 121
Quoted 9 lines
arch/arm64/include/asm/kvm_mmu.h +6
-0@@ -112,13 +113,28 @@ void kvm_compute_layout(void);u32 kvm_hyp_va_bits (void );void kvm_apply_hyp_relocations (void ); +static __always_inline bool kvm_hyp_init_uses_ttbr1(void) +{ +BUILD_BUG_ON (__is_defined (__KVM_NVHE_HYPERVISOR__)); + +return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_HVHE ); +}[Severity: Medium]
Will this lead to a fatal hVHE layout mismatch on hardware that does not support VHE?
Quoted 6 lines
If a user specifies the arm64.sw_feature_override=hvhe command-line parameter on an unsupported CPU, early initialization will use the TTBR1 layout based on this override. However, the final CPU capabilities check in
kvm_hyp_uses_ttbr1()evaluates to false, causing a desynchronization between early-boot and runtime page table layouts.
But match_options(), which ultimately calls hvhe_filter(), is invoked before we start using kvm_hyp_init_uses_ttbr1()
Quoted 34 lines
[ ... ]
Quoted 17 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +2
-1@@ -127,13 +127,22 @@ SYM_CODE_START_LOCAL(___kvm_hyp_init) ldr x1, [x0, #NVHE_INIT_VTCR] msr vtcr_el2, x1 - ldr x1, [x0, #NVHE_INIT_PGD_PA] + ldr x1, [x0, #NVHE_INIT_TTBR0_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP orr x2, x2, #TTBRx_EL1_CnP alternative_else_nop_endif msr ttbr0_el2, x2 + ldr x1, [x0, #NVHE_INIT_TTBR1_PGD_PA]arch/arm64/kvm/hyp/nvhe/hyp-init.S SYM_CODE_START_LOCAL(___kvm_hyp_init) · line 147
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +8
-0+ cbz x1, 2f + phys_to_ttbr x2, x1 +alternative_if ARM64_HAS_CNP + orr x2, x2, #TTBRx_EL1_CnP +alternative_else_nop_endif + msr_s SYS_TTBR1_EL2, x2 +2: + ldr x0, [x0, #NVHE_INIT_TCR_EL2] msr tcr_el2, x0[Severity: High]
Does the write toSYS_TTBR1_EL2require an immediate isb?
Quoted 5 lines
Subsystem architectural invariants require every write to a control-plane system register to be followed by an isb as the very next instruction. Since it is delayed here behind register loads and another msr, intervening instructions might execute against an undefined pipeline state.
But we can also batch multilple msr?
The full change looks as below. Isn't that safe?
ldr x1, [x0, #NVHE_INIT_TTBR0_PGD_PA] phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP orr x2, x2, #TTBRx_EL1_CnP alternative_else_nop_endif msr ttbr0_el2, x2 ldr x1, [x0, #NVHE_INIT_TTBR1_PGD_PA] cbz x1, 2f phys_to_ttbr x2, x1 alternative_if ARM64_HAS_CNP orr x2, x2, #TTBRx_EL1_CnP alternative_else_nop_endif msr_s SYS_TTBR1_EL2, x2
2:
ldr x0, [x0, #NVHE_INIT_TCR_EL2] msr tcr_el2, x0 isb <<<--- this should be enough /* Invalidate the stale TLBs from Bootloader */ tlbi alle2 tlbi alle1 dsb sy
Quoted 27 lines
[ ... ]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +5
-0@@ -284,14 +295,22 @@ alternative_if ARM64_HAS_CNP alternative_else_nop_endif msr ttbr0_el2, x5 + cbz x1, 1f + phys_to_ttbr x5, x1 +alternative_if ARM64_HAS_CNP + orr x5, x5, #TTBRx_EL1_CnP +alternative_else_nop_endifarch/arm64/kvm/hyp/nvhe/hyp-init.S alternative_if ARM64_HAS_CNP · line 311
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/hyp-init.S +4
-1+ msr_s SYS_TTBR1_EL2, x5 +1: + /* Set the new stack pointer */ - mov sp, x1 + mov sp, x2 /* And turn the MMU back on! */ dsb nsh isb[Severity: High]
Could the lack of an immediate isb aftermsr_sSYS_TTBR1_EL2cause unpredictable behavior here as well?
Quoted 2 lines
The isb is placed after the stack pointer assignment and dsb, which leaves a window where instructions execute against an undefined pipeline state.
isn't isb before turning the mmu back on sufficient?
isb set_sctlr_el2 x3
Related full change looks as below:
cbz x1, 1f phys_to_ttbr x5, x1 alternative_if ARM64_HAS_CNP orr x5, x5, #TTBRx_EL1_CnP alternative_else_nop_endif msr_s SYS_TTBR1_EL2, x5
1:
/* Set the new stack pointer */ mov sp, x2 /* And turn the MMU back on! */ dsb nsh isb set_sctlr_el2 x3 ret x6
Quoted 80 lines
[ ... ]
Quoted 9 lines
arch/arm64/kvm/hyp/nvhe/mm.c +1
-0@@ -431,6 +432,22 @@ int hyp_create_fixmap(void)return create_fixblock (); } +static int pkvm_map_idmap_text (unsigned long start,unsigned long end)arch/arm64/kvm/hyp/nvhe/mm.c int hyp_create_fixmap(void) · line 445
Quoted 10 lines
arch/arm64/kvm/hyp/nvhe/mm.c +10
-0+{ +unsigned long idmap_start, idmap_end; + + idmap_start =__hyp_symbol_pa (__hyp_idmap_text_start); + idmap_start =ALIGN_DOWN (idmap_start,PAGE_SIZE ); + idmap_end =__hyp_symbol_pa (__hyp_idmap_text_end); + idmap_end =ALIGN (idmap_end,PAGE_SIZE ); + +if (WARN_ON (start != idmap_start || end != idmap_end)) +return -EINVAL ;[Severity: Medium]
Is the error-handling branch following thisWARN_ONdead code?At EL2 in nVHE/pKVM,
WARN_ONexpands to a fatal BRK instruction which is caught by the hypervisor panic handler. The subsystem does not support warn-and-return semantics, meaning the return-EINVALwill never execute.[ ... ]
arch/arm64/kvm/hyp/nvhe/setup.c int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas · line 399
Quoted 11 lines
arch/arm64/kvm/hyp/nvhe/setup.c +3
-0@@ -371,8 +394,12 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long *per_cpu_bas/* Jump in the idmap page to switch to the new page-tables */ params =this_cpu_ptr (&kvm_init_params); +if (WARN_ON (!pkvm_init_ttbrs_valid (params))) +return -EINVAL ; +[Severity: Medium]
Similarly here, is this return-EINVALunreachable?The
WARN_ONwill trigger a hypervisor panic, so the recovery path cannot be executed.[ ... ]
arch/arm64/kvm/va_layout.c void kvm_patch_vector_branch(struct alt_instr *alt, · line 253
Quoted 13 lines
arch/arm64/kvm/va_layout.c +1
-1@@ -225,7 +247,7 @@ void kvm_patch_vector_branch(struct alt_instr *alt,/* * Compute HYP VA by using the same computation as kern_hyp_va() */ - addr =__early_kern_hyp_va ((u64 )kvm_hyp_kimg_kaddr (__kvm_hyp_vector)); + addr =__early_kern_sym_hyp_va (__kvm_hyp_vector);/* Use PC[10:7] to branch to the same vector in KVM */ addr |= ((u64 )origptr &GENMASK_ULL (10 ,7 ));[Severity: High]
Does the Spectre V3A vector hardening branch synthesis truncate TTBR1 canonical addresses?The
AARCH64_INSN_MOVEWIDE_ZEROinstruction generated below clears bits 16-63 to zero:insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0, (u16)addr, 0, AARCH64_INSN_VARIANT_64BIT, AARCH64_INSN_MOVEWIDE_ZERO); *updptr++ = cpu_to_le32(insn);
Quoted 5 lines
This branch synthesis relies on a TTBR0 address format. Since a TTBR1 address requires bits 48-63 to be set to 1, the address becomes corrupted. When the hypervisor exits to the host, this truncated address could lead to a fatal EL2 Instruction Abort.
I need to look at this more closely. It does seem like this could be a problem.
-aneesh