KVM: arm64: Fix host access to the EL2 stacks
Subsystems (3)
Fuad Tabba
reviewercover KVM: arm64: Fix host access to the EL2 stacks
Hi folks,
The per-CPU EL2 stacks are initialised as host-owned after pKVM init. fix_host_ownership() walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.
Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.
Patch 2 is a fix in its own right, but to a different bug. fix_host_ownership_walker() matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.
Based on Linux 7.3-rc2 (df2908090cda3).
Cheers,
/fuad
Fuad Tabba (4):
KVM: arm64: Transfer the hyp stack pages out of the host stage-2
KVM: arm64: Match hyp text by physical address in fix_host_ownership()
KVM: arm64: Move the private VA allocation cursor to __io_map_next
KVM: arm64: Check every private mapping is hyp-owned at pKVM init
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/include/nvhe/mm.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 20 +++++ arch/arm64/kvm/hyp/nvhe/mm.c | 79 ++++++++++++++++--- arch/arm64/kvm/hyp/nvhe/setup.c | 16 +++- 5 files changed, 108 insertions(+), 10 deletions(-)
base-commit: df2908090cda368b01ff43709f51890076c56157
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH 1/4 KVM: arm64: Transfer the hyp stack pages out of the host stage-2
fix_host_ownership() walks only the linear-map alias of each memblock region, and the per-CPU hyp stack, mapped in the private VA range for its guard page, has none.
Walk each stack's VA range with the same walker.
--- arch/arm64/kvm/hyp/nvhe/setup.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/kvm/hyp/nvhe/setup.c +10 -0
@@ -269,6 +269,16 @@ static int fix_host_ownership(void)return ret; } +/* The stacks sit in the private VA range, not the linear map. */ +for (i =0 ; i < hyp_nr_cpus; i++) { +struct kvm_nvhe_init_params *params =per_cpu_ptr (&kvm_init_params, i); +u64 start = params->stack_hyp_va -NVHE_STACK_SIZE ; + + ret =kvm_pgtable_walk (&pkvm_pgtable, start,NVHE_STACK_SIZE , &walker); +if (ret) +return ret; + } +return 0 ; }
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH 2/4 KVM: arm64: Match hyp text by physical address in fix_host_ownership()
In reply to Fuad TabbaOn a non-hVHE host, fix_host_ownership_walker()'s test for PAGE_HYP_EXEC never matches: KVM_PGTABLE_PROT_UX is cleared at map time and only PX is reported on read-back. Hyp text is therefore donated rather than left read-only in the host stage-2, and the instruction dump in nvhe_hyp_panic_handler() reads a page the host has no access to.
Match the text by physical address instead, in a helper a later patch reuses. A test on the permissions would leave any other executable mapping host-readable too.
--- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 8 ++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- 3 files changed, 10 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +1 -0
@@ -52,6 +52,7 @@ int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pages, bool mkold, structint __pkvm_host_mkyoung_guest (u64 gfn,struct pkvm_hyp_vcpu *vcpu);bool addr_is_memory (phys_addr_t phys); +bool addr_is_hyp_text (phys_addr_t phys);int host_stage2_idmap_locked (phys_addr_t addr,u64 size,enum kvm_pgtable_prot prot);int host_stage2_set_owner_locked (phys_addr_t addr,u64 size,u8 owner_id);int kvm_host_prepare_stage2 (void *pgt_pool_base);
arch/arm64/kvm/hyp/nvhe/mem_protect.c +8 -0
@@ -450,6 +450,14 @@ bool addr_is_memory(phys_addr_t phys)return !!find_mem_range (phys, &range); } +bool addr_is_hyp_text (phys_addr_t phys) +{ +phys_addr_t start =ALIGN_DOWN (__hyp_pa (__hyp_text_start),PAGE_SIZE ); +phys_addr_t end =PAGE_ALIGN (__hyp_pa (__hyp_text_end)); + +return phys >= start && phys < end; +} +static bool is_in_mem_range (u64 addr,struct kvm_mem_range *range) {return range->start <= addr && addr < range->end;
arch/arm64/kvm/hyp/nvhe/setup.c +1 -1
@@ -217,7 +217,7 @@ static int fix_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,case PKVM_PAGE_OWNED :set_hyp_state (page,PKVM_PAGE_OWNED );/* hyp text is RO in the host stage-2 to be inspected on panic. */ -if (prot ==PAGE_HYP_EXEC ) { +if (addr_is_hyp_text (phys)) {set_host_state (page,PKVM_NOPAGE );return host_stage2_idmap_locked (phys,PAGE_SIZE ,KVM_PGTABLE_PROT_R ); }else {
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH 3/4 KVM: arm64: Move the private VA allocation cursor to __io_map_next
In reply to Fuad Tabba__io_map_base is the start of the private VA range only until the first allocation from it, after which it is the allocation cursor. Keep it as the start and move the cursor to __io_map_next, for the walk of the range the next patch adds.
No functional change intended.
--- arch/arm64/kvm/hyp/nvhe/mm.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
arch/arm64/kvm/hyp/nvhe/mm.c +11 -9
@@ -25,6 +25,7 @@ struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];unsigned int hyp_memblock_nr;static u64 __io_map_base; +static u64 __io_map_next;struct hyp_fixmap_slot {u64 addr;
@@ -50,7 +51,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)hyp_assert_lock_held (&pkvm_pgd_lock); -if (!start || start < __io_map_base) +if (!start || start < __io_map_next)return -EINVAL ;/* The allocated size is always a multiple of PAGE_SIZE */
@@ -60,7 +61,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)if (cur > __hyp_vmemmap)return -ENOMEM ; - __io_map_base = cur; + __io_map_next = cur;return 0 ; }
@@ -70,7 +71,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)* @size: The size of the VA range to reserve. * @haddr: The hypervisor virtual start address of the allocation. * -* The private virtual address (VA) range is allocated above __io_map_base +* The private virtual address (VA) range is allocated above __io_map_next * and aligned based on the order of @size. * * Return: 0 on success or negative error code on failure.
@@ -81,7 +82,7 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr)int ret;hyp_spin_lock (&pkvm_pgd_lock); - addr = __io_map_base; + addr = __io_map_next; ret =__pkvm_alloc_private_va_range (addr, size);hyp_spin_unlock (&pkvm_pgd_lock);
@@ -341,7 +342,7 @@ static int create_fixblock(void)return -EINVAL ;hyp_spin_lock (&pkvm_pgd_lock); - addr =ALIGN (__io_map_base,PMD_SIZE ); + addr =ALIGN (__io_map_next,PMD_SIZE ); ret =__pkvm_alloc_private_va_range (addr,PMD_SIZE );if (ret) goto unlock;
@@ -426,6 +427,7 @@ int hyp_create_idmap(u32 hyp_va_bits)*/ __io_map_base = start &BIT (hyp_va_bits -2 ); __io_map_base ^=BIT (hyp_va_bits -2 ); + __io_map_next = __io_map_base; __hyp_vmemmap = __io_map_base |BIT (hyp_va_bits -3 );return __pkvm_create_mappings (start, end - start, start,PAGE_HYP_EXEC );
@@ -433,19 +435,19 @@ int hyp_create_idmap(u32 hyp_va_bits)int pkvm_create_stack (phys_addr_t phys,unsigned long *haddr) { -unsigned long addr, prev_base; +unsigned long addr, prev_next;size_t size;int ret;hyp_spin_lock (&pkvm_pgd_lock); - prev_base = __io_map_base; + prev_next = __io_map_next;/* * Efficient stack verification using the NVHE_STACK_SHIFT bit implies * an alignment of our allocation on the order of the size. */ size =NVHE_STACK_SIZE *2 ; - addr =ALIGN (__io_map_base, size); + addr =ALIGN (__io_map_next, size); ret =__pkvm_alloc_private_va_range (addr, size);if (!ret) {
@@ -461,7 +463,7 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr) ret =kvm_pgtable_hyp_map (&pkvm_pgtable, addr +NVHE_STACK_SIZE ,NVHE_STACK_SIZE , phys,PAGE_HYP );if (ret) - __io_map_base = prev_base; + __io_map_next = prev_next; }hyp_spin_unlock (&pkvm_pgd_lock);
Signature · 1 line
2.39.5
Fuad Tabba
reviewerPATCH 4/4 KVM: arm64: Check every private mapping is hyp-owned at pKVM init
In reply to Fuad Tabbafix_host_ownership() transfers only what it walks, so a hyp mapping outside the linear map is not manipulated by the walk.
Walk the quarter of the VA space holding the private range and the vmemmap once the transfer is done, and fail init unless every valid leaf is hyp-owned: in the vmemmap when the page is memory, and in the host stage-2, where hyp text may instead be mapped without write access. A leaf that is not memory has no vmemmap entry and is checked against the host stage-2 alone. Hyp text is matched by physical address, since the only executable mapping in the range is the Spectre-v3a vectors, whose VA is a private allocation, and an executable mapping of anything else must not be host-readable. The vmemmap can be block-mapped, so the walker checks each page of a leaf.
--- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/include/nvhe/mm.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 12 ++++ arch/arm64/kvm/hyp/nvhe/mm.c | 59 +++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 4 ++ 5 files changed, 77 insertions(+)
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +1 -0
@@ -55,6 +55,7 @@ bool addr_is_memory(phys_addr_t phys);bool addr_is_hyp_text (phys_addr_t phys);int host_stage2_idmap_locked (phys_addr_t addr,u64 size,enum kvm_pgtable_prot prot);int host_stage2_set_owner_locked (phys_addr_t addr,u64 size,u8 owner_id); +bool host_stage2_pte_is_hyp_owned (kvm_pte_t pte);int kvm_host_prepare_stage2 (void *pgt_pool_base);int kvm_guest_prepare_stage2 (struct pkvm_hyp_vm *vm,void *pgd);void kvm_guest_destroy_stage2 (struct pkvm_hyp_vm *vm);
arch/arm64/kvm/hyp/include/nvhe/mm.h +1 -0
@@ -29,6 +29,7 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,enum kvm_pgtable_prot prot, unsigned long *haddr);int pkvm_create_stack (phys_addr_t phys,unsigned long *haddr); +int pkvm_check_host_ownership (void );int pkvm_alloc_private_va_range (size_t size,unsigned long *haddr);#endif /* __KVM_HYP_MM_H */
arch/arm64/kvm/hyp/nvhe/mem_protect.c +12 -0
@@ -641,6 +641,18 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)return ret; } +bool host_stage2_pte_is_hyp_owned (kvm_pte_t pte) +{ +if (kvm_pte_valid (pte)) +return false; + +if (FIELD_GET (KVM_INVALID_PTE_TYPE_MASK , pte) != +KVM_HOST_INVALID_PTE_TYPE_DONATION ) +return false; + +return FIELD_GET (KVM_HOST_DONATION_PTE_OWNER_MASK , pte) ==PKVM_ID_HYP ; +} +#define KVM_HOST_PTE_OWNER_GUEST_HANDLE_MASK GENMASK(15, 0)/* We need 40 bits for the GFN to cover a 52-bit IPA with 4k pages and LPA2 */ #define KVM_HOST_PTE_OWNER_GUEST_GFN_MASK GENMASK(55, 16)
arch/arm64/kvm/hyp/nvhe/mm.c +59 -0
65 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/setup.c +4 -0
@@ -334,6 +334,10 @@ void __noreturn __pkvm_init_finalise(void)if (ret) goto out; + ret =pkvm_check_host_ownership (); +if (ret) + goto out; + ret =hyp_ffa_init (ffa_proxy_pages);if (ret) goto out;
Signature · 1 line
2.39.5
Quoted 14 lines · Fuad Tabba
Hi folks,
The per-CPU EL2 stacks are initialised as host-owned after pKVM init.
fix_host_ownership()walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.
Quoted 9 lines
Patch 2 is a fix in its own right, but to a different bug.
fix_host_ownership_walker()matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 42 lines · Fuad Tabba
Hi folks,
The per-CPU EL2 stacks are initialised as host-owned after pKVM init.
fix_host_ownership()walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.
Patch 2 is a fix in its own right, but to a different bug.
fix_host_ownership_walker()matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.Based on Linux 7.3-rc2 (
df2908090cda3).Cheers,
/fuadFuad Tabba (4):
KVM: arm64: Transfer the hyp stack pages out of the host stage-2
KVM: arm64: Match hyp text by physical address infix_host_ownership()
KVM: arm64: Move the private VA allocation cursor to__io_map_next
KVM: arm64: Check every private mapping is hyp-owned at pKVM initarch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/include/nvhe/mm.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 20 +++++ arch/arm64/kvm/hyp/nvhe/mm.c | 79 ++++++++++++++++--- arch/arm64/kvm/hyp/nvhe/setup.c | 16 +++- 5 files changed, 108 insertions(+), 10 deletions(-)
Quoted 4 lines
base-commit:
df2908090cda368b01ff43709f51890076c56157Signature · 1 line
2.39.5
Signature · 1 line
Vincent
Quoted 5 lines · Fuad Tabba
The per-CPU EL2 stacks are initialised as host-owned after pKVM init.
fix_host_ownership()walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.
Quoted 7 lines
Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.
[...]
Applied to fixes, thanks!
[1/4] KVM: arm64: Transfer the hyp stack pages out of the host stage-2
https://git.kernel.org/kvmarm/kvmarm/c/3a8c562892b9 [2/4] KVM: arm64: Match hyp text by physical address in fix_host_ownership() https://git.kernel.org/kvmarm/kvmarm/c/5a8b505ede13 [3/4] KVM: arm64: Move the private VA allocation cursor to __io_map_next https://git.kernel.org/kvmarm/kvmarm/c/224584140114 [4/4] KVM: arm64: Check every private mapping is hyp-owned at pKVM init https://git.kernel.org/kvmarm/kvmarm/c/cfe80c3837f9
--
Best,
Oliver