Skip to contentSkip to conversation

KVM: arm64: Fix host access to the EL2 stacks

PATCH Applied Fuad Tabba 4 participants 8 messages Sep 8 – Sep 15

Subsystems (3)
lore
Share & tools
lore

Fuad Tabba

author reviewer
#0 lore raw

cover KVM: arm64: Fix host access to the EL2 stacks

Hi folks,

The per-CPU EL2 stacks are initialised as host-owned after pKVM init. fix_host_ownership() walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.

Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.

Patch 2 is a fix in its own right, but to a different bug. fix_host_ownership_walker() matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.

Based on Linux 7.3-rc2 (df2908090cda3).

Cheers,
/fuad

Fuad Tabba (4):
KVM: arm64: Transfer the hyp stack pages out of the host stage-2
KVM: arm64: Match hyp text by physical address in fix_host_ownership()
KVM: arm64: Move the private VA allocation cursor to __io_map_next
KVM: arm64: Check every private mapping is hyp-owned at pKVM init

 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  2 +
 arch/arm64/kvm/hyp/include/nvhe/mm.h          |  1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 20 +++++
 arch/arm64/kvm/hyp/nvhe/mm.c                  | 79 ++++++++++++++++---
 arch/arm64/kvm/hyp/nvhe/setup.c               | 16 +++-
 5 files changed, 108 insertions(+), 10 deletions(-)

base-commit: df2908090cda368b01ff43709f51890076c56157

Signature · 1 line
2.39.5
author reviewer
#1 lore raw

PATCH 1/4 KVM: arm64: Transfer the hyp stack pages out of the host stage-2

fix_host_ownership() walks only the linear-map alias of each memblock region, and the per-CPU hyp stack, mapped in the private VA range for its guard page, has none.

Walk each stack's VA range with the same walker.

Fixes 1a919b17ef012 ("KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack")
---
 arch/arm64/kvm/hyp/nvhe/setup.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/kvm/hyp/nvhe/setup.c +10 -0
index 75b00c3233102..362f2891cb32e 100644
@@ -269,6 +269,16 @@ static int fix_host_ownership(void)
 			return ret;
 	}
 
+	/* The stacks sit in the private VA range, not the linear map. */
+	for (i = 0; i < hyp_nr_cpus; i++) {
+		struct kvm_nvhe_init_params *params = per_cpu_ptr(&kvm_init_params, i);
+		u64 start = params->stack_hyp_va - NVHE_STACK_SIZE;
+
+		ret = kvm_pgtable_walk(&pkvm_pgtable, start, NVHE_STACK_SIZE, &walker);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
Signature · 1 line
2.39.5
author reviewer
#2 lore raw

PATCH 2/4 KVM: arm64: Match hyp text by physical address in fix_host_ownership()

In reply to Fuad Tabba

On a non-hVHE host, fix_host_ownership_walker()'s test for PAGE_HYP_EXEC never matches: KVM_PGTABLE_PROT_UX is cleared at map time and only PX is reported on read-back. Hyp text is therefore donated rather than left read-only in the host stage-2, and the instruction dump in nvhe_hyp_panic_handler() reads a page the host has no access to.

Match the text by physical address instead, in a helper a later patch reuses. A test on the permissions would leave any other executable mapping host-readable too.

Fixes 80cbfd7174f31 ("KVM: arm64: Honor UX/PX attributes for EL2 S1 mappings")
---
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 8 ++++++++
 arch/arm64/kvm/hyp/nvhe/setup.c               | 2 +-
 3 files changed, 10 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +1 -0
index 29935c7da1dec..cab27f7bd423a 100644
@@ -52,6 +52,7 @@ int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pages, bool mkold, struct
 int __pkvm_host_mkyoung_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu);
 
 bool addr_is_memory(phys_addr_t phys);
+bool addr_is_hyp_text(phys_addr_t phys);
 int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot prot);
 int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id);
 int kvm_host_prepare_stage2(void *pgt_pool_base);
arch/arm64/kvm/hyp/nvhe/mem_protect.c +8 -0
index 39aa8911f62c1..d026f446bd8ef 100644
@@ -450,6 +450,14 @@ bool addr_is_memory(phys_addr_t phys)
 	return !!find_mem_range(phys, &range);
 }
 
+bool addr_is_hyp_text(phys_addr_t phys)
+{
+	phys_addr_t start = ALIGN_DOWN(__hyp_pa(__hyp_text_start), PAGE_SIZE);
+	phys_addr_t end = PAGE_ALIGN(__hyp_pa(__hyp_text_end));
+
+	return phys >= start && phys < end;
+}
+
 static bool is_in_mem_range(u64 addr, struct kvm_mem_range *range)
 {
 	return range->start <= addr && addr < range->end;
arch/arm64/kvm/hyp/nvhe/setup.c +1 -1
index 362f2891cb32e..bb667cd7080b4 100644
@@ -217,7 +217,7 @@ static int fix_host_ownership_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	case PKVM_PAGE_OWNED:
 		set_hyp_state(page, PKVM_PAGE_OWNED);
 		/* hyp text is RO in the host stage-2 to be inspected on panic. */
-		if (prot == PAGE_HYP_EXEC) {
+		if (addr_is_hyp_text(phys)) {
 			set_host_state(page, PKVM_NOPAGE);
 			return host_stage2_idmap_locked(phys, PAGE_SIZE, KVM_PGTABLE_PROT_R);
 		} else {
Signature · 1 line
2.39.5
author reviewer
#3 lore raw

PATCH 3/4 KVM: arm64: Move the private VA allocation cursor to __io_map_next

In reply to Fuad Tabba

__io_map_base is the start of the private VA range only until the first allocation from it, after which it is the allocation cursor. Keep it as the start and move the cursor to __io_map_next, for the walk of the range the next patch adds.

No functional change intended.

---
 arch/arm64/kvm/hyp/nvhe/mm.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
arch/arm64/kvm/hyp/nvhe/mm.c +11 -9
index 3b0bee496bffb..422ee57be9560 100644
@@ -25,6 +25,7 @@ struct memblock_region hyp_memory[HYP_MEMBLOCK_REGIONS];
 unsigned int hyp_memblock_nr;
 
 static u64 __io_map_base;
+static u64 __io_map_next;
 
 struct hyp_fixmap_slot {
 	u64 addr;
@@ -50,7 +51,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
 
 	hyp_assert_lock_held(&pkvm_pgd_lock);
 
-	if (!start || start < __io_map_base)
+	if (!start || start < __io_map_next)
 		return -EINVAL;
 
 	/* The allocated size is always a multiple of PAGE_SIZE */
@@ -60,7 +61,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
 	if (cur > __hyp_vmemmap)
 		return -ENOMEM;
 
-	__io_map_base = cur;
+	__io_map_next = cur;
 
 	return 0;
 }
@@ -70,7 +71,7 @@ static int __pkvm_alloc_private_va_range(unsigned long start, size_t size)
  * @size:	The size of the VA range to reserve.
  * @haddr:	The hypervisor virtual start address of the allocation.
  *
- * The private virtual address (VA) range is allocated above __io_map_base
+ * The private virtual address (VA) range is allocated above __io_map_next
  * and aligned based on the order of @size.
  *
  * Return: 0 on success or negative error code on failure.
@@ -81,7 +82,7 @@ int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr)
 	int ret;
 
 	hyp_spin_lock(&pkvm_pgd_lock);
-	addr = __io_map_base;
+	addr = __io_map_next;
 	ret = __pkvm_alloc_private_va_range(addr, size);
 	hyp_spin_unlock(&pkvm_pgd_lock);
 
@@ -341,7 +342,7 @@ static int create_fixblock(void)
 		return -EINVAL;
 
 	hyp_spin_lock(&pkvm_pgd_lock);
-	addr = ALIGN(__io_map_base, PMD_SIZE);
+	addr = ALIGN(__io_map_next, PMD_SIZE);
 	ret = __pkvm_alloc_private_va_range(addr, PMD_SIZE);
 	if (ret)
 		goto unlock;
@@ -426,6 +427,7 @@ int hyp_create_idmap(u32 hyp_va_bits)
 	 */
 	__io_map_base = start & BIT(hyp_va_bits - 2);
 	__io_map_base ^= BIT(hyp_va_bits - 2);
+	__io_map_next = __io_map_base;
 	__hyp_vmemmap = __io_map_base | BIT(hyp_va_bits - 3);
 
 	return __pkvm_create_mappings(start, end - start, start, PAGE_HYP_EXEC);
@@ -433,19 +435,19 @@ int hyp_create_idmap(u32 hyp_va_bits)
 
 int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
 {
-	unsigned long addr, prev_base;
+	unsigned long addr, prev_next;
 	size_t size;
 	int ret;
 
 	hyp_spin_lock(&pkvm_pgd_lock);
 
-	prev_base = __io_map_base;
+	prev_next = __io_map_next;
 	/*
 	 * Efficient stack verification using the NVHE_STACK_SHIFT bit implies
 	 * an alignment of our allocation on the order of the size.
 	 */
 	size = NVHE_STACK_SIZE * 2;
-	addr = ALIGN(__io_map_base, size);
+	addr = ALIGN(__io_map_next, size);
 
 	ret = __pkvm_alloc_private_va_range(addr, size);
 	if (!ret) {
@@ -461,7 +463,7 @@ int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr)
 		ret = kvm_pgtable_hyp_map(&pkvm_pgtable, addr + NVHE_STACK_SIZE,
 					  NVHE_STACK_SIZE, phys, PAGE_HYP);
 		if (ret)
-			__io_map_base = prev_base;
+			__io_map_next = prev_next;
 	}
 	hyp_spin_unlock(&pkvm_pgd_lock);
 
Signature · 1 line
2.39.5
author reviewer
#4 lore raw

PATCH 4/4 KVM: arm64: Check every private mapping is hyp-owned at pKVM init

In reply to Fuad Tabba

fix_host_ownership() transfers only what it walks, so a hyp mapping outside the linear map is not manipulated by the walk.

Walk the quarter of the VA space holding the private range and the vmemmap once the transfer is done, and fail init unless every valid leaf is hyp-owned: in the vmemmap when the page is memory, and in the host stage-2, where hyp text may instead be mapped without write access. A leaf that is not memory has no vmemmap entry and is checked against the host stage-2 alone. Hyp text is matched by physical address, since the only executable mapping in the range is the Spectre-v3a vectors, whose VA is a private allocation, and an executable mapping of anything else must not be host-readable. The vmemmap can be block-mapped, so the walker checks each page of a leaf.

---
 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  1 +
 arch/arm64/kvm/hyp/include/nvhe/mm.h          |  1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 12 ++++
 arch/arm64/kvm/hyp/nvhe/mm.c                  | 59 +++++++++++++++++++
 arch/arm64/kvm/hyp/nvhe/setup.c               |  4 ++
 5 files changed, 77 insertions(+)
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +1 -0
index cab27f7bd423a..ec85a95471207 100644
@@ -55,6 +55,7 @@ bool addr_is_memory(phys_addr_t phys);
 bool addr_is_hyp_text(phys_addr_t phys);
 int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_prot prot);
 int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id);
+bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte);
 int kvm_host_prepare_stage2(void *pgt_pool_base);
 int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd);
 void kvm_guest_destroy_stage2(struct pkvm_hyp_vm *vm);
arch/arm64/kvm/hyp/include/nvhe/mm.h +1 -0
index 6e83ce35c2f2e..31cae95ddb716 100644
@@ -29,6 +29,7 @@ int __pkvm_create_private_mapping(phys_addr_t phys, size_t size,
 				  enum kvm_pgtable_prot prot,
 				  unsigned long *haddr);
 int pkvm_create_stack(phys_addr_t phys, unsigned long *haddr);
+int pkvm_check_host_ownership(void);
 int pkvm_alloc_private_va_range(size_t size, unsigned long *haddr);
 
 #endif /* __KVM_HYP_MM_H */
arch/arm64/kvm/hyp/nvhe/mem_protect.c +12 -0
index d026f446bd8ef..a6a47c1e058b3 100644
@@ -641,6 +641,18 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)
 	return ret;
 }
 
+bool host_stage2_pte_is_hyp_owned(kvm_pte_t pte)
+{
+	if (kvm_pte_valid(pte))
+		return false;
+
+	if (FIELD_GET(KVM_INVALID_PTE_TYPE_MASK, pte) !=
+	    KVM_HOST_INVALID_PTE_TYPE_DONATION)
+		return false;
+
+	return FIELD_GET(KVM_HOST_DONATION_PTE_OWNER_MASK, pte) == PKVM_ID_HYP;
+}
+
 #define KVM_HOST_PTE_OWNER_GUEST_HANDLE_MASK	GENMASK(15, 0)
 /* We need 40 bits for the GFN to cover a 52-bit IPA with 4k pages and LPA2 */
 #define KVM_HOST_PTE_OWNER_GUEST_GFN_MASK	GENMASK(55, 16)
arch/arm64/kvm/hyp/nvhe/mm.c +59 -0

65 lines not on this page · raw message

arch/arm64/kvm/hyp/nvhe/setup.c +4 -0
index bb667cd7080b4..45ac5f2ba4f7a 100644
@@ -334,6 +334,10 @@ void __noreturn __pkvm_init_finalise(void)
 	if (ret)
 		goto out;
 
+	ret = pkvm_check_host_ownership();
+	if (ret)
+		goto out;
+
 	ret = hyp_ffa_init(ffa_proxy_pages);
 	if (ret)
 		goto out;
Signature · 1 line
2.39.5
maintainer Reviewed-by on the v1 series
#5 lore raw
In reply to Fuad Tabba
Quoted 14 lines · Fuad Tabba

Hi folks,

The per-CPU EL2 stacks are initialised as host-owned after pKVM init. fix_host_ownership() walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.

Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.

Quoted 9 lines

Patch 2 is a fix in its own right, but to a different bug. fix_host_ownership_walker() matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Reviewed-by Tested-by on the v1 series
#6 lore raw
In reply to Fuad Tabba
Quoted 42 lines · Fuad Tabba

Hi folks,

The per-CPU EL2 stacks are initialised as host-owned after pKVM init. fix_host_ownership() walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.

Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.

Patch 2 is a fix in its own right, but to a different bug. fix_host_ownership_walker() matches hyp text on its stage-1 permissions, and that test hasn't matched on a non-hVHE host since 7.0, so hyp text is donated there rather than left read-only. Nothing crashes, but the instruction dump on hyp panic reads a page the host can't access, so the Code: line comes out as question marks. It matches by physical address instead, which is what patch 4 needs too. Patch 3 splits the start of the range from the allocation cursor, with no functional change, so patch 4 knows where the range begins.

Based on Linux 7.3-rc2 (df2908090cda3).

Cheers,
/fuad

Fuad Tabba (4):
KVM: arm64: Transfer the hyp stack pages out of the host stage-2
KVM: arm64: Match hyp text by physical address in fix_host_ownership()
KVM: arm64: Move the private VA allocation cursor to __io_map_next
KVM: arm64: Check every private mapping is hyp-owned at pKVM init

 arch/arm64/kvm/hyp/include/nvhe/mem_protect.h |  2 +
 arch/arm64/kvm/hyp/include/nvhe/mm.h          |  1 +
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         | 20 +++++
 arch/arm64/kvm/hyp/nvhe/mm.c                  | 79 ++++++++++++++++---
 arch/arm64/kvm/hyp/nvhe/setup.c               | 16 +++-
 5 files changed, 108 insertions(+), 10 deletions(-)
Quoted 4 lines

base-commit: df2908090cda368b01ff43709f51890076c56157

Signature · 1 line
2.39.5
Signature · 1 line
Vincent
maintainer Applied
#7 lore raw
In reply to Fuad Tabba
Quoted 5 lines · Fuad Tabba

The per-CPU EL2 stacks are initialised as host-owned after pKVM init. fix_host_ownership() walks only the linear-map alias of each memblock region, and the stacks, mapped in the private VA range for their guard pages, have none.

Quoted 7 lines

Patch 1 is the fix, and it needs none of the other three. Patch 4 adds the check that would have caught it and would catch similar instances in the future: once the transfer is done, walk the private VA range and fail init unless every valid leaf is hyp-owned, or is hyp text mapped without write access.

[...]

Applied to fixes, thanks!

[1/4] KVM: arm64: Transfer the hyp stack pages out of the host stage-2

      https://git.kernel.org/kvmarm/kvmarm/c/3a8c562892b9
[2/4] KVM: arm64: Match hyp text by physical address in fix_host_ownership()
      https://git.kernel.org/kvmarm/kvmarm/c/5a8b505ede13
[3/4] KVM: arm64: Move the private VA allocation cursor to __io_map_next
      https://git.kernel.org/kvmarm/kvmarm/c/224584140114
[4/4] KVM: arm64: Check every private mapping is hyp-owned at pKVM init
      https://git.kernel.org/kvmarm/kvmarm/c/cfe80c3837f9

--
Best,
Oliver