Skip to contentSkip to conversation

ARM: KVM: Revamping the HYP init code for fun and profit

PATCH New Marc Zyngier 4 participants 16 messages Apr 12, 2013 – Apr 27, 2013 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

v4 cover ARM: KVM: Revamping the HYP init code for fun and profit

Over the past few weeks, I've gradually realized how broken our HYP idmap code is. Badly broken.

The main problem is about supporting CPU hotplug. Imagine a CPU being initialized normally, running VMs, and then being powered down. So far, so good. Now mentally bring it back online. The CPU will come back via the secondary CPU boot path, and then what? We cannot use it anymore, because we need an idmap which is long gone, and because our page tables are now live, containing the world-switch code, VM structures, and other bits and pieces.

Another fun issue is that we don't have any TLB invalidation in the HYP init code. And guess what? we cannot do it! HYP TLB invalidation has to occur in HYP, and once we've installed the runtime page tables, it is already too late. It is actually fairly easy to construct a scenario where idmap and runtime pages have colliding translations.

The nail on the coffin was provided by Catalin Marinas who told me how much he disliked the arm64 HYP idmap code, and made me realize that we already have all the necessary code in arch/arm/kvm/mmu.c. It just needs a tiny bit of care and affection. With a chainsaw.

The solution to the first two issues is a bit tricky, but doesn't involve a lot of code. The hotplug problem mandates that we keep two sets of page tables (boot and runtime). The TLB problem mandates that we're able to transition from one PGD to another while in HYP, invalidating the TLBs in the process.

To be able to do this, we need to share a page between the two page tables. A page that will have the same VA in both configurations. All we need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text

The vectors page VA seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory, it is unlikely to use the last 64kB (I doubt we'll ever support KVM on a system with something like 4MB of RAM, but patches are very welcome).

Let's call this VA the trampoline VA.

Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd

The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd, runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).

Once we have this infrastructure in place, supporting CPU hot-plug is a piece of cake. Just wire a cpu-notifier in the existing code.

This has been tested on both arm (VE TC2) and arm64 (Foundation Model).

Quoted 1 line

From v3:

- Remove stray conflict left in the middle of patch 6/7. Always send the patch series you're actually testing...

Quoted 1 line

From v2:

- Change the init.S calling convention to pass the PGD in r2-r3, removing the horrible endianess-dependant split of the 64bit physical address.
- Fix the HYP page table freeing code by getting rid of it. Our stage-2 code does it a lot better, and I'd rather fix bugs only once.
- Flush HYP page tables to the Point of Coherency. This ensures that the incoming CPU will see the page tables, even with cache disabled (spotted by Will).
- Use kmalloc instead of kzalloc for the bounce page (spotted by Christoffer).

Quoted 1 line

From v1:

- Reduce the mandatory alignement of the init code page, and use a bounce page if the code is crossing a page boundary.
- Add some more comments to the code (as requested by Christoffer)
- Fixed compilation issue with older toolchains (spotted by Geoff)

Marc Zyngier (7):
ARM: KVM: simplify HYP mapping population
ARM: KVM: fix HYP mapping limitations around zero
ARM: KVM: move to a KVM provided HYP idmap
ARM: KVM: enforce maximum size for identity mapped code
ARM: KVM: rework HYP page table freeing
ARM: KVM: switch to a dual-step HYP init code
ARM: KVM: perform HYP initilization for hotplugged CPUs

 arch/arm/include/asm/idmap.h    |   1 -
 arch/arm/include/asm/kvm_host.h |  31 +--
 arch/arm/include/asm/kvm_mmu.h  |  28 ++-
 arch/arm/kernel/vmlinux.lds.S   |   7 +-
 arch/arm/kvm/arm.c              |  62 ++++--
 arch/arm/kvm/init.S             |  78 +++++--
 arch/arm/kvm/mmu.c              | 455 +++++++++++++++++++++++-----------------
 arch/arm/mm/idmap.c             |  32 +--
 8 files changed, 404 insertions(+), 290 deletions(-)
Signature · 1 line
1.8.1.4
#1 lore raw

PATCH v4 1/7 ARM: KVM: simplify HYP mapping population

The way we populate HYP mappings is a bit convoluted, to say the least. Passing a pointer around to keep track of the current PFN is quite odd, and we end-up having two different PTE accessors for no good reason.

Simplify the whole thing by unifying the two PTE accessors, passing a pgprot_t around, and moving the various validity checks to the upper layers.

---
 arch/arm/kvm/mmu.c | 102 ++++++++++++++++++++++-------------------------------
 1 file changed, 42 insertions(+), 60 deletions(-)
arch/arm/kvm/mmu.c +42 -60
index 2f12e40..3003321 100644
@@ -125,54 +125,34 @@ void free_hyp_pmds(void)
 }
 
 static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
-				    unsigned long end)
+				    unsigned long end, unsigned long pfn,
+				    pgprot_t prot)
 {
 	pte_t *pte;
 	unsigned long addr;
-	struct page *page;
 
-	for (addr = start & PAGE_MASK; addr < end; addr += PAGE_SIZE) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-
-		pte = pte_offset_kernel(pmd, hyp_addr);
-		BUG_ON(!virt_addr_valid(addr));
-		page = virt_to_page(addr);
-		kvm_set_pte(pte, mk_pte(page, PAGE_HYP));
-	}
-}
-
-static void create_hyp_io_pte_mappings(pmd_t *pmd, unsigned long start,
-				       unsigned long end,
-				       unsigned long *pfn_base)
-{
-	pte_t *pte;
-	unsigned long addr;
-
-	for (addr = start & PAGE_MASK; addr < end; addr += PAGE_SIZE) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-
-		pte = pte_offset_kernel(pmd, hyp_addr);
-		BUG_ON(pfn_valid(*pfn_base));
-		kvm_set_pte(pte, pfn_pte(*pfn_base, PAGE_HYP_DEVICE));
-		(*pfn_base)++;
+	for (addr = start; addr < end; addr += PAGE_SIZE) {
+		pte = pte_offset_kernel(pmd, addr);
+		kvm_set_pte(pte, pfn_pte(pfn, prot));
+		pfn++;
 	}
 }
 
 static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
-				   unsigned long end, unsigned long *pfn_base)
+				   unsigned long end, unsigned long pfn,
+				   pgprot_t prot)
 {
 	pmd_t *pmd;
 	pte_t *pte;
 	unsigned long addr, next;
 
 	for (addr = start; addr < end; addr = next) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-		pmd = pmd_offset(pud, hyp_addr);
+		pmd = pmd_offset(pud, addr);
 
 		BUG_ON(pmd_sect(*pmd));
 
 		if (pmd_none(*pmd)) {
-			pte = pte_alloc_one_kernel(NULL, hyp_addr);
+			pte = pte_alloc_one_kernel(NULL, addr);
 			if (!pte) {
 				kvm_err("Cannot allocate Hyp pte\n");
 				return -ENOMEM;
@@ -182,25 +162,17 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
 
 		next = pmd_addr_end(addr, end);
 
-		/*
-		 * If pfn_base is NULL, we map kernel pages into HYP with the
-		 * virtual address. Otherwise, this is considered an I/O
-		 * mapping and we map the physical region starting at
-		 * *pfn_base to [start, end[.
-		 */
-		if (!pfn_base)
-			create_hyp_pte_mappings(pmd, addr, next);
-		else
-			create_hyp_io_pte_mappings(pmd, addr, next, pfn_base);
+		create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
+		pfn += (next - addr) >> PAGE_SHIFT;
 	}
 
 	return 0;
 }
 
-static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
+static int __create_hyp_mappings(pgd_t *pgdp,
+				 unsigned long start, unsigned long end,
+				 unsigned long pfn, pgprot_t prot)
 {
-	unsigned long start = (unsigned long)from;
-	unsigned long end = (unsigned long)to;
 	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
@@ -209,21 +181,14 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 
 	if (start >= end)
 		return -EINVAL;
-	/* Check for a valid kernel memory mapping */
-	if (!pfn_base && (!virt_addr_valid(from) || !virt_addr_valid(to - 1)))
-		return -EINVAL;
-	/* Check for a valid kernel IO mapping */
-	if (pfn_base && (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1)))
-		return -EINVAL;
 
 	mutex_lock(&kvm_hyp_pgd_mutex);
-	for (addr = start; addr < end; addr = next) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-		pgd = hyp_pgd + pgd_index(hyp_addr);
-		pud = pud_offset(pgd, hyp_addr);
+	for (addr = start & PAGE_MASK; addr < end; addr = next) {
+		pgd = pgdp + pgd_index(addr);
+		pud = pud_offset(pgd, addr);
 
 		if (pud_none_or_clear_bad(pud)) {
-			pmd = pmd_alloc_one(NULL, hyp_addr);
+			pmd = pmd_alloc_one(NULL, addr);
 			if (!pmd) {
 				kvm_err("Cannot allocate Hyp pmd\n");
 				err = -ENOMEM;
@@ -233,9 +198,10 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 		}
 
 		next = pgd_addr_end(addr, end);
-		err = create_hyp_pmd_mappings(pud, addr, next, pfn_base);
+		err = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);
 		if (err)
 			goto out;
+		pfn += (next - addr) >> PAGE_SHIFT;
 	}
 out:
 	mutex_unlock(&kvm_hyp_pgd_mutex);
@@ -255,22 +221,38 @@ out:
  */
 int create_hyp_mappings(void *from, void *to)
 {
-	return __create_hyp_mappings(from, to, NULL);
+	unsigned long phys_addr = virt_to_phys(from);
+	unsigned long start = KERN_TO_HYP((unsigned long)from);
+	unsigned long end = KERN_TO_HYP((unsigned long)to);
+
+	/* Check for a valid kernel memory mapping */
+	if (!virt_addr_valid(from) || !virt_addr_valid(to - 1))
+		return -EINVAL;
+
+	return __create_hyp_mappings(hyp_pgd, start, end,
+				     __phys_to_pfn(phys_addr), PAGE_HYP);
 }
 
 /**
  * create_hyp_io_mappings - duplicate a kernel IO mapping into Hyp mode
  * @from:	The kernel start VA of the range
  * @to:		The kernel end VA of the range (exclusive)
- * @addr:	The physical start address which gets mapped
+ * @phys_addr:	The physical start address which gets mapped
  *
  * The resulting HYP VA is the same as the kernel VA, modulo
  * HYP_PAGE_OFFSET.
  */
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t addr)
+int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
 {
-	unsigned long pfn = __phys_to_pfn(addr);
-	return __create_hyp_mappings(from, to, &pfn);
+	unsigned long start = KERN_TO_HYP((unsigned long)from);
+	unsigned long end = KERN_TO_HYP((unsigned long)to);
+
+	/* Check for a valid kernel IO mapping */
+	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
+		return -EINVAL;
+
+	return __create_hyp_mappings(hyp_pgd, start, end,
+				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
 }
 
 /**
Signature · 1 line
1.8.1.4
#2 lore raw

PATCH v4 1/7 ARM: KVM: simplify HYP mapping population

Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index 2f12e40..3003321 100644

...

Quoted 37 lines
arch/arm/kvm/mmu.c +7 -14
-static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
+static int __create_hyp_mappings(pgd_t *pgdp,
+				 unsigned long start, unsigned long end,
+				 unsigned long pfn, pgprot_t prot)
 {
-	unsigned long start = (unsigned long)from;
-	unsigned long end = (unsigned long)to;
 	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
@@ -209,21 +181,14 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 
 	if (start >= end)
 		return -EINVAL;
-	/* Check for a valid kernel memory mapping */
-	if (!pfn_base && (!virt_addr_valid(from) || !virt_addr_valid(to - 1)))
-		return -EINVAL;
-	/* Check for a valid kernel IO mapping */
-	if (pfn_base && (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1)))
-		return -EINVAL;
 
 	mutex_lock(&kvm_hyp_pgd_mutex);
-	for (addr = start; addr < end; addr = next) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-		pgd = hyp_pgd + pgd_index(hyp_addr);
-		pud = pud_offset(pgd, hyp_addr);
+	for (addr = start & PAGE_MASK; addr < end; addr = next) {
+		pgd = pgdp + pgd_index(addr);
+		pud = pud_offset(pgd, addr);
 
 		if (pud_none_or_clear_bad(pud)) {
-			pmd = pmd_alloc_one(NULL, hyp_addr);
+			pmd = pmd_alloc_one(NULL, addr);
 			if (!pmd) {
 				kvm_err("Cannot allocate Hyp pmd\n");
 				err = -ENOMEM;

arch/arm/kvm/mmu.c static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base) · line 201

Quoted 5 lines
arch/arm/kvm/mmu.c +1 -1
@@ -233,9 +198,10 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 		}
 
 		next = pgd_addr_end(addr, end);
-		err = create_hyp_pmd_mappings(pud, addr, next, pfn_base);
+		err = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);

I would add another indirection function here like create_hyp_pud_mappings() if we are to share this code with arm64. If written correctly, the compiler should turn it into no-op. But here you just make the assumption that pgd/pud are the same.

(it can be additional patch at some point if you already prepared these patches for upstream)

Signature · 1 line
Catalin
#3 lore raw

PATCH v4 1/7 ARM: KVM: simplify HYP mapping population

Quoted 45 lines · Catalin Marinas
Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index 2f12e40..3003321 100644

...

Quoted 39 lines
arch/arm/kvm/mmu.c +7 -14
-static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
+static int __create_hyp_mappings(pgd_t *pgdp,
+				 unsigned long start, unsigned long end,
+				 unsigned long pfn, pgprot_t prot)
 {
-	unsigned long start = (unsigned long)from;
-	unsigned long end = (unsigned long)to;
 	pgd_t *pgd;
 	pud_t *pud;
 	pmd_t *pmd;
@@ -209,21 +181,14 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 
 	if (start >= end)
 		return -EINVAL;
-	/* Check for a valid kernel memory mapping */
-	if (!pfn_base && (!virt_addr_valid(from) || !virt_addr_valid(to - 1)))
-		return -EINVAL;
-	/* Check for a valid kernel IO mapping */
-	if (pfn_base && (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1)))
-		return -EINVAL;
 
 	mutex_lock(&kvm_hyp_pgd_mutex);
-	for (addr = start; addr < end; addr = next) {
-		unsigned long hyp_addr = KERN_TO_HYP(addr);
-		pgd = hyp_pgd + pgd_index(hyp_addr);
-		pud = pud_offset(pgd, hyp_addr);
+	for (addr = start & PAGE_MASK; addr < end; addr = next) {
+		pgd = pgdp + pgd_index(addr);
+		pud = pud_offset(pgd, addr);
 
 		if (pud_none_or_clear_bad(pud)) {
-			pmd = pmd_alloc_one(NULL, hyp_addr);
+			pmd = pmd_alloc_one(NULL, addr);
 			if (!pmd) {
 				kvm_err("Cannot allocate Hyp pmd\n");
 				err = -ENOMEM;
@@ -233,9 +198,10 @@ static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base)
 		}
 
Quoted 8 lines

arch/arm/kvm/mmu.c static int __create_hyp_mappings(void *from, void *to, unsigned long *pfn_base) · line 201

Quoted 3 lines
arch/arm/kvm/mmu.c +1 -1
 		next = pgd_addr_end(addr, end);
-		err = create_hyp_pmd_mappings(pud, addr, next, pfn_base);
+		err = create_hyp_pmd_mappings(pud, addr, next, pfn, prot);

I would add another indirection function here like create_hyp_pud_mappings() if we are to share this code with arm64. If written correctly, the compiler should turn it into no-op. But here you just make the assumption that pgd/pud are the same.

Yes, it is a problem that is present across most of KVM/arm[64]. In general, the code assumes that PUD is folded into PGD.

0> (it can be additional patch at some point if you already prepared these

Quoted 1 line

patches for upstream)

It is my intention to have an additional patch series that adds proper PUD handling, so we could properly support 4 level page tables without any other change.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#4 lore raw

PATCH v4 2/7 ARM: KVM: fix HYP mapping limitations around zero

In reply to Marc Zyngier

The current code for creating HYP mapping doesn't like to wrap around zero, which prevents from mapping anything into the last page of the virtual address space.

It doesn't take much effort to remove this limitation, making the code more consistent with the rest of the kernel in the process.

---
 arch/arm/kvm/mmu.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)
arch/arm/kvm/mmu.c +10 -11
index 3003321..96d61da 100644
@@ -131,11 +131,12 @@ static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
 	pte_t *pte;
 	unsigned long addr;
 
-	for (addr = start; addr < end; addr += PAGE_SIZE) {
+	addr = start;
+	do {
 		pte = pte_offset_kernel(pmd, addr);
 		kvm_set_pte(pte, pfn_pte(pfn, prot));
 		pfn++;
-	}
+	} while (addr += PAGE_SIZE, addr != end);
 }
 
 static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
@@ -146,7 +147,8 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
 	pte_t *pte;
 	unsigned long addr, next;
 
-	for (addr = start; addr < end; addr = next) {
+	addr = start;
+	do {
 		pmd = pmd_offset(pud, addr);
 
 		BUG_ON(pmd_sect(*pmd));
@@ -164,7 +166,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,
 
 		create_hyp_pte_mappings(pmd, addr, next, pfn, prot);
 		pfn += (next - addr) >> PAGE_SHIFT;
-	}
+	} while (addr = next, addr != end);
 
 	return 0;
 }
@@ -179,11 +181,10 @@ static int __create_hyp_mappings(pgd_t *pgdp,
 	unsigned long addr, next;
 	int err = 0;
 
-	if (start >= end)
-		return -EINVAL;
-
 	mutex_lock(&kvm_hyp_pgd_mutex);
-	for (addr = start & PAGE_MASK; addr < end; addr = next) {
+	addr = start & PAGE_MASK;
+	end = PAGE_ALIGN(end);
+	do {
 		pgd = pgdp + pgd_index(addr);
 		pud = pud_offset(pgd, addr);
 
@@ -202,7 +203,7 @@ static int __create_hyp_mappings(pgd_t *pgdp,
 		if (err)
 			goto out;
 		pfn += (next - addr) >> PAGE_SHIFT;
-	}
+	} while (addr = next, addr != end);
 out:
 	mutex_unlock(&kvm_hyp_pgd_mutex);
 	return err;
@@ -216,8 +217,6 @@ out:
  * The same virtual address as the kernel virtual address is also used
  * in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
  * physical pages.
- *
- * Note: Wrapping around zero in the "to" address is not supported.
  */
 int create_hyp_mappings(void *from, void *to)
 {
Signature · 1 line
1.8.1.4
#5 lore raw

PATCH v4 3/7 ARM: KVM: move to a KVM provided HYP idmap

In reply to Marc Zyngier

After the HYP page table rework, it is pretty easy to let the KVM code provide its own idmap, rather than expecting the kernel to provide it. It takes actually less code to do so.

---
 arch/arm/include/asm/idmap.h   |  1 -
 arch/arm/include/asm/kvm_mmu.h |  1 -
 arch/arm/kvm/mmu.c             | 24 +++++++++++++++++++++++-
 arch/arm/mm/idmap.c            | 32 +-------------------------------
 4 files changed, 24 insertions(+), 34 deletions(-)
arch/arm/include/asm/idmap.h +0 -1
index 1a66f907..bf863ed 100644
@@ -8,7 +8,6 @@
 #define __idmap __section(.idmap.text) noinline notrace
 
 extern pgd_t *idmap_pgd;
-extern pgd_t *hyp_pgd;
 
 void setup_mm_for_reboot(void);
 
arch/arm/include/asm/kvm_mmu.h +0 -1
index 970f3b5..3c71a1d 100644
@@ -21,7 +21,6 @@
 
 #include <asm/cacheflush.h>
 #include <asm/pgalloc.h>
-#include <asm/idmap.h>
 
 /*
  * We directly use the kernel VA for the HYP, as we can directly share
arch/arm/kvm/mmu.c +23 -1
index 96d61da..bfc5927 100644
@@ -32,6 +32,7 @@
 
 extern char  __hyp_idmap_text_start[], __hyp_idmap_text_end[];
 
+static pgd_t *hyp_pgd;
 static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
 
 static void kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa)
@@ -715,12 +716,33 @@ phys_addr_t kvm_mmu_get_httbr(void)
 
 int kvm_mmu_init(void)
 {
+	unsigned long hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
+	unsigned long hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
+	int err;
+
+	hyp_pgd = kzalloc(PTRS_PER_PGD * sizeof(pgd_t), GFP_KERNEL);
 	if (!hyp_pgd) {
 		kvm_err("Hyp mode PGD not allocated\n");
-		return -ENOMEM;
+		err = -ENOMEM;
+		goto out;
+	}
+
+	/* Create the idmap in the boot page tables */
+	err = 	__create_hyp_mappings(boot_hyp_pgd,
+				      hyp_idmap_start, hyp_idmap_end,
+				      __phys_to_pfn(hyp_idmap_start),
+				      PAGE_HYP);
+
+	if (err) {
+		kvm_err("Failed to idmap %lx-%lx\n",
+			hyp_idmap_start, hyp_idmap_end);
+		goto out;
 	}
 
 	return 0;
+out:
+	kfree(hyp_pgd);
+	return err;
 }
 
 /**
arch/arm/mm/idmap.c +1 -31
index 5ee505c..83cb3ac 100644
@@ -8,7 +8,6 @@
 #include <asm/pgtable.h>
 #include <asm/sections.h>
 #include <asm/system_info.h>
-#include <asm/virt.h>
 
 pgd_t *idmap_pgd;
 
@@ -83,37 +82,10 @@ static void identity_mapping_add(pgd_t *pgd, const char *text_start,
 	} while (pgd++, addr = next, addr != end);
 }
 
-#if defined(CONFIG_ARM_VIRT_EXT) && defined(CONFIG_ARM_LPAE)
-pgd_t *hyp_pgd;
-
-extern char  __hyp_idmap_text_start[], __hyp_idmap_text_end[];
-
-static int __init init_static_idmap_hyp(void)
-{
-	hyp_pgd = kzalloc(PTRS_PER_PGD * sizeof(pgd_t), GFP_KERNEL);
-	if (!hyp_pgd)
-		return -ENOMEM;
-
-	pr_info("Setting up static HYP identity map for 0x%p - 0x%p\n",
-		__hyp_idmap_text_start, __hyp_idmap_text_end);
-	identity_mapping_add(hyp_pgd, __hyp_idmap_text_start,
-			     __hyp_idmap_text_end, PMD_SECT_AP1);
-
-	return 0;
-}
-#else
-static int __init init_static_idmap_hyp(void)
-{
-	return 0;
-}
-#endif
-
 extern char  __idmap_text_start[], __idmap_text_end[];
 
 static int __init init_static_idmap(void)
 {
-	int ret;
-
 	idmap_pgd = pgd_alloc(&init_mm);
 	if (!idmap_pgd)
 		return -ENOMEM;
@@ -123,12 +95,10 @@ static int __init init_static_idmap(void)
 	identity_mapping_add(idmap_pgd, __idmap_text_start,
 			     __idmap_text_end, 0);
 
-	ret = init_static_idmap_hyp();
-
 	/* Flush L1 for the hardware to see this page table content */
 	flush_cache_louis();
 
-	return ret;
+	return 0;
 }
 early_initcall(init_static_idmap);
 
Signature · 1 line
1.8.1.4
#6 lore raw

PATCH v4 3/7 ARM: KVM: move to a KVM provided HYP idmap

Quoted 149 lines · Marc Zyngier

After the HYP page table rework, it is pretty easy to let the KVM code provide its own idmap, rather than expecting the kernel to provide it. It takes actually less code to do so.

---
 arch/arm/include/asm/idmap.h   |  1 -
 arch/arm/include/asm/kvm_mmu.h |  1 -
 arch/arm/kvm/mmu.c             | 24 +++++++++++++++++++++++-
 arch/arm/mm/idmap.c            | 32 +-------------------------------
 4 files changed, 24 insertions(+), 34 deletions(-)
arch/arm/include/asm/idmap.h +0 -1
index 1a66f907..bf863ed 100644
@@ -8,7 +8,6 @@
 #define __idmap __section(.idmap.text) noinline notrace
 
 extern pgd_t *idmap_pgd;
-extern pgd_t *hyp_pgd;
 
 void setup_mm_for_reboot(void);
 
arch/arm/include/asm/kvm_mmu.h +0 -1
index 970f3b5..3c71a1d 100644
@@ -21,7 +21,6 @@
 
 #include <asm/cacheflush.h>
 #include <asm/pgalloc.h>
-#include <asm/idmap.h>
 
 /*
  * We directly use the kernel VA for the HYP, as we can directly share
arch/arm/kvm/mmu.c +23 -1
index 96d61da..bfc5927 100644
@@ -32,6 +32,7 @@
 
 extern char  __hyp_idmap_text_start[], __hyp_idmap_text_end[];
 
+static pgd_t *hyp_pgd;
 static DEFINE_MUTEX(kvm_hyp_pgd_mutex);
 
 static void kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa)
@@ -715,12 +716,33 @@ phys_addr_t kvm_mmu_get_httbr(void)
 
 int kvm_mmu_init(void)
 {
+       unsigned long hyp_idmap_start = virt_to_phys(__hyp_idmap_text_start);
+       unsigned long hyp_idmap_end = virt_to_phys(__hyp_idmap_text_end);
+       int err;
+
+       hyp_pgd = kzalloc(PTRS_PER_PGD * sizeof(pgd_t), GFP_KERNEL);
        if (!hyp_pgd) {
                kvm_err("Hyp mode PGD not allocated\n");
-               return -ENOMEM;
+               err = -ENOMEM;
+               goto out;
+       }
+
+       /* Create the idmap in the boot page tables */
+       err =   __create_hyp_mappings(boot_hyp_pgd,
+                                     hyp_idmap_start, hyp_idmap_end,
+                                     __phys_to_pfn(hyp_idmap_start),
+                                     PAGE_HYP);
+
+       if (err) {
+               kvm_err("Failed to idmap %lx-%lx\n",
+                       hyp_idmap_start, hyp_idmap_end);
+               goto out;
        }
 
        return 0;
+out:
+       kfree(hyp_pgd);
+       return err;
 }
 
 /**
arch/arm/mm/idmap.c +1 -32
index 5ee505c..83cb3ac 100644
@@ -8,7 +8,6 @@
 #include <asm/pgtable.h>
 #include <asm/sections.h>
 #include <asm/system_info.h>
-#include <asm/virt.h>
 
 pgd_t *idmap_pgd;
 
@@ -83,37 +82,10 @@ static void identity_mapping_add(pgd_t *pgd, const char *text_start,
        } while (pgd++, addr = next, addr != end);
 }
 
-#if defined(CONFIG_ARM_VIRT_EXT) && defined(CONFIG_ARM_LPAE)
-pgd_t *hyp_pgd;
-
-extern char  __hyp_idmap_text_start[], __hyp_idmap_text_end[];
-
-static int __init init_static_idmap_hyp(void)
-{
-       hyp_pgd = kzalloc(PTRS_PER_PGD * sizeof(pgd_t), GFP_KERNEL);
-       if (!hyp_pgd)
-               return -ENOMEM;
-
-       pr_info("Setting up static HYP identity map for 0x%p - 0x%p\n",
-               __hyp_idmap_text_start, __hyp_idmap_text_end);
-       identity_mapping_add(hyp_pgd, __hyp_idmap_text_start,
-                            __hyp_idmap_text_end, PMD_SECT_AP1);
-
-       return 0;
-}
-#else
-static int __init init_static_idmap_hyp(void)
-{
-       return 0;
-}
-#endif
-
 extern char  __idmap_text_start[], __idmap_text_end[];
 
 static int __init init_static_idmap(void)
 {
-       int ret;
-
        idmap_pgd = pgd_alloc(&init_mm);
        if (!idmap_pgd)
                return -ENOMEM;
@@ -123,12 +95,10 @@ static int __init init_static_idmap(void)
        identity_mapping_add(idmap_pgd, __idmap_text_start,
                             __idmap_text_end, 0);
 
-       ret = init_static_idmap_hyp();
-
        /* Flush L1 for the hardware to see this page table content */
        flush_cache_louis();
 
-       return ret;
+       return 0;
 }
 early_initcall(init_static_idmap);
 
--
Quoted 2 lines

1.8.1.4

Russell,

I would appreciate your ack on this one so we can merge this through the KVM tree, or alternatively let us know if this needs to go your tree, so we can plan accordingly for the next merge window.

Thanks a bunch,
-Christoffer

#7 lore raw

PATCH v4 3/7 ARM: KVM: move to a KVM provided HYP idmap

Quoted 13 lines · Christoffer Dall
Quoted 12 lines · Marc Zyngier

After the HYP page table rework, it is pretty easy to let the KVM code provide its own idmap, rather than expecting the kernel to provide it. It takes actually less code to do so.

---
 arch/arm/include/asm/idmap.h   |  1 -
 arch/arm/include/asm/kvm_mmu.h |  1 -
 arch/arm/kvm/mmu.c             | 24 +++++++++++++++++++++++-
 arch/arm/mm/idmap.c            | 32 +-------------------------------
 4 files changed, 24 insertions(+), 34 deletions(-)

[...]

Quoted 5 lines

Russell,

I would appreciate your ack on this one so we can merge this through the KVM tree, or alternatively let us know if this needs to go your tree, so we can plan accordingly for the next merge window.

The core ARM part of this is basically just deleting code that KVM doesn't require any more. I'm not aware of any other patches in this area at the moment so, unless Russell objects, I don't think there would be a problem with this going via the kvm tree providing that it sits in -next until it's in mainline.

Will

#8 lore raw

PATCH v4 4/7 ARM: KVM: enforce maximum size for identity mapped code

In reply to Marc Zyngier

We're about to move to an init procedure where we rely on the fact that the init code fits in a single page. Make sure we align the idmap text on a vector alignment, and that the code is not bigger than a single page.

---
 arch/arm/kernel/vmlinux.lds.S | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm/kernel/vmlinux.lds.S +6 -1
index b571484..a871b8e 100644
@@ -20,7 +20,7 @@
 	VMLINUX_SYMBOL(__idmap_text_start) = .;				\
 	*(.idmap.text)							\
 	VMLINUX_SYMBOL(__idmap_text_end) = .;				\
-	ALIGN_FUNCTION();						\
+	. = ALIGN(32);							\
 	VMLINUX_SYMBOL(__hyp_idmap_text_start) = .;			\
 	*(.hyp.idmap.text)						\
 	VMLINUX_SYMBOL(__hyp_idmap_text_end) = .;
@@ -315,3 +315,8 @@ SECTIONS
  */
 ASSERT((__proc_info_end - __proc_info_begin), "missing CPU support")
 ASSERT((__arch_info_end - __arch_info_begin), "no machine record defined")
+/*
+ * The HYP init code can't be more than a page long.
+ * The above comment applies as well.
+ */
+ASSERT(((__hyp_idmap_text_end - __hyp_idmap_text_start) <= PAGE_SIZE), "HYP init code too big")
Signature · 1 line
1.8.1.4
#9 lore raw

PATCH v4 5/7 ARM: KVM: rework HYP page table freeing

In reply to Marc Zyngier

There is no point in freeing HYP page tables differently from Stage-2. They now have the same requirements, and should be dealt with the same way.

Promote unmap_stage2_range to be The One True Way, and get rid of a number of nasty bugs in the process (goo thing we never actually called free_hyp_pmds before...).

---
 arch/arm/include/asm/kvm_mmu.h |   2 +-
 arch/arm/kvm/arm.c             |   2 +-
 arch/arm/kvm/mmu.c             | 181 ++++++++++++++++++-----------------------
 3 files changed, 82 insertions(+), 103 deletions(-)
arch/arm/include/asm/kvm_mmu.h +1 -1
index 3c71a1d..92eb20d 100644
@@ -32,7 +32,7 @@
 
 int create_hyp_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
-void free_hyp_pmds(void);
+void free_hyp_pgds(void);
 
 int kvm_alloc_stage2_pgd(struct kvm *kvm);
 void kvm_free_stage2_pgd(struct kvm *kvm);
arch/arm/kvm/arm.c +1 -1
index 23538ed..8992f05 100644
@@ -937,7 +937,7 @@ static int init_hyp_mode(void)
 out_free_context:
 	free_percpu(kvm_host_cpu_state);
 out_free_mappings:
-	free_hyp_pmds();
+	free_hyp_pgds();
 out_free_stack_pages:
 	for_each_possible_cpu(cpu)
 		free_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
arch/arm/kvm/mmu.c +80 -101

246 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#10 lore raw

PATCH v4 5/7 ARM: KVM: rework HYP page table freeing

Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index bfc5927..7464824 100644

...

Quoted 7 lines
arch/arm/kvm/mmu.c +7 -0
+static void clear_pmd_entry(pmd_t *pmd)
+{
+	pte_t *pte_table = pte_offset_kernel(pmd, 0);
+	pmd_clear(pmd);
+	pte_free_kernel(NULL, pte_table);
+	put_page(virt_to_page(pmd));
+}

...

Quoted 1 line

static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size)

Is there a chance that this function (or the other unmapping function for Hyp pages) is called on an active stage 2 table (VTTBR pointing to this pgd)? If yes, than you probably have to follow the mmu_gather mechanism of freeing page table pages to avoid speculative loads. Basically flushing the TLB between pmd_clear and pte_free_kernel.

Signature · 1 line
Catalin
#11 lore raw

PATCH v4 5/7 ARM: KVM: rework HYP page table freeing

Quoted 14 lines · Catalin Marinas
Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index bfc5927..7464824 100644

...

Quoted 7 lines
arch/arm/kvm/mmu.c +7 -0
+static void clear_pmd_entry(pmd_t *pmd)
+{
+	pte_t *pte_table = pte_offset_kernel(pmd, 0);
+	pmd_clear(pmd);
+	pte_free_kernel(NULL, pte_table);
+	put_page(virt_to_page(pmd));
+}

...

Quoted 7 lines
Quoted 1 line

static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size)

Is there a chance that this function (or the other unmapping function for Hyp pages) is called on an active stage 2 table (VTTBR pointing to this pgd)? If yes, than you probably have to follow the mmu_gather mechanism of freeing page table pages to avoid speculative loads. Basically flushing the TLB between pmd_clear and pte_free_kernel.

Blah. You're right, we got it wrong.

We need to move our TLB invalidation out of kvm_unmap_hva_handler, and put it in clear_pmd_entry. I'll cook a patch.

Thanks for reviewing.

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#12 lore raw

PATCH v4 5/7 ARM: KVM: rework HYP page table freeing

Quoted 23 lines · Marc Zyngier
Quoted 14 lines · Catalin Marinas
Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index bfc5927..7464824 100644

...

Quoted 7 lines
arch/arm/kvm/mmu.c +7 -0
+static void clear_pmd_entry(pmd_t *pmd)
+{
+    pte_t *pte_table = pte_offset_kernel(pmd, 0);
+    pmd_clear(pmd);
+    pte_free_kernel(NULL, pte_table);
+    put_page(virt_to_page(pmd));
+}

...

Quoted 7 lines
Quoted 1 line

static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size)

Is there a chance that this function (or the other unmapping function for Hyp pages) is called on an active stage 2 table (VTTBR pointing to this pgd)? If yes, than you probably have to follow the mmu_gather mechanism of freeing page table pages to avoid speculative loads. Basically flushing the TLB between pmd_clear and pte_free_kernel.

Quoted 7 lines

Blah. You're right, we got it wrong.

We need to move our TLB invalidation out of kvm_unmap_hva_handler, and put it in clear_pmd_entry. I'll cook a patch.

Thanks for reviewing.

Ah, because clean_pmd_entry doesn't flush stage2 TLB, that's the issue?

#13 lore raw

PATCH v4 5/7 ARM: KVM: rework HYP page table freeing

Quoted 24 lines · Christoffer Dall
Quoted 23 lines · Marc Zyngier
Quoted 21 lines · Catalin Marinas
Quoted 4 lines · Marc Zyngier
arch/arm/kvm/mmu.c +0 -0
index bfc5927..7464824 100644

...

Quoted 7 lines
arch/arm/kvm/mmu.c +7 -0
+static void clear_pmd_entry(pmd_t *pmd)
+{
+    pte_t *pte_table = pte_offset_kernel(pmd, 0);
+    pmd_clear(pmd);
+    pte_free_kernel(NULL, pte_table);
+    put_page(virt_to_page(pmd));
+}

...

Quoted 1 line

static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size)

Is there a chance that this function (or the other unmapping function for Hyp pages) is called on an active stage 2 table (VTTBR pointing to this pgd)? If yes, than you probably have to follow the mmu_gather mechanism of freeing page table pages to avoid speculative loads. Basically flushing the TLB between pmd_clear and pte_free_kernel.

Quoted 7 lines
Quoted 6 lines

Blah. You're right, we got it wrong.

We need to move our TLB invalidation out of kvm_unmap_hva_handler, and put it in clear_pmd_entry. I'll cook a patch.

Thanks for reviewing.

Ah, because clean_pmd_entry doesn't flush stage2 TLB, that's the issue?

Yes. After clearing a pmd entry you need to flush the stage 2 TLB before freeing the pte page it was pointing to. Otherwise you can get other CPUs loading the TLB with invalid data (either because of intermediate level caching in the TLB or simply because they haven't observed the actual pmd clearing).

Catalin

#14 lore raw

PATCH v4 6/7 ARM: KVM: switch to a dual-step HYP init code

In reply to Marc Zyngier

Our HYP init code suffers from two major design issues:
- it cannot support CPU hotplug, as we tear down the idmap very early
- it cannot perform a TLB invalidation when switching from init to runtime mappings, as pages are manipulated from PL1 exclusively

The hotplug problem mandates that we keep two sets of page tables (boot and runtime). The TLB problem mandates that we're able to transition from one PGD to another while in HYP, invalidating the TLBs in the process.

To be able to do this, we need to share a page between the two page tables. A page that will have the same VA in both configurations. All we need is a VA that has the following properties:
- This VA can't be used to represent a kernel mapping.
- This VA will not conflict with the physical address of the kernel text

The vectors page seems to satisfy this requirement:
- The kernel never maps anything else there
- The kernel text being copied at the beginning of the physical memory, it is unlikely to use the last 64kB (I doubt we'll ever support KVM on a system with something like 4MB of RAM, but patches are very welcome).

Let's call this VA the trampoline VA.

Now, we map our init page at 3 locations:
- idmap in the boot pgd
- trampoline VA in the boot pgd
- trampoline VA in the runtime pgd

The init scenario is now the following:
- We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd, runtime stack, runtime vectors
- Enable the MMU with the boot pgd
- Jump to a target into the trampoline page (remember, this is the same physical page!)
- Now switch to the runtime pgd (same VA, and still the same physical page!)
- Invalidate TLBs
- Set stack and vectors
- Profit! (or eret, if you only care about the code).

Note that we keep the boot mapping permanently (it is not strictly an idmap anymore) to allow for CPU hotplug in later patches.

---
 arch/arm/include/asm/kvm_host.h |  31 ++++++----
 arch/arm/include/asm/kvm_mmu.h  |  24 +++++++-
 arch/arm/kvm/arm.c              |  11 ++--
 arch/arm/kvm/init.S             |  78 ++++++++++++++++++------
 arch/arm/kvm/mmu.c              | 132 ++++++++++++++++++++++++++++------------
 5 files changed, 197 insertions(+), 79 deletions(-)
arch/arm/include/asm/kvm_host.h +19 -12
index a2907fb..57cb786 100644
@@ -190,23 +190,30 @@ int kvm_arm_coproc_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *);
 int handle_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
 		int exception_index);
 
-static inline void __cpu_init_hyp_mode(unsigned long long pgd_ptr,
+static inline void __cpu_init_hyp_mode(unsigned long long boot_pgd_ptr,
+				       unsigned long long pgd_ptr,
 				       unsigned long hyp_stack_ptr,
 				       unsigned long vector_ptr)
 {
-	unsigned long pgd_low, pgd_high;
-
-	pgd_low = (pgd_ptr & ((1ULL << 32) - 1));
-	pgd_high = (pgd_ptr >> 32ULL);
-
 	/*
-	 * Call initialization code, and switch to the full blown
-	 * HYP code. The init code doesn't need to preserve these registers as
-	 * r1-r3 and r12 are already callee save according to the AAPCS.
-	 * Note that we slightly misuse the prototype by casing the pgd_low to
-	 * a void *.
+	 * Call initialization code, and switch to the full blown HYP
+	 * code. The init code doesn't need to preserve these
+	 * registers as r0-r3 are already callee saved according to
+	 * the AAPCS.
+	 * Note that we slightly misuse the prototype by casing the
+	 * stack pointer to a void *.
+	 *
+	 * We don't have enough registers to perform the full init in
+	 * one go.  Install the boot PGD first, and then install the
+	 * runtime PGD, stack pointer and vectors. The PGDs are always
+	 * passed as the third argument, in order to be passed into
+	 * r2-r3 to the init code (yes, this is compliant with the
+	 * PCS!).
 	 */
-	kvm_call_hyp((void *)pgd_low, pgd_high, hyp_stack_ptr, vector_ptr);
+
+	kvm_call_hyp(NULL, 0, boot_pgd_ptr);
+
+	kvm_call_hyp((void*)hyp_stack_ptr, vector_ptr, pgd_ptr);
 }
 
 static inline int kvm_arch_dev_ioctl_check_extension(long ext)
arch/arm/include/asm/kvm_mmu.h +21 -3
index 92eb20d..24b767a 100644
@@ -19,17 +19,29 @@
 #ifndef __ARM_KVM_MMU_H__
 #define __ARM_KVM_MMU_H__
 
-#include <asm/cacheflush.h>
-#include <asm/pgalloc.h>
+#include <asm/memory.h>
+#include <asm/page.h>
 
 /*
  * We directly use the kernel VA for the HYP, as we can directly share
  * the mapping (HTTBR "covers" TTBR1).
  */
-#define HYP_PAGE_OFFSET_MASK	(~0UL)
+#define HYP_PAGE_OFFSET_MASK	UL(~0)
 #define HYP_PAGE_OFFSET		PAGE_OFFSET
 #define KERN_TO_HYP(kva)	(kva)
 
+/*
+ * Our virtual mapping for the boot-time MMU-enable code. Must be
+ * shared across all the page-tables. Conveniently, we use the vectors
+ * page, where no kernel data will ever be shared with HYP.
+ */
+#define TRAMPOLINE_VA		UL(CONFIG_VECTORS_BASE)
+
+#ifndef __ASSEMBLY__
+
+#include <asm/cacheflush.h>
+#include <asm/pgalloc.h>
+
 int create_hyp_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
 void free_hyp_pgds(void);
@@ -44,6 +56,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
 void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
 
 phys_addr_t kvm_mmu_get_httbr(void);
+phys_addr_t kvm_mmu_get_boot_httbr(void);
+phys_addr_t kvm_get_idmap_vector(void);
 int kvm_mmu_init(void);
 void kvm_clear_hyp_idmap(void);
 
@@ -113,4 +127,8 @@ static inline void coherent_icache_guest_page(struct kvm *kvm, gfn_t gfn)
 	}
 }
 
+#define kvm_flush_dcache_to_poc(a,l)	__cpuc_flush_dcache_area((a), (l))
+
+#endif	/* !__ASSEMBLY__ */
+
 #endif /* __ARM_KVM_MMU_H__ */
arch/arm/kvm/arm.c +4 -7
index 8992f05..f8fbab3 100644
@@ -796,20 +796,22 @@ long kvm_arch_vm_ioctl(struct file *filp,
 
 static void cpu_init_hyp_mode(void *vector)
 {
+	unsigned long long boot_pgd_ptr;
 	unsigned long long pgd_ptr;
 	unsigned long hyp_stack_ptr;
 	unsigned long stack_page;
 	unsigned long vector_ptr;
 
 	/* Switch from the HYP stub to our own HYP init vector */
-	__hyp_set_vectors((unsigned long)vector);
+	__hyp_set_vectors(kvm_get_idmap_vector());
 
+	boot_pgd_ptr = (unsigned long long)kvm_mmu_get_boot_httbr();
 	pgd_ptr = (unsigned long long)kvm_mmu_get_httbr();
 	stack_page = __get_cpu_var(kvm_arm_hyp_stack_page);
 	hyp_stack_ptr = stack_page + PAGE_SIZE;
 	vector_ptr = (unsigned long)__kvm_hyp_vector;
 
-	__cpu_init_hyp_mode(pgd_ptr, hyp_stack_ptr, vector_ptr);
+	__cpu_init_hyp_mode(boot_pgd_ptr, pgd_ptr, hyp_stack_ptr, vector_ptr);
 }
 
 /**
@@ -863,11 +865,6 @@ static int init_hyp_mode(void)
 	}
 
 	/*
-	 * Unmap the identity mapping
-	 */
-	kvm_clear_hyp_idmap();
-
-	/*
 	 * Map the Hyp-code called directly from the host
 	 */
 	err = create_hyp_mappings(__kvm_hyp_code_start, __kvm_hyp_code_end);
arch/arm/kvm/init.S +59 -19

118 lines not on this page · raw message

arch/arm/kvm/mmu.c +94 -38

199 lines not on this page · raw message

Signature · 1 line
1.8.1.4
#15 lore raw

PATCH v4 7/7 ARM: KVM: perform HYP initilization for hotplugged CPUs

In reply to Marc Zyngier

Now that we have the necessary infrastructure to boot a hotplugged CPU at any point in time, wire a CPU notifier that will perform the HYP init for the incoming CPU.

Note that this depends on the platform code and/or firmware to boot the incoming CPU with HYP mode enabled and return to the kernel by following the normal boot path (HYP stub installed).

---
 arch/arm/include/asm/kvm_mmu.h |  1 +
 arch/arm/kvm/arm.c             | 49 +++++++++++++++++++++++++++++-------------
 arch/arm/kvm/mmu.c             | 35 ++++++++++++++++++++++++------
 3 files changed, 63 insertions(+), 22 deletions(-)
arch/arm/include/asm/kvm_mmu.h +1 -0
index 24b767a..472ac70 100644
@@ -44,6 +44,7 @@
 
 int create_hyp_mappings(void *from, void *to);
 int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
+void free_boot_hyp_pgd(void);
 void free_hyp_pgds(void);
 
 int kvm_alloc_stage2_pgd(struct kvm *kvm);
arch/arm/kvm/arm.c +34 -15

96 lines not on this page · raw message

arch/arm/kvm/mmu.c +28 -7
index 4646c17..9657065 100644
@@ -156,6 +156,31 @@ static void unmap_range(pgd_t *pgdp, unsigned long long start, u64 size)
 }
 
 /**
+ * free_boot_hyp_pgd - free HYP boot page tables
+ *
+ * Free the HYP boot page tables. The bounce page is also freed.
+ */
+void free_boot_hyp_pgd(void)
+{
+	mutex_lock(&kvm_hyp_pgd_mutex);
+
+	if (boot_hyp_pgd) {
+		unmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
+		unmap_range(boot_hyp_pgd, TRAMPOLINE_VA, PAGE_SIZE);
+		kfree(boot_hyp_pgd);
+		boot_hyp_pgd = NULL;
+	}
+
+	if (hyp_pgd)
+		unmap_range(hyp_pgd, TRAMPOLINE_VA, PAGE_SIZE);
+
+	kfree(init_bounce_page);
+	init_bounce_page = NULL;
+
+	mutex_unlock(&kvm_hyp_pgd_mutex);
+}
+
+/**
  * free_hyp_pgds - free Hyp-mode page tables
  *
  * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
@@ -169,13 +194,9 @@ void free_hyp_pgds(void)
 {
 	unsigned long addr;
 
-	mutex_lock(&kvm_hyp_pgd_mutex);
+	free_boot_hyp_pgd();
 
-	if (boot_hyp_pgd) {
-		unmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
-		unmap_range(boot_hyp_pgd, TRAMPOLINE_VA, PAGE_SIZE);
-		kfree(boot_hyp_pgd);
-	}
+	mutex_lock(&kvm_hyp_pgd_mutex);
 
 	if (hyp_pgd) {
 		for (addr = PAGE_OFFSET; virt_addr_valid(addr); addr += PGDIR_SIZE)
@@ -183,9 +204,9 @@ void free_hyp_pgds(void)
 		for (addr = VMALLOC_START; is_vmalloc_addr((void*)addr); addr += PGDIR_SIZE)
 			unmap_range(hyp_pgd, KERN_TO_HYP(addr), PGDIR_SIZE);
 		kfree(hyp_pgd);
+		hyp_pgd = NULL;
 	}
 
-	kfree(init_bounce_page);
 	mutex_unlock(&kvm_hyp_pgd_mutex);
 }
 
Signature · 1 line
1.8.1.4