KVM/ARM Implementation

From: Marc Zyngier <[email protected]>

The KVM code sometimes uses CONFIG_HAVE_KVM_IRQCHIP to protect code that is related to IRQ routing, which not all in-kernel irqchips may support.

Use KVM_CAP_IRQ_ROUTING instead.

This is a preparatory patch for the KVM/ARM implementation. KVM/ARM will use the KVM_IRQ_LINE ioctl, which is currently conditional on __KVM_HAVE_IOAPIC, but ARM obviously doesn't have any IOAPIC support and we need a separate define.

From: Marc Zyngier <[email protected]>

In order to avoid compilation failure when KVM is not compiled in, guard the mmu_notifier specific sections with both CONFIG_MMU_NOTIFIER and KVM_ARCH_WANT_MMU_NOTIFIER, like it is being done in the rest of the KVM code.

include/linux/kvm_host.h +1 -1

-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
+#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	struct mmu_notifier mmu_notifier;
 	unsigned long mmu_notifier_seq;
 	long mmu_notifier_count;

Quoted 17 lines · Christoffer Dall

From: Marc Zyngier <[email protected]>

In order to avoid compilation failure when KVM is not compiled in, guard the mmu_notifier specific sections with both CONFIG_MMU_NOTIFIER and KVM_ARCH_WANT_MMU_NOTIFIER, like it is being done in the rest of the KVM code.

(quoted diff) +1 -1

-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
+#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)

      struct mmu_notifier mmu_notifier;
      unsigned long mmu_notifier_seq;
      long mmu_notifier_count;

include/linux/kvm_host.h +0 -0

@@ -780,7 +780,7 @@ struct kvm_stats_debugfs_item {
  extern struct kvm_stats_debugfs_item debugfs_entries[];
  extern struct dentry *kvm_debugfs_dir;
 

include/linux/kvm_host.h struct kvm_stats_debugfs_item { · line 785

Quoted 4 lines

include/linux/kvm_host.h +1 -1

-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
+#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
  static inline int mmu_notifier_retry(struct kvm_vcpu *vcpu, unsigned long mmu_seq)
  {

Why not have Kconfig select CONFIG_MMU_NOTIFIER?

Quoted 18 lines · Avi Kivity

Quoted 17 lines · Christoffer Dall

From: Marc Zyngier <[email protected]>

In order to avoid compilation failure when KVM is not compiled in, guard the mmu_notifier specific sections with both CONFIG_MMU_NOTIFIER and KVM_ARCH_WANT_MMU_NOTIFIER, like it is being done in the rest of the KVM code.

include/linux/kvm_host.h +1 -1

-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
+#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
      struct mmu_notifier mmu_notifier;
      unsigned long mmu_notifier_seq;
      long mmu_notifier_count;

@@ -780,7 +780,7 @@ struct kvm_stats_debugfs_item {
 extern struct kvm_stats_debugfs_item debugfs_entries[];
 extern struct dentry *kvm_debugfs_dir;
 

Quoted 8 lines

include/linux/kvm_host.h struct kvm_stats_debugfs_item { · line 785

Quoted 4 lines

include/linux/kvm_host.h +1 -1

-#ifdef KVM_ARCH_WANT_MMU_NOTIFIER
+#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline int mmu_notifier_retry(struct kvm_vcpu *vcpu, unsigned long mmu_seq)
 {

Why not have Kconfig select CONFIG_MMU_NOTIFIER?

Not sure I understand. Where would you select this option?

We do select this option when choosing to compile KVM on, but when we do _not_, then other includes of kvm_host.h fails.

From: Marc Zyngier <[email protected]>

In order to avoid compilation failure when KVM is not compiled in, guard the mmu_notifier specific sections with both CONFIG_MMU_NOTIFIER and KVM_ARCH_WANT_MMU_NOTIFIER, like it is being done in the rest of the KVM code.

Signed-off-by Marc Zyngier <[email protected]>

Signed-off-by Christoffer Dall <[email protected]>

---
 include/linux/kvm_host.h |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Adds support in the identity mapping feature that allows KVM to setup identity mapping for the Hyp mode with the AP[1] bit set as required by the specification and also supports freeing created sub pmd's after finished use.

These two functions:
- hyp_idmap_add(pgd, addr, end);
- hyp_idmap_del(pgd, addr, end);
are essentially calls to the same function as the non-hyp versions but with a different argument value. KVM calls these functions to setup and teardown the identity mapping used to initialize the hypervisor.

Note, the hyp-version of the _del function actually frees the pmd's pointed to by the pgd as opposed to the non-hyp version which just clears them.

Quoted 12 lines · Christoffer Dall

Adds support in the identity mapping feature that allows KVM to setup identity mapping for the Hyp mode with the AP[1] bit set as required by the specification and also supports freeing created sub pmd's after finished use.

These two functions:
 - hyp_idmap_add(pgd, addr, end);
 - hyp_idmap_del(pgd, addr, end);
are essentially calls to the same function as the non-hyp versions but with a different argument value. KVM calls these functions to setup and teardown the identity mapping used to initialize the hypervisor.

Quoted 3 lines

Note, the hyp-version of the _del function actually frees the pmd's pointed to by the pgd as opposed to the non-hyp version which just clears them.

I asked previously what happens if two data structures share a page, and one of them is removed.  Is that handled now?  How?

Why not just identity map all memory?  You can use large pages so it's fast and doesn't consume a lot of page table memory.

Quoted 13 lines · Avi Kivity

Quoted 12 lines · Christoffer Dall

Adds support in the identity mapping feature that allows KVM to setup identity mapping for the Hyp mode with the AP[1] bit set as required by the specification and also supports freeing created sub pmd's after finished use.

These two functions:
- hyp_idmap_add(pgd, addr, end);
- hyp_idmap_del(pgd, addr, end);
are essentially calls to the same function as the non-hyp versions but with a different argument value. KVM calls these functions to setup and teardown the identity mapping used to initialize the hypervisor.

Quoted 8 lines

Quoted 3 lines

Note, the hyp-version of the _del function actually frees the pmd's pointed to by the pgd as opposed to the non-hyp version which just clears them.

I asked previously what happens if two data structures share a page, and one of them is removed. Is that handled now? How?

I think you asked previously for the general hyp-mode mappings, not the identity mappings. For the general hyp-mode mappings we simply don't unmap the data structures, potentially leaking a few pages for the page tables themselves.

This is only for initialization, so there are not really any data structures mapped, only one/two pages to initialize the hypervisor mode.

Quoted 2 lines

Why not just identity map all memory? You can use large pages so it's fast and doesn't consume a lot of page table memory.

That's an option, but it still seems like an awful waste since it's only used once (unless you unload and re-load the module) and there's really no problem with data structures here.

The truth is that this is going to go away, and the code will be put in a section that's idmapped from kernel start. There's a patch under way from Marc taking care of this which I assmue we'll merge for v9.

Sets up the required registers to run code in HYP-mode from the kernel.

By setting the HVBAR the kernel can execute code in Hyp-mode with the MMU disabled. The HVBAR initially points to initialization code, which initializes other Hyp-mode registers and enables the MMU for Hyp-mode. Afterwards, the HVBAR is changed to point to KVM Hyp vectors used to catch guest faults and to switch to Hyp mode to perform a world-switch into a KVM guest.

Also provides memory mapping code to map required code pages and data structures accessed in Hyp mode at the same virtual address as the host kernel virtual addresses, but which conforms to the architectural requirements for translations in Hyp mode. This interface is added in arch/arm/kvm/arm_mmu.c and is comprised of:
- create_hyp_mappings(hyp_pgd, start, end);
- free_hyp_pmds(pgd_hyp);

Note: The initialization mechanism currently relies on an SMC #0 call to the secure monitor, which was merely a fast way of getting to the hypervisor. Dave Marting and Rusty Russel have patches out to make the boot-wrapper and the kernel boot in Hyp-mode and setup a generic way for hypervisors to get access to Hyp-mode if the boot-loader allows such access.

Signed-off-by Christoffer Dall <[email protected]>

---
 arch/arm/include/asm/kvm_arm.h              |  117 +++++++++++++++++++
 arch/arm/include/asm/kvm_asm.h              |   22 +++
 arch/arm/include/asm/kvm_mmu.h              |   37 ++++++
 arch/arm/include/asm/pgtable-3level-hwdef.h |    4 +
 arch/arm/include/asm/pgtable-3level.h       |    4 +
 arch/arm/include/asm/pgtable.h              |    1 
 arch/arm/kvm/arm.c                          |  167 +++++++++++++++++++++++++++
 arch/arm/kvm/exports.c                      |   15 ++
 arch/arm/kvm/init.S                         |   99 ++++++++++++++++
 arch/arm/kvm/interrupts.S                   |   47 +++++++
 arch/arm/kvm/mmu.c                          |  170 +++++++++++++++++++++++++++
 mm/memory.c                                 |    2 
 12 files changed, 685 insertions(+)
 create mode 100644 arch/arm/include/asm/kvm_arm.h
 create mode 100644 arch/arm/include/asm/kvm_mmu.h

arch/arm/include/asm/kvm_arm.h new file +117 -0

new file mode 100644
index 0000000..7f30cbd
@@ -0,0 +1,117 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ */
+
+#ifndef __KVM_ARM_H__
+#define __KVM_ARM_H__
+
+#include <asm/types.h>
+
+/* Hyp Configuration Register (HCR) bits */
+#define HCR_TGE		(1 << 27)
+#define HCR_TVM		(1 << 26)
+#define HCR_TTLB	(1 << 25)
+#define HCR_TPU		(1 << 24)
+#define HCR_TPC		(1 << 23)
+#define HCR_TSW		(1 << 22)
+#define HCR_TAC		(1 << 21)
+#define HCR_TIDCP	(1 << 20)
+#define HCR_TSC		(1 << 19)
+#define HCR_TID3	(1 << 18)
+#define HCR_TID2	(1 << 17)
+#define HCR_TID1	(1 << 16)
+#define HCR_TID0	(1 << 15)
+#define HCR_TWE		(1 << 14)
+#define HCR_TWI		(1 << 13)
+#define HCR_DC		(1 << 12)
+#define HCR_BSU		(3 << 10)
+#define HCR_BSU_IS	(1 << 10)
+#define HCR_FB		(1 << 9)
+#define HCR_VA		(1 << 8)
+#define HCR_VI		(1 << 7)
+#define HCR_VF		(1 << 6)
+#define HCR_AMO		(1 << 5)
+#define HCR_IMO		(1 << 4)
+#define HCR_FMO		(1 << 3)
+#define HCR_PTW		(1 << 2)
+#define HCR_SWIO	(1 << 1)
+#define HCR_VM		1
+
+/*
+ * The bits we set in HCR:
+ * TAC:		Trap ACTLR
+ * TSC:		Trap SMC
+ * TWI:		Trap WFI
+ * BSU_IS:	Upgrade barriers to the inner shareable domain
+ * FB:		Force broadcast of all maintainance operations
+ * AMO:		Override CPSR.A and enable signaling with VA
+ * IMO:		Override CPSR.I and enable signaling with VI
+ * FMO:		Override CPSR.F and enable signaling with VF
+ * SWIO:	Turn set/way invalidates into set/way clean+invalidate
+ */
+#define HCR_GUEST_MASK (HCR_TSC | HCR_TWI | HCR_VM | HCR_BSU_IS | HCR_FB | \
+			HCR_AMO | HCR_IMO | HCR_FMO | HCR_FMO | HCR_SWIO)
+
+/* Hyp System Control Register (HSCTLR) bits */
+#define HSCTLR_TE	(1 << 30)
+#define HSCTLR_EE	(1 << 25)
+#define HSCTLR_FI	(1 << 21)
+#define HSCTLR_WXN	(1 << 19)
+#define HSCTLR_I	(1 << 12)
+#define HSCTLR_C	(1 << 2)
+#define HSCTLR_A	(1 << 1)
+#define HSCTLR_M	1
+#define HSCTLR_MASK	(HSCTLR_M | HSCTLR_A | HSCTLR_C | HSCTLR_I | \
+			 HSCTLR_WXN | HSCTLR_FI | HSCTLR_EE | HSCTLR_TE)
+
+/* TTBCR and HTCR Registers bits */
+#define TTBCR_EAE	(1 << 31)
+#define TTBCR_IMP	(1 << 30)
+#define TTBCR_SH1	(3 << 28)
+#define TTBCR_ORGN1	(3 << 26)
+#define TTBCR_IRGN1	(3 << 24)
+#define TTBCR_EPD1	(1 << 23)
+#define TTBCR_A1	(1 << 22)
+#define TTBCR_T1SZ	(3 << 16)
+#define TTBCR_SH0	(3 << 12)
+#define TTBCR_ORGN0	(3 << 10)
+#define TTBCR_IRGN0	(3 << 8)
+#define TTBCR_EPD0	(1 << 7)
+#define TTBCR_T0SZ	3
+#define HTCR_MASK	(TTBCR_T0SZ | TTBCR_IRGN0 | TTBCR_ORGN0 | TTBCR_SH0)
+
+
+/* Virtualization Translation Control Register (VTCR) bits */
+#define VTCR_SH0	(3 << 12)
+#define VTCR_ORGN0	(3 << 10)
+#define VTCR_IRGN0	(3 << 8)
+#define VTCR_SL0	(3 << 6)
+#define VTCR_S		(1 << 4)
+#define VTCR_T0SZ	3
+#define VTCR_MASK	(VTCR_SH0 | VTCR_ORGN0 | VTCR_IRGN0 | VTCR_SL0 | \
+			 VTCR_S | VTCR_T0SZ | VTCR_MASK)
+#define VTCR_HTCR_SH	(VTCR_SH0 | VTCR_ORGN0 | VTCR_IRGN0)
+#define VTCR_SL_L2	0		/* Starting-level: 2 */
+#define VTCR_SL_L1	(1 << 6)	/* Starting-level: 1 */
+#define VTCR_GUEST_SL	VTCR_SL_L1
+#define VTCR_GUEST_T0SZ	0
+#if VTCR_GUEST_SL == 0
+#define VTTBR_X		(14 - VTCR_GUEST_T0SZ)
+#else
+#define VTTBR_X		(5 - VTCR_GUEST_T0SZ)
+#endif
+
+
+#endif /* __KVM_ARM_H__ */

arch/arm/include/asm/kvm_asm.h +22 -0

index c3d4458..69afdf3 100644
@@ -24,5 +24,27 @@
 #define ARM_EXCEPTION_DATA_ABORT  4
 #define ARM_EXCEPTION_IRQ	  5
 #define ARM_EXCEPTION_FIQ	  6
+#define ARM_EXCEPTION_HVC	  7
+
+/*
+ * SMC Hypervisor API call number
+ */
+#define SMCHYP_HVBAR_W 0xfffffff0
+
+#ifndef __ASSEMBLY__
+struct kvm_vcpu;
+
+extern char __kvm_hyp_init[];
+extern char __kvm_hyp_init_end[];
+
+extern char __kvm_hyp_vector[];
+
+extern char __kvm_hyp_code_start[];
+extern char __kvm_hyp_code_end[];
+
+extern void __kvm_flush_vm_context(void);
+
+extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+#endif
 
 #endif /* __ARM_KVM_ASM_H__ */

arch/arm/include/asm/kvm_mmu.h new file +37 -0

new file mode 100644
index 0000000..1aa1af4
@@ -0,0 +1,37 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ */
+
+#ifndef __ARM_KVM_MMU_H__
+#define __ARM_KVM_MMU_H__
+
+/*
+ * The architecture supports 40-bit IPA as input to the 2nd stage translations
+ * and PTRS_PER_PGD2 could therefore be 1024.
+ *
+ * To save a bit of memory and to avoid alignment issues we assume 39-bit IPA
+ * for now, but remember that the level-1 table must be aligned to its size.
+ */
+#define PTRS_PER_PGD2	512
+#define PGD2_ORDER	get_order(PTRS_PER_PGD2 * sizeof(pgd_t))
+
+int create_hyp_mappings(void *from, void *to);
+void free_hyp_pmds(void);
+
+int kvm_hyp_pgd_alloc(void);
+pgd_t *kvm_hyp_pgd_get(void);
+void kvm_hyp_pgd_free(void);
+
+#endif /* __ARM_KVM_MMU_H__ */

arch/arm/include/asm/pgtable-3level-hwdef.h +4 -0

index a2d404e..18f5cef 100644
@@ -32,6 +32,9 @@
 #define PMD_TYPE_SECT		(_AT(pmdval_t, 1) << 0)
 #define PMD_BIT4		(_AT(pmdval_t, 0))
 #define PMD_DOMAIN(x)		(_AT(pmdval_t, 0))
+#define PMD_APTABLE_SHIFT	(61)
+#define PMD_APTABLE		(_AT(pgdval_t, 3) << PGD_APTABLE_SHIFT)
+#define PMD_PXNTABLE		(_AT(pgdval_t, 1) << 59)
 
 /*
  *   - section

@@ -41,6 +44,7 @@
 #define PMD_SECT_S		(_AT(pmdval_t, 3) << 8)
 #define PMD_SECT_AF		(_AT(pmdval_t, 1) << 10)
 #define PMD_SECT_nG		(_AT(pmdval_t, 1) << 11)
+#define PMD_SECT_PXN		(_AT(pmdval_t, 1) << 53)
 #define PMD_SECT_XN		(_AT(pmdval_t, 1) << 54)
 #define PMD_SECT_AP_WRITE	(_AT(pmdval_t, 0))
 #define PMD_SECT_AP_READ	(_AT(pmdval_t, 0))

arch/arm/include/asm/pgtable-3level.h +4 -0

index b249035..1169a8a 100644
@@ -107,6 +107,10 @@
 #define pud_none(pud)		(!pud_val(pud))
 #define pud_bad(pud)		(!(pud_val(pud) & 2))
 #define pud_present(pud)	(pud_val(pud))
+#define pmd_table(pmd)		((pmd_val(pmd) & PMD_TYPE_MASK) == \
+						 PMD_TYPE_TABLE)
+#define pmd_sect(pmd)		((pmd_val(pmd) & PMD_TYPE_MASK) == \
+						 PMD_TYPE_SECT)
 
 #define pud_clear(pudp)			\
 	do {				\

arch/arm/include/asm/pgtable.h +1 -0

index c7bd809..4b72287 100644
@@ -82,6 +82,7 @@ extern pgprot_t		pgprot_kernel;
 #define PAGE_READONLY_EXEC	_MOD_PROT(pgprot_user, L_PTE_USER | L_PTE_RDONLY)
 #define PAGE_KERNEL		_MOD_PROT(pgprot_kernel, L_PTE_XN)
 #define PAGE_KERNEL_EXEC	pgprot_kernel
+#define PAGE_HYP		_MOD_PROT(pgprot_kernel, L_PTE_USER)
 
 #define __PAGE_NONE		__pgprot(_L_PTE_DEFAULT | L_PTE_RDONLY | L_PTE_XN)
 #define __PAGE_SHARED		__pgprot(_L_PTE_DEFAULT | L_PTE_USER | L_PTE_XN)

arch/arm/kvm/arm.c +137 -0

index 5992d90..4c61d3c 100644
@@ -31,6 +31,12 @@
 #include <asm/uaccess.h>
 #include <asm/ptrace.h>
 #include <asm/mman.h>
+#include <asm/tlbflush.h>
+#include <asm/kvm_arm.h>
+#include <asm/kvm_asm.h>
+#include <asm/kvm_mmu.h>
+
+static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
 
 int kvm_arch_hardware_enable(void *garbage)
 {

@@ -255,13 +261,174 @@ long kvm_arch_vm_ioctl(struct file *filp,
 	return -EINVAL;
 }
 
+static void cpu_set_vector(void *vector)
+{
+	unsigned long vector_ptr;
+	unsigned long smc_hyp_nr;
+
+	vector_ptr = (unsigned long)vector;
+	smc_hyp_nr = SMCHYP_HVBAR_W;
+
+	/*
+	 * Set the HVBAR
+	 */
+	asm volatile (
+		"mov	r0, %[vector_ptr]\n\t"
+		"mov	r7, %[smc_hyp_nr]\n\t"
+		"smc	#0\n\t" : :
+		[vector_ptr] "r" (vector_ptr),
+		[smc_hyp_nr] "r" (smc_hyp_nr) :
+		"r0", "r7");
+}
+
+static void cpu_init_hyp_mode(void *vector)
+{
+	unsigned long pgd_ptr;
+	unsigned long hyp_stack_ptr;
+	unsigned long stack_page;
+
+	cpu_set_vector(vector);
+
+	pgd_ptr = virt_to_phys(kvm_hyp_pgd_get());
+	stack_page = __get_cpu_var(kvm_arm_hyp_stack_page);
+	hyp_stack_ptr = stack_page + PAGE_SIZE;
+
+	/*
+	 * Call initialization code
+	 */
+	asm volatile (
+		"mov	r0, %[pgd_ptr]\n\t"
+		"mov	r1, %[hyp_stack_ptr]\n\t"
+		"hvc	#0\n\t" : :
+		[pgd_ptr] "r" (pgd_ptr),
+		[hyp_stack_ptr] "r" (hyp_stack_ptr) :
+		"r0", "r1");
+}
+
+/**
+ * Inits Hyp-mode on all online CPUs
+ */
+static int init_hyp_mode(void)
+{
+	phys_addr_t init_phys_addr, init_end_phys_addr;
+	int cpu;
+	int err = 0;
+
+	/*
+	 * Allocate stack pages for Hypervisor-mode
+	 */
+	for_each_possible_cpu(cpu) {
+		unsigned long stack_page;
+
+		stack_page = __get_free_page(GFP_KERNEL);
+		if (!stack_page) {
+			err = -ENOMEM;
+			goto out_free_stack_pages;
+		}
+
+		per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page;
+	}
+
+	/*
+	 * Allocate Hyp level-1 page table
+	 */
+	err = kvm_hyp_pgd_alloc();
+	if (err)
+		goto out_free_stack_pages;
+
+	init_phys_addr = virt_to_phys(__kvm_hyp_init);
+	init_end_phys_addr = virt_to_phys(__kvm_hyp_init_end);
+	BUG_ON(init_phys_addr & 0x1f);
+
+	/*
+	 * Create identity mapping for the init code.
+	 */
+	hyp_idmap_add(kvm_hyp_pgd_get(),
+		      (unsigned long)init_phys_addr,
+		      (unsigned long)init_end_phys_addr);
+
+	/*
+	 * Execute the init code on each CPU.
+	 *
+	 * Note: The stack is not mapped yet, so don't do anything else than
+	 * initializing the hypervisor mode on each CPU using a local stack
+	 * space for temporary storage.
+	 */
+	for_each_online_cpu(cpu) {
+		smp_call_function_single(cpu, cpu_init_hyp_mode,
+					 (void *)(long)init_phys_addr, 1);
+	}
+
+	/*
+	 * Unmap the identity mapping
+	 */
+	hyp_idmap_del(kvm_hyp_pgd_get(),
+		      (unsigned long)init_phys_addr,
+		      (unsigned long)init_end_phys_addr);
+
+	/*
+	 * Map the Hyp-code called directly from the host
+	 */
+	err = create_hyp_mappings(__kvm_hyp_code_start, __kvm_hyp_code_end);
+	if (err) {
+		kvm_err("Cannot map world-switch code\n");
+		goto out_free_mappings;
+	}
+
+	/*
+	 * Map the Hyp stack pages
+	 */
+	for_each_possible_cpu(cpu) {
+		char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu);
+		err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE);
+
+		if (err) {
+			kvm_err("Cannot map hyp stack\n");
+			goto out_free_mappings;
+		}
+	}
+
+	/*
+	 * Set the HVBAR to the virtual kernel address
+	 */
+	for_each_online_cpu(cpu)

Quoted 445 lines · Christoffer Dall

Sets up the required registers to run code in HYP-mode from the kernel.

By setting the HVBAR the kernel can execute code in Hyp-mode with the MMU disabled. The HVBAR initially points to initialization code, which initializes other Hyp-mode registers and enables the MMU for Hyp-mode. Afterwards, the HVBAR is changed to point to KVM Hyp vectors used to catch guest faults and to switch to Hyp mode to perform a world-switch into a KVM guest.

Also provides memory mapping code to map required code pages and data structures accessed in Hyp mode at the same virtual address as the host kernel virtual addresses, but which conforms to the architectural requirements for translations in Hyp mode. This interface is added in arch/arm/kvm/arm_mmu.c and is comprised of:
 - create_hyp_mappings(hyp_pgd, start, end);
 - free_hyp_pmds(pgd_hyp);

Note: The initialization mechanism currently relies on an SMC #0 call to the secure monitor, which was merely a fast way of getting to the hypervisor. Dave Marting and Rusty Russel have patches out to make the boot-wrapper and the kernel boot in Hyp-mode and setup a generic way for hypervisors to get access to Hyp-mode if the boot-loader allows such access.

Signed-off-by Christoffer Dall <[email protected]>

---

 arch/arm/include/asm/kvm_arm.h              |  117 +++++++++++++++++++
 arch/arm/include/asm/kvm_asm.h              |   22 +++
 arch/arm/include/asm/kvm_mmu.h              |   37 ++++++

 arch/arm/include/asm/pgtable-3level-hwdef.h |    4 + arch/arm/include/asm/pgtable-3level.h       |    4 + arch/arm/include/asm/pgtable.h              |    1 arch/arm/kvm/arm.c                          |  167 +++++++++++++++++++++++++++ arch/arm/kvm/exports.c                      |   15 ++ arch/arm/kvm/init.S                         |   99 ++++++++++++++++ arch/arm/kvm/interrupts.S                   |   47 +++++++ arch/arm/kvm/mmu.c                          |  170 +++++++++++++++++++++++++++ mm/memory.c                                 |    2

 12 files changed, 685 insertions(+)

 create mode 100644 arch/arm/include/asm/kvm_arm.h create mode 100644 arch/arm/include/asm/kvm_mmu.h

arch/arm/include/asm/kvm_arm.h new file +117 -0

new file mode 100644
index 0000000..7f30cbd
@@ -0,0 +1,117 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ */
+
+#ifndef __KVM_ARM_H__
+#define __KVM_ARM_H__
+
+#include <asm/types.h>
+
+/* Hyp Configuration Register (HCR) bits */
+#define HCR_TGE              (1 << 27)
+#define HCR_TVM              (1 << 26)
+#define HCR_TTLB     (1 << 25)
+#define HCR_TPU              (1 << 24)
+#define HCR_TPC              (1 << 23)
+#define HCR_TSW              (1 << 22)
+#define HCR_TAC              (1 << 21)
+#define HCR_TIDCP    (1 << 20)
+#define HCR_TSC              (1 << 19)
+#define HCR_TID3     (1 << 18)
+#define HCR_TID2     (1 << 17)
+#define HCR_TID1     (1 << 16)
+#define HCR_TID0     (1 << 15)
+#define HCR_TWE              (1 << 14)
+#define HCR_TWI              (1 << 13)
+#define HCR_DC               (1 << 12)
+#define HCR_BSU              (3 << 10)
+#define HCR_BSU_IS   (1 << 10)
+#define HCR_FB               (1 << 9)
+#define HCR_VA               (1 << 8)
+#define HCR_VI               (1 << 7)
+#define HCR_VF               (1 << 6)
+#define HCR_AMO              (1 << 5)
+#define HCR_IMO              (1 << 4)
+#define HCR_FMO              (1 << 3)
+#define HCR_PTW              (1 << 2)
+#define HCR_SWIO     (1 << 1)
+#define HCR_VM               1
+
+/*
+ * The bits we set in HCR:
+ * TAC:              Trap ACTLR
+ * TSC:              Trap SMC
+ * TWI:              Trap WFI
+ * BSU_IS:   Upgrade barriers to the inner shareable domain
+ * FB:               Force broadcast of all maintainance operations
+ * AMO:              Override CPSR.A and enable signaling with VA
+ * IMO:              Override CPSR.I and enable signaling with VI
+ * FMO:              Override CPSR.F and enable signaling with VF
+ * SWIO:     Turn set/way invalidates into set/way clean+invalidate
+ */
+#define HCR_GUEST_MASK (HCR_TSC | HCR_TWI | HCR_VM | HCR_BSU_IS | HCR_FB | \
+                     HCR_AMO | HCR_IMO | HCR_FMO | HCR_FMO | HCR_SWIO)
+
+/* Hyp System Control Register (HSCTLR) bits */
+#define HSCTLR_TE    (1 << 30)
+#define HSCTLR_EE    (1 << 25)
+#define HSCTLR_FI    (1 << 21)
+#define HSCTLR_WXN   (1 << 19)
+#define HSCTLR_I     (1 << 12)
+#define HSCTLR_C     (1 << 2)
+#define HSCTLR_A     (1 << 1)
+#define HSCTLR_M     1
+#define HSCTLR_MASK  (HSCTLR_M | HSCTLR_A | HSCTLR_C | HSCTLR_I | \
+                      HSCTLR_WXN | HSCTLR_FI | HSCTLR_EE | HSCTLR_TE)
+
+/* TTBCR and HTCR Registers bits */
+#define TTBCR_EAE    (1 << 31)
+#define TTBCR_IMP    (1 << 30)
+#define TTBCR_SH1    (3 << 28)
+#define TTBCR_ORGN1  (3 << 26)
+#define TTBCR_IRGN1  (3 << 24)
+#define TTBCR_EPD1   (1 << 23)
+#define TTBCR_A1     (1 << 22)
+#define TTBCR_T1SZ   (3 << 16)
+#define TTBCR_SH0    (3 << 12)
+#define TTBCR_ORGN0  (3 << 10)
+#define TTBCR_IRGN0  (3 << 8)
+#define TTBCR_EPD0   (1 << 7)
+#define TTBCR_T0SZ   3
+#define HTCR_MASK    (TTBCR_T0SZ | TTBCR_IRGN0 | TTBCR_ORGN0 | TTBCR_SH0)
+
+
+/* Virtualization Translation Control Register (VTCR) bits */
+#define VTCR_SH0     (3 << 12)
+#define VTCR_ORGN0   (3 << 10)
+#define VTCR_IRGN0   (3 << 8)
+#define VTCR_SL0     (3 << 6)
+#define VTCR_S               (1 << 4)
+#define VTCR_T0SZ    3
+#define VTCR_MASK    (VTCR_SH0 | VTCR_ORGN0 | VTCR_IRGN0 | VTCR_SL0 | \
+                      VTCR_S | VTCR_T0SZ | VTCR_MASK)
+#define VTCR_HTCR_SH (VTCR_SH0 | VTCR_ORGN0 | VTCR_IRGN0)
+#define VTCR_SL_L2   0               /* Starting-level: 2 */
+#define VTCR_SL_L1   (1 << 6)        /* Starting-level: 1 */
+#define VTCR_GUEST_SL        VTCR_SL_L1
+#define VTCR_GUEST_T0SZ      0
+#if VTCR_GUEST_SL == 0
+#define VTTBR_X              (14 - VTCR_GUEST_T0SZ)
+#else
+#define VTTBR_X              (5 - VTCR_GUEST_T0SZ)
+#endif
+
+
+#endif /* __KVM_ARM_H__ */

arch/arm/include/asm/kvm_asm.h +22 -0

index c3d4458..69afdf3 100644
@@ -24,5 +24,27 @@
  #define ARM_EXCEPTION_DATA_ABORT  4
  #define ARM_EXCEPTION_IRQ      5
  #define ARM_EXCEPTION_FIQ      6
+#define ARM_EXCEPTION_HVC      7
+
+/*
+ * SMC Hypervisor API call number
+ */
+#define SMCHYP_HVBAR_W 0xfffffff0
+
+#ifndef __ASSEMBLY__
+struct kvm_vcpu;
+
+extern char __kvm_hyp_init[];
+extern char __kvm_hyp_init_end[];
+
+extern char __kvm_hyp_vector[];
+
+extern char __kvm_hyp_code_start[];
+extern char __kvm_hyp_code_end[];
+
+extern void __kvm_flush_vm_context(void);
+
+extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
+#endif
 
  #endif /* __ARM_KVM_ASM_H__ */

arch/arm/include/asm/kvm_mmu.h new file +37 -0

new file mode 100644
index 0000000..1aa1af4
@@ -0,0 +1,37 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ *
+ */
+
+#ifndef __ARM_KVM_MMU_H__
+#define __ARM_KVM_MMU_H__
+
+/*
+ * The architecture supports 40-bit IPA as input to the 2nd stage translations
+ * and PTRS_PER_PGD2 could therefore be 1024.
+ *
+ * To save a bit of memory and to avoid alignment issues we assume 39-bit IPA
+ * for now, but remember that the level-1 table must be aligned to its size.
+ */
+#define PTRS_PER_PGD2        512
+#define PGD2_ORDER   get_order(PTRS_PER_PGD2 * sizeof(pgd_t))
+
+int create_hyp_mappings(void *from, void *to);
+void free_hyp_pmds(void);
+
+int kvm_hyp_pgd_alloc(void);
+pgd_t *kvm_hyp_pgd_get(void);
+void kvm_hyp_pgd_free(void);
+
+#endif /* __ARM_KVM_MMU_H__ */

arch/arm/include/asm/pgtable-3level-hwdef.h +4 -0

index a2d404e..18f5cef 100644
@@ -32,6 +32,9 @@
  #define PMD_TYPE_SECT                (_AT(pmdval_t, 1) << 0)
  #define PMD_BIT4             (_AT(pmdval_t, 0))
  #define PMD_DOMAIN(x)                (_AT(pmdval_t, 0))
+#define PMD_APTABLE_SHIFT    (61)
+#define PMD_APTABLE          (_AT(pgdval_t, 3) << PGD_APTABLE_SHIFT)
+#define PMD_PXNTABLE         (_AT(pgdval_t, 1) << 59)
 
  /*
   *   - section

@@ -41,6 +44,7 @@
  #define PMD_SECT_S           (_AT(pmdval_t, 3) << 8)
  #define PMD_SECT_AF          (_AT(pmdval_t, 1) << 10)
  #define PMD_SECT_nG          (_AT(pmdval_t, 1) << 11)
+#define PMD_SECT_PXN         (_AT(pmdval_t, 1) << 53)
  #define PMD_SECT_XN          (_AT(pmdval_t, 1) << 54)
  #define PMD_SECT_AP_WRITE    (_AT(pmdval_t, 0))
  #define PMD_SECT_AP_READ     (_AT(pmdval_t, 0))

arch/arm/include/asm/pgtable-3level.h +4 -0

index b249035..1169a8a 100644
@@ -107,6 +107,10 @@
  #define pud_none(pud)                (!pud_val(pud))
  #define pud_bad(pud)         (!(pud_val(pud) & 2))
  #define pud_present(pud)     (pud_val(pud))
+#define pmd_table(pmd)               ((pmd_val(pmd) & PMD_TYPE_MASK) == \
+                                              PMD_TYPE_TABLE)
+#define pmd_sect(pmd)                ((pmd_val(pmd) & PMD_TYPE_MASK) == \
+                                              PMD_TYPE_SECT)
 
  #define pud_clear(pudp)                      \
       do {                            \

arch/arm/include/asm/pgtable.h +1 -0

index c7bd809..4b72287 100644
@@ -82,6 +82,7 @@ extern pgprot_t             pgprot_kernel;
  #define PAGE_READONLY_EXEC   _MOD_PROT(pgprot_user, L_PTE_USER | L_PTE_RDONLY)
  #define PAGE_KERNEL          _MOD_PROT(pgprot_kernel, L_PTE_XN)
  #define PAGE_KERNEL_EXEC     pgprot_kernel
+#define PAGE_HYP             _MOD_PROT(pgprot_kernel, L_PTE_USER)
 
  #define __PAGE_NONE          __pgprot(_L_PTE_DEFAULT | L_PTE_RDONLY | L_PTE_XN)
  #define __PAGE_SHARED                __pgprot(_L_PTE_DEFAULT | L_PTE_USER | L_PTE_XN)

arch/arm/kvm/arm.c +141 -0

index 5992d90..4c61d3c 100644
@@ -31,6 +31,12 @@
  #include <asm/uaccess.h>
  #include <asm/ptrace.h>
  #include <asm/mman.h>
+#include <asm/tlbflush.h>
+#include <asm/kvm_arm.h>
+#include <asm/kvm_asm.h>
+#include <asm/kvm_mmu.h>
+
+static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
 
  int kvm_arch_hardware_enable(void *garbage)
  {

@@ -255,13 +261,174 @@ long kvm_arch_vm_ioctl(struct file *filp,
       return -EINVAL;
  }
 
+static void cpu_set_vector(void *vector)
+{
+     unsigned long vector_ptr;
+     unsigned long smc_hyp_nr;
+
+     vector_ptr = (unsigned long)vector;
+     smc_hyp_nr = SMCHYP_HVBAR_W;
+
+     /*
+      * Set the HVBAR
+      */
+     asm volatile (
+             "mov    r0, %[vector_ptr]\n\t"
+             "mov    r7, %[smc_hyp_nr]\n\t"
+             "smc    #0\n\t" : :
+             [vector_ptr] "r" (vector_ptr),
+             [smc_hyp_nr] "r" (smc_hyp_nr) :
+             "r0", "r7");
+}
+
+static void cpu_init_hyp_mode(void *vector)
+{
+     unsigned long pgd_ptr;
+     unsigned long hyp_stack_ptr;
+     unsigned long stack_page;
+
+     cpu_set_vector(vector);
+
+     pgd_ptr = virt_to_phys(kvm_hyp_pgd_get());
+     stack_page = __get_cpu_var(kvm_arm_hyp_stack_page);
+     hyp_stack_ptr = stack_page + PAGE_SIZE;
+
+     /*
+      * Call initialization code
+      */
+     asm volatile (
+             "mov    r0, %[pgd_ptr]\n\t"
+             "mov    r1, %[hyp_stack_ptr]\n\t"
+             "hvc    #0\n\t" : :
+             [pgd_ptr] "r" (pgd_ptr),
+             [hyp_stack_ptr] "r" (hyp_stack_ptr) :
+             "r0", "r1");
+}
+
+/**
+ * Inits Hyp-mode on all online CPUs
+ */
+static int init_hyp_mode(void)
+{
+     phys_addr_t init_phys_addr, init_end_phys_addr;
+     int cpu;
+     int err = 0;
+
+     /*
+      * Allocate stack pages for Hypervisor-mode
+      */
+     for_each_possible_cpu(cpu) {
+             unsigned long stack_page;
+
+             stack_page = __get_free_page(GFP_KERNEL);
+             if (!stack_page) {
+                     err = -ENOMEM;
+                     goto out_free_stack_pages;
+             }
+
+             per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page;
+     }
+
+     /*
+      * Allocate Hyp level-1 page table
+      */
+     err = kvm_hyp_pgd_alloc();
+     if (err)
+             goto out_free_stack_pages;
+
+     init_phys_addr = virt_to_phys(__kvm_hyp_init);
+     init_end_phys_addr = virt_to_phys(__kvm_hyp_init_end);
+     BUG_ON(init_phys_addr & 0x1f);
+
+     /*
+      * Create identity mapping for the init code.
+      */
+     hyp_idmap_add(kvm_hyp_pgd_get(),
+                   (unsigned long)init_phys_addr,
+                   (unsigned long)init_end_phys_addr);
+
+     /*
+      * Execute the init code on each CPU.
+      *
+      * Note: The stack is not mapped yet, so don't do anything else than
+      * initializing the hypervisor mode on each CPU using a local stack
+      * space for temporary storage.
+      */
+     for_each_online_cpu(cpu) {
+             smp_call_function_single(cpu, cpu_init_hyp_mode,
+                                      (void *)(long)init_phys_addr, 1);
+     }
+
+     /*
+      * Unmap the identity mapping
+      */
+     hyp_idmap_del(kvm_hyp_pgd_get(),
+                   (unsigned long)init_phys_addr,
+                   (unsigned long)init_end_phys_addr);
+
+     /*
+      * Map the Hyp-code called directly from the host
+      */
+     err = create_hyp_mappings(__kvm_hyp_code_start, __kvm_hyp_code_end);
+     if (err) {
+             kvm_err("Cannot map world-switch code\n");
+             goto out_free_mappings;
+     }
+
+     /*
+      * Map the Hyp stack pages
+      */
+     for_each_possible_cpu(cpu) {
+             char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu);
+             err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE);
+
+             if (err) {
+                     kvm_err("Cannot map hyp stack\n");
+                     goto out_free_mappings;
+             }
+     }
+
+     /*
+      * Set the HVBAR to the virtual kernel address
+      */
+     for_each_online_cpu(cpu)
+             smp_call_function_single(cpu, cpu_set_vector,
+                                      __kvm_hyp_vector, 1);
+
+     return 0;

arch/arm/kvm/arm.c long kvm_arch_vm_ioctl(struct file *filp, · line 404

Quoted 6 lines

arch/arm/kvm/arm.c +6 -0

+out_free_mappings:
+     free_hyp_pmds();
+     kvm_hyp_pgd_free();
+out_free_stack_pages:
+     for_each_possible_cpu(cpu)
+             free_page(per_cpu(kvm_arm_hyp_stack_page, cpu));

should assign per_cpu(kvm_arm_hyp_stack_page, cpu) to NULL.

Is there CPU hotplug support on ARM?

Quoted 2 lines

should assign per_cpu(kvm_arm_hyp_stack_page, cpu) to NULL.

why? this is run as part of the init code and thus the only way it could ever run again would be to have the module unloaded in which case the variable would be re-initialized to zero as per the static declaration, no?

From: Christoffer Dall <[email protected]>

This commit introduces the framework for guest memory management through the use of 2nd stage translation. Each VM has a pointer to a level-1 table (the pgd field in struct kvm_arch) which is used for the 2nd stage translations. Entries are added when handling guest faults (later patch) and the table itself can be allocated and freed through the following functions implemented in arch/arm/kvm/arm_mmu.c:
- kvm_alloc_stage2_pgd(struct kvm *kvm);
- kvm_free_stage2_pgd(struct kvm *kvm);

Further, each entry in TLBs and caches are tagged with a VMID identifier in addition to ASIDs. The VMIDs are assigned consecutively to VMs in the order that VMs are executed, and caches and tlbs are invalidated when the VMID space has been used to allow for more than 255 simultaenously running guests.

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is freed in kvm_arch_destroy_vm(). Both functions are called from the main KVM code.

Signed-off-by Christoffer Dall <[email protected]>

---
 arch/arm/include/asm/kvm_arm.h |    2 -
 arch/arm/include/asm/kvm_mmu.h |    5 ++
 arch/arm/kvm/arm.c             |   65 ++++++++++++++++++++++---
 arch/arm/kvm/mmu.c             |  103 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 166 insertions(+), 9 deletions(-)

arch/arm/include/asm/kvm_arm.h +1 -1

index 7f30cbd..257242f 100644
@@ -62,7 +62,7 @@
  * SWIO:	Turn set/way invalidates into set/way clean+invalidate
  */
 #define HCR_GUEST_MASK (HCR_TSC | HCR_TWI | HCR_VM | HCR_BSU_IS | HCR_FB | \
-			HCR_AMO | HCR_IMO | HCR_FMO | HCR_FMO | HCR_SWIO)
+			HCR_TAC | HCR_AMO | HCR_IMO | HCR_FMO | HCR_SWIO)
 
 /* Hyp System Control Register (HSCTLR) bits */
 #define HSCTLR_TE	(1 << 30)

arch/arm/include/asm/kvm_mmu.h +5 -0

index 1aa1af4..d95662eb 100644
@@ -34,4 +34,9 @@ int kvm_hyp_pgd_alloc(void);
 pgd_t *kvm_hyp_pgd_get(void);
 void kvm_hyp_pgd_free(void);
 
+int kvm_alloc_stage2_pgd(struct kvm *kvm);
+void kvm_free_stage2_pgd(struct kvm *kvm);
+
+int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
+
 #endif /* __ARM_KVM_MMU_H__ */

arch/arm/kvm/arm.c +57 -8

index efe130c..81babe9 100644
@@ -38,6 +38,13 @@
 
 static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
 
+/* The VMID used in the VTTBR */
+#define VMID_BITS               8
+#define VMID_MASK               ((1 << VMID_BITS) - 1)
+#define VMID_FIRST_GENERATION	(1 << VMID_BITS)
+static u64 next_vmid;		/* The next available VMID in the sequence */
+DEFINE_SPINLOCK(kvm_vmid_lock);
+
 int kvm_arch_hardware_enable(void *garbage)
 {
 	return 0;

@@ -70,14 +77,6 @@ void kvm_arch_sync_events(struct kvm *kvm)
 {
 }
 
-int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
-{
-	if (type)
-		return -EINVAL;
-
-	return 0;
-}
-
 int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)
 {
 	return VM_FAULT_SIGBUS;

@@ -93,10 +92,46 @@ int kvm_arch_create_memslot(struct kvm_memory_slot *slot, unsigned long npages)
 	return 0;
 }
 
+/**
+ * kvm_arch_init_vm - initializes a VM data structure
+ * @kvm:	pointer to the KVM struct
+ */
+int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
+{
+	int ret = 0;
+
+	if (type)
+		return -EINVAL;
+
+	ret = kvm_alloc_stage2_pgd(kvm);
+	if (ret)
+		goto out_fail_alloc;
+	mutex_init(&kvm->arch.pgd_mutex);
+
+	ret = create_hyp_mappings(kvm, kvm + 1);
+	if (ret)
+		goto out_free_stage2_pgd;
+
+	/* Mark the initial VMID invalid */
+	kvm->arch.vmid = 0;
+
+	return ret;
+out_free_stage2_pgd:
+	kvm_free_stage2_pgd(kvm);
+out_fail_alloc:
+	return ret;
+}
+
+/**
+ * kvm_arch_destroy_vm - destroy the VM data structure
+ * @kvm:	pointer to the KVM struct
+ */
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	int i;
 
+	kvm_free_stage2_pgd(kvm);
+
 	for (i = 0; i < KVM_MAX_VCPUS; ++i) {
 		if (kvm->vcpus[i]) {
 			kvm_arch_vcpu_free(kvm->vcpus[i]);

@@ -172,6 +207,10 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
 	if (err)
 		goto free_vcpu;
 
+	err = create_hyp_mappings(vcpu, vcpu + 1);
+	if (err)
+		goto free_vcpu;
+
 	return vcpu;
 free_vcpu:
 	kmem_cache_free(kvm_vcpu_cache, vcpu);

@@ -181,6 +220,7 @@ out:
 
 void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
 {
+	kmem_cache_free(kvm_vcpu_cache, vcpu);
 }
 
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)

@@ -416,6 +456,15 @@ int kvm_arch_init(void *opaque)
 	if (err)
 		goto out_err;
 
+	/*
+	 * The upper 56 bits of VMIDs are used to identify the generation
+	 * counter, so VMIDs initialized to 0, having generation == 0, will
+	 * never be considered valid and therefor a new VMID must always be
+	 * assigned. Whent he VMID generation rolls over, we start from
+	 * VMID_FIRST_GENERATION again.
+	 */
+	next_vmid = VMID_FIRST_GENERATION;
+
 	return 0;
 out_err:
 	return err;

arch/arm/kvm/mmu.c +23 -0

index a320b56a..b256540 100644
@@ -159,6 +159,109 @@ out:
 	return err;
 }
 
+/**
+ * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation.
+ * @kvm:	The KVM struct pointer for the VM.
+ *
+ * Allocates the 1st level table only of size defined by PGD2_ORDER (can
+ * support either full 40-bit input addresses or limited to 32-bit input
+ * addresses). Clears the allocated pages.
+ */
+int kvm_alloc_stage2_pgd(struct kvm *kvm)
+{
+	pgd_t *pgd;
+
+	if (kvm->arch.pgd != NULL) {
+		kvm_err("kvm_arch already initialized?\n");
+		return -EINVAL;
+	}
+
+	pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD2_ORDER);
+	if (!pgd)
+		return -ENOMEM;
+
+	memset(pgd, 0, PTRS_PER_PGD2 * sizeof(pgd_t));
+	kvm->arch.pgd = pgd;

arch/arm/kvm/mmu.c +66 -0

+next_page:
+		pte++;
+	}
+}
 
+
+static void free_stage2_ptes(pmd_t *pmd, unsigned long addr)
+{
+	unsigned int i;
+	pte_t *pte;
+	struct page *page;
+
+	for (i = 0; i < PTRS_PER_PMD; i++, addr += PMD_SIZE) {
+		BUG_ON(pmd_sect(*pmd));
+		if (!pmd_none(*pmd) && pmd_table(*pmd)) {
+			pte = pte_offset_kernel(pmd, addr);
+			free_guest_pages(pte, addr);
+			page = virt_to_page((void *)pte);
+			WARN_ON(atomic_read(&page->_count) != 1);
+			pte_free_kernel(NULL, pte);
+		}
+		pmd++;
+	}
+}
+
+/**
+ * kvm_free_stage2_pgd - free all stage-2 tables
+ * @kvm:	The KVM struct pointer for the VM.
+ *
+ * Walks the level-1 page table pointed to by kvm->arch.pgd and frees all
+ * underlying level-2 and level-3 tables before freeing the actual level-1 table
+ * and setting the struct pointer to NULL.
+ */
+void kvm_free_stage2_pgd(struct kvm *kvm)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pmd_t *pmd;
+	unsigned long long i, addr;
+
+	if (kvm->arch.pgd == NULL)
+		return;
+
+	/*
+	 * We do this slightly different than other places, since we need more
+	 * than 32 bits and for instance pgd_addr_end converts to unsigned long.
+	 */
+	addr = 0;
+	for (i = 0; i < PTRS_PER_PGD2; i++) {
+		addr = i * (unsigned long long)PGDIR_SIZE;
+		pgd = kvm->arch.pgd + i;
+		pud = pud_offset(pgd, addr);
+
+		if (pud_none(*pud))
+			continue;
+
+		BUG_ON(pud_bad(*pud));
+
+		pmd = pmd_offset(pud, addr);
+		free_stage2_ptes(pmd, addr);
+		pmd_free(NULL, pmd);
+	}
+
+	free_pages((unsigned long)kvm->arch.pgd, PGD2_ORDER);
+	kvm->arch.pgd = NULL;
+}
+
 int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {

return -EINVAL;

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 206 lines · Christoffer Dall

From: Christoffer Dall <[email protected]>

This commit introduces the framework for guest memory management through the use of 2nd stage translation. Each VM has a pointer to a level-1 table (the pgd field in struct kvm_arch) which is used for the 2nd stage translations. Entries are added when handling guest faults (later patch) and the table itself can be allocated and freed through the following functions implemented in arch/arm/kvm/arm_mmu.c:
 - kvm_alloc_stage2_pgd(struct kvm *kvm);
 - kvm_free_stage2_pgd(struct kvm *kvm);

Further, each entry in TLBs and caches are tagged with a VMID identifier in addition to ASIDs. The VMIDs are assigned consecutively to VMs in the order that VMs are executed, and caches and tlbs are invalidated when the VMID space has been used to allow for more than 255 simultaenously running guests.

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is freed in kvm_arch_destroy_vm(). Both functions are called from the main KVM code.

Signed-off-by Christoffer Dall <[email protected]>

---
 arch/arm/include/asm/kvm_arm.h |    2 - arch/arm/include/asm/kvm_mmu.h |    5 ++
 arch/arm/kvm/arm.c             |   65 ++++++++++++++++++++++--- arch/arm/kvm/mmu.c             |  103 ++++++++++++++++++++++++++++++++++++++++

 4 files changed, 166 insertions(+), 9 deletions(-)

arch/arm/include/asm/kvm_arm.h +1 -1

index 7f30cbd..257242f 100644
@@ -62,7 +62,7 @@
   * SWIO:     Turn set/way invalidates into set/way clean+invalidate
   */
  #define HCR_GUEST_MASK (HCR_TSC | HCR_TWI | HCR_VM | HCR_BSU_IS | HCR_FB | \
-                     HCR_AMO | HCR_IMO | HCR_FMO | HCR_FMO | HCR_SWIO)
+                     HCR_TAC | HCR_AMO | HCR_IMO | HCR_FMO | HCR_SWIO)
 
  /* Hyp System Control Register (HSCTLR) bits */
  #define HSCTLR_TE    (1 << 30)

arch/arm/include/asm/kvm_mmu.h +5 -0

index 1aa1af4..d95662eb 100644
@@ -34,4 +34,9 @@ int kvm_hyp_pgd_alloc(void);
  pgd_t *kvm_hyp_pgd_get(void);
  void kvm_hyp_pgd_free(void);
 
+int kvm_alloc_stage2_pgd(struct kvm *kvm);
+void kvm_free_stage2_pgd(struct kvm *kvm);
+
+int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
+
  #endif /* __ARM_KVM_MMU_H__ */

arch/arm/kvm/arm.c +47 -8

index efe130c..81babe9 100644
@@ -38,6 +38,13 @@
 
  static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
 
+/* The VMID used in the VTTBR */
+#define VMID_BITS               8
+#define VMID_MASK               ((1 << VMID_BITS) - 1)
+#define VMID_FIRST_GENERATION        (1 << VMID_BITS)
+static u64 next_vmid;                /* The next available VMID in the sequence */
+DEFINE_SPINLOCK(kvm_vmid_lock);
+
  int kvm_arch_hardware_enable(void *garbage)
  {
       return 0;

@@ -70,14 +77,6 @@ void kvm_arch_sync_events(struct kvm *kvm)
  {
  }
 
-int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
-{
-     if (type)
-             return -EINVAL;
-
-     return 0;
-}
-
  int kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf)
  {
       return VM_FAULT_SIGBUS;

@@ -93,10 +92,46 @@ int kvm_arch_create_memslot(struct kvm_memory_slot *slot, unsigned long npages)
       return 0;
  }
 
+/**
+ * kvm_arch_init_vm - initializes a VM data structure
+ * @kvm:     pointer to the KVM struct
+ */
+int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
+{
+     int ret = 0;
+
+     if (type)
+             return -EINVAL;
+
+     ret = kvm_alloc_stage2_pgd(kvm);
+     if (ret)
+             goto out_fail_alloc;
+     mutex_init(&kvm->arch.pgd_mutex);
+
+     ret = create_hyp_mappings(kvm, kvm + 1);
+     if (ret)
+             goto out_free_stage2_pgd;
+
+     /* Mark the initial VMID invalid */
+     kvm->arch.vmid = 0;
+
+     return ret;
+out_free_stage2_pgd:
+     kvm_free_stage2_pgd(kvm);
+out_fail_alloc:
+     return ret;
+}
+
+/**
+ * kvm_arch_destroy_vm - destroy the VM data structure
+ * @kvm:     pointer to the KVM struct
+ */
  void kvm_arch_destroy_vm(struct kvm *kvm)
  {
       int i;
 
+     kvm_free_stage2_pgd(kvm);
+
       for (i = 0; i < KVM_MAX_VCPUS; ++i) {
               if (kvm->vcpus[i]) {
                       kvm_arch_vcpu_free(kvm->vcpus[i]);

@@ -172,6 +207,10 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
       if (err)
               goto free_vcpu;
 
+     err = create_hyp_mappings(vcpu, vcpu + 1);
+     if (err)
+             goto free_vcpu;
+
       return vcpu;

 free_vcpu:
      kmem_cache_free(kvm_vcpu_cache, vcpu);

arch/arm/kvm/arm.c +10 -0

@@ -181,6 +220,7 @@ out:
 
  void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
  {
+     kmem_cache_free(kvm_vcpu_cache, vcpu);
  }
 
  void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)

@@ -416,6 +456,15 @@ int kvm_arch_init(void *opaque)
       if (err)
               goto out_err;
 
+     /*
+      * The upper 56 bits of VMIDs are used to identify the generation
+      * counter, so VMIDs initialized to 0, having generation == 0, will
+      * never be considered valid and therefor a new VMID must always be
+      * assigned. Whent he VMID generation rolls over, we start from
+      * VMID_FIRST_GENERATION again.
+      */
+     next_vmid = VMID_FIRST_GENERATION;
+
       return 0;

 out_err:
      return err;

arch/arm/kvm/mmu.c +26 -0

index a320b56a..b256540 100644
@@ -159,6 +159,109 @@ out:
       return err;
  }
 
+/**
+ * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation.
+ * @kvm:     The KVM struct pointer for the VM.
+ *
+ * Allocates the 1st level table only of size defined by PGD2_ORDER (can
+ * support either full 40-bit input addresses or limited to 32-bit input
+ * addresses). Clears the allocated pages.
+ */
+int kvm_alloc_stage2_pgd(struct kvm *kvm)
+{
+     pgd_t *pgd;
+
+     if (kvm->arch.pgd != NULL) {
+             kvm_err("kvm_arch already initialized?\n");
+             return -EINVAL;
+     }
+
+     pgd = (pgd_t *)__get_free_pages(GFP_KERNEL, PGD2_ORDER);
+     if (!pgd)
+             return -ENOMEM;
+
+     memset(pgd, 0, PTRS_PER_PGD2 * sizeof(pgd_t));
+     kvm->arch.pgd = pgd;
+
+     return 0;
+}

arch/arm/kvm/mmu.c out: · line 194

Quoted 7 lines

arch/arm/kvm/mmu.c +7 -0

+
+static void free_guest_pages(pte_t *pte, unsigned long addr)
+{
+     unsigned int i;
+     struct page *page;
+
+     for (i = 0; i < PTRS_PER_PTE; i++, addr += PAGE_SIZE) {

Hmm, "addr" is not used.

arch/arm/kvm/mmu.c out: · line 196

Quoted 2 lines

arch/arm/kvm/mmu.c +2 -0

+             if (!pte_present(*pte))
+                     goto next_page;

Why goto instead of:

             if(pte_present(*pte)) {

arch/arm/kvm/mmu.c out: · line 198

Quoted 2 lines

arch/arm/kvm/mmu.c +2 -0

+             page = pfn_to_page(pte_pfn(*pte));
+             put_page(page);

             }

Quoted 71 lines

arch/arm/kvm/mmu.c +66 -0

+next_page:
+             pte++;
+     }
+}
 
+
+static void free_stage2_ptes(pmd_t *pmd, unsigned long addr)
+{
+     unsigned int i;
+     pte_t *pte;
+     struct page *page;
+
+     for (i = 0; i < PTRS_PER_PMD; i++, addr += PMD_SIZE) {
+             BUG_ON(pmd_sect(*pmd));
+             if (!pmd_none(*pmd) && pmd_table(*pmd)) {
+                     pte = pte_offset_kernel(pmd, addr);
+                     free_guest_pages(pte, addr);
+                     page = virt_to_page((void *)pte);
+                     WARN_ON(atomic_read(&page->_count) != 1);
+                     pte_free_kernel(NULL, pte);
+             }
+             pmd++;
+     }
+}
+
+/**
+ * kvm_free_stage2_pgd - free all stage-2 tables
+ * @kvm:     The KVM struct pointer for the VM.
+ *
+ * Walks the level-1 page table pointed to by kvm->arch.pgd and frees all
+ * underlying level-2 and level-3 tables before freeing the actual level-1 table
+ * and setting the struct pointer to NULL.
+ */
+void kvm_free_stage2_pgd(struct kvm *kvm)
+{
+     pgd_t *pgd;
+     pud_t *pud;
+     pmd_t *pmd;
+     unsigned long long i, addr;
+
+     if (kvm->arch.pgd == NULL)
+             return;
+
+     /*
+      * We do this slightly different than other places, since we need more
+      * than 32 bits and for instance pgd_addr_end converts to unsigned long.
+      */
+     addr = 0;
+     for (i = 0; i < PTRS_PER_PGD2; i++) {
+             addr = i * (unsigned long long)PGDIR_SIZE;
+             pgd = kvm->arch.pgd + i;
+             pud = pud_offset(pgd, addr);
+
+             if (pud_none(*pud))
+                     continue;
+
+             BUG_ON(pud_bad(*pud));
+
+             pmd = pmd_offset(pud, addr);
+             free_stage2_ptes(pmd, addr);
+             pmd_free(NULL, pmd);
+     }
+
+     free_pages((unsigned long)kvm->arch.pgd, PGD2_ORDER);
+     kvm->arch.pgd = NULL;
+}
+
  int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run)
  {

      return -EINVAL;

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
                       Gleb.

From: Christoffer Dall <[email protected]>

This commit introduces the framework for guest memory management through the use of 2nd stage translation. Each VM has a pointer to a level-1 table (the pgd field in struct kvm_arch) which is used for the 2nd stage translations. Entries are added when handling guest faults (later patch) and the table itself can be allocated and freed through the following functions implemented in arch/arm/kvm/arm_mmu.c:
- kvm_alloc_stage2_pgd(struct kvm *kvm);
- kvm_free_stage2_pgd(struct kvm *kvm);

Further, each entry in TLBs and caches are tagged with a VMID identifier in addition to ASIDs. The VMIDs are assigned consecutively to VMs in the order that VMs are executed, and caches and tlbs are invalidated when the VMID space has been used to allow for more than 255 simultaenously running guests.

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is freed in kvm_arch_destroy_vm(). Both functions are called from the main KVM code.

Signed-off-by Christoffer Dall <[email protected]>

Quoted 18 lines · Christoffer Dall

From: Christoffer Dall <[email protected]>

This commit introduces the framework for guest memory management through the use of 2nd stage translation. Each VM has a pointer to a level-1 table (the pgd field in struct kvm_arch) which is used for the 2nd stage translations. Entries are added when handling guest faults (later patch) and the table itself can be allocated and freed through the following functions implemented in arch/arm/kvm/arm_mmu.c:
 - kvm_alloc_stage2_pgd(struct kvm *kvm);
 - kvm_free_stage2_pgd(struct kvm *kvm);

Further, each entry in TLBs and caches are tagged with a VMID identifier in addition to ASIDs. The VMIDs are assigned consecutively to VMs in the order that VMs are executed, and caches and tlbs are invalidated when the VMID space has been used to allow for more than 255 simultaenously running guests.

Quoted 5 lines

The 2nd stage pgd is allocated in kvm_arch_init_vm(). The table is freed in kvm_arch_destroy_vm(). Both functions are called from the main KVM code.

Signed-off-by Christoffer Dall <[email protected]>

Can you explain on a high level how the IPA -> PA mappings work?

From: Christoffer Dall <[email protected]>

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl. This ioctl is used since the sematics are in fact two lines that can be either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or IRQ line is raised/lowered. Hence both pieces of information is packed in the kvm_irq_level->irq field. The irq fild value will be:
IRQ: vcpu_index << 1

  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the corresponding irq_line field on the VCPU struct, which will cause the world-switch code to raise/lower virtual interrupts when running the guest on next switch. The wait_for_interrupt flag is also cleared for raised IRQs or FIQs causing an idle VCPU to become active again. CPUs in guest mode are kicked to make sure they refresh their interrupt status.

arch/arm/kvm/arm.c +24 -0

+static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
+				      struct kvm_irq_level *irq_level)
+{
+	int mask;
+	unsigned int vcpu_idx;
+	struct kvm_vcpu *vcpu;
+	unsigned long old, new, *ptr;
+
+	vcpu_idx = irq_level->irq >> 1;
+	if (vcpu_idx >= KVM_MAX_VCPUS)
+		return -EINVAL;
+
+	vcpu = kvm_get_vcpu(kvm, vcpu_idx);
+	if (!vcpu)
+		return -EINVAL;
+
+	if ((irq_level->irq & 1) == KVM_ARM_IRQ_LINE)
+		mask = HCR_VI;
+	else /* KVM_ARM_FIQ_LINE */
+		mask = HCR_VF;
+
+	trace_kvm_set_irq(irq_level->irq, irq_level->level, 0);
+
+	ptr = (unsigned long *)&vcpu->arch.irq_lines;

arch/arm/kvm/arm.c +9 -1

+
+	return 0;
+}
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {

@@ -298,7 +345,20 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log)
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
-	return -EINVAL;
+	struct kvm *kvm = filp->private_data;
+	void __user *argp = (void __user *)arg;
+
+	switch (ioctl) {
+	case KVM_IRQ_LINE: {

Quoted 46 lines · Christoffer Dall

From: Christoffer Dall <[email protected]>

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl. This ioctl is used since the sematics are in fact two lines that can be either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or IRQ line is raised/lowered. Hence both pieces of information is packed in the kvm_irq_level->irq field. The irq fild value will be:
  IRQ: vcpu_index << 1
  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the corresponding irq_line field on the VCPU struct, which will cause the world-switch code to raise/lower virtual interrupts when running the guest on next switch. The wait_for_interrupt flag is also cleared for raised IRQs or FIQs causing an idle VCPU to become active again. CPUs in guest mode are kicked to make sure they refresh their interrupt status.

(quoted diff) +24 -0

+static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
+                                   struct kvm_irq_level *irq_level)
+{
+     int mask;
+     unsigned int vcpu_idx;
+     struct kvm_vcpu *vcpu;
+     unsigned long old, new, *ptr;
+
+     vcpu_idx = irq_level->irq >> 1;
+     if (vcpu_idx >= KVM_MAX_VCPUS)
+             return -EINVAL;
+
+     vcpu = kvm_get_vcpu(kvm, vcpu_idx);
+     if (!vcpu)
+             return -EINVAL;
+
+     if ((irq_level->irq & 1) == KVM_ARM_IRQ_LINE)
+             mask = HCR_VI;
+     else /* KVM_ARM_FIQ_LINE */
+             mask = HCR_VF;
+
+     trace_kvm_set_irq(irq_level->irq, irq_level->level, 0);
+
+     ptr = (unsigned long *)&vcpu->arch.irq_lines;

Review

Quoted 10 lines

(quoted diff) +10 -0

+     do {
+             old = ACCESS_ONCE(*ptr);
+             if (irq_level->level)
+                     new = old | mask;
+             else
+                     new = old & ~mask;
+
+             if (new == old)
+                     return 0; /* no change */
+     } while (cmpxchg(ptr, old, new) != old);

Isn't this a complicated

  if (level)
      set_bit()
  else
      clear_bit()

?

Quoted 7 lines

+
+     /*
+      * The vcpu irq_lines field was updated, wake up sleeping VCPUs and
+      * trigger a world-switch round on the running physical CPU to set the
+      * virtual IRQ/FIQ fields in the HCR appropriately.
+      */
+     kvm_vcpu_kick(vcpu);

No need to wake when the line is asserted so you can make this conditional on level.

Quoted 21 lines

(quoted diff) +4 -0

+
+     return 0;
+}
+

 long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) {

(quoted diff) +9 -1

@@ -298,7 +345,20 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log)
  long kvm_arch_vm_ioctl(struct file *filp,
                      unsigned int ioctl, unsigned long arg)
  {
-     return -EINVAL;
+     struct kvm *kvm = filp->private_data;
+     void __user *argp = (void __user *)arg;
+
+     switch (ioctl) {
+     case KVM_IRQ_LINE: {
+             struct kvm_irq_level irq_event;
+
+             if (copy_from_user(&irq_event, argp, sizeof irq_event))
+                     return -EFAULT;

Review int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log) · line 362

Quoted 6 lines

(quoted diff) +5 -0

+             return kvm_arch_vm_ioctl_irq_line(kvm, &irq_event);
+     }
+     default:
+             return -EINVAL;
+     }
  }

Should be in common code guarded by the define introduced previously.

Quoted 66 lines · Avi Kivity

Quoted 46 lines · Christoffer Dall

From: Christoffer Dall <[email protected]>

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl. This ioctl is used since the sematics are in fact two lines that can be either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or IRQ line is raised/lowered. Hence both pieces of information is packed in the kvm_irq_level->irq field. The irq fild value will be:
IRQ: vcpu_index << 1

  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the corresponding irq_line field on the VCPU struct, which will cause the world-switch code to raise/lower virtual interrupts when running the guest on next switch. The wait_for_interrupt flag is also cleared for raised IRQs or FIQs causing an idle VCPU to become active again. CPUs in guest mode are kicked to make sure they refresh their interrupt status.

(quoted diff) +24 -0

+static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
+                                   struct kvm_irq_level *irq_level)
+{
+     int mask;
+     unsigned int vcpu_idx;
+     struct kvm_vcpu *vcpu;
+     unsigned long old, new, *ptr;
+
+     vcpu_idx = irq_level->irq >> 1;
+     if (vcpu_idx >= KVM_MAX_VCPUS)
+             return -EINVAL;
+
+     vcpu = kvm_get_vcpu(kvm, vcpu_idx);
+     if (!vcpu)
+             return -EINVAL;
+
+     if ((irq_level->irq & 1) == KVM_ARM_IRQ_LINE)
+             mask = HCR_VI;
+     else /* KVM_ARM_FIQ_LINE */
+             mask = HCR_VF;
+
+     trace_kvm_set_irq(irq_level->irq, irq_level->level, 0);
+
+     ptr = (unsigned long *)&vcpu->arch.irq_lines;

Review

Quoted 10 lines

(quoted diff) +10 -0

+     do {
+             old = ACCESS_ONCE(*ptr);
+             if (irq_level->level)
+                     new = old | mask;
+             else
+                     new = old & ~mask;
+
+             if (new == old)
+                     return 0; /* no change */
+     } while (cmpxchg(ptr, old, new) != old);

Isn't this a complicated

if (level)

      set_bit()
  else
      clear_bit()

Quoted 2 lines

?

we need to atomically know if we changed either the FIQ/IRQ fields and atomically update without locking.

(I think you suggested this
approach, in fact).

Quoted 9 lines

Quoted 8 lines

(quoted diff) +7 -0

+
+     /*
+      * The vcpu irq_lines field was updated, wake up sleeping VCPUs and
+      * trigger a world-switch round on the running physical CPU to set the
+      * virtual IRQ/FIQ fields in the HCR appropriately.
+      */
+     kvm_vcpu_kick(vcpu);

Quoted 3 lines

No need to wake when the line is asserted so you can make this conditional on level.

we need to trigger a world switch to update the virtualized register from the actual running physical CPU if we changed any of the IRQ/FIQ fields. We return in the loop above if we didn't change anything.

Quoted 22 lines

Quoted 22 lines

arch/x86/kvm/x86.c +13 -1

+
+     return 0;
+}
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
                       unsigned int ioctl, unsigned long arg)
 {

@@ -298,7 +345,20 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log)
 long kvm_arch_vm_ioctl(struct file *filp,
                     unsigned int ioctl, unsigned long arg)
 {
-     return -EINVAL;
+     struct kvm *kvm = filp->private_data;
+     void __user *argp = (void __user *)arg;
+
+     switch (ioctl) {
+     case KVM_IRQ_LINE: {
+             struct kvm_irq_level irq_event;
+
+             if (copy_from_user(&irq_event, argp, sizeof irq_event))
+                     return -EFAULT;

Quoted 10 lines

Review int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log) · line 362

Quoted 6 lines

(quoted diff) +5 -0

+             return kvm_arch_vm_ioctl_irq_line(kvm, &irq_event);
+     }
+     default:
+             return -EINVAL;
+     }
 }

Should be in common code guarded by the define introduced previously.

you mean like this: ?

arch/arm/kvm/arm.c +1 -2

index a9f209b..5bf2193 100644
@@ -535,8 +535,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu,
 struct kvm_run *run)
 	return ret;
 }
 
-static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
-				      struct kvm_irq_level *irq_level)
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level)
 {
 	int mask;

arch/arm/kvm/arm.c +1 -14

 	unsigned int vcpu_idx;

@@ -596,20 +595,7 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 struct kvm_dirty_log *log)
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
-	struct kvm *kvm = filp->private_data;
-	void __user *argp = (void __user *)arg;
-
-	switch (ioctl) {
-	case KVM_IRQ_LINE: {
-		struct kvm_irq_level irq_event;
-
-		if (copy_from_user(&irq_event, argp, sizeof irq_event))
-			return -EFAULT;
-		return kvm_arch_vm_ioctl_irq_line(kvm, &irq_event);
-	}
-	default:
-		return -EINVAL;
-	}
+	return -EINVAL;
 }
 

static void cpu_set_vector(void *vector)

arch/ia64/kvm/kvm-ia64.c +10 -0

index bd77cb5..122a4b2 100644
@@ -924,6 +924,16 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu
 *vcpu, struct kvm_regs *regs)
 	return 0;
 }
 
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event)
+{
+	if (!irqchip_in_kernel(kvm))
+		return -ENXIO;
+
+	irq_event->statusstatus = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
+					      irq_event->irq, irq_event->level);
+	return 0;
+}
+
 long kvm_arch_vm_ioctl(struct file *filp,
 		unsigned int ioctl, unsigned long arg)

{

arch/ia64/kvm/kvm-ia64.c +0 -23

@@ -963,29 +973,6 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			goto out;
 		}
 		break;
-	case KVM_IRQ_LINE_STATUS:
-	case KVM_IRQ_LINE: {
-		struct kvm_irq_level irq_event;
-
-		r = -EFAULT;
-		if (copy_from_user(&irq_event, argp, sizeof irq_event))
-			goto out;
-		r = -ENXIO;
-		if (irqchip_in_kernel(kvm)) {
-			__s32 status;
-			status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
-				    irq_event.irq, irq_event.level);
-			if (ioctl == KVM_IRQ_LINE_STATUS) {
-				r = -EFAULT;
-				irq_event.status = status;
-				if (copy_to_user(argp, &irq_event,
-							sizeof irq_event))
-					goto out;
-			}
-			r = 0;
-		}
-		break;
-		}
 	case KVM_GET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip chip;

arch/x86/kvm/x86.c +10 -23

index a01a424..c9c4186 100644
@@ -3144,6 +3144,16 @@ out:
 	return r;
 }
 
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event)
+{
+	if (!irqchip_in_kernel(kvm))
+		return -ENXIO;
+
+	irq_event->status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
+					irq_event->irq, irq_event->level);
+	return 0;
+}
+
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {

@@ -3250,29 +3260,6 @@ long kvm_arch_vm_ioctl(struct file *filp,
 	create_pit_unlock:
 		mutex_unlock(&kvm->slots_lock);
 		break;
-	case KVM_IRQ_LINE_STATUS:
-	case KVM_IRQ_LINE: {
-		struct kvm_irq_level irq_event;
-
-		r = -EFAULT;
-		if (copy_from_user(&irq_event, argp, sizeof irq_event))
-			goto out;
-		r = -ENXIO;
-		if (irqchip_in_kernel(kvm)) {
-			__s32 status;
-			status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
-					irq_event.irq, irq_event.level);
-			if (ioctl == KVM_IRQ_LINE_STATUS) {
-				r = -EFAULT;
-				irq_event.status = status;
-				if (copy_to_user(argp, &irq_event,
-							sizeof irq_event))
-					goto out;
-			}
-			r = 0;
-		}
-		break;
-	}
 	case KVM_GET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip *chip;

include/linux/kvm_host.h +1 -0

index e3c86f8..96aa7fb 100644
@@ -494,6 +494,7 @@ int kvm_vm_ioctl_set_memory_region(struct kvm *kvm,
 				   struct
 				   kvm_userspace_memory_region *mem,
 				   int user_alloc);
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level);
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg);
 

virt/kvm/kvm_main.c +12 -0

index 636bd08..1d33877 100644
@@ -2093,6 +2093,25 @@ static long kvm_vm_ioctl(struct file *filp,
 		break;
 	}
 #endif
+#ifdef __KVM_HAVE_IRQ_LINE
+	case KVM_IRQ_LINE_STATUS:
+	case KVM_IRQ_LINE: {
+		struct kvm_irq_level irq_event;
+
+		r = -EFAULT;
+		if (copy_from_user(&irq_event, argp, sizeof irq_event))
+			goto out;
+
+		r = kvm_vm_ioctl_irq_line(kvm, &irq_event);
+
+		if (ioctl == KVM_IRQ_LINE_STATUS) {

Quoted 67 lines · Christoffer Dall

Quoted 66 lines · Avi Kivity

Quoted 46 lines · Christoffer Dall

From: Christoffer Dall <[email protected]>

Userspace can inject IRQs and FIQs through the KVM_IRQ_LINE VM ioctl. This ioctl is used since the sematics are in fact two lines that can be either raised or lowered on the VCPU - the IRQ and FIQ lines.

KVM needs to know which VCPU it must operate on and whether the FIQ or IRQ line is raised/lowered. Hence both pieces of information is packed in the kvm_irq_level->irq field. The irq fild value will be:
  IRQ: vcpu_index << 1
  FIQ: (vcpu_index << 1) | 1

This is documented in Documentation/kvm/api.txt.

The effect of the ioctl is simply to simply raise/lower the corresponding irq_line field on the VCPU struct, which will cause the world-switch code to raise/lower virtual interrupts when running the guest on next switch. The wait_for_interrupt flag is also cleared for raised IRQs or FIQs causing an idle VCPU to become active again. CPUs in guest mode are kicked to make sure they refresh their interrupt status.

(quoted diff) +24 -0

+static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
+                                   struct kvm_irq_level *irq_level)
+{
+     int mask;
+     unsigned int vcpu_idx;
+     struct kvm_vcpu *vcpu;
+     unsigned long old, new, *ptr;
+
+     vcpu_idx = irq_level->irq >> 1;
+     if (vcpu_idx >= KVM_MAX_VCPUS)
+             return -EINVAL;
+
+     vcpu = kvm_get_vcpu(kvm, vcpu_idx);
+     if (!vcpu)
+             return -EINVAL;
+
+     if ((irq_level->irq & 1) == KVM_ARM_IRQ_LINE)
+             mask = HCR_VI;
+     else /* KVM_ARM_FIQ_LINE */
+             mask = HCR_VF;
+
+     trace_kvm_set_irq(irq_level->irq, irq_level->level, 0);
+
+     ptr = (unsigned long *)&vcpu->arch.irq_lines;

Review

Quoted 10 lines

(quoted diff) +10 -0

+     do {
+             old = ACCESS_ONCE(*ptr);
+             if (irq_level->level)
+                     new = old | mask;
+             else
+                     new = old & ~mask;
+
+             if (new == old)
+                     return 0; /* no change */
+     } while (cmpxchg(ptr, old, new) != old);

Isn't this a complicated

  if (level)
      set_bit()
  else
      clear_bit()

Quoted 5 lines

Quoted 2 lines

?

we need to atomically know if we changed either the FIQ/IRQ fields and atomically update without locking.

So use test_and_set_bit()/test_and_clear_bit().

Quoted 2 lines

(I think you suggested this
approach, in fact).

I think so too, but it only makes sense if you need to consider both fields simultaneously (which you don't here).  For example, if IRQ was asserted while FIQ was already raised, you don't need to kick.  But that's not the case according to the below.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

+
+     /*
+      * The vcpu irq_lines field was updated, wake up sleeping VCPUs and
+      * trigger a world-switch round on the running physical CPU to set the
+      * virtual IRQ/FIQ fields in the HCR appropriately.
+      */
+     kvm_vcpu_kick(vcpu);

Quoted 7 lines

Quoted 3 lines

No need to wake when the line is asserted so you can make this conditional on level.

we need to trigger a world switch to update the virtualized register from the actual running physical CPU if we changed any of the IRQ/FIQ fields. We return in the loop above if we didn't change anything.

Okay.

Quoted 216 lines

Quoted 22 lines

Quoted 22 lines

arch/x86/kvm/x86.c +4 -0

+
+     return 0;
+}
+

 long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) {

(quoted diff) +9 -1

@@ -298,7 +345,20 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log)
  long kvm_arch_vm_ioctl(struct file *filp,
                      unsigned int ioctl, unsigned long arg)
  {
-     return -EINVAL;
+     struct kvm *kvm = filp->private_data;
+     void __user *argp = (void __user *)arg;
+
+     switch (ioctl) {
+     case KVM_IRQ_LINE: {
+             struct kvm_irq_level irq_event;
+
+             if (copy_from_user(&irq_event, argp, sizeof irq_event))
+                     return -EFAULT;

Quoted 10 lines

Review int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log) · line 362

Quoted 6 lines

(quoted diff) +5 -0

+             return kvm_arch_vm_ioctl_irq_line(kvm, &irq_event);
+     }
+     default:
+             return -EINVAL;
+     }
  }

Should be in common code guarded by the define introduced previously.

you mean like this: ?

arch/arm/kvm/arm.c +1 -2

index a9f209b..5bf2193 100644
@@ -535,8 +535,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu,
 struct kvm_run *run)
       return ret;
  }
 
-static int kvm_arch_vm_ioctl_irq_line(struct kvm *kvm,
-                                   struct kvm_irq_level *irq_level)
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level)
  {
       int mask;

      unsigned int vcpu_idx;

arch/arm/kvm/arm.c +1 -14

@@ -596,20 +595,7 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 struct kvm_dirty_log *log)
  long kvm_arch_vm_ioctl(struct file *filp,
                      unsigned int ioctl, unsigned long arg)
  {
-     struct kvm *kvm = filp->private_data;
-     void __user *argp = (void __user *)arg;
-
-     switch (ioctl) {
-     case KVM_IRQ_LINE: {
-             struct kvm_irq_level irq_event;
-
-             if (copy_from_user(&irq_event, argp, sizeof irq_event))
-                     return -EFAULT;
-             return kvm_arch_vm_ioctl_irq_line(kvm, &irq_event);
-     }
-     default:
-             return -EINVAL;
-     }
+     return -EINVAL;
  }
 

 static void cpu_set_vector(void *vector)

arch/ia64/kvm/kvm-ia64.c +10 -0

index bd77cb5..122a4b2 100644
@@ -924,6 +924,16 @@ int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu
 *vcpu, struct kvm_regs *regs)
       return 0;
  }
 
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event)
+{
+     if (!irqchip_in_kernel(kvm))
+             return -ENXIO;
+
+     irq_event->statusstatus = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
+                                           irq_event->irq, irq_event->level);
+     return 0;
+}
+
  long kvm_arch_vm_ioctl(struct file *filp,
               unsigned int ioctl, unsigned long arg)

 {

arch/ia64/kvm/kvm-ia64.c +0 -23

@@ -963,29 +973,6 @@ long kvm_arch_vm_ioctl(struct file *filp,
                       goto out;
               }
               break;
-     case KVM_IRQ_LINE_STATUS:
-     case KVM_IRQ_LINE: {
-             struct kvm_irq_level irq_event;
-
-             r = -EFAULT;
-             if (copy_from_user(&irq_event, argp, sizeof irq_event))
-                     goto out;
-             r = -ENXIO;
-             if (irqchip_in_kernel(kvm)) {
-                     __s32 status;
-                     status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
-                                 irq_event.irq, irq_event.level);
-                     if (ioctl == KVM_IRQ_LINE_STATUS) {
-                             r = -EFAULT;
-                             irq_event.status = status;
-                             if (copy_to_user(argp, &irq_event,
-                                                     sizeof irq_event))
-                                     goto out;
-                     }
-                     r = 0;
-             }
-             break;
-             }
       case KVM_GET_IRQCHIP: {
               /* 0: PIC master, 1: PIC slave, 2: IOAPIC */
               struct kvm_irqchip chip;

arch/x86/kvm/x86.c +10 -0

index a01a424..c9c4186 100644
@@ -3144,6 +3144,16 @@ out:
       return r;
  }
 
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event)
+{
+     if (!irqchip_in_kernel(kvm))
+             return -ENXIO;
+
+     irq_event->status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
+                                     irq_event->irq, irq_event->level);
+     return 0;
+}
+
  long kvm_arch_vm_ioctl(struct file *filp,
                      unsigned int ioctl, unsigned long arg)
  {

@@ -3250,29 +3260,6 @@ long kvm_arch_vm_ioctl(struct file *filp,

      create_pit_unlock:
              mutex_unlock(&kvm->slots_lock);
              break;
-     case KVM_IRQ_LINE_STATUS:
-     case KVM_IRQ_LINE: {
-             struct kvm_irq_level irq_event;
-
-             r = -EFAULT;
-             if (copy_from_user(&irq_event, argp, sizeof irq_event))
-                     goto out;
-             r = -ENXIO;
-             if (irqchip_in_kernel(kvm)) {
-                     __s32 status;
-                     status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
-                                     irq_event.irq, irq_event.level);
-                     if (ioctl == KVM_IRQ_LINE_STATUS) {
-                             r = -EFAULT;
-                             irq_event.status = status;
-                             if (copy_to_user(argp, &irq_event,
-                                                     sizeof irq_event))
-                                     goto out;
-                     }
-                     r = 0;
-             }
-             break;
-     }
      case KVM_GET_IRQCHIP: {
              /* 0: PIC master, 1: PIC slave, 2: IOAPIC */ struct kvm_irqchip *chip;

include/linux/kvm_host.h +0 -0

index e3c86f8..96aa7fb 100644
@@ -494,6 +494,7 @@ int kvm_vm_ioctl_set_memory_region(struct kvm *kvm,

                                 struct kvm_userspace_memory_region *mem, int user_alloc);
+int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level); long kvm_arch_vm_ioctl(struct file *filp,
                     unsigned int ioctl, unsigned long arg);

virt/kvm/kvm_main.c +19 -0

index 636bd08..1d33877 100644
@@ -2093,6 +2093,25 @@ static long kvm_vm_ioctl(struct file *filp,
               break;
       }
  #endif
+#ifdef __KVM_HAVE_IRQ_LINE
+     case KVM_IRQ_LINE_STATUS:
+     case KVM_IRQ_LINE: {
+             struct kvm_irq_level irq_event;
+
+             r = -EFAULT;
+             if (copy_from_user(&irq_event, argp, sizeof irq_event))
+                     goto out;
+
+             r = kvm_vm_ioctl_irq_line(kvm, &irq_event);
+
+             if (ioctl == KVM_IRQ_LINE_STATUS) {
+                     if (copy_to_user(argp, &irq_event, sizeof irq_event))
+                             r = -EFAULT;
+             }
+
+             break;
+     }
+#endif

Quoted 3 lines

      default:
              r = kvm_arch_vm_ioctl(filp, ioctl, arg); if (r == -ENOTTY)

Yup.  Note it brings in KVM_IRQ_LINE_STATUS for ARM.  I think it makes sense if you have guests that do timekeeping by counting interrupts (as opposed to reading a wall clock register).

--
error compiling committee.c: too many arguments to function