Skip to contentSkip to conversation

KVM: ARM: Get rid of hardcoded VGIC addresses

PATCH New Christoffer Dall 1 participant 4 messages Oct 20, 2012

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: ARM: Get rid of hardcoded VGIC addresses

We need a way to specify the address at which we expect VMs to access the interrupt controller (both the emulated distributor and the hardware interface supporting virtualization). User space should decide on this address as user space decides on an emulated board and loads a device tree describing these details directly to the guest.

We introduce a new ioctl, KVM_SET_DEVICE_ADDRESS, that lets user space provide a base address for a device based on exported device ids. For now, this is only supported for the ARM vgic. User space provides this address after creating the IRQ chip and KVM performs the required mappings for a VM on the first execution of a VCPU.

To avoid users mistakenly supplying overlapping io address ranges we check for existing mappings at io mappings for guests and explicitly raise an error if address space conflicts occur.

Changelog[2]:
- Check for page aligned addresses on vgic_set_devic_address
- Move vgic_cpu_base for host to vgic_hyp_init
- Fix irqchip_in_kernel to be VM-specific

---

Christoffer Dall (3):
KVM: ARM: Check for overlaps of mapped io addresses
KVM: ARM: Introduce KVM_SET_DEVICE_ADDRESS ioctl
KVM: ARM: Defer parts of the vgic init until first KVM_RUN

 Documentation/virtual/kvm/api.txt |   37 +++++++++++
 arch/arm/include/asm/kvm.h        |   13 ++++
 arch/arm/include/asm/kvm_mmu.h    |    2 +
 arch/arm/include/asm/kvm_vgic.h   |   28 +++++++--
 arch/arm/kvm/arm.c                |   41 ++++++++++++-
 arch/arm/kvm/mmu.c                |   11 ++-
 arch/arm/kvm/vgic.c               |  121 +++++++++++++++++++++++++++++++------
 include/linux/kvm.h               |    8 ++
 8 files changed, 231 insertions(+), 30 deletions(-)
Signature · 1 line
Signature
#1 lore raw

PATCH v2 1/3 KVM: ARM: Check for overlaps of mapped io addresses

When calling stage2_set_pte from kvm_phys_addr_ioremap we pass an argument to say that this is an IO mapping, and that we expect the adress range to be free, otherwise return an error.

This should catch errors earlier when user space supplies guest physical addresses that overlap.

---
 arch/arm/kvm/mmu.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/arm/kvm/mmu.c +7 -4
index 0ab098e..e5ace0e 100644
@@ -422,7 +422,7 @@ static void stage2_clear_pte(struct kvm *kvm, phys_addr_t addr)
 }
 
 static void stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
-			   phys_addr_t addr, const pte_t *new_pte)
+			   phys_addr_t addr, const pte_t *new_pte, bool iomap)
 {
 	pgd_t *pgd;
 	pud_t *pud;
@@ -454,6 +454,9 @@ static void stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
 	} else
 		pte = pte_offset_kernel(pmd, addr);
 
+	if (iomap && pte_present(old_pte))
+		return -EFAULT;
+
 	/* Create 2nd stage page table mapping - Level 3 */
 	old_pte = *pte;
 	set_pte_ext(pte, *new_pte, 0);
@@ -489,7 +492,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 		if (ret)
 			goto out;
 		spin_lock(&kvm->mmu_lock);
-		stage2_set_pte(kvm, &cache, addr, &pte);
+		stage2_set_pte(kvm, &cache, addr, &pte, true);
 		spin_unlock(&kvm->mmu_lock);
 
 		pfn++;
@@ -565,7 +568,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 		pte_val(new_pte) |= L_PTE_S2_RDWR;
 		kvm_set_pfn_dirty(pfn);
 	}
-	stage2_set_pte(vcpu->kvm, memcache, fault_ipa, &new_pte);
+	stage2_set_pte(vcpu->kvm, memcache, fault_ipa, &new_pte, false);
 
 out_unlock:
 	spin_unlock(&vcpu->kvm->mmu_lock);
@@ -716,7 +719,7 @@ static void kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, void *data)
 {
 	pte_t *pte = (pte_t *)data;
 
-	stage2_set_pte(kvm, NULL, gpa, pte);
+	stage2_set_pte(kvm, NULL, gpa, pte, false);
 }
 
 
#2 lore raw

PATCH v2 2/3 KVM: ARM: Introduce KVM_SET_DEVICE_ADDRESS ioctl

In reply to Christoffer Dall

On ARM (and possibly other architectures) some bits are specific to the model being emulated for the guest and user space needs a way to tell the kernel about those bits. An example is mmio device base addresses, where KVM must know the base address for a given device to properly emulate mmio accesses within a certain address range or directly map a device with virtualiation extensions into the guest address space.

We try to make this API slightly more generic than for our specific use, but so far only the VGIC uses this feature.

---
 Documentation/virtual/kvm/api.txt |   37 +++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/kvm.h        |   13 +++++++++++++
 arch/arm/include/asm/kvm_mmu.h    |    2 ++
 arch/arm/include/asm/kvm_vgic.h   |    6 ++++++
 arch/arm/kvm/arm.c                |   31 ++++++++++++++++++++++++++++++-
 arch/arm/kvm/vgic.c               |   28 ++++++++++++++++++++++++++++
 include/linux/kvm.h               |    8 ++++++++
 7 files changed, 124 insertions(+), 1 deletion(-)
Documentation/virtual/kvm/api.txt +37 -0
index 0aa4d83..dae4f05 100644
@@ -2102,6 +2102,43 @@ This ioctl returns the guest registers that are supported for the
 KVM_GET_ONE_REG/KVM_SET_ONE_REG calls.
 
 
+4.80 KVM_SET_DEVICE_ADDRESS
+
+Capability: KVM_CAP_SET_DEVICE_ADDRESS
+Architectures: arm
+Type: vm ioctl
+Parameters: struct kvm_device_address (in)
+Returns: 0 on success, -1 on error
+Errors:
+  ENODEV: The device id is unknown
+  ENXIO:  Device not supported on current system
+  EEXIST: Address already set
+  E2BIG:  Address outside guest physical address space
+
+struct kvm_device_address {
+	__u32 id;
+	__u64 addr;
+};
+
+Specify a device address in the guest's physical address space where guests
+can access emulated or directly exposed devices, which the host kernel needs
+to know about. The id field is an architecture specific identifier for a
+specific device.
+
+ARM divides the id field into two parts, a device id and an address type id
+specific to the individual device.
+
+  bits:  | 31    ...    16 | 15    ...    0 |
+  field: |     device id   |  addr type id  |
+
+ARM currently only require this when using the in-kernel GIC support for the
+hardware vGIC features, using KVM_ARM_DEVICE_VGIC_V2 as the device id.  When
+setting the base address for the guest's mapping of the vGIC virtual CPU
+and distributor interface, the ioctl must be called after calling
+KVM_CREATE_IRQCHIP, but before calling KVM_RUN on any of the VCPUs.  Calling
+this ioctl twice for any of the base addresses will return -EEXIST.
+
+
 5. The kvm_run structure
 ------------------------
 
arch/arm/include/asm/kvm.h +13 -0
index fb41608..a7ae073 100644
@@ -42,6 +42,19 @@ struct kvm_regs {
 #define KVM_ARM_TARGET_CORTEX_A15	0
 #define KVM_ARM_NUM_TARGETS		1
 
+/* KVM_SET_DEVICE_ADDRESS ioctl id encoding */
+#define KVM_DEVICE_TYPE_SHIFT		0
+#define KVM_DEVICE_TYPE_MASK		(0xffff << KVM_DEVICE_TYPE_SHIFT)
+#define KVM_DEVICE_ID_SHIFT		16
+#define KVM_DEVICE_ID_MASK		(0xffff << KVM_DEVICE_ID_SHIFT)
+
+/* Supported device IDs */
+#define KVM_ARM_DEVICE_VGIC_V2		0
+
+/* Supported VGIC address types  */
+#define KVM_VGIC_V2_ADDR_TYPE_DIST	0
+#define KVM_VGIC_V2_ADDR_TYPE_CPU	1
+
 struct kvm_vcpu_init {
 	__u32 target;
 	__u32 features[7];
arch/arm/include/asm/kvm_mmu.h +2 -0
index 9bd0508..0800531 100644
@@ -26,6 +26,8 @@
  * To save a bit of memory and to avoid alignment issues we assume 39-bit IPA
  * for now, but remember that the level-1 table must be aligned to its size.
  */
+#define KVM_PHYS_SHIFT	(38)
+#define KVM_PHYS_MASK	((1ULL << KVM_PHYS_SHIFT) - 1)
 #define PTRS_PER_PGD2	512
 #define PGD2_ORDER	get_order(PTRS_PER_PGD2 * sizeof(pgd_t))
 
arch/arm/include/asm/kvm_vgic.h +6 -0
index 588c637..a688132 100644
@@ -242,6 +242,7 @@ struct kvm_exit_mmio;
 
 #ifdef CONFIG_KVM_ARM_VGIC
 int kvm_vgic_hyp_init(void);
+int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr);
 int kvm_vgic_init(struct kvm *kvm);
 void kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu);
 void kvm_vgic_sync_to_cpu(struct kvm_vcpu *vcpu);
@@ -261,6 +262,11 @@ static inline int kvm_vgic_hyp_init(void)
 	return 0;
 }
 
+static inline int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
+{
+	return 0;
+}
+
 static inline int kvm_vgic_init(struct kvm *kvm)
 {
 	return 0;
arch/arm/kvm/arm.c +30 -1
index d552b94..282794e 100644
@@ -206,6 +206,9 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_COALESCED_MMIO:
 		r = KVM_COALESCED_MMIO_PAGE_OFFSET;
 		break;
+	case KVM_CAP_SET_DEVICE_ADDR:
+		r = 1;
+		break;
 	default:
 		r = 0;
 		break;
@@ -858,20 +861,46 @@ int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log)
 	return -EINVAL;
 }
 
+static int kvm_vm_ioctl_set_device_address(struct kvm *kvm,
+					   struct kvm_device_address *dev_addr)
+{
+	unsigned long dev_id, type;
+
+	dev_id = (dev_addr->id & KVM_DEVICE_ID_MASK) >> KVM_DEVICE_ID_SHIFT;
+	type = (dev_addr->id & KVM_DEVICE_TYPE_MASK) >> KVM_DEVICE_TYPE_SHIFT;
+
+	switch (dev_id) {
+	case KVM_ARM_DEVICE_VGIC_V2:
+		if (!vgic_present)
+			return -ENXIO;
+		return kvm_vgic_set_addr(kvm, type, dev_addr->addr);
+	default:
+		return -ENODEV;
+	}
+}
+
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
+	struct kvm *kvm = filp->private_data;
+	void __user *argp = (void __user *)arg;
 
 	switch (ioctl) {
 #ifdef CONFIG_KVM_ARM_VGIC
 	case KVM_CREATE_IRQCHIP: {
-		struct kvm *kvm = filp->private_data;
 		if (vgic_present)
 			return kvm_vgic_init(kvm);
 		else
 			return -EINVAL;
 	}
 #endif
+	case KVM_SET_DEVICE_ADDRESS: {
+		struct kvm_device_address dev_addr;
+
+		if (copy_from_user(&dev_addr, argp, sizeof(dev_addr)))
+			return -EFAULT;
+		return kvm_vm_ioctl_set_device_address(kvm, &dev_addr);
+	}
 	default:
 		return -EINVAL;
 	}
arch/arm/kvm/vgic.c +28 -0
index b669b85..82fba58 100644
@@ -1140,3 +1140,31 @@ out:
 
 	return ret;
 }
+
+int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr)
+{
+	int r = 0;
+
+	if (addr & ~KVM_PHYS_MASK)
+		return -E2BIG;
+
+	if (addr & ~PAGE_MASK)
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+	switch (type) {
+	case KVM_VGIC_V2_ADDR_TYPE_DIST:
+		if (addr != VGIC_DIST_BASE)
+			return -EINVAL;
+		break;
+	case KVM_VGIC_V2_ADDR_TYPE_CPU:
+		if (addr != VGIC_CPU_BASE)
+			return -EINVAL;
+		break;
+	default:
+		r = -ENODEV;
+	}
+
+	mutex_unlock(&kvm->lock);
+	return r;
+}
include/linux/kvm.h +8 -0
index 172cc10..72d5594 100644
@@ -626,6 +626,7 @@ struct kvm_ppc_smmu_info {
 #ifdef __KVM_HAVE_READONLY_MEM
 #define KVM_CAP_READONLY_MEM 81
 #endif
+#define KVM_CAP_SET_DEVICE_ADDR 82
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -764,6 +765,11 @@ struct kvm_msi {
 	__u8  pad[16];
 };
 
+struct kvm_device_address {
+	__u32 id;
+	__u64 addr;
+};
+
 /*
  * ioctls for VM fds
  */
@@ -844,6 +850,8 @@ struct kvm_s390_ucas_mapping {
 #define KVM_PPC_GET_SMMU_INFO	  _IOR(KVMIO,  0xa6, struct kvm_ppc_smmu_info)
 /* Available with KVM_CAP_PPC_ALLOC_HTAB */
 #define KVM_PPC_ALLOCATE_HTAB	  _IOWR(KVMIO, 0xa7, __u32)
+/* Available with KVM_CAP_SET_DEVICE_ADDR */
+#define KVM_SET_DEVICE_ADDRESS	  _IOW(KVMIO,  0xa8, struct kvm_device_address)
 
 /*
  * ioctls for vcpu fds
#3 lore raw

PATCH v2 3/3 KVM: ARM: Defer parts of the vgic init until first KVM_RUN

In reply to Christoffer Dall

The vgic virtual cpu and emulated distributor interfaces must be mapped at a given physical address in the guest. This address is provided through the KVM_SET_DEVICE_ADDRESS ioctl, which happens after the KVM_CREATE_IRQCHIP ioctl is called, but before the first VCPU is excuted thorugh KVM_RUN. We create the vgic on KVM_CREATE_IRQCHIP, but query kvm_vgic_ready(kvm), which checks if the vgic.vctrl_base field has been set, before we execute a VCPU, and if it has not been set, we call kvm_vgic_init, which takes care of the remaining setup.

We use the IS_VGIC_ADDR_UNDEF() macro, which compares to the VGIC_ADDR_UNDEF constant, to check if an address has been set; it's unlikely that a device will sit on address 0, but since this is a part of main kernel boot procedure if this stuff is enabled in the config, I'm being paranoid.

The distributor and vcpu base addresses used to be a per-host setting global for all VMs, but this is not a requirement and when we want to emulate several boards on a single host, we need the flexibility of storing these guest addresses on a per-VM basis.

---
 arch/arm/include/asm/kvm_vgic.h |   22 +++++++--
 arch/arm/kvm/arm.c              |   10 ++++
 arch/arm/kvm/vgic.c             |   97 ++++++++++++++++++++++++++++++---------
 3 files changed, 102 insertions(+), 27 deletions(-)
arch/arm/include/asm/kvm_vgic.h +18 -4
index a688132..3c71c79 100644
@@ -154,13 +154,14 @@ static inline void vgic_bytemap_set_irq_val(struct vgic_bytemap *x,
 struct vgic_dist {
 #ifdef CONFIG_KVM_ARM_VGIC
 	spinlock_t		lock;
+	bool			ready;
 
 	/* Virtual control interface mapping */
 	void __iomem		*vctrl_base;
 
-	/* Distributor mapping in the guest */
-	unsigned long		vgic_dist_base;
-	unsigned long		vgic_dist_size;
+	/* Distributor and vcpu interface mapping in the guest */
+	phys_addr_t		vgic_dist_base;
+	phys_addr_t		vgic_cpu_base;
 
 	/* Distributor enabled */
 	u32			enabled;
@@ -243,6 +244,7 @@ struct kvm_exit_mmio;
 #ifdef CONFIG_KVM_ARM_VGIC
 int kvm_vgic_hyp_init(void);
 int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 addr);
+int kvm_vgic_create(struct kvm *kvm);
 int kvm_vgic_init(struct kvm *kvm);
 void kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu);
 void kvm_vgic_sync_to_cpu(struct kvm_vcpu *vcpu);
@@ -252,8 +254,10 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int irq_num,
 int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu);
 bool vgic_handle_mmio(struct kvm_vcpu *vcpu, struct kvm_run *run,
 		      struct kvm_exit_mmio *mmio);
+bool irqchip_in_kernel(struct kvm *kvm);
 
-#define irqchip_in_kernel(k)	(!!((k)->arch.vgic.vctrl_base))
+#define irqchip_in_kernel(k)	((k)->arch.vgic.vctrl_base)
+#define vgic_initialized(k)	((k)->arch.vgic.ready)
 #define vgic_active_irq(v)	(atomic_read(&(v)->arch.vgic_cpu.irq_active_count) == 0)
 
 #else
@@ -267,6 +271,11 @@ static inline int kvm_vgic_set_addr(struct kvm *kvm, unsigned long type, u64 add
 	return 0;
 }
 
+static inline int kvm_vgic_create(struct kvm *kvm)
+{
+	return 0;
+}
+
 static inline int kvm_vgic_init(struct kvm *kvm)
 {
 	return 0;
@@ -298,6 +307,11 @@ static inline int irqchip_in_kernel(struct kvm *kvm)
 	return 0;
 }
 
+static inline bool kvm_vgic_initialized(struct kvm *kvm)
+{
+	return true;
+}
+
 static inline int vgic_active_irq(struct kvm_vcpu *vcpu)
 {
 	return 0;
arch/arm/kvm/arm.c +9 -1
index 282794e..d64783e 100644
@@ -636,6 +636,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 	if (unlikely(vcpu->arch.target < 0))
 		return -ENOEXEC;
 
+	/* Initalize the VGIC before running the vcpu */
+	if (unlikely(irqchip_in_kernel(vcpu->kvm) &&
+		     !vgic_initialized(vcpu->kvm))) {
+		ret = kvm_vgic_init(vcpu->kvm);
+		if (ret)
+			return ret;
+	}
+
 	if (run->exit_reason == KVM_EXIT_MMIO) {
 		ret = kvm_handle_mmio_return(vcpu, vcpu->run);
 		if (ret)
@@ -889,7 +897,7 @@ long kvm_arch_vm_ioctl(struct file *filp,
 #ifdef CONFIG_KVM_ARM_VGIC
 	case KVM_CREATE_IRQCHIP: {
 		if (vgic_present)
-			return kvm_vgic_init(kvm);
+			return kvm_vgic_create(kvm);
 		else
 			return -EINVAL;
 	}
arch/arm/kvm/vgic.c +75 -22

177 lines not on this page · raw message