add initial kvm dev passhtrough support

This is the initial device pass through support.
At this time host == guest only is supported.
Basic Operation:

- QEMU parameters: -device kvm-device-assign,host=<device name> for example - kvm-device-assign,host='arm-sp804'. Essentially any device that does PIO should be supported.

Is there any particular reason you're not going down that path for your ARM implementation?

On the embedded PPC side we've been discussing vfio and how it fits into a device tree, non-PCI world for a while. If you like, we can dive into more detail on that, either via email or via phone.

Alex

Quoted 2 lines · Alexander Graf

Is there any particular reason you're not going down that path for your ARM implementation?

We see this as a good starting point to build on, we need baseline numbers for performance, latency, interrupt throughput on real hardware ASAP to build competency for NFV, which has demanding Dev. Passthrough requirements. Over time we plan contributing to SMMU and VFIO as well (we're looking into this now).

FYI NFV is an initiative wireless/fixed network operators are working towards - to virtualize Core, likely Radia Access and even Home Network equipment, this is a epic undertaking (i.e. Network Function Virtualization). So far VMware has taken the lead (mostly x86).

Quoted 1 line

Quoted 1 line

On the embedded PPC side we've been discussing vfio and how it fits into a device tree, non-PCI world for a while. If you like, we can dive into more detail on that, either via email or via phone.

I'll email you offline, I'd like to know more what you've done on this and see where we can align/leverage the effort.

Quoted 17 lines · Mario Smarduch

Quoted 2 lines · Alexander Graf

Is there any particular reason you're not going down that path for your ARM implementation?

We see this as a good starting point to build on, we need baseline numbers for performance, latency, interrupt throughput on real hardware ASAP to build competency for NFV, which has demanding Dev. Passthrough requirements. Over time we plan contributing to SMMU and VFIO as well (we're looking into this now).

FYI NFV is an initiative wireless/fixed network operators are working towards - to virtualize Core, likely Radia Access and even Home Network equipment, this is a epic undertaking (i.e. Network Function Virtualization). So far VMware has taken the lead (mostly x86).

Quoted 1 line

Quoted 4 lines

Quoted 1 line

On the embedded PPC side we've been discussing vfio and how it fits into a device tree, non-PCI world for a while. If you like, we can dive into more detail on that, either via email or via phone.

I'll email you offline, I'd like to know more what you've done on this and see where we can align/leverage the effort.

Yes, please let's use VFIO rather than continue to use or invent new device assignment interfaces for KVM. Antonios Motakis (cc'd) already contacted me about VFIO for ARM. IIRC, his initial impression was that the IOMMU backend was almost entirely reusable for ARM (a couple PCI assumptions implicit in the IOMMU API to handle) and my hope was that ARM and PPC could work together on a common VFIO device tree backend. Thanks,

Alex

arch/arm/include/asm/kvm_host.h +7 -0

+struct kvm_arm_assigned_dev_kernel {
+       struct list_head list;
+       struct kvm_arm_assigned_device dev;
+       irqreturn_t (*irq_handler)(int, void *);
+       void *irq_arg;
+};
+

Instead of irq_arg, isn't something such as target_vcpu more clear?

diff --git a/arch/arm/kvm/vgic.c b/arch/arm/kvm/vgic.c
index 17c5ac7..f4cb804 100644
--- a/arch/arm/kvm/vgic.c
+++ b/arch/arm/kvm/vgic.c
@@ -449,6 +449,41 @@ static u32 vgic_get_target_reg(struct kvm *kvm, int irq)
return val;
}

+/* Follow the IRQ vCPU affinity so passthrough device interrupts are injected
+ * on physical CPU they execute.
+ */
+static void vgic_set_passthru_affinity(struct kvm *kvm, int irq, u32 target) +{
+ struct list_head *dev_list_ptr = &kvm->arch.assigned_dev_head;
+ struct list_head *ptr;
+ struct kvm_arm_assigned_dev_kernel *assigned_dev;
+ struct vgic_dist *dist = &kvm->arch.vgic;
+ char *buf;
+ int cpu, hwirq;
+
+ mutex_lock(&kvm->arch.dev_pasthru_lock);
+ list_for_each(ptr, dev_list_ptr) {
+ assigned_dev = list_entry(ptr,
+ struct kvm_arm_assigned_dev_kernel, list);
+ if (assigned_dev->dev.guest_res.girq == irq) {
+ if (assigned_dev->irq_arg)
+ free_irq(irq, assigned_dev->irq_arg);
+ cpu = kvm->vcpus[target]->cpu;
+ hwirq = assigned_dev->dev.dev_res.hostirq.hwirq;
+ irq_set_affinity(hwirq, cpumask_of(cpu));
+ assigned_dev->irq_arg = kvm->vcpus[target];
+ buf = assigned_dev->dev.dev_res.hostirq.host_name;
+ sprintf(buf, "%s-KVM Pass-through",
+ assigned_dev->dev.dev_res.devname);
+ gic_spi_set_priodrop(hwirq);
+ dist->guest_irq[hwirq - VGIC_NR_PRIVATE_IRQS] = irq;
+ request_irq(hwirq, assigned_dev->irq_handler, 0, buf,
+ assigned_dev->irq_arg);
+ }
+ }
+ mutex_unlock(&kvm->arch.dev_pasthru_lock);
+}
+

Maybe vgic_set_pasthru_affinity is not an ideal name for the function, since you do more than that here.

After looking at your code I think things will be much easier if you decouple the host irq affinity bits from here. After that there is not much stopping from affinity following the CPU a vCPU will execute.

I would rename this to something to reflect that you enable priodrop for this IRQ here, for example only vgic_set_passthrough could suffice (I'm don't like the pasthru abbreviation a lot). Then the affinity bits can be put in a different function.

In arch/arm/kvm/arm.c kvm_arch_vcpu_load() you can follow up whenever a vcpu is moved to a different cpu. However in practice I don't know if the additional complexity of having the irq affinity follow the vcpu significantly improves irq latency.

Signature · 5 lines

-- 

*Antonios Motakis*, Virtual Open Systems*
*/Open Source KVM Virtualization Development
/www.virtualopensystems.com <http://www.virtualopensystems.com>