Device/Interrupt Passthrough - 3.9rc5

Sorry for the lengthy email, it covers both the patch submitted here and high-level approach for Device Passthrough.

For the overall Device Passthrough approach please scroll down.

The Patch
=========
This patch by itself does not serve much purpose without the additional QEMU/KVM support, the patch has been tested with those features working Those patches will be rolled out soon. Given this patch touches critical GIC code required by Device Passthrough it needs to go in first.

The interrupt pass-through patch supports priority drop/deactivation. This is a first patch intended to facilitate overall device (interrupt) passthrough to guest.

ARM GICv2 describes recommended way to handle interrupt pass-through (5.2.2 - 1., 2.) - this patch sets Host GICC_CTLR.EOImodeNS=1 and for injecting interrupts GICH_LRn.HW=1. Guest sets its CTLR.EOImodeNS=0 executing without changes.

New config param is defined (arm/kvm/Kconfig' KVM_ARM_INT_PRIO_DROP) which is enabled only if Virtualization && KVM_ARM_VGIC && OF are enabled.

If the parameter is enabled, an IRQ can be marked for pass-through, in that case host writes to EOIR to lower IRQ priority and Guest EOIR deactivates the IRQ. For host interrupts writes to both EOIR/DIR are done. If the config param is diabled then nothing changes.

The patch is strictly intended for Virtualization & KVM device passthrough, there may be other uses for IRQ prio drop/deactivation they're not obvious or considered at this time.

Currently the patch assumes if primary CPU has HYP mode enabled and if PRIO_DROP is defined EOImodeNS is set. It might turn out that subsequent CPU(s) may not boot in HYP mode in that case all CPUs will be configured to execute in PRIO DROP mode which practically has no overhead.

What's been tested -
- we pass-through sp804 timer and event devices from guest to host, this tests QEMU, kernel, ARM-KVM memory and interrupt passthrough. We're working on lan 911x passthrough and then Arndale NIC passthrough
- all combinations of host and guest have been tested, i.e. Host Virt=on/off, PrioDrop=on/off; Guest virt=on/of, PD=on/off Any combination of Guest/Host config is supported.

DEVICE PASSTHROUGH overall Architecture
---------------------------------------

To pass through a device following data is required:
o Host - Device HPA, Size, Perms (r; r/w), Attr (dev, mem), host irq - [optional: CPU IRQ affinity, vCPU affinity - dynamic affiniy although affinity for RT/performance a require option] - these values come from Device Tree and command line parameters.
o Guest - Device GPA, IRQ - at run-time select memory and IRQ from range predefined for Device passthrough for the emulated machine model and patch guest DT. The ranges should be predefined to prevent future conflicts between MMIO and devices passed through.

Overall Device Passthrough Design summary - simple usage -
"-device kvm-device-assign,host="device name" -
device as known in DT. Although passing all the parameters is also supported.

- This patch being first to lay the groundwork and given it touches generic GIC code we would like early feedback.
- Device I/O or Memory Passthrough - combination of new QEMU (kvm-device-assign) device and ARM KVM ioctl hooks to map device in 2nd stage tables. Dynamic discovery and/or allocation of resources. With SMMU availability this should be where you map/pin the Guest.
- Interrupt passthrough - reuse current ioctl QEMU/KVM ioctl infrastracture in combination with this patch KVM Guest can mark a device for interrupt passthrough. Both host/guest IRQs run-time discovered/allocated.
o eventual run-time IRQ -> CPU -> vCPU framework support, although CPU/vCPU affinity for some applications is useful as well.
- Support for 1:1 memory mapping to support DMA with no IOMMU. Eventually with SMMU secure DMA will be supported until then - provide contiguous memory to guest and map 1:1 2nd stage tables - Guest IPA == HPA. This approach has some merits and has been utilized by propriotary solutions primarily in Wireless where guests are trusted (some devices like SEC engines may not be behind SMMU and DMA to memory look-aside crypto).

Mario.

---
 arch/arm/kvm/Kconfig            |   14 ++++
 drivers/irqchip/irq-gic.c       |  148 ++++++++++++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic.h |    6 ++
 3 files changed, 165 insertions(+), 3 deletions(-)

index 49dd64e..3fcc18e 100644
@@ -59,6 +59,20 @@ config KVM_ARM_VGIC
 	---help---
 	  Adds support for a hardware assisted, in-kernel GIC emulation.
 
+config KVM_ARM_INT_PRIO_DROP
+	bool "KVM support for Interrupt pass-through"
+	depends on KVM_ARM_VGIC && OF
+	default n
+	---help---
+	  Seperates interrupt priority drop and deactivation to enable device
+	  pass-through to Guests. Enabling this config seperates priority drop
+	  from deactivation of the interrupt (active->inactive or pending).
+	  This requires write to EOIR and DIR registers. For interrupts passed
+	  through to Guest, the host does the EOIR and Guest EOIR deactivates
+	  the interrupt. For interrupts not passed through. The host writes to
+	  EOIR and DIR registers. For host to access the DIR reg additional
+	  page must be mapped for GICC (i.e. 2 pages total)
+
 config KVM_ARM_TIMER
 	bool "KVM support for Architected Timers"
 	depends on KVM_ARM_VGIC && ARM_ARCH_TIMER

index 3fd8e42..5d387ce 100644
@@ -17,6 +17,7 @@
 #define GIC_CPU_EOI			0x10
 #define GIC_CPU_RUNNINGPRI		0x14
 #define GIC_CPU_HIGHPRI			0x18
+#define GIC_CPU_DIR                     0x1000
 
 #define GIC_DIST_CTRL			0x000
 #define GIC_DIST_CTR			0x004

@@ -74,6 +75,11 @@ static inline void gic_init(unsigned int nr, int start,
 	gic_init_bases(nr, start, dist, cpu, 0, NULL);
 }
 
+/* Functions to manage interrupt pass-through */
+void gic_spi_set_priodrop(int);
+void gic_spi_clr_priodrop(int);
+int gic_spi_get_priodrop(int);
+
 #endif /* __ASSEMBLY */
 
 #endif

1.7.9.5