qemu-kvm: MSI-X support

Here's a draft MSI-X support patch. Among missing features:
save/load support, and command-line flag to control the feature. This is on top of qemu-kvm: msi-x is disabled without kvm interrupt injection support for now.

Michael S. Tsirkin (2):
qemu-kvm: add MSI-X support
qemu-kvm: use common code for assigned msix

 Makefile.target        |    2 +-
 hw/device-assignment.c |  336 +++++++++++---------------------------------
 hw/device-assignment.h |    8 +-
 hw/msix.c              |  371 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/msix.h              |   33 +++++
 hw/pci.c               |   35 ++++--
 hw/pci.h               |   57 +++++++-
 hw/virtio-balloon.c    |    2 +-
 hw/virtio-blk.c        |    3 +-
 hw/virtio-console.c    |    3 +-
 hw/virtio-net.c        |    3 +-
 hw/virtio.c            |  167 +++++++++++++++++-----
 hw/virtio.h            |    4 +-
 13 files changed, 709 insertions(+), 315 deletions(-)
 create mode 100644 hw/msix.c
 create mode 100644 hw/msix.h

What's your impression of how much work would be to get this going on top of upstream QEMU?

I'm willing to borrow a few cycles to help out here. I'd really like to see this series go in via QEMU if possible.

Regards,

Anthony Liguori

What's your impression of how much work would be to get this going on top of upstream QEMU?

I'm willing to borrow a few cycles to help out here. I'd really like to see this series go in via QEMU if possible.

Regards,

Anthony Liguori

It seems that if I just call apic_deliver_irq each time I want to send MSI, things will work.

However, large part of the msix code is managing IRQs versus kernel, and I'm not sure it's a wise investment of effort to rip it all out. So IMHO, what's missing is API that abstracts managing irq routes in kvm, specifically abstract this stuff in some way:
kvm_get_irq_route_gsi
kvm_add_routing_entry
kvm_del_routing_entry
kvm_commit_irq_routes
kvm_set_irq
How hard is that?

For now, this API could be a stub that just stores the routes somewhere, and set_irq would call the local apic emulation, along the lines of:

	uint8_t dest = (addr_lo & MSI_ADDR_DEST_ID_MASK)
		>> MSI_ADDR_DEST_ID_SHIFT;
	uint8_t vector = (addr_hi & MSI_DATA_VECTOR_MASK)
		>> MSI_DATA_VECTOR_SHIFT;
	uint8_t dest_mode = (addr_lo >> MSI_ADDR_DEST_MODE_SHIFT) & 0x1;
	uint8_t trigger_mode = (data >> MSI_DATA_TRIGGER_SHIFT) & 0x1;
	uint8_t delivery_mode = (data >> MSI_DATA_DELIVERY_MODE_SHIFT) &
		0x7;
	apic_deliver_irq(dest, dest_mode, delivery_mode, vector, 0,
			 trigger_mode);

I would be happy to port my msix code to work on top of this API. Willing to help?

MST

It seems that if I just call apic_deliver_irq each time I want to send MSI, things will work.

However, large part of the msix code is managing IRQs versus kernel, and I'm not sure it's a wise investment of effort to rip it all out. So IMHO, what's missing is API that abstracts managing irq routes in kvm, specifically abstract this stuff in some way:
kvm_get_irq_route_gsi
kvm_add_routing_entry
kvm_del_routing_entry
kvm_commit_irq_routes
kvm_set_irq
How hard is that?

For now, this API could be a stub that just stores the routes somewhere, and set_irq would call the local apic emulation, along the lines of:

	uint8_t dest = (addr_lo & MSI_ADDR_DEST_ID_MASK)
		>> MSI_ADDR_DEST_ID_SHIFT;
	uint8_t vector = (addr_hi & MSI_DATA_VECTOR_MASK)
		>> MSI_DATA_VECTOR_SHIFT;
	uint8_t dest_mode = (addr_lo >> MSI_ADDR_DEST_MODE_SHIFT) & 0x1;
	uint8_t trigger_mode = (data >> MSI_DATA_TRIGGER_SHIFT) & 0x1;
	uint8_t delivery_mode = (data >> MSI_DATA_DELIVERY_MODE_SHIFT) &
		0x7;
	apic_deliver_irq(dest, dest_mode, delivery_mode, vector, 0,
			 trigger_mode);

I would be happy to port my msix code to work on top of this API. Willing to help?

MST

All these are just games with qemu_irq objects. Should be a lot simpler in userspace.

qemu_set_irq().

Should be pretty easy, once you get the hang of qemu_irq.

qemu_set_irq() eventually calls a callback that you specify; just set it do look up the entry and call apic_deliver_irq.

Do not meddle in the internals of kernels, for they are subtle and quick to panic.