Skip to contentSkip to conversation

KVM platform device passthrough

RFC Superseded Eric Auger 4 participants 28 messages Jun 2, 2014 – Jun 26, 2014

v3 of 18 Superseded Latest is v18 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v3 cover KVM platform device passthrough

This RFC series aims at enabling KVM platform device passthrough. It implements a VFIO platform device and offers the capability to instantiate this VFIO device in mach-virt.

The VFIO platform device uses an host VFIO platform driver which must be bound to the assigned device prior to the QEMU system start.

- the guest can directly access the device register space
- assigned device IRQs are transparently routed to the guest by QEMU/KVM (2 methods currently are supported)
- iommu is transparently programmed to prevent the device from accessing physical pages outside of the guest address space

The patch series is made of the following patch files

1) provides a mach_virt implementation where a VFIO device is instantiated at a fixed location

2) addresses A.Williamson's comment to have the platform device code separated from the PCI device code. The VFIO device is moved in a new directory hw/vfio/

3) provides a VFIO platform device that supports MMIO direct accesses. the vfio device was reworked to factorize at best the code between the PCI device and the platform device.

4) simplifies pci device trace calls using common "name" field

5) provides initial IRQ support. The device IRQ now are routed to the guest. IRQ handling is based on eventfds handled on user side. End of interrupt is detected by trapping guest access to MMIO. Functional but suffers from some performance limitations.

6) enables the QEMU end-user to dynamically assign the device from command line, using -device option. Request a minimal knowledge from the end-user (vfio driver name and compatibility). From that point on the VFIO platform device becomes fully generic. a single compat string is supported. A single MMIO region is supported.

7) regions are IOMMU mapped as executable. This feature is requested for some DMA devices that fetch code from some regions (typically the PL330).

8) Add support for multi compat strings. This feature is requested for Primecell devices

9) forces eventfd notifying mechanism

10) Introduces a new way of IRQ routing (based on KVM irqfd/GSI routing). This method is far more performant than the one introduced in 4) since eventfds are handled on host kernel side and also interrupt completion is trapped at GIC level.

v1 (Kim Phillips):
Initial versions for 1, 2, 3

v1->v2 changes (Kim Phillips, Eric Auger):
- reworked split between PCI and platform (3)
- IRQ initial support (5)
- dynamic instantiation (6)

v2->v3 changes (Alvise Rigo, Eric Auger):
- Following Alex recommandations, further efforts to factorize the code between PCI, platform usage of VFIOPlatform and VFIORegion as base classes (3, 4)
- cleanup following Kim's comments
- multiple IRQ support mechanics should be in place although not tested
- Better handling of MMIO multiple regions
- New features and fixes by Alvise (7, 8, 9)
- irqfd support (10)

This patch has the following dependencies on kernel side:

- [RFC Patch v5 0/11] VFIO support for platform devices
http://www.spinics.net/lists/kvm/msg102309.html
- [Patch] ARM: KVM: Handle IPA unmapping on memory region deletion
https://patches.linaro.org/27691/
- [PATCH v2] ARM: KVM: add irqfd and irq routing support
https://patches.linaro.org/29896/
- [PATCH] ARM: KVM: Enable the KVM-VFIO device
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-March/008629.html
- [PATCH] ARM: KVM: user_mem_abort: support stage 2 MMIO page mapping
https://lists.cs.columbia.edu/pipermail/kvmarm/2014-March/008630.html

The patch series was tested on Calxeda Midway (ARMv7) where one xgmac is assigned to KVM host while the second one is assigned to the guest.

Tentative Plan:
- further IRQ handling optimizations (removal of maintenance IRQ)
- unbind/migration/reset problematics
- multi-instantiation testing
- multiple IRQ testing
- management of platform devices with more complex device tree node

Here are the instructions to test on a Calxeda Midway:

https://wiki.linaro.org/LEG/Engineering/Virtualization/Platform_Device_Passthrough_on_Midway

git://git.linaro.org/people/eric.auger/linux.git (branch irqfd_integ_v2)
git://git.linaro.org/people/eric.auger/qemu.git (branch vfio-dev-integ-RFCv3)

Best Regards

Eric

Alvise Rigo (3):
Add EXEC_FLAG to VFIO DMA mappings
Add AMBA devices support to VFIO
Always use eventfd as notifying mechanism

Eric Auger (4):
vfio: simplifed DPRINTF calls using device name
vfio: Add initial IRQ support in platform device
virt: Assign a VFIO platform device with -device option
vfio: Add irqfd support in platform device

Kim Phillips (3):
hw/arm/virt: add a xgmac device
vfio: move hw/misc/vfio.c to hw/vfio/pci.c
vfio: add vfio-platform support

 LICENSE                        |    2 +-
 MAINTAINERS                    |    2 +-
 hw/Makefile.objs               |    1 +
 hw/arm/virt.c                  |  238 +++++-
 hw/intc/arm_gic_kvm.c          |    1 +
 hw/misc/Makefile.objs          |    1 -
 hw/vfio/Makefile.objs          |    5 +
 hw/vfio/common.c               |  854 ++++++++++++++++++++++
 hw/{misc/vfio.c => vfio/pci.c} | 1562 ++++++++++------------------------------
 hw/vfio/platform.c             |  733 +++++++++++++++++++
 hw/vfio/vfio-common.h          |  153 ++++
 linux-headers/linux/vfio.h     |    3 +
 12 files changed, 2378 insertions(+), 1177 deletions(-)
 create mode 100644 hw/vfio/Makefile.objs
 create mode 100644 hw/vfio/common.c
 rename hw/{misc/vfio.c => vfio/pci.c} (65%)
 create mode 100644 hw/vfio/platform.c
 create mode 100644 hw/vfio/vfio-common.h
Signature · 1 line
1.8.3.2
author
#1 lore raw

RFC v3 1/10 hw/arm/virt: add a xgmac device

From: Kim Phillips <[email protected]>

This is a hack and only serves as an example of what needs to be done to make the next RFC - add vfio-platform support - work for development purposes on a Calxeda Midway system. We don't want mach-virt to always create this ethernet device - DO NOT APPLY, etc.

Initial attempts to convince QEMU to create a memory mapped device on the command line (e.g., -device vfio-platform,name=fff51000.ethernet) would fail with "Parameter 'driver' expects pluggable device type". Any guidance as to how to overcome this apparent design limitation is welcome.

RAM is reduced from 30 to 1GiB such as to not overlap the xgmac device's physical address. Not sure if the 30GiB RAM (or whatever the user sets it to with -m) could be set up above 0x1_0000_0000, but there is probably extra work needed to resolve this type of conflict.

note: vfio-platform interrupt support development may want interrupt property data filled; here it's omitted for the time being.

---
 hw/arm/virt.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
hw/arm/virt.c +23 -1
index ea4f02d..becd76b 100644
@@ -65,6 +65,7 @@ enum {
     VIRT_GIC_CPU,
     VIRT_UART,
     VIRT_MMIO,
+    VIRT_ETHERNET,
 };
 
 typedef struct MemMapEntry {
@@ -104,7 +105,8 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_MMIO] = { 0xa000000, 0x200 },
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     /* 0x10000000 .. 0x40000000 reserved for PCI */
-    [VIRT_MEM] = { 0x40000000, 30ULL * 1024 * 1024 * 1024 },
+    [VIRT_MEM] = { 0x40000000, 1ULL * 1024 * 1024 * 1024 },
+    [VIRT_ETHERNET] = { 0xfff51000, 0x1000 },
 };
 
 static const int a15irqmap[] = {
@@ -340,6 +342,25 @@ static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic)
     g_free(nodename);
 }
 
+static void create_ethernet(const VirtBoardInfo *vbi, qemu_irq *pic)
+{
+    char *nodename;
+    hwaddr base = vbi->memmap[VIRT_ETHERNET].base;
+    hwaddr size = vbi->memmap[VIRT_ETHERNET].size;
+    const char compat[] = "calxeda,hb-xgmac";
+
+    sysbus_create_simple("vfio-platform", base, NULL);
+
+    nodename = g_strdup_printf("/ethernet@%" PRIx64, base);
+    qemu_fdt_add_subnode(vbi->fdt, nodename);
+
+    /* Note that we can't use setprop_string because of the embedded NUL */
+    qemu_fdt_setprop(vbi->fdt, nodename, "compatible", compat, sizeof(compat));
+    qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "reg", 2, base, 2, size);
+
+    g_free(nodename);
+}
+
 static void create_virtio_devices(const VirtBoardInfo *vbi, qemu_irq *pic)
 {
     int i;
@@ -454,6 +475,7 @@ static void machvirt_init(QEMUMachineInitArgs *args)
     create_gic(vbi, pic);
 
     create_uart(vbi, pic);
+    create_ethernet(vbi, pic);
 
     /* Create mmio transports, so the user can create virtio backends
      * (which will be automatically plugged in to the transports). If
Signature · 1 line
1.8.3.2
author
#2 lore raw

RFC v3 2/10 vfio: move hw/misc/vfio.c to hw/vfio/pci.c

In reply to Eric Auger

From: Kim Phillips <[email protected]>

This is done in preparation for the addition of VFIO platform device support.

---
 LICENSE                        | 2 +-
 MAINTAINERS                    | 2 +-
 hw/Makefile.objs               | 1 +
 hw/misc/Makefile.objs          | 1 -
 hw/vfio/Makefile.objs          | 3 +++
 hw/{misc/vfio.c => vfio/pci.c} | 0
 6 files changed, 6 insertions(+), 3 deletions(-)
 create mode 100644 hw/vfio/Makefile.objs
 rename hw/{misc/vfio.c => vfio/pci.c} (100%)
LICENSE +1 -1
index da70e94..0e0b4b9 100644
@@ -11,7 +11,7 @@ option) any later version.
 
 As of July 2013, contributions under version 2 of the GNU General Public
 License (and no later version) are only accepted for the following files
-or directories: bsd-user/, linux-user/, hw/misc/vfio.c, hw/xen/xen_pt*.
+or directories: bsd-user/, linux-user/, hw/vfio/, hw/xen/xen_pt*.
 
 3) The Tiny Code Generator (TCG) is released under the BSD license
    (see license headers in files).
MAINTAINERS +1 -1
index 97c9fa1..0b58184 100644
@@ -613,7 +613,7 @@ F: tests/usb-hcd-ehci-test.c
 VFIO
 M: Alex Williamson <[email protected]>
 S: Supported
-F: hw/misc/vfio.c
+F: hw/vfio/*
 
 vhost
 M: Michael S. Tsirkin <[email protected]>
hw/Makefile.objs +1 -0
index d178b65..b16dada 100644
@@ -26,6 +26,7 @@ devices-dirs-$(CONFIG_SOFTMMU) += ssi/
 devices-dirs-$(CONFIG_SOFTMMU) += timer/
 devices-dirs-$(CONFIG_TPM) += tpm/
 devices-dirs-$(CONFIG_SOFTMMU) += usb/
+devices-dirs-$(CONFIG_SOFTMMU) += vfio/
 devices-dirs-$(CONFIG_VIRTIO) += virtio/
 devices-dirs-$(CONFIG_SOFTMMU) += watchdog/
 devices-dirs-$(CONFIG_SOFTMMU) += xen/
hw/misc/Makefile.objs +0 -1
index f674365..656570c 100644
@@ -21,7 +21,6 @@ common-obj-$(CONFIG_MACIO) += macio/
 
 ifeq ($(CONFIG_PCI), y)
 obj-$(CONFIG_KVM) += ivshmem.o
-obj-$(CONFIG_LINUX) += vfio.o
 endif
 
 obj-$(CONFIG_REALVIEW) += arm_sysctl.o
hw/vfio/Makefile.objs new file +3 -0
new file mode 100644
index 0000000..31c7dab
@@ -0,0 +1,3 @@
+ifeq ($(CONFIG_LINUX), y)
+obj-$(CONFIG_PCI) += pci.o
+endif
hw/vfio/pci.c renamed from hw/misc/vfio.c +0 -0
similarity index 100%
rename from hw/misc/vfio.c
rename to hw/vfio/pci.c
Signature · 1 line
1.8.3.2
author
#3 lore raw

RFC v3 3/10 vfio: add vfio-platform support

In reply to Eric Auger

From: Kim Phillips <[email protected]>

Functions for which PCI and platform device support share are moved into common.c. The common vfio_{get,put}_group() get an additional argument, a pointer to a vfio_reset_handler(), for which to pass on to qemu_register_reset, but only if it exists (the platform device code currently passes a NULL as its reset_handler).

For the platform device code, we basically use SysBusDevice instead of PCIDevice. Since realize() returns void, unlike PCIDevice's initfn, error codes are moved into the error message text with %m.

Currently only MMIO access is supported at this time.

The perceived path for future QEMU development is:

- add support for interrupts
- verify and test platform dev unmap path
- test existing PCI path for any regressions
- add support for creating platform devices on the qemu command line
- currently device address specification is hardcoded for test development on Calxeda Midway's fff51000.ethernet device
- reset is not supported and registration of reset functions is bypassed for platform devices.
- there is no standard means of resetting a platform device, unsure if it suffices to be handled at device--VFIO binding time

[1] http://www.spinics.net/lists/kvm-arm/msg08195.html

Changes (v2 -> v3):
[work done by Eric Auger]

This new version introduces 2 separate VFIO Device objects:
- VFIOPCIDevice
- VFIOPlatformDevice

Both objects share a VFIODevice struct.

Also a VFIORegion shared struct was created. It is embedded in VFIOBAR struct. VFIOPlatformDevice uses VFIORegion directly.

Introducing those base classes induced quite a lot of tiny changes in the PCI code. Theoretically PCI and platform devices can be supported simultaneously. PCI modifications currently are not tested.

The VFIODevice is not a QOM object due to the single inheritance model limitation.

The VFIODevice struct embeds an ops structure which is specialized in each VFIO leaf device. This makes possible to call device specific functions in common parts, hence achieving better factorization.

Reset handling typically is handled that way where a unique generic ResetHandler (in common.c) is used for both derived classes. It calls device specific methods.

As in the original contribution, only MMIO is supported in that patch file (in mmap mode). IRQ support comes in a subsequent patch.

---
 hw/vfio/Makefile.objs      |    2 +
 hw/vfio/common.c           |  849 ++++++++++++++++++++++++++++
 hw/vfio/pci.c              | 1316 ++++++++++----------------------------------
 hw/vfio/platform.c         |  267 +++++++++
 hw/vfio/vfio-common.h      |  143 +++++
 linux-headers/linux/vfio.h |    1 +
 6 files changed, 1565 insertions(+), 1013 deletions(-)
 create mode 100644 hw/vfio/common.c
 create mode 100644 hw/vfio/platform.c
 create mode 100644 hw/vfio/vfio-common.h
hw/vfio/Makefile.objs +2 -0
index 31c7dab..c5c76fe 100644
@@ -1,3 +1,5 @@
 ifeq ($(CONFIG_LINUX), y)
+obj-$(CONFIG_SOFTMMU) += common.o
 obj-$(CONFIG_PCI) += pci.o
+obj-$(CONFIG_SOFTMMU) += platform.o
 endif
hw/vfio/common.c new file +849 -0

849 lines not on this page · raw message

hw/vfio/pci.c +303 -1013

2,146 lines not on this page · raw message

hw/vfio/platform.c new file +267 -0

267 lines not on this page · raw message

hw/vfio/vfio-common.h new file +143 -0

143 lines not on this page · raw message

linux-headers/linux/vfio.h +1 -0
index 26c218e..ef4815d 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
1.8.3.2

Part of the diff is not shown here. View the raw message.

#4 lore raw

Re: [Qemu-devel] [RFC v3 03/10] vfio: add vfio-platform support

Quoted 64 lines · Eric Auger

From: Kim Phillips <[email protected]>

Functions for which PCI and platform device support share are moved into common.c. The common vfio_{get,put}_group() get an additional argument, a pointer to a vfio_reset_handler(), for which to pass on to qemu_register_reset, but only if it exists (the platform device code currently passes a NULL as its reset_handler).

For the platform device code, we basically use SysBusDevice instead of PCIDevice. Since realize() returns void, unlike PCIDevice's initfn, error codes are moved into the error message text with %m.

Currently only MMIO access is supported at this time.

The perceived path for future QEMU development is:

- add support for interrupts
- verify and test platform dev unmap path
- test existing PCI path for any regressions
- add support for creating platform devices on the qemu command line
- currently device address specification is hardcoded for test development on Calxeda Midway's fff51000.ethernet device
- reset is not supported and registration of reset functions is bypassed for platform devices.
- there is no standard means of resetting a platform device, unsure if it suffices to be handled at device--VFIO binding time

[1] http://www.spinics.net/lists/kvm-arm/msg08195.html

Changes (v2 -> v3):
[work done by Eric Auger]

This new version introduces 2 separate VFIO Device objects:
- VFIOPCIDevice
- VFIOPlatformDevice

Both objects share a VFIODevice struct.

Also a VFIORegion shared struct was created. It is embedded in VFIOBAR struct. VFIOPlatformDevice uses VFIORegion directly.

Introducing those base classes induced quite a lot of tiny changes in the PCI code. Theoretically PCI and platform devices can be supported simultaneously. PCI modifications currently are not tested.

The VFIODevice is not a QOM object due to the single inheritance model limitation.

The VFIODevice struct embeds an ops structure which is specialized in each VFIO leaf device. This makes possible to call device specific functions in common parts, hence achieving better factorization.

Reset handling typically is handled that way where a unique generic ResetHandler (in common.c) is used for both derived classes. It calls device specific methods.

As in the original contribution, only MMIO is supported in that patch file (in mmap mode). IRQ support comes in a subsequent patch.

Quoted 8 lines
---
  hw/vfio/Makefile.objs      |    2 +
  hw/vfio/common.c           |  849 ++++++++++++++++++++++++++++
  hw/vfio/pci.c              | 1316 ++++++++++----------------------------------
  hw/vfio/platform.c         |  267 +++++++++
  hw/vfio/vfio-common.h      |  143 +++++
  linux-headers/linux/vfio.h |    1 +
  6 files changed, 1565 insertions(+), 1013 deletions(-)

This patch is impossible to review. Please split it out into a part (or maybe even multiple patches) that separate pci specific VFIO support into its own file and then another set of patches implementing platform support.

Alex

author
#5 lore raw

Re: [Qemu-devel] [RFC v3 03/10] vfio: add vfio-platform support

Quoted 76 lines · Alexander Graf
Quoted 64 lines · Eric Auger

From: Kim Phillips <[email protected]>

Functions for which PCI and platform device support share are moved into common.c. The common vfio_{get,put}_group() get an additional argument, a pointer to a vfio_reset_handler(), for which to pass on to qemu_register_reset, but only if it exists (the platform device code currently passes a NULL as its reset_handler).

For the platform device code, we basically use SysBusDevice instead of PCIDevice. Since realize() returns void, unlike PCIDevice's initfn, error codes are moved into the error message text with %m.

Currently only MMIO access is supported at this time.

The perceived path for future QEMU development is:

- add support for interrupts
- verify and test platform dev unmap path
- test existing PCI path for any regressions
- add support for creating platform devices on the qemu command line
- currently device address specification is hardcoded for test development on Calxeda Midway's fff51000.ethernet device
- reset is not supported and registration of reset functions is bypassed for platform devices.
- there is no standard means of resetting a platform device, unsure if it suffices to be handled at device--VFIO binding time

[1] http://www.spinics.net/lists/kvm-arm/msg08195.html

Changes (v2 -> v3):
[work done by Eric Auger]

This new version introduces 2 separate VFIO Device objects:
- VFIOPCIDevice
- VFIOPlatformDevice

Both objects share a VFIODevice struct.

Also a VFIORegion shared struct was created. It is embedded in VFIOBAR struct. VFIOPlatformDevice uses VFIORegion directly.

Introducing those base classes induced quite a lot of tiny changes in the PCI code. Theoretically PCI and platform devices can be supported simultaneously. PCI modifications currently are not tested.

The VFIODevice is not a QOM object due to the single inheritance model limitation.

The VFIODevice struct embeds an ops structure which is specialized in each VFIO leaf device. This makes possible to call device specific functions in common parts, hence achieving better factorization.

Reset handling typically is handled that way where a unique generic ResetHandler (in common.c) is used for both derived classes. It calls device specific methods.

As in the original contribution, only MMIO is supported in that patch file (in mmap mode). IRQ support comes in a subsequent patch.

Quoted 9 lines
---
  hw/vfio/Makefile.objs      |    2 +
  hw/vfio/common.c           |  849 ++++++++++++++++++++++++++++
  hw/vfio/pci.c              | 1316
++++++++++----------------------------------
  hw/vfio/platform.c         |  267 +++++++++
  hw/vfio/vfio-common.h      |  143 +++++
  linux-headers/linux/vfio.h |    1 +
  6 files changed, 1565 insertions(+), 1013 deletions(-)
Quoted 4 lines

This patch is impossible to review. Please split it out into a part (or maybe even multiple patches) that separate pci specific VFIO support into its own file and then another set of patches implementing platform support.

Hi Alex,

OK I will reorganize the patch as you suggest.

Alex (both of you), do you think the general split direction looks reasonable, ie. having this VFIODevice struct and VFIORegion struct? or do you think it induces too many (and possibly cumbersome) changes in the PCI code?

wrt VFIOPlatformDevice I changed the code to inherit from PlatformDevice instead of SysBusDevice - maybe too early but future will say ... - .

Best Regards

Eric

Quoted 4 lines

Alex

#6 lore raw

Re: [Qemu-devel] [RFC v3 03/10] vfio: add vfio-platform support

Quoted 79 lines · Eric Auger
Quoted 74 lines · Alexander Graf
Quoted 64 lines · Eric Auger

From: Kim Phillips <[email protected]>

Functions for which PCI and platform device support share are moved into common.c. The common vfio_{get,put}_group() get an additional argument, a pointer to a vfio_reset_handler(), for which to pass on to qemu_register_reset, but only if it exists (the platform device code currently passes a NULL as its reset_handler).

For the platform device code, we basically use SysBusDevice instead of PCIDevice. Since realize() returns void, unlike PCIDevice's initfn, error codes are moved into the error message text with %m.

Currently only MMIO access is supported at this time.

The perceived path for future QEMU development is:

- add support for interrupts
- verify and test platform dev unmap path
- test existing PCI path for any regressions
- add support for creating platform devices on the qemu command line
- currently device address specification is hardcoded for test development on Calxeda Midway's fff51000.ethernet device
- reset is not supported and registration of reset functions is bypassed for platform devices.
- there is no standard means of resetting a platform device, unsure if it suffices to be handled at device--VFIO binding time

[1] http://www.spinics.net/lists/kvm-arm/msg08195.html

Changes (v2 -> v3):
[work done by Eric Auger]

This new version introduces 2 separate VFIO Device objects:
- VFIOPCIDevice
- VFIOPlatformDevice

Both objects share a VFIODevice struct.

Also a VFIORegion shared struct was created. It is embedded in VFIOBAR struct. VFIOPlatformDevice uses VFIORegion directly.

Introducing those base classes induced quite a lot of tiny changes in the PCI code. Theoretically PCI and platform devices can be supported simultaneously. PCI modifications currently are not tested.

The VFIODevice is not a QOM object due to the single inheritance model limitation.

The VFIODevice struct embeds an ops structure which is specialized in each VFIO leaf device. This makes possible to call device specific functions in common parts, hence achieving better factorization.

Reset handling typically is handled that way where a unique generic ResetHandler (in common.c) is used for both derived classes. It calls device specific methods.

As in the original contribution, only MMIO is supported in that patch file (in mmap mode). IRQ support comes in a subsequent patch.

Quoted 9 lines
---
   hw/vfio/Makefile.objs      |    2 +
   hw/vfio/common.c           |  849 ++++++++++++++++++++++++++++
   hw/vfio/pci.c              | 1316
++++++++++----------------------------------
   hw/vfio/platform.c         |  267 +++++++++
   hw/vfio/vfio-common.h      |  143 +++++
   linux-headers/linux/vfio.h |    1 +
   6 files changed, 1565 insertions(+), 1013 deletions(-)
Quoted 4 lines

This patch is impossible to review. Please split it out into a part (or maybe even multiple patches) that separate pci specific VFIO support into its own file and then another set of patches implementing platform support.

Quoted 8 lines

Hi Alex,

OK I will reorganize the patch as you suggest.

Alex (both of you), do you think the general split direction looks reasonable, ie. having this VFIODevice struct and VFIORegion struct? or do you think it induces too many (and possibly cumbersome) changes in the PCI code?

I don't think we'll be able to tell before the implications are visible - and that basically means splitting the patch :(.

Quoted 2 lines

wrt VFIOPlatformDevice I changed the code to inherit from PlatformDevice instead of SysBusDevice - maybe too early but future will say ... - .

I'll check out which way to go forward there today. But either way, the two aren't incredibly different.

Alex

author
#7 lore raw

RFC v3 4/10 vfio: simplifed DPRINTF calls using device name

In reply to Eric Auger

This patch gets benefit from the new VFIODevice name field.

Occurences of
DPRINTF("%s(%04x:%02x:%02x.%x) ...", __func__, vdev->host.domain,

        vdev->host.bus, vdev->host.slot, vdev->host.function, ...)

are replaced by
DPRINTF("%s(%s ...", __func__, vdev->vdev.name, ...).

name is built using "%04x:%02x:%02x.%01x" format string.

---
 hw/vfio/pci.c | 227 ++++++++++++++++++++--------------------------------------
 1 file changed, 78 insertions(+), 149 deletions(-)
hw/vfio/pci.c +78 -149
index a9e4d97..ad0c2a0 100644
@@ -212,9 +212,7 @@ static void vfio_intx_interrupt(void *opaque)
         return;
     }
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) Pin %c\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function,
-            'A' + vdev->intx.pin);
+    DPRINTF("%s(%s) Pin %c\n", __func__, vdev->vdev.name, 'A' + vdev->intx.pin);
 
     vdev->intx.pending = true;
     pci_irq_assert(&vdev->pdev);
@@ -233,8 +231,7 @@ static void vfio_pci_eoi(VFIODevice *vdev)
         return;
     }
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) EOI\n", __func__, vpcidev->host.domain,
-            vpcidev->host.bus, vpcidev->host.slot, vpcidev->host.function);
+    DPRINTF("%s(%s) EOI\n", __func__, vdev->name);
 
     vpcidev->intx.pending = false;
     pci_irq_deassert(&vpcidev->pdev);
@@ -303,9 +300,7 @@ static void vfio_enable_intx_kvm(VFIOPCIDevice *vdev)
 
     vdev->intx.kvm_accel = true;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) KVM INTx accel enabled\n",
-            __func__, vdev->host.domain, vdev->host.bus,
-            vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s) KVM INTx accel enabled\n", __func__, vdev->vdev.name);
 
     return;
 
@@ -357,9 +352,7 @@ static void vfio_disable_intx_kvm(VFIOPCIDevice *vdev)
     /* If we've missed an event, let it re-fire through QEMU */
     vfio_unmask_irqindex(&vdev->vdev, VFIO_PCI_INTX_IRQ_INDEX);
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) KVM INTx accel disabled\n",
-            __func__, vdev->host.domain, vdev->host.bus,
-            vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s) KVM INTx accel disabled\n", __func__, vdev->vdev.name);
 #endif
 }
 
@@ -378,9 +371,8 @@ static void vfio_update_irq(PCIDevice *pdev)
         return; /* Nothing changed */
     }
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) IRQ moved %d -> %d\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, vdev->intx.route.irq, route.irq);
+    DPRINTF("%s(%s) IRQ moved %d -> %d\n", __func__, vdev->vdev.name,
+            vdev->intx.route.irq, route.irq);
 
     vfio_disable_intx_kvm(vdev);
 
@@ -456,8 +448,7 @@ static int vfio_enable_intx(VFIOPCIDevice *vdev)
 
     vdev->interrupt = VFIO_INT_INTx;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 
     return 0;
 }
@@ -479,8 +470,7 @@ static void vfio_disable_intx(VFIOPCIDevice *vdev)
 
     vdev->interrupt = VFIO_INT_NONE;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 }
 
 /*
@@ -507,9 +497,8 @@ static void vfio_msi_interrupt(void *opaque)
         abort();
     }
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) vector %d 0x%"PRIx64"/0x%x\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, nr, msg.address, msg.data);
+    DPRINTF("%s(%s) vector %d 0x%"PRIx64"/0x%x\n", __func__,
+            vdev->vdev.name, nr, msg.address, msg.data);
 #endif
 
     if (vdev->interrupt == VFIO_INT_MSIX) {
@@ -560,9 +549,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,
     VFIOMSIVector *vector;
     int ret;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) vector %d used\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, nr);
+    DPRINTF("%s(%s) vector %d used\n", __func__, vdev->vdev.name, nr);
 
     vector = &vdev->msi_vectors[nr];
     vector->vdev = vdev;
@@ -645,9 +632,7 @@ static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr)
     struct vfio_irq_set *irq_set;
     int32_t *pfd;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) vector %d released\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, nr);
+    DPRINTF("%s(%s) vector %d released\n", __func__, vdev->vdev.name, nr);
 
     /*
      * XXX What's the right thing to do here?  This turns off the interrupt
@@ -716,8 +701,7 @@ static void vfio_enable_msix(VFIOPCIDevice *vdev)
         error_report("vfio: msix_set_vector_notifiers failed");
     }
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 }
 
 static void vfio_enable_msi(VFIOPCIDevice *vdev)
@@ -792,9 +776,8 @@ retry:
 
     vdev->interrupt = VFIO_INT_MSI;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) Enabled %d MSI vectors\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, vdev->nr_vectors);
+    DPRINTF("%s(%s) Enabled %d MSI vectors\n", __func__, vdev->vdev.name,
+            vdev->nr_vectors);
 }
 
 static void vfio_disable_msi_common(VFIOPCIDevice *vdev)
@@ -829,8 +812,7 @@ static void vfio_disable_msix(VFIOPCIDevice *vdev)
 
     vfio_disable_msi_common(vdev);
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 }
 
 static void vfio_disable_msi(VFIOPCIDevice *vdev)
@@ -861,8 +843,7 @@ static void vfio_disable_msi(VFIOPCIDevice *vdev)
 
     vfio_disable_msi_common(vdev);
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 }
 
 static void vfio_update_msi(VFIOPCIDevice *vdev)
@@ -882,9 +863,8 @@ static void vfio_update_msi(VFIOPCIDevice *vdev)
         if (msg.address != vector->msg.address ||
             msg.data != vector->msg.data) {
 
-            DPRINTF("%s(%04x:%02x:%02x.%x) MSI vector %d changed\n",
-                    __func__, vdev->host.domain, vdev->host.bus,
-                    vdev->host.slot, vdev->host.function, i);
+            DPRINTF("%s(%s) MSI vector %d changed\n", __func__,
+                    vdev->vdev.name, i);
 
             kvm_irqchip_update_msi_route(kvm_state, vector->virq, msg);
             vector->msg = msg;
@@ -918,8 +898,7 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev)
         return;
     }
 
-    DPRINTF("Device %04x:%02x:%02x.%x ROM:\n", vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("Device %s ROM:\n", vdev->vdev.name);
     DPRINTF("  size: 0x%lx, offset: 0x%lx, flags: 0x%lx\n",
             (unsigned long)reg_info.size, (unsigned long)reg_info.offset,
             (unsigned long)reg_info.flags);
@@ -929,10 +908,7 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev)
 
     if (!vdev->rom_size) {
         vdev->rom_read_failed = true;
-        error_report("vfio-pci: Cannot read device rom at "
-                    "%04x:%02x:%02x.%x",
-                    vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                    vdev->host.function);
+        error_report("vfio-pci: Cannot read device rom at %s", vdev->vdev.name);
         error_printf("Device option ROM contents are probably invalid "
                     "(check dmesg).\nSkip option ROM probe with rombar=0, "
                     "or load from file with romfile=\n");
@@ -972,9 +948,8 @@ static uint64_t vfio_rom_read(void *opaque, hwaddr addr, unsigned size)
     memcpy(&val, vdev->rom + addr,
            (addr < vdev->rom_size) ? MIN(size, vdev->rom_size - addr) : 0);
 
-    DPRINTF("%s(%04x:%02x:%02x.%x, 0x%"HWADDR_PRIx", 0x%x) = 0x%"PRIx64"\n",
-            __func__, vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, addr, size, val);
+    DPRINTF("%s(%s, 0x%"HWADDR_PRIx", 0x%x) = 0x%"PRIx64"\n",
+            __func__, vdev->vdev.name, addr, size, val);
 
     return val;
 }
@@ -1021,12 +996,11 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)
     if (vdev->pdev.romfile || !vdev->pdev.rom_bar) {
         /* Since pci handles romfile, just print a message and return */
         if (vfio_blacklist_opt_rom(vdev) && vdev->pdev.romfile) {
-            error_printf("Warning : Device at %04x:%02x:%02x.%x "
+            error_printf("Warning : Device at %s "
                          "is known to cause system instability issues during "
                          "option rom execution. "
                          "Proceeding anyway since user specified romfile\n",
-                         vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                         vdev->host.function);
+                         vdev->vdev.name);
         }
         return;
     }
@@ -1039,9 +1013,7 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)
         pwrite(fd, &size, 4, offset) != 4 ||
         pread(fd, &size, 4, offset) != 4 ||
         pwrite(fd, &orig, 4, offset) != 4) {
-        error_report("%s(%04x:%02x:%02x.%x) failed: %m",
-                     __func__, vdev->host.domain, vdev->host.bus,
-                     vdev->host.slot, vdev->host.function);
+        error_report("%s(%s) failed: %m", __func__, vdev->vdev.name);
         return;
     }
 
@@ -1053,30 +1025,25 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev)
 
     if (vfio_blacklist_opt_rom(vdev)) {
         if (dev->opts && qemu_opt_get(dev->opts, "rombar")) {
-            error_printf("Warning : Device at %04x:%02x:%02x.%x "
+            error_printf("Warning : Device at %s "
                          "is known to cause system instability issues during "
                          "option rom execution. "
                          "Proceeding anyway since user specified non zero value for "
                          "rombar\n",
-                         vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                         vdev->host.function);
+                         vdev->vdev.name);
         } else {
             error_printf("Warning : Rom loading for device at "
-                         "%04x:%02x:%02x.%x has been disabled due to "
+                         "%s has been disabled due to "
                          "system instability issues. "
                          "Specify rombar=1 or romfile to force\n",
-                         vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                         vdev->host.function);
+                         vdev->vdev.name);
             return;
         }
     }
 
-    DPRINTF("%04x:%02x:%02x.%x ROM size 0x%x\n", vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function, size);
+    DPRINTF("%s ROM size 0x%x\n", vdev->vdev.name, size);
 
-    snprintf(name, sizeof(name), "vfio[%04x:%02x:%02x.%x].rom",
-             vdev->host.domain, vdev->host.bus, vdev->host.slot,
-             vdev->host.function);
+    snprintf(name, sizeof(name), "vfio[%s].rom", vdev->vdev.name);
 
     memory_region_init_io(&vdev->pdev.rom, OBJECT(vdev),
                           &vfio_rom_ops, vdev, name, size);
@@ -1207,9 +1174,8 @@ static uint64_t vfio_generic_window_quirk_read(void *opaque,
         data = vfio_pci_read_config(&vdev->pdev,
                                     quirk->data.address_val + offset, size);
 
-        DPRINTF("%s read(%04x:%02x:%02x.%x:BAR%d+0x%"HWADDR_PRIx", %d) = 0x%"
-                PRIx64"\n", memory_region_name(&quirk->mem), vdev->host.domain,
-                vdev->host.bus, vdev->host.slot, vdev->host.function,
+        DPRINTF("%s read(%s:BAR%d+0x%"HWADDR_PRIx", %d) = 0x%"
+                PRIx64"\n", memory_region_name(&quirk->mem), vdev->vdev.name,
                 quirk->data.bar, addr, size, data);
     } else {
         data = vfio_region_read(&vdev->bars[quirk->data.bar].region,
@@ -1256,10 +1222,9 @@ static void vfio_generic_window_quirk_write(void *opaque, hwaddr addr,
 
         vfio_pci_write_config(&vdev->pdev,
                               quirk->data.address_val + offset, data, size);
-        DPRINTF("%s write(%04x:%02x:%02x.%x:BAR%d+0x%"HWADDR_PRIx", 0x%"
+        DPRINTF("%s write(%s:BAR%d+0x%"HWADDR_PRIx", 0x%"
                 PRIx64", %d)\n", memory_region_name(&quirk->mem),
-                vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                vdev->host.function, quirk->data.bar, addr, data, size);
+                vdev->vdev.name, quirk->data.bar, addr, data, size);
         return;
     }
 
@@ -1292,9 +1257,8 @@ static uint64_t vfio_generic_quirk_read(void *opaque,
 
         data = vfio_pci_read_config(&vdev->pdev, addr - offset, size);
 
-        DPRINTF("%s read(%04x:%02x:%02x.%x:BAR%d+0x%"HWADDR_PRIx", %d) = 0x%"
-                PRIx64"\n", memory_region_name(&quirk->mem), vdev->host.domain,
-                vdev->host.bus, vdev->host.slot, vdev->host.function,
+        DPRINTF("%s read(%s:BAR%d+0x%"HWADDR_PRIx", %d) = 0x%"
+                PRIx64"\n", memory_region_name(&quirk->mem), vdev->vdev.name,
                 quirk->data.bar, addr + base, size, data);
     } else {
         data = vfio_region_read(&vdev->bars[quirk->data.bar].region,
@@ -1322,10 +1286,9 @@ static void vfio_generic_quirk_write(void *opaque, hwaddr addr,
 
         vfio_pci_write_config(&vdev->pdev, addr - offset, data, size);
 
-        DPRINTF("%s write(%04x:%02x:%02x.%x:BAR%d+0x%"HWADDR_PRIx", 0x%"
+        DPRINTF("%s write(%s:BAR%d+0x%"HWADDR_PRIx", 0x%"
                 PRIx64", %d)\n", memory_region_name(&quirk->mem),
-                vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                vdev->host.function, quirk->data.bar, addr + base, data, size);
+                vdev->vdev.name, quirk->data.bar, addr + base, data, size);
     } else {
         vfio_region_write(&vdev->bars[quirk->data.bar].region, addr + base,
                             data, size);
@@ -1396,9 +1359,7 @@ static void vfio_vga_probe_ati_3c3_quirk(VFIOPCIDevice *vdev)
     QLIST_INSERT_HEAD(&vdev->vga.region[QEMU_PCI_VGA_IO_HI].quirks,
                       quirk, next);
 
-    DPRINTF("Enabled ATI/AMD quirk 0x3c3 BAR4for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled ATI/AMD quirk 0x3c3 BAR4for device %s\n", vdev->vdev.name);
 }
 
 /*
@@ -1439,9 +1400,8 @@ static void vfio_probe_ati_bar4_window_quirk(VFIOPCIDevice *vdev, int nr)
 
     QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next);
 
-    DPRINTF("Enabled ATI/AMD BAR4 window quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled ATI/AMD BAR4 window quirk for device %s\n",
+            vdev->vdev.name);
 }
 
 /*
@@ -1474,9 +1434,8 @@ static void vfio_probe_ati_bar2_4000_quirk(VFIOPCIDevice *vdev, int nr)
 
     QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next);
 
-    DPRINTF("Enabled ATI/AMD BAR2 0x4000 quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled ATI/AMD BAR2 0x4000 quirk for device %s\n",
+            vdev->vdev.name);
 }
 
 /*
@@ -1609,9 +1568,7 @@ static void vfio_vga_probe_nvidia_3d0_quirk(VFIOPCIDevice *vdev)
     QLIST_INSERT_HEAD(&vdev->vga.region[QEMU_PCI_VGA_IO_HI].quirks,
                       quirk, next);
 
-    DPRINTF("Enabled NVIDIA VGA 0x3d0 quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled NVIDIA VGA 0x3d0 quirk for device %s\n", vdev->vdev.name);
 }
 
 /*
@@ -1700,9 +1657,8 @@ static void vfio_probe_nvidia_bar5_window_quirk(VFIOPCIDevice *vdev, int nr)
 
     QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next);
 
-    DPRINTF("Enabled NVIDIA BAR5 window quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled NVIDIA BAR5 window quirk for device %s\n",
+            vdev->vdev.name);
 }
 
 static void vfio_nvidia_88000_quirk_write(void *opaque, hwaddr addr,
@@ -1769,9 +1725,8 @@ static void vfio_probe_nvidia_bar0_88000_quirk(VFIOPCIDevice *vdev, int nr)
 
     QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next);
 
-    DPRINTF("Enabled NVIDIA BAR0 0x88000 quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled NVIDIA BAR0 0x88000 quirk for device %s\n",
+            vdev->vdev.name);
 }
 
 /*
@@ -1808,9 +1763,8 @@ static void vfio_probe_nvidia_bar0_1800_quirk(VFIOPCIDevice *vdev, int nr)
 
     QLIST_INSERT_HEAD(&vdev->bars[nr].quirks, quirk, next);
 
-    DPRINTF("Enabled NVIDIA BAR0 0x1800 quirk for device %04x:%02x:%02x.%x\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("Enabled NVIDIA BAR0 0x1800 quirk for device %s\n",
+            vdev->vdev.name);
 }
 
 /*
@@ -1885,9 +1839,8 @@ static uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len)
 
         ret = pread(vdev->vdev.fd, &phys_val, len, vdev->config_offset + addr);
         if (ret != len) {
-            error_report("%s(%04x:%02x:%02x.%x, 0x%x, 0x%x) failed: %m",
-                         __func__, vdev->host.domain, vdev->host.bus,
-                         vdev->host.slot, vdev->host.function, addr, len);
+            error_report("%s(%s, 0x%x, 0x%x) failed: %m",
+                         __func__, vdev->vdev.name, addr, len);
             return -errno;
         }
         phys_val = le32_to_cpu(phys_val);
@@ -1895,9 +1848,8 @@ static uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len)
 
     val = (emu_val & emu_bits) | (phys_val & ~emu_bits);
 
-    DPRINTF("%s(%04x:%02x:%02x.%x, @0x%x, len=0x%x) %x\n", __func__,
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, addr, len, val);
+    DPRINTF("%s(%s, @0x%x, len=0x%x) %x\n", __func__, vdev->vdev.name,
+            addr, len, val);
 
     return val;
 }
@@ -1912,11 +1864,10 @@ static void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr,
             addr, val, len);
 
     /* Write everything to VFIO, let it filter out what we can't write */
-    if (pwrite(vdev->vdev.fd, &val_le, len, 
+    if (pwrite(vdev->vdev.fd, &val_le, len,
                 vdev->config_offset + addr) != len) {
-        error_report("%s(%04x:%02x:%02x.%x, 0x%x, 0x%x, 0x%x) failed: %m",
-                     __func__, vdev->host.domain, vdev->host.bus,
-                     vdev->host.slot, vdev->host.function, addr, val, len);
+        error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m",
+                    __func__, vdev->vdev.name, addr, val, len);
     }
 
     /* MSI/MSI-X Enabling/Disabling */
@@ -1992,8 +1943,7 @@ static int vfio_setup_msi(VFIOPCIDevice *vdev, int pos)
     msi_maskbit = !!(ctrl & PCI_MSI_FLAGS_MASKBIT);
     entries = 1 << ((ctrl & PCI_MSI_FLAGS_QMASK) >> 1);
 
-    DPRINTF("%04x:%02x:%02x.%x PCI MSI CAP @0x%x\n", vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function, pos);
+    DPRINTF("%s PCI MSI CAP @0x%x\n", vdev->vdev.name, pos);
 
     ret = msi_init(&vdev->pdev, pos, entries, msi_64bit, msi_maskbit);
     if (ret < 0) {
@@ -2054,10 +2004,8 @@ static int vfio_early_setup_msix(VFIOPCIDevice *vdev)
     vdev->msix->pba_offset = pba & ~PCI_MSIX_FLAGS_BIRMASK;
     vdev->msix->entries = (ctrl & PCI_MSIX_FLAGS_QSIZE) + 1;
 
-    DPRINTF("%04x:%02x:%02x.%x "
-            "PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function, pos, vdev->msix->table_bar,
+    DPRINTF("%s PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d\n",
+            vdev->vdev.name, pos, vdev->msix->table_bar,
             vdev->msix->table_offset, vdev->msix->entries);
 
     return 0;
@@ -2151,9 +2099,7 @@ static void vfio_map_bar(VFIOPCIDevice *vdev, int nr)
         return;
     }
 
-    snprintf(name, sizeof(name), "VFIO %04x:%02x:%02x.%x BAR %d",
-             vdev->host.domain, vdev->host.bus, vdev->host.slot,
-             vdev->host.function, nr);
+    snprintf(name, sizeof(name), "VFIO %s BAR %d", vdev->vdev.name, nr);
 
     /* Determine what type of BAR this is for registration */
     ret = pread(vdev->vdev.fd, &pci_bar, sizeof(pci_bar),
@@ -2397,9 +2343,7 @@ static void vfio_check_pcie_flr(VFIOPCIDevice *vdev, uint8_t pos)
     uint32_t cap = pci_get_long(vdev->pdev.config + pos + PCI_EXP_DEVCAP);
 
     if (cap & PCI_EXP_DEVCAP_FLR) {
-        DPRINTF("%04x:%02x:%02x.%x Supports FLR via PCIe cap\n",
-                vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                vdev->host.function);
+        DPRINTF("%s Supports FLR via PCIe cap\n", vdev->vdev.name);
         vdev->has_flr = true;
     }
 }
@@ -2409,9 +2353,7 @@ static void vfio_check_pm_reset(VFIOPCIDevice *vdev, uint8_t pos)
     uint16_t csr = pci_get_word(vdev->pdev.config + pos + PCI_PM_CTRL);
 
     if (!(csr & PCI_PM_CTRL_NO_SOFT_RESET)) {
-        DPRINTF("%04x:%02x:%02x.%x Supports PM reset\n",
-                vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                vdev->host.function);
+        DPRINTF("%s Supports PM reset\n", vdev->vdev.name);
         vdev->has_pm_reset = true;
     }
 }
@@ -2421,9 +2363,7 @@ static void vfio_check_af_flr(VFIOPCIDevice *vdev, uint8_t pos)
     uint8_t cap = pci_get_byte(vdev->pdev.config + pos + PCI_AF_CAP);
 
     if ((cap & PCI_AF_CAP_TP) && (cap & PCI_AF_CAP_FLR)) {
-        DPRINTF("%04x:%02x:%02x.%x Supports FLR via AF cap\n",
-                vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                vdev->host.function);
+        DPRINTF("%s Supports FLR via AF cap\n", vdev->vdev.name);
         vdev->has_flr = true;
     }
 }
@@ -2493,9 +2433,8 @@ static int vfio_add_std_cap(VFIOPCIDevice *vdev, uint8_t pos)
     }
 
     if (ret < 0) {
-        error_report("vfio: %04x:%02x:%02x.%x Error adding PCI capability "
-                     "0x%x[0x%x]@0x%x: %d", vdev->host.domain,
-                     vdev->host.bus, vdev->host.slot, vdev->host.function,
+        error_report("vfio: %s Error adding PCI capability "
+                     "0x%x[0x%x]@0x%x: %d", vdev->vdev.name,
                      cap_id, size, pos, ret);
         return ret;
     }
@@ -2575,9 +2514,7 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
     bool multi = false;
     int fd = vdev->vdev.fd;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x) %s\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function,
-            single ? "one" : "multi");
+    DPRINTF("%s(%s) %s\n", __func__, vdev->vdev.name, single ? "one" : "multi");
 
     vfio_pci_pre_reset(vdev);
     vdev->vdev.needs_reset = false;
@@ -2589,9 +2526,8 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
     if (ret && errno != ENOSPC) {
         ret = -errno;
         if (!vdev->has_pm_reset) {
-            error_report("vfio: Cannot reset device %04x:%02x:%02x.%x, "
-                         "no available reset mechanism.", vdev->host.domain,
-                         vdev->host.bus, vdev->host.slot, vdev->host.function);
+            error_report("vfio: Cannot reset device %s, "
+                         "no available reset mechanism.", vdev->vdev.name);
         }
         goto out_single;
     }
@@ -2608,9 +2544,7 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
         goto out_single;
     }
 
-    DPRINTF("%04x:%02x:%02x.%x: hot reset dependent devices:\n",
-            vdev->host.domain, vdev->host.bus, vdev->host.slot,
-            vdev->host.function);
+    DPRINTF("%s: hot reset dependent devices:\n", vdev->vdev.name);
 
     /* Verify that we have all the groups required */
     for (i = 0; i < info->count; i++) {
@@ -2638,10 +2572,9 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
 
         if (!group) {
             if (!vdev->has_pm_reset) {
-                error_report("vfio: Cannot reset device %04x:%02x:%02x.%x, "
+                error_report("vfio: Cannot reset device %s, "
                              "depends on group %d which is not owned.",
-                             vdev->host.domain, vdev->host.bus, vdev->host.slot,
-                             vdev->host.function, devices[i].group_id);
+                             vdev->vdev.name, devices[i].group_id);
             }
             ret = -EPERM;
             goto out;
@@ -2704,9 +2637,7 @@ static int vfio_pci_hot_reset(VFIOPCIDevice *vdev, bool single)
     ret = ioctl(fd, VFIO_DEVICE_PCI_HOT_RESET, reset);
     g_free(reset);
 
-    DPRINTF("%04x:%02x:%02x.%x hot reset: %s\n", vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function,
-            ret ? "%m" : "Success");
+    DPRINTF("%s hot reset: %s\n", vdev->vdev.name, ret ? "%m" : "Success");
 
 out:
     /* Re-enable INTx on affected devices */
@@ -3198,15 +3129,13 @@ static void vfio_pci_reset(DeviceState *dev)
     VFIOPCIDevice *vdev = container_of(pdev, VFIOPCIDevice, pdev);
     int fd = vdev->vdev.fd;
 
-    DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+    DPRINTF("%s(%s)\n", __func__, vdev->vdev.name);
 
     vfio_pci_pre_reset(vdev);
 
     if (vdev->vdev.reset_works && (vdev->has_flr || !vdev->has_pm_reset) &&
-        !ioctl(vdev->vdev.fd, VFIO_DEVICE_RESET)) {
-        DPRINTF("%04x:%02x:%02x.%x FLR/VFIO_DEVICE_RESET\n", vdev->host.domain,
-            vdev->host.bus, vdev->host.slot, vdev->host.function);
+        !ioctl(fd, VFIO_DEVICE_RESET)) {
+        DPRINTF("%s FLR/VFIO_DEVICE_RESET\n", vdev->vdev.name);
         goto post_reset;
     }
 
Signature · 1 line
1.8.3.2
#8 lore raw

Re: [Qemu-devel] [RFC v3 04/10] vfio: simplifed DPRINTF calls using device name

Quoted 11 lines · Eric Auger

This patch gets benefit from the new VFIODevice name field.

Occurences of
DPRINTF("%s(%04x:%02x:%02x.%x) ...", __func__, vdev->host.domain,

         vdev->host.bus, vdev->host.slot, vdev->host.function, ...)

are replaced by
DPRINTF("%s(%s ...", __func__, vdev->vdev.name, ...).

name is built using "%04x:%02x:%02x.%01x" format string.

Just convert them to trace points?

Alex

author
#9 lore raw

RFC v3 5/10 vfio: Add initial IRQ support in platform device

In reply to Eric Auger

This patch brings a first support for device IRQ assignment to a KVM guest. Code is inspired of PCI INTx code.

General principle of IRQ handling:

when a physical IRQ occurs, VFIO driver signals an eventfd that was registered by the QEMU VFIO platform device. The eventfd handler (vfio_intp_interrupt) injects the IRQ through QEMU/KVM and also disables MMIO region fast path (where MMIO regions are mapped as RAM). The purpose is to trap the IRQ status register guest reset. The physical interrupt is unmasked on the first read/write in any MMIO region. It was masked in the VFIO driver at the instant it signaled the eventfd.

A single IRQ can be forwarded to the guest at a time, ie. before a new virtual IRQ to be injected, the previous active one must have completed.

When no IRQ is pending anymore, fast path can be restored. This is done on mmap_timer scheduling.

irqfd support will be added in a subsequent patch. irqfd brings a framework where the eventfd is handled on kernel side instead of in user-side as currently done, hence improving the performance.

Although the code is prepared to support multiple IRQs, this is not tested at that stage.

Tested on Calxeda Midway xgmac which can be directly assigned to one guest (unfortunately only the main IRQ is exercised). A KVM patch is required to invalidate stage2 entries on RAM memory region destruction
(https://patches.linaro.org/27691/). Without that patch, slow/fast path switch cannot work.

change v2 -> v3:

- Move mmap_timer and mmap_timeout in new VFIODevice struct as PCI/platform factorization.
- multiple IRQ handling (a pending IRQ queue is added) - not tested -
- create vfio_mmap_set_enabled as in PCI code
- name of irq changed in virt

---
 hw/arm/virt.c         |  13 +-
 hw/vfio/pci.c         |  22 ++--
 hw/vfio/platform.c    | 323 ++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/vfio/vfio-common.h |  10 +-
 4 files changed, 346 insertions(+), 22 deletions(-)
hw/arm/virt.c +12 -1
index becd76b..f5693aa 100644
@@ -112,6 +112,7 @@ static const MemMapEntry a15memmap[] = {
 static const int a15irqmap[] = {
     [VIRT_UART] = 1,
     [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */
+    [VIRT_ETHERNET] = 77,
 };
 
 static VirtBoardInfo machines[] = {
@@ -348,8 +349,14 @@ static void create_ethernet(const VirtBoardInfo *vbi, qemu_irq *pic)
     hwaddr base = vbi->memmap[VIRT_ETHERNET].base;
     hwaddr size = vbi->memmap[VIRT_ETHERNET].size;
     const char compat[] = "calxeda,hb-xgmac";
+    int main_irq = vbi->irqmap[VIRT_ETHERNET];
+    int power_irq = main_irq+1;
+    int low_power_irq = main_irq+2;
 
-    sysbus_create_simple("vfio-platform", base, NULL);
+    sysbus_create_varargs("vfio-platform", base,
+                          pic[main_irq],
+                          pic[power_irq],
+                          pic[low_power_irq], NULL);
 
     nodename = g_strdup_printf("/ethernet@%" PRIx64, base);
     qemu_fdt_add_subnode(vbi->fdt, nodename);
@@ -357,6 +364,10 @@ static void create_ethernet(const VirtBoardInfo *vbi, qemu_irq *pic)
     /* Note that we can't use setprop_string because of the embedded NUL */
     qemu_fdt_setprop(vbi->fdt, nodename, "compatible", compat, sizeof(compat));
     qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "reg", 2, base, 2, size);
+    qemu_fdt_setprop_cells(vbi->fdt, nodename, "interrupts",
+                                0x0, main_irq, 0x4,
+                                0x0, power_irq, 0x4,
+                                0x0, low_power_irq, 0x4);
 
     g_free(nodename);
 }
hw/vfio/pci.c +10 -12
index ad0c2a0..1b49205 100644
@@ -83,8 +83,6 @@ typedef struct VFIOINTx {
     EventNotifier interrupt; /* eventfd triggered on interrupt */
     EventNotifier unmask; /* eventfd for unmask on QEMU bypass */
     PCIINTxRoute route; /* routing info for QEMU bypass */
-    uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
-    QEMUTimer *mmap_timer; /* enable mmaps after periods w/o interrupts */
 } VFIOINTx;
 
 typedef struct VFIOMSIVector {
@@ -196,8 +194,8 @@ static void vfio_intx_mmap_enable(void *opaque)
     VFIOPCIDevice *vdev = opaque;
 
     if (vdev->intx.pending) {
-        timer_mod(vdev->intx.mmap_timer,
-               qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + vdev->intx.mmap_timeout);
+        timer_mod(vdev->vdev.mmap_timer,
+               qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + vdev->vdev.mmap_timeout);
         return;
     }
 
@@ -217,9 +215,9 @@ static void vfio_intx_interrupt(void *opaque)
     vdev->intx.pending = true;
     pci_irq_assert(&vdev->pdev);
     vfio_mmap_set_enabled(vdev, false);
-    if (vdev->intx.mmap_timeout) {
-        timer_mod(vdev->intx.mmap_timer,
-               qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + vdev->intx.mmap_timeout);
+    if (vdev->vdev.mmap_timeout) {
+        timer_mod(vdev->vdev.mmap_timer,
+               qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + vdev->vdev.mmap_timeout);
     }
 }
 
@@ -457,7 +455,7 @@ static void vfio_disable_intx(VFIOPCIDevice *vdev)
 {
     int fd;
 
-    timer_del(vdev->intx.mmap_timer);
+    timer_del(vdev->vdev.mmap_timer);
     vfio_disable_intx_kvm(vdev);
     vfio_disable_irqindex(&vdev->vdev, VFIO_PCI_INTX_IRQ_INDEX);
     vdev->intx.pending = false;
@@ -3079,7 +3077,7 @@ static int vfio_initfn(PCIDevice *pdev)
     }
 
     if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) {
-        vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL,
+        vdev->vdev.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL,
                                                   vfio_intx_mmap_enable, vdev);
         pci_device_set_intx_routing_notifier(&vdev->pdev, vfio_update_irq);
         ret = vfio_enable_intx(vdev);
@@ -3112,8 +3110,8 @@ static void vfio_exitfn(PCIDevice *pdev)
     vfio_unregister_err_notifier(vdev);
     pci_device_set_intx_routing_notifier(&vdev->pdev, NULL);
     vfio_disable_interrupts(vdev);
-    if (vdev->intx.mmap_timer) {
-        timer_free(vdev->intx.mmap_timer);
+    if (vdev->vdev.mmap_timer) {
+        timer_free(vdev->vdev.mmap_timer);
     }
     vfio_teardown_msi(vdev);
     vfio_unmap_bars(vdev);
@@ -3158,7 +3156,7 @@ post_reset:
 static Property vfio_pci_dev_properties[] = {
     DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host),
     DEFINE_PROP_UINT32("x-intx-mmap-timeout-ms", VFIOPCIDevice,
-                       intx.mmap_timeout, 1100),
+                       vdev.mmap_timeout, 1100),
     DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features,
                     VFIO_FEATURE_ENABLE_VGA_BIT, false),
     DEFINE_PROP_INT32("bootindex", VFIOPCIDevice, bootindex, -1),
hw/vfio/platform.c +315 -8

375 lines not on this page · raw message

hw/vfio/vfio-common.h +9 -1
index 2699fba..7139d81 100644
@@ -42,6 +42,13 @@ enum {
     VFIO_DEVICE_TYPE_PLATFORM = 1,
 };
 
+enum {
+    VFIO_IRQ_INACTIVE = 0,
+    VFIO_IRQ_PENDING = 1,
+    VFIO_IRQ_ACTIVE = 2,
+    /* VFIO_IRQ_ACTIVE_AND_PENDING cannot happen with VFIO */
+};
+
 struct VFIOGroup;
 struct VFIODevice;
 
@@ -61,7 +68,6 @@ typedef struct VFIORegion {
     uint8_t nr; /* cache the region number for debug */
 } VFIORegion;
 
-
 /* Base Class for a VFIO device */
 
 typedef struct VFIODevice {
@@ -75,6 +81,8 @@ typedef struct VFIODevice {
     int type;
     bool reset_works;
     bool needs_reset;
+    uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
+    QEMUTimer *mmap_timer; /* enable mmaps after periods w/o interrupts */
     VFIODeviceOps *ops;
 } VFIODevice;
 
Signature · 1 line
1.8.3.2
#10 lore raw

Re: [Qemu-devel] [RFC v3 05/10] vfio: Add initial IRQ support in platform device

Quoted 5 lines · Eric Auger

This patch brings a first support for device IRQ assignment to a KVM guest. Code is inspired of PCI INTx code.

General principle of IRQ handling:

Quoted 8 lines

when a physical IRQ occurs, VFIO driver signals an eventfd that was registered by the QEMU VFIO platform device. The eventfd handler (vfio_intp_interrupt) injects the IRQ through QEMU/KVM and also disables MMIO region fast path (where MMIO regions are mapped as RAM). The purpose is to trap the IRQ status register guest reset. The physical interrupt is unmasked on the first read/write in any MMIO region. It was masked in the VFIO driver at the instant it signaled the eventfd.

This doesn't sound like a very promising generic scheme to me. I can easily see devices requiring 2 or 3 or more accesses until they're pulling down the IRQ line. During that time interrupts will keep firing, queue up in the irqfd and get at us as spurious interrupts.

Can't we handle it like PCI where we require devices to not share an interrupt line? Then we can just wait until the EOI in the interrupt controller.

Alex

Quoted 30 lines

A single IRQ can be forwarded to the guest at a time, ie. before a new virtual IRQ to be injected, the previous active one must have completed.

When no IRQ is pending anymore, fast path can be restored. This is done on mmap_timer scheduling.

irqfd support will be added in a subsequent patch. irqfd brings a framework where the eventfd is handled on kernel side instead of in user-side as currently done, hence improving the performance.

Although the code is prepared to support multiple IRQs, this is not tested at that stage.

Tested on Calxeda Midway xgmac which can be directly assigned to one guest (unfortunately only the main IRQ is exercised). A KVM patch is required to invalidate stage2 entries on RAM memory region destruction
(https://patches.linaro.org/27691/). Without that patch, slow/fast path switch cannot work.

change v2 -> v3:

- Move mmap_timer and mmap_timeout in new VFIODevice struct as PCI/platform factorization.
- multiple IRQ handling (a pending IRQ queue is added) - not tested -
- create vfio_mmap_set_enabled as in PCI code
- name of irq changed in virt

#11 lore raw

Re: [Qemu-devel] [RFC v3 05/10] vfio: Add initial IRQ support in platform device

Quoted 15 lines · Alexander Graf
Quoted 5 lines · Eric Auger

This patch brings a first support for device IRQ assignment to a KVM guest. Code is inspired of PCI INTx code.

General principle of IRQ handling:

Quoted 8 lines

when a physical IRQ occurs, VFIO driver signals an eventfd that was registered by the QEMU VFIO platform device. The eventfd handler (vfio_intp_interrupt) injects the IRQ through QEMU/KVM and also disables MMIO region fast path (where MMIO regions are mapped as RAM). The purpose is to trap the IRQ status register guest reset. The physical interrupt is unmasked on the first read/write in any MMIO region. It was masked in the VFIO driver at the instant it signaled the eventfd.

Quoted 8 lines

This doesn't sound like a very promising generic scheme to me. I can easily see devices requiring 2 or 3 or more accesses until they're pulling down the IRQ line. During that time interrupts will keep firing, queue up in the irqfd and get at us as spurious interrupts.

Can't we handle it like PCI where we require devices to not share an interrupt line? Then we can just wait until the EOI in the interrupt controller.

QEMU's interrupt abstraction makes this really difficult and something that's not generally necessary outside of device assignment. I spent a long time trying to figure out how we'd do it for PCI before I came up with this super generic hack that works surprisingly well. Yes, we may get additional spurious interrupts, but from a host perspective they're rate limited by the guest poking hardware, so there's a feedback loop. Also note that assuming this is the same approach we take for PCI, this mode is only used for the non-KVM accelerated path. When we have a KVM irqchip that supports a resampling irqfd then we can get an eventfd signal back at the point when we should unmask the interrupt on the host. Creating a cross-architecture QEMU interface to give you a callback when the architecture's notion of a resampling event occurs is not a trivial undertaking. Thanks,

Alex

author
#12 lore raw

Re: [Qemu-devel] [RFC v3 05/10] vfio: Add initial IRQ support in platform device

Quoted 16 lines · Alex Williamson
Quoted 15 lines · Alexander Graf
Quoted 13 lines · Eric Auger

This patch brings a first support for device IRQ assignment to a KVM guest. Code is inspired of PCI INTx code.

General principle of IRQ handling:

when a physical IRQ occurs, VFIO driver signals an eventfd that was registered by the QEMU VFIO platform device. The eventfd handler (vfio_intp_interrupt) injects the IRQ through QEMU/KVM and also disables MMIO region fast path (where MMIO regions are mapped as RAM). The purpose is to trap the IRQ status register guest reset. The physical interrupt is unmasked on the first read/write in any MMIO region. It was masked in the VFIO driver at the instant it signaled the eventfd.

Quoted 8 lines
Quoted 8 lines

This doesn't sound like a very promising generic scheme to me. I can easily see devices requiring 2 or 3 or more accesses until they're pulling down the IRQ line. During that time interrupts will keep firing, queue up in the irqfd and get at us as spurious interrupts.

Can't we handle it like PCI where we require devices to not share an interrupt line? Then we can just wait until the EOI in the interrupt controller.

Hi Alex,

Actually I transposed what was done for PCI INTx. For sure the virtual IRQ completion instant is not precise but as Alex says latter on irqfd should be used whenever possible for both precision aspects and performance. Given the perf of this legacy solution for IRQ intensive IP I would discourage to use that mode anyway. This is why I did not plan to invest more on this mode.

Quoted 9 lines

QEMU's interrupt abstraction makes this really difficult and something that's not generally necessary outside of device assignment. I spent a long time trying to figure out how we'd do it for PCI before I came up with this super generic hack that works surprisingly well. Yes, we may get additional spurious interrupts, but from a host perspective they're rate limited by the guest poking hardware, so there's a feedback loop. Also note that assuming this is the same approach we take for PCI, this mode is only used for the non-KVM accelerated path.

Yes this is again exactly the same approach as for PCI. We now have full irqfd + resamplefd support.

Best Regards

Eric

Quoted 9 lines

When we have a KVM
irqchip that supports a resampling irqfd then we can get an eventfd signal back at the point when we should unmask the interrupt on the host. Creating a cross-architecture QEMU interface to give you a callback when the architecture's notion of a resampling event occurs is not a trivial undertaking. Thanks,

Alex

author
#13 lore raw

RFC v3 6/10 virt: Assign a VFIO platform device with -device option

In reply to Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

The QEMU platform device becomes generic.

Current choice is to reuse the "-device" option.

For example when assigning Calxeda Midway xgmac device this option is used:
-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=1000

where
- fff51000.ethernet is the name of the device in /sys/bus/platform/devices/
- calxeda/hb-xgma is the compatibility where the standard comma separator is replaced by "/" since coma is specifically used by QEMU command line parser
- mmap-timeout-ms is minimal amount of time (ms) during which the IP register space stays MMIO mapped after an IRQ triggers in order to trap the end of interrupt (EOI). This is an optional parameter (default value set to 1100 ms).

mach-virt was modified to interpret this line and automatically

- map the device at a chosen guest physical address in [0xa004000, 0x10000000],
- map the device IRQs after 48,
- create the associated guest device tree with the provided compatibility.

The "-device" option underlying implementation is not standard which can be argued. Indeed normaly it induces the call to the QEMU device realize function once after the virtual machine init execution.

In vl.c the following sequence is implemented:
1) machine init machine_class->init(&current_machine->init_args);
2) init of devices added with -device option qemu_opts_foreach(qemu_find_opts("device"), device_init_func, NULL, 1)

The issue with that sequence is that the device tree is built in mach-virt and at that stage we miss information about the VFIO device (IRQ number, region number and size). Those information only are collected when the VFIO device is realized.

For that reason it is decided to interpret the -device option line in mach-virt and also to call the VFIO realize function there.

since vl.c is not changed by this patch, this means the VFIO realize function is called twice, once from mach-virt and once from vl.c The second call returns immediatly since the QEMU device is recognized as already attached to the group.

changes v2 -> v3
- retrieve device properties through standard functions
- support of multiple "reg" tuples (not tested)

Acknowledgements:
- a single compatibility currently is supported
- More complex device nodes will request specialized code
- cases where multiple VFIO devices are assigned could not be tested

---
 hw/arm/virt.c         | 222 +++++++++++++++++++++++++++++++++++++++++---------
 hw/vfio/common.c      |  10 +--
 hw/vfio/pci.c         |  17 ++++
 hw/vfio/platform.c    |  39 ++++++---
 hw/vfio/vfio-common.h |   1 +
 5 files changed, 233 insertions(+), 56 deletions(-)
hw/arm/virt.c +184 -38
index f5693aa..8de6d1a 100644
@@ -40,6 +40,8 @@
 #include "exec/address-spaces.h"
 #include "qemu/bitops.h"
 #include "qemu/error-report.h"
+#include "monitor/qdev.h"
+#include "qemu/config-file.h"
 
 #define NUM_VIRTIO_TRANSPORTS 32
 
@@ -65,7 +67,7 @@ enum {
     VIRT_GIC_CPU,
     VIRT_UART,
     VIRT_MMIO,
-    VIRT_ETHERNET,
+    VIRT_VFIO,
 };
 
 typedef struct MemMapEntry {
@@ -77,7 +79,10 @@ typedef struct VirtBoardInfo {
     struct arm_boot_info bootinfo;
     const char *cpu_model;
     const MemMapEntry *memmap;
+    qemu_irq pic[NUM_IRQS];
     const int *irqmap;
+    hwaddr avail_vfio_base;
+    int avail_vfio_irq;
     int smp_cpus;
     void *fdt;
     int fdt_size;
@@ -103,16 +108,16 @@ static const MemMapEntry a15memmap[] = {
     [VIRT_GIC_CPU] = { 0x8010000, 0x10000 },
     [VIRT_UART] = { 0x9000000, 0x1000 },
     [VIRT_MMIO] = { 0xa000000, 0x200 },
+    [VIRT_VFIO] = { 0xa004000, 0x0 }, /* size is dynamically populated */
     /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */
     /* 0x10000000 .. 0x40000000 reserved for PCI */
-    [VIRT_MEM] = { 0x40000000, 1ULL * 1024 * 1024 * 1024 },
-    [VIRT_ETHERNET] = { 0xfff51000, 0x1000 },
+    [VIRT_MEM] = { 0x40000000, 30ULL * 1024 * 1024 * 1024 },
 };
 
 static const int a15irqmap[] = {
     [VIRT_UART] = 1,
     [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */
-    [VIRT_ETHERNET] = 77,
+    [VIRT_VFIO] = 48,
 };
 
 static VirtBoardInfo machines[] = {
@@ -265,7 +270,7 @@ static void fdt_add_gic_node(const VirtBoardInfo *vbi)
     qemu_fdt_setprop_cell(vbi->fdt, "/intc", "phandle", gic_phandle);
 }
 
-static void create_gic(const VirtBoardInfo *vbi, qemu_irq *pic)
+static void create_gic(VirtBoardInfo *vbi)
 {
     /* We create a standalone GIC v2 */
     DeviceState *gicdev;
@@ -309,13 +314,13 @@ static void create_gic(const VirtBoardInfo *vbi, qemu_irq *pic)
     }
 
     for (i = 0; i < NUM_IRQS; i++) {
-        pic[i] = qdev_get_gpio_in(gicdev, i);
+        vbi->pic[i] = qdev_get_gpio_in(gicdev, i);
     }
 
     fdt_add_gic_node(vbi);
 }
 
-static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic)
+static void create_uart(const VirtBoardInfo *vbi)
 {
     char *nodename;
     hwaddr base = vbi->memmap[VIRT_UART].base;
@@ -324,7 +329,7 @@ static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic)
     const char compat[] = "arm,pl011\0arm,primecell";
     const char clocknames[] = "uartclk\0apb_pclk";
 
-    sysbus_create_simple("pl011", base, pic[irq]);
+    sysbus_create_simple("pl011", base, vbi->pic[irq]);
 
     nodename = g_strdup_printf("/pl011@%" PRIx64, base);
     qemu_fdt_add_subnode(vbi->fdt, nodename);
@@ -343,36 +348,177 @@ static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic)
     g_free(nodename);
 }
 
-static void create_ethernet(const VirtBoardInfo *vbi, qemu_irq *pic)
+/*
+ * Function called for each vfio-platform device option found in the
+ * qemu user command line:
+ * -device vfio-platform,vfio-device="<device>",compat"<compat>"
+ * for instance <device> can be fff51000.ethernet (device unbound from
+ * original driver and bound to vfio driver)
+ * for instance <compat> can be calxeda/hb-xgmac
+ * note "/" replaces normal ",". Indeed "," would be interpreted by QEMU as
+ * a separator
+ */
+
+static int vfio_init_func(QemuOpts *opts, void *opaque)
 {
+    const char *driver;
+    DeviceState *dev;
+    SysBusDevice *s;
+    VirtBoardInfo *vbi = (VirtBoardInfo *)opaque;
+    driver = qemu_opt_get(opts, "driver");
+    int irq_start = vbi->avail_vfio_irq;
+    hwaddr vfio_base = vbi->avail_vfio_base;
     char *nodename;
-    hwaddr base = vbi->memmap[VIRT_ETHERNET].base;
-    hwaddr size = vbi->memmap[VIRT_ETHERNET].size;
-    const char compat[] = "calxeda,hb-xgmac";
-    int main_irq = vbi->irqmap[VIRT_ETHERNET];
-    int power_irq = main_irq+1;
-    int low_power_irq = main_irq+2;
-
-    sysbus_create_varargs("vfio-platform", base,
-                          pic[main_irq],
-                          pic[power_irq],
-                          pic[low_power_irq], NULL);
-
-    nodename = g_strdup_printf("/ethernet@%" PRIx64, base);
-    qemu_fdt_add_subnode(vbi->fdt, nodename);
+    char *corrected_compat, *compat, *name;
+    int num_irqs, num_regions;
+    MemoryRegion *mr;
+    int i, ret;
+    uint32_t *irq_attr;
+    uint64_t *reg_attr;
+    uint64_t size;
+    Error *errp = NULL;
+
+    if (!driver) {
+        qerror_report(QERR_MISSING_PARAMETER, "driver");
+        return -1 ;
+    }
 
-    /* Note that we can't use setprop_string because of the embedded NUL */
-    qemu_fdt_setprop(vbi->fdt, nodename, "compatible", compat, sizeof(compat));
-    qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "reg", 2, base, 2, size);
-    qemu_fdt_setprop_cells(vbi->fdt, nodename, "interrupts",
-                                0x0, main_irq, 0x4,
-                                0x0, power_irq, 0x4,
-                                0x0, low_power_irq, 0x4);
+    if (strcasecmp(driver, "vfio-platform") == 0) {
+        dev = qdev_device_add(opts);
+        if (!dev) {
+            return -1;
+        }
+        s = SYS_BUS_DEVICE(dev);
 
-    g_free(nodename);
+        name = object_property_get_str(OBJECT(s), "vfio_device", &errp);
+        if (errp != NULL || (name == NULL)) {
+            error_report("Couldn't retrieve vfio device name: %s\n",
+                         error_get_pretty(errp));
+            exit(1);
+           }
+        compat = object_property_get_str(OBJECT(s), "compat", &errp);
+        if ((errp != NULL) || (name == NULL)) {
+            error_report("Couldn't retrieve VFIO device compat: %s\n",
+                         error_get_pretty(errp));
+            exit(1);
+           }
+        num_irqs = object_property_get_int(OBJECT(s), "num_irqs", &errp);
+        if (errp != NULL) {
+            error_report("Couldn't retrieve VFIO IRQ number: %s\n",
+                         error_get_pretty(errp));
+            exit(1);
+           }
+        num_regions = object_property_get_int(OBJECT(s), "num_regions", &errp);
+        if ((errp != NULL) || (num_regions == 0)) {
+            error_report("Couldn't retrieve VFIO region number: %s\n",
+                         error_get_pretty(errp));
+            exit(1);
+           }
+
+        /*
+         * collect region info and build reg property as tuplets
+         * 2 base 2 size
+         * 2 being the number of cells for base and size
+         */
+        reg_attr = g_new(uint64_t, num_regions*4);
+
+        for (i = 0; i < num_regions; i++) {
+            mr = sysbus_mmio_get_region(s, i);
+            size = memory_region_size(mr);
+            reg_attr[4*i] = 2;
+            reg_attr[4*i+1] = vbi->avail_vfio_base;
+            reg_attr[4*i+2] = 2;
+            reg_attr[4*i+3] = size;
+            vbi->avail_vfio_base += size;
+        }
+
+        if (vbi->avail_vfio_base >= 0x10000000) {
+            /* VFIO region size exceeds remaining VFIO space */
+            qerror_report(QERR_DEVICE_INIT_FAILED, name);
+        } else if (irq_start + num_irqs >= NUM_IRQS) {
+            /* VFIO IRQ number exceeded */
+            qerror_report(QERR_DEVICE_INIT_FAILED, name);
+        }
+
+        /*
+         * process compatibility property string passed by end-user
+         * replaces / by ,
+         * currently a single property compatibility value is supported!
+         */
+        corrected_compat = g_strdup(compat);
+        char *slash = strchr(corrected_compat, '/');
+        if (slash != NULL) {
+            *slash = ',';
+        } else {
+            error_report("Wrong compat string %s, should contain a /\n",
+                         compat);
+            exit(1);
+        }
+
+        sysbus_mmio_map(s, 0, vfio_base);
+        nodename = g_strdup_printf("/%s@%" PRIx64,
+                                   name, vfio_base);
+
+        qemu_fdt_add_subnode(vbi->fdt, nodename);
+
+        qemu_fdt_setprop(vbi->fdt, nodename, "compatible",
+                             corrected_compat, strlen(corrected_compat));
+
+        ret = qemu_fdt_setprop_sized_cells_from_array(vbi->fdt, nodename, "reg",
+                         num_regions*2, reg_attr);
+        if (ret < 0) {
+            error_report("could not set reg property of node %s", nodename);
+        }
+
+        irq_attr = g_new(uint32_t, num_irqs*3);
+        for (i = 0; i < num_irqs; i++) {
+            sysbus_connect_irq(s, i, vbi->pic[irq_start+i]);
+
+            irq_attr[3*i] = cpu_to_be32(0);
+            irq_attr[3*i+1] = cpu_to_be32(irq_start+i);
+            irq_attr[3*i+2] = cpu_to_be32(0x4);
+        }
+
+        ret = qemu_fdt_setprop(vbi->fdt, nodename, "interrupts",
+                         irq_attr, num_irqs*3*sizeof(uint32_t));
+        if (ret < 0) {
+            error_report("could not set interrupts property of node %s",
+                         nodename);
+        }
+
+        vbi->avail_vfio_irq += num_irqs;
+
+        g_free(nodename);
+        g_free(corrected_compat);
+        g_free(irq_attr);
+        g_free(reg_attr);
+
+        object_unref(OBJECT(dev));
+
+    }
+
+  return 0;
 }
 
-static void create_virtio_devices(const VirtBoardInfo *vbi, qemu_irq *pic)
+/*
+ * parses the option line and look for -device option
+ * for each of time vfio_init_func is called.
+ * this later only applies to -device vfio-platform ones
+ */
+
+static void create_vfio_devices(VirtBoardInfo *vbi)
+{
+    vbi->avail_vfio_base = vbi->memmap[VIRT_VFIO].base;
+    vbi->avail_vfio_irq =  vbi->irqmap[VIRT_VFIO];
+
+    if (qemu_opts_foreach(qemu_find_opts("device"),
+                        vfio_init_func, (void *)vbi, 1) != 0) {
+        exit(1);
+    }
+}
+
+
+static void create_virtio_devices(const VirtBoardInfo *vbi)
 {
     int i;
     hwaddr size = vbi->memmap[VIRT_MMIO].size;
@@ -386,7 +532,7 @@ static void create_virtio_devices(const VirtBoardInfo *vbi, qemu_irq *pic)
         int irq = vbi->irqmap[VIRT_MMIO] + i;
         hwaddr base = vbi->memmap[VIRT_MMIO].base + i * size;
 
-        sysbus_create_simple("virtio-mmio", base, pic[irq]);
+        sysbus_create_simple("virtio-mmio", base, vbi->pic[irq]);
     }
 
     for (i = NUM_VIRTIO_TRANSPORTS - 1; i >= 0; i--) {
@@ -417,7 +563,6 @@ static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size)
 
 static void machvirt_init(QEMUMachineInitArgs *args)
 {
-    qemu_irq pic[NUM_IRQS];
     MemoryRegion *sysmem = get_system_memory();
     int n;
     MemoryRegion *ram = g_new(MemoryRegion, 1);
@@ -483,16 +628,17 @@ static void machvirt_init(QEMUMachineInitArgs *args)
     vmstate_register_ram_global(ram);
     memory_region_add_subregion(sysmem, vbi->memmap[VIRT_MEM].base, ram);
 
-    create_gic(vbi, pic);
+    create_gic(vbi);
+
+    create_uart(vbi);
 
-    create_uart(vbi, pic);
-    create_ethernet(vbi, pic);
+    create_vfio_devices(vbi);
 
     /* Create mmio transports, so the user can create virtio backends
      * (which will be automatically plugged in to the transports). If
      * no backend is created the transport will just sit harmlessly idle.
      */
-    create_virtio_devices(vbi, pic);
+    create_virtio_devices(vbi);
 
     vbi->bootinfo.ram_size = args->ram_size;
     vbi->bootinfo.kernel_filename = args->kernel_filename;
hw/vfio/common.c +3 -7
index 07dc409..28d29de 100644
@@ -732,7 +732,6 @@ void vfio_put_base_device(VFIODevice *vdev)
 
 int vfio_base_device_init(VFIODevice *vdev, int type)
 {
-    VFIODevice *tmp;
     VFIOGroup *group;
     char path[PATH_MAX], iommu_group_path[PATH_MAX], *group_name;
     ssize_t len;
@@ -792,12 +791,9 @@ int vfio_base_device_init(VFIODevice *vdev, int type)
 
     snprintf(path, sizeof(path), "%s", vdev->name);
 
-    QLIST_FOREACH(tmp, &group->device_list, next) {
-        if (strcmp(tmp->name, vdev->name) == 0) {
-            error_report("vfio: error: device %s is already attached", path);
-            vfio_put_group(group, vfio_reset_handler);
-            return -EBUSY;
-        }
+    if (vdev->ops->vfio_is_device_already_attached(vdev, group)) {
+        vfio_put_group(group, vfio_reset_handler);
+        return -EBUSY;
     }
 
     ret = vfio_get_base_device(group, path, vdev);
hw/vfio/pci.c +17 -0
index 1b49205..c86bef9 100644
@@ -2988,6 +2988,22 @@ static void vfio_unregister_err_notifier(VFIOPCIDevice *vdev)
     event_notifier_cleanup(&vdev->err_notifier);
 }
 
+static bool vfio_pci_is_device_already_attached(VFIODevice *vdev,
+                                                VFIOGroup *group)
+{
+    VFIODevice *tmp;
+
+    QLIST_FOREACH(tmp, &group->device_list, next) {
+        if (strcmp(tmp->name, vdev->name) == 0) {
+            error_report("vfio: error: device %s is already attached",
+                         vdev->name);
+            return true;
+        }
+    }
+    return false;
+}
+
+
 
 static VFIODeviceOps vfio_pci_ops = {
     .vfio_eoi = vfio_pci_eoi,
@@ -2996,6 +3012,7 @@ static VFIODeviceOps vfio_pci_ops = {
     .vfio_check_device = vfio_pci_check_device,
     .vfio_get_device_regions = vfio_pci_get_device_regions,
     .vfio_get_device_interrupts = vfio_pci_get_device_interrupts,
+    .vfio_is_device_already_attached = vfio_pci_is_device_already_attached,
 };
 
 static int vfio_initfn(PCIDevice *pdev)
hw/vfio/platform.c +28 -11
index 5b9451f..377783b 100644
@@ -43,6 +43,7 @@ typedef struct VFIOPlatformDevice {
     QLIST_HEAD(, VFIOINTp) intp_list; /* list of IRQ */
     /* queue of pending IRQ */
     QSIMPLEQ_HEAD(pending_intp_queue, VFIOINTp) pending_intp_queue;
+    char *compat; /* compatibility string */
 } VFIOPlatformDevice;
 
 
@@ -394,11 +395,6 @@ static int vfio_platform_get_device_interrupts(VFIODevice *vdev)
     int i, ret;
     VFIOPlatformDevice *vplatdev = container_of(vdev, VFIOPlatformDevice, vdev);
 
-    /*
-     * mmap timeout = 1100 ms, PCI default value
-     * this will become a user-defined value in subsequent patch
-     */
-    vdev->mmap_timeout = 1100;
     vdev->mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL,
                                     vfio_intp_mmap_enable, vdev);
 
@@ -446,6 +442,19 @@ static void vfio_disable_intp(VFIODevice *vdev)
 
 }
 
+static bool vfio_platform_is_device_already_attached(VFIODevice *vdev,
+                                                     VFIOGroup *group)
+{
+    VFIODevice *tmp;
+
+    QLIST_FOREACH(tmp, &group->device_list, next) {
+        if (strcmp(tmp->name, vdev->name) == 0) {
+            return true;
+        }
+    }
+    return false;
+}
+
 
 static VFIODeviceOps vfio_platform_ops = {
     .vfio_eoi = vfio_platform_eoi,
@@ -454,6 +463,7 @@ static VFIODeviceOps vfio_platform_ops = {
     .vfio_check_device = vfio_platform_check_device,
     .vfio_get_device_regions = vfio_platform_get_device_regions,
     .vfio_get_device_interrupts = vfio_platform_get_device_interrupts,
+    .vfio_is_device_already_attached = vfio_platform_is_device_already_attached,
 };
 
 
@@ -466,9 +476,7 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp)
 
     vbasedev->ops = &vfio_platform_ops;
 
-    /* TODO: pass device name on command line */
-    vbasedev->name = malloc(PATH_MAX);
-    snprintf(vbasedev->name, PATH_MAX, "%s", "fff51000.ethernet");
+    DPRINTF("vfio device %s, compat = %s\n", vbasedev->name, vdev->compat);
 
     ret = vfio_base_device_init(vbasedev, VFIO_DEVICE_TYPE_PLATFORM);
     if (ret < 0) {
@@ -531,8 +539,8 @@ static const VMStateDescription vfio_platform_vmstate = {
 
 typedef struct VFIOPlatformDeviceClass {
     DeviceClass parent_class;
+    void (*init)(VFIOPlatformDevice *vdev);
 
-    int (*init)(VFIODevice *dev);
 } VFIOPlatformDeviceClass;
 
 #define VFIO_PLATFORM_DEVICE(obj) \
@@ -542,19 +550,28 @@ typedef struct VFIOPlatformDeviceClass {
 #define VFIO_PLATFORM_DEVICE_GET_CLASS(obj) \
      OBJECT_GET_CLASS(VFIOPlatformDeviceClass, (obj), TYPE_VFIO_PLATFORM)
 
+static Property vfio_platform_dev_properties[] = {
+DEFINE_PROP_STRING("vfio_device", VFIOPlatformDevice, vdev.name),
+DEFINE_PROP_STRING("compat", VFIOPlatformDevice, compat),
+DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
+                   vdev.mmap_timeout, 1100),
+DEFINE_PROP_UINT32("num_irqs", VFIOPlatformDevice, vdev.num_irqs, 0),
+DEFINE_PROP_UINT32("num_regions", VFIOPlatformDevice, vdev.num_regions, 0),
+DEFINE_PROP_END_OF_LIST(),
+};
 
 
 static void vfio_platform_dev_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
     VFIOPlatformDeviceClass *vdc = VFIO_PLATFORM_DEVICE_CLASS(klass);
-
+    dc->props = vfio_platform_dev_properties;
     dc->realize = vfio_platform_realize;
     dc->unrealize = vfio_platform_unrealize;
     dc->vmsd = &vfio_platform_vmstate;
     dc->desc = "VFIO-based platform device assignment";
     set_bit(DEVICE_CATEGORY_MISC, dc->categories);
-
+    dc->cannot_instantiate_with_device_add_yet = false;
     vdc->init = NULL;
 }
 
hw/vfio/vfio-common.h +1 -0
index 7139d81..5412acd 100644
@@ -123,6 +123,7 @@ struct VFIODeviceOps {
     int (*vfio_check_device)(VFIODevice *vdev);
     int (*vfio_get_device_regions)(VFIODevice *vdev);
     int (*vfio_get_device_interrupts)(VFIODevice *vdev);
+    bool (*vfio_is_device_already_attached)(VFIODevice *vdev, VFIOGroup*);
 };
 
 
Signature · 1 line
1.8.3.2
#14 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 4 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

Quoted 8 lines

The QEMU platform device becomes generic.

Current choice is to reuse the "-device" option.

For example when assigning Calxeda Midway xgmac device this option is used:
-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=1000

I think we're walking into the right direction, but there is one major nit I have. I don't think we should have a -device vfio-platform. I think we should have a -device vfio-xgmac that maybe inherits from an abstrace vfio-platform class.

That way machine code can assemble the device tree according to the device and you can also implement hardware specific hacks or dependencies if you need them - for example the MMIO masking to find an EOI you did earlier.

Alex

author
#15 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 20 lines · Alexander Graf
Quoted 4 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

Quoted 8 lines

The QEMU platform device becomes generic.

Current choice is to reuse the "-device" option.

For example when assigning Calxeda Midway xgmac device this option is used:
-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=1000

I think we're walking into the right direction, but there is one major nit I have. I don't think we should have a -device vfio-platform. I think we should have a -device vfio-xgmac that maybe inherits from an abstrace vfio-platform class.

Quoted 4 lines

That way machine code can assemble the device tree according to the device and you can also implement hardware specific hacks or dependencies if you need them - for example the MMIO masking to find an EOI you did earlier.

I must admit I am lacking experience of other devices than my dear xgmac. I can just say that for the time beeing the approach seems to fit some ARM Amba devices like PL330 DMA. We need to go further to identity the limits of this generic approach.

Best Regards

Eric

Quoted 4 lines

Alex

#16 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 19 lines · Eric Auger
Quoted 18 lines · Alexander Graf
Quoted 4 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

Quoted 8 lines

The QEMU platform device becomes generic.

Current choice is to reuse the "-device" option.

For example when assigning Calxeda Midway xgmac device this option is used:
-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=1000

I think we're walking into the right direction, but there is one major nit I have. I don't think we should have a -device vfio-platform. I think we should have a -device vfio-xgmac that maybe inherits from an abstrace vfio-platform class.

Quoted 8 lines
Quoted 4 lines

That way machine code can assemble the device tree according to the device and you can also implement hardware specific hacks or dependencies if you need them - for example the MMIO masking to find an EOI you did earlier.

I must admit I am lacking experience of other devices than my dear xgmac. I can just say that for the time beeing the approach seems to fit some ARM Amba devices like PL330 DMA. We need to go further to identity the limits of this generic approach.

No, I think we're better off not faking anything generic at all, because I'm 99.9% sure it will never be generic in real-world device cases.

And if we start doing things generically, people will soon want to have other mad additions to do device specific things in generic code, such as "take the device tree from the host, but modify property x, y and z". Better be clear about our limits from the beginning :).

Imagine vfio-platform as a transport, similar to TCP. We have ports and moving data from left to right is always the same, but whether you need to open 2 ports to get a working FTP data transfer is up to the implementation of the protocol above. Same thing here.

Alex

author
#17 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 38 lines · Alexander Graf
Quoted 19 lines · Eric Auger
Quoted 18 lines · Alexander Graf
Quoted 4 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

Quoted 8 lines

The QEMU platform device becomes generic.

Current choice is to reuse the "-device" option.

For example when assigning Calxeda Midway xgmac device this option is used:
-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=1000

I think we're walking into the right direction, but there is one major nit I have. I don't think we should have a -device vfio-platform. I think we should have a -device vfio-xgmac that maybe inherits from an abstrace vfio-platform class.

Quoted 8 lines
Quoted 4 lines

That way machine code can assemble the device tree according to the device and you can also implement hardware specific hacks or dependencies if you need them - for example the MMIO masking to find an EOI you did earlier.

I must admit I am lacking experience of other devices than my dear xgmac. I can just say that for the time beeing the approach seems to fit some ARM Amba devices like PL330 DMA. We need to go further to identity the limits of this generic approach.

No, I think we're better off not faking anything generic at all, because I'm 99.9% sure it will never be generic in real-world device cases.

And if we start doing things generically, people will soon want to have other mad additions to do device specific things in generic code, such as "take the device tree from the host, but modify property x, y and z". Better be clear about our limits from the beginning :).

Quoted 4 lines

Imagine vfio-platform as a transport, similar to TCP. We have ports and moving data from left to right is always the same, but whether you need to open 2 ports to get a working FTP data transfer is up to the implementation of the protocol above. Same thing here.

OK you convinced me. I will investigate that way then. Eric

Quoted 4 lines

Alex

#18 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

In reply to Eric Auger
Quoted 5 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

 hw/arm/virt.c         | 222 +++++++++++++++++++++++++++++++++++++++++---------

This is way too much code to be adding to virt.c. I really don't want to be dealing with VFIO related code in board models beyond an absolute minimal "go do vfio stuff if the user asked for it" set of hooks.

thanks
-- PMM

#19 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 9 lines · Peter Maydell
Quoted 4 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

  hw/arm/virt.c         | 222 +++++++++++++++++++++++++++++++++++++++++---------

This is way too much code to be adding to virt.c. I really don't want to be dealing with VFIO related code in board models beyond an absolute minimal "go do vfio stuff if the user asked for it" set of hooks.

And device tree chunks :).

Alex

author
#20 lore raw

Re: [Qemu-devel] [RFC v3 06/10] virt: Assign a VFIO platform device with -device option

Quoted 8 lines · Alexander Graf
Quoted 6 lines · Peter Maydell
Quoted 5 lines · Eric Auger

This patch aims at allowing the end-user to specify the device he wants to directly assign to his mach-virt guest in the QEMU command line.

  hw/arm/virt.c         | 222

+++++++++++++++++++++++++++++++++++++++++---------

Quoted 4 lines
Quoted 4 lines

This is way too much code to be adding to virt.c. I really don't want to be dealing with VFIO related code in board models beyond an absolute minimal "go do vfio stuff if the user asked for it" set of hooks.

Hi Alex, Peter,

Thanks for your comments. I am currently preparing v4 where I use the same technique as Alex did in
[PATCH 4/5] PPC: e500: Support platform devices
(http://lists.gnu.org/archive/html/qemu-devel/2014-06/msg00847.html), ie. using
- qemu_add_machine_init_done_notifier
- qemu_register_reset, which seems to be the right clean way to do what I tried to achieve here.
Alex, actually I am reusing the code you put in e500, moved it in a separate helper file. That way, effectively, I will be able to have very few vfio specific code in the machine file.

Best Regards

Eric

Quoted 6 lines

And device tree chunks :).

Alex

author
#21 lore raw

RFC v3 7/10 Add EXEC_FLAG to VFIO DMA mappings

In reply to Eric Auger

From: Alvise Rigo <[email protected]>

The flag is mandatory for the ARM SMMU so we always add it if the MMIO handles it.

---
 hw/vfio/common.c           | 9 +++++++++
 hw/vfio/vfio-common.h      | 1 +
 linux-headers/linux/vfio.h | 2 ++
 3 files changed, 12 insertions(+)
hw/vfio/common.c +9 -0
index 28d29de..8b25380 100644
@@ -82,6 +82,11 @@ static int vfio_dma_map(VFIOContainer *container, hwaddr iova,
         map.flags |= VFIO_DMA_MAP_FLAG_WRITE;
     }
 
+    /* add exec flag */
+    if (container->iommu_data.has_exec_cap) {
+        map.flags |= VFIO_DMA_MAP_FLAG_EXEC;
+    }
+
     /*
      * Try the mapping, if it fails with EBUSY, unmap the region and try
      * again.  This shouldn't be necessary, but we sometimes see it in
@@ -327,6 +332,10 @@ static int vfio_connect_container(VFIOGroup *group)
             return -errno;
         }
 
+        if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_IOMMU_PROT_EXEC)) {
+            container->iommu_data.has_exec_cap = true;
+        }
+
         container->iommu_data.type1.listener = vfio_memory_listener;
         container->iommu_data.release = vfio_listener_release;
 
hw/vfio/vfio-common.h +1 -0
index 5412acd..dcd7ddd 100644
@@ -100,6 +100,7 @@ typedef struct VFIOContainer {
         union {
             VFIOType1 type1;
         };
+        bool has_exec_cap; /* support of exec capability by the IOMMU */
         void (*release)(struct VFIOContainer *);
     } iommu_data;
     QLIST_HEAD(, VFIOGroup) group_list;
linux-headers/linux/vfio.h +2 -0
index ef4815d..e96e14d 100644
@@ -30,6 +30,7 @@
  */
 #define VFIO_DMA_CC_IOMMU		4
 
+#define VFIO_IOMMU_PROT_EXEC		5
 /*
  * The IOCTL interface is designed for extensibility by embedding the
  * structure length (argsz) and flags into structures passed between
@@ -399,6 +400,7 @@ struct vfio_iommu_type1_dma_map {
 	__u32	flags;
 #define VFIO_DMA_MAP_FLAG_READ (1 << 0)		/* readable from device */
 #define VFIO_DMA_MAP_FLAG_WRITE (1 << 1)	/* writable from device */
+#define VFIO_DMA_MAP_FLAG_EXEC (1 << 2)	/* executable from device */
 	__u64	vaddr;				/* Process virtual address */
 	__u64	iova;				/* IO virtual address */
 	__u64	size;				/* Size of mapping (bytes) */
Signature · 1 line
1.8.3.2
author
#22 lore raw

RFC v3 8/10 Add AMBA devices support to VFIO

In reply to Eric Auger

From: Alvise Rigo <[email protected]>

The impossibility to add more then one compatibility property to the device tree node was not permitting to bind AMBA devices.
Now we can add an arbitrary number of compatibility property values divided by the character ";".

If the compatibility string contains the substring "arm,primecell", a clock property will be added to the device tree node in order to allow the AMBA bus code to probe the device.

[Eric Auger]
put str_ptr in the declaration part and rename pcompat into compat

---
 hw/arm/virt.c | 45 +++++++++++++++++++++++++++++++++++----------
 1 file changed, 35 insertions(+), 10 deletions(-)
hw/arm/virt.c +35 -10
index 8de6d1a..bc561b5 100644
@@ -369,6 +369,7 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
     int irq_start = vbi->avail_vfio_irq;
     hwaddr vfio_base = vbi->avail_vfio_base;
     char *nodename;
+    char *str_ptr;
     char *corrected_compat, *compat, *name;
     int num_irqs, num_regions;
     MemoryRegion *mr;
@@ -377,6 +378,8 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
     uint64_t *reg_attr;
     uint64_t size;
     Error *errp = NULL;
+    bool is_amba = false;
+    int compat_str_len;
 
     if (!driver) {
         qerror_report(QERR_MISSING_PARAMETER, "driver");
@@ -442,17 +445,30 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
 
         /*
          * process compatibility property string passed by end-user
-         * replaces / by ,
-         * currently a single property compatibility value is supported!
+         * replaces / by , and ; by NUL character
          */
         corrected_compat = g_strdup(compat);
-        char *slash = strchr(corrected_compat, '/');
-        if (slash != NULL) {
-            *slash = ',';
-        } else {
-            error_report("Wrong compat string %s, should contain a /\n",
-                         compat);
-            exit(1);
+        /*
+         * the total length of the string has to include also the last
+         * NUL char.
+         */
+        compat_str_len = strlen(corrected_compat) + 1;
+
+        str_ptr = corrected_compat;
+        while ((str_ptr = strchr(str_ptr, '/')) != NULL) {
+            *str_ptr = ',';
+        }
+
+        /* check if is an AMBA device */
+        str_ptr = corrected_compat;
+        if (strstr(str_ptr, "arm,primecell") != NULL) {
+            is_amba = true;
+        }
+
+        /* substitute ";" with the NUL char */
+        str_ptr = corrected_compat;
+        while ((str_ptr = strchr(str_ptr, ';')) != NULL) {
+            *str_ptr = '\0';
         }
 
         sysbus_mmio_map(s, 0, vfio_base);
@@ -462,7 +478,7 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
         qemu_fdt_add_subnode(vbi->fdt, nodename);
 
         qemu_fdt_setprop(vbi->fdt, nodename, "compatible",
-                             corrected_compat, strlen(corrected_compat));
+                             corrected_compat, compat_str_len);
 
         ret = qemu_fdt_setprop_sized_cells_from_array(vbi->fdt, nodename, "reg",
                          num_regions*2, reg_attr);
@@ -471,6 +487,15 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
         }
 
         irq_attr = g_new(uint32_t, num_irqs*3);
+
+        if (is_amba) {
+            qemu_fdt_setprop_cells(vbi->fdt, nodename, "clocks",
+                                   vbi->clock_phandle);
+            char clock_names[] = "apb_pclk";
+            qemu_fdt_setprop(vbi->fdt, nodename, "clock-names", clock_names,
+                                                       sizeof(clock_names));
+        }
+
         for (i = 0; i < num_irqs; i++) {
             sysbus_connect_irq(s, i, vbi->pic[irq_start+i]);
 
Signature · 1 line
1.8.3.2
author
#23 lore raw

RFC v3 9/10 Always use eventfd as notifying mechanism

In reply to Eric Auger

From: Alvise Rigo <[email protected]>

When eventfd is not configured the method event_notifier_init fallbacks to the pipe/pipe2 system call, causing an error in VFIO_DEVICE_SET_IRQS since we pass to the kernel a file descriptor which is not created by eventfd.

---
 hw/vfio/platform.c | 5 +++++
 1 file changed, 5 insertions(+)
hw/vfio/platform.c +5 -0
index 377783b..56dde5f 100644
@@ -346,6 +346,11 @@ static int vfio_enable_intp(VFIODevice *vdev, unsigned int index)
     sysbus_init_irq(sbdev, &intp->qemuirq);
 
     ret = event_notifier_init(&intp->interrupt, 0);
+    if (!ret && (intp->interrupt.rfd != intp->interrupt.wfd)) {
+        /* event_notifier_init created a pipe instead of eventfd */
+        ret = -1;
+    }
+
     if (ret) {
         error_report("vfio: Error: event_notifier_init failed ");
         return ret;
Signature · 1 line
1.8.3.2
author
#24 lore raw

RFC v3 10/10 vfio: Add irqfd support in platform device

In reply to Eric Auger

This patch aims at optimizing IRQ handling using irqfd framework. It brings significant performance improvement over "traditional" IRQ handling introduced in :
"vfio: Add initial IRQ support in platform device".

This new IRQ handling method depends on kernel KVM irqfd/GSI routing capability.

The IRQ handling method can be dynamically chosen (default is irqfd, if kernel supports it obviously). For example to disable irqfd handling, use:

-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=110,irqfd=false\

Performances are improved for the following reasons:
- eventfds signalled by the VFIO platform driver are handled on kernel side by the KVM irqfd framework.
- the end of interrupt(EOI) is trapped at GIC level and not at MMIO region level. As a reminder, in traditional IRQ handling QEMU assumed the first guest access to a device MMIO region after IRQ hit was the IRQ status register reset. This trap was approximate and obliged to swap to slow path after IRQ hit. A mmap timer mechanism enabled to swap back to fast path after the mmap period introducing extra complexity. Now GIC detects the completion of the virtual IRQ and signals a resampler eventfd on maintenance IRQ. The corresponding handler re-enables the physical IRQ.

Next optimization step consists in attempting to remove EOI trap (ie. maintenance IRQ). This should be covered by another patch in near future.

This work was tested with Calxeda Midway xgmac.

---
 hw/arm/virt.c         |  14 +++++
 hw/intc/arm_gic_kvm.c |   1 +
 hw/vfio/platform.c    | 165 +++++++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 166 insertions(+), 14 deletions(-)
hw/arm/virt.c +14 -0
index bc561b5..de1b885 100644
@@ -43,6 +43,9 @@
 #include "monitor/qdev.h"
 #include "qemu/config-file.h"
 
+#define ENABLE_IRQFD 1
+void vfio_setup_irqfd(SysBusDevice *s, int index, int virq);
+
 #define NUM_VIRTIO_TRANSPORTS 32
 
 /* Number of external interrupt lines to configure the GIC with */
@@ -380,6 +383,7 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
     Error *errp = NULL;
     bool is_amba = false;
     int compat_str_len;
+    bool irqfd_allowed;
 
     if (!driver) {
         qerror_report(QERR_MISSING_PARAMETER, "driver");
@@ -417,6 +421,13 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
                          error_get_pretty(errp));
             exit(1);
            }
+        irqfd_allowed = object_property_get_bool(OBJECT(s), "irqfd", &errp);
+        if (errp != NULL) {
+            error_report("Couldn't retrieve irqfd flag: %s\n",
+                         error_get_pretty(errp));
+            exit(1);
+           }
+
 
         /*
          * collect region info and build reg property as tuplets
@@ -502,6 +513,9 @@ static int vfio_init_func(QemuOpts *opts, void *opaque)
             irq_attr[3*i] = cpu_to_be32(0);
             irq_attr[3*i+1] = cpu_to_be32(irq_start+i);
             irq_attr[3*i+2] = cpu_to_be32(0x4);
+            if (irqfd_allowed) {
+                vfio_setup_irqfd(s, i, irq_start+i);
+            }
         }
 
         ret = qemu_fdt_setprop(vbi->fdt, nodename, "interrupts",
hw/intc/arm_gic_kvm.c +1 -0
index 5038885..18a6204 100644
@@ -576,6 +576,7 @@ static void kvm_arm_gic_realize(DeviceState *dev, Error **errp)
                             KVM_DEV_ARM_VGIC_GRP_ADDR,
                             KVM_VGIC_V2_ADDR_TYPE_CPU,
                             s->dev_fd);
+    kvm_irqfds_allowed = true;
 }
 
 static void kvm_arm_gic_class_init(ObjectClass *klass, void *data)
hw/vfio/platform.c +151 -14

229 lines not on this page · raw message

Signature · 1 line
1.8.3.2
#25 lore raw

Re: [Qemu-devel] [RFC v3 10/10] vfio: Add irqfd support in platform device

Quoted 15 lines · Eric Auger

This patch aims at optimizing IRQ handling using irqfd framework. It brings significant performance improvement over "traditional" IRQ handling introduced in :
"vfio: Add initial IRQ support in platform device".

This new IRQ handling method depends on kernel KVM irqfd/GSI routing capability.

The IRQ handling method can be dynamically chosen (default is irqfd, if kernel supports it obviously). For example to disable irqfd handling, use:

-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=110,irqfd=false\

Quoted 12 lines

Performances are improved for the following reasons:
- eventfds signalled by the VFIO platform driver are handled on kernel side by the KVM irqfd framework.
- the end of interrupt(EOI) is trapped at GIC level and not at MMIO region level. As a reminder, in traditional IRQ handling QEMU assumed the first guest access to a device MMIO region after IRQ hit was the IRQ status register reset. This trap was approximate and obliged to swap to slow path after IRQ hit. A mmap timer mechanism enabled to swap back to fast path after the mmap period introducing extra complexity. Now GIC detects the completion of the virtual IRQ and signals a resampler eventfd on maintenance IRQ. The corresponding handler re-enables the physical IRQ.

Ah, so if you're using irqfd you do unmask the interrupt on EOI. Why not without irqfd? And if the answer is "because it's too difficult" - why support VFIO without irqfd at all then?

Alex

#26 lore raw

Re: [Qemu-devel] [RFC v3 10/10] vfio: Add irqfd support in platform device

Quoted 29 lines · Alexander Graf
Quoted 15 lines · Eric Auger

This patch aims at optimizing IRQ handling using irqfd framework. It brings significant performance improvement over "traditional" IRQ handling introduced in :
"vfio: Add initial IRQ support in platform device".

This new IRQ handling method depends on kernel KVM irqfd/GSI routing capability.

The IRQ handling method can be dynamically chosen (default is irqfd, if kernel supports it obviously). For example to disable irqfd handling, use:

-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=110,irqfd=false\

Quoted 12 lines

Performances are improved for the following reasons:
- eventfds signalled by the VFIO platform driver are handled on kernel side by the KVM irqfd framework.
- the end of interrupt(EOI) is trapped at GIC level and not at MMIO region level. As a reminder, in traditional IRQ handling QEMU assumed the first guest access to a device MMIO region after IRQ hit was the IRQ status register reset. This trap was approximate and obliged to swap to slow path after IRQ hit. A mmap timer mechanism enabled to swap back to fast path after the mmap period introducing extra complexity. Now GIC detects the completion of the virtual IRQ and signals a resampler eventfd on maintenance IRQ. The corresponding handler re-enables the physical IRQ.

Quoted 3 lines

Ah, so if you're using irqfd you do unmask the interrupt on EOI. Why not without irqfd? And if the answer is "because it's too difficult" - why support VFIO without irqfd at all then?

Yes, it's too difficult, or at least it doesn't have sufficient ROI to try to plumb it through QEMU. What the hack in patch 5 enables is IRQ support that doesn't rely on a KVM irqchip. By not supporting it, we drop any hope of running tcg targets. I'll admit that a tcg target with an assigned device is mostly an unsupportable toy, but it's potentially useful for development and things like driver or even hardware debugging. IMHO, the EOI on device access is a nice, self contained solution, even if it ends up being rather sloppy with interrupts. Thanks,

Alex

#27 lore raw

Re: [Qemu-devel] [RFC v3 10/10] vfio: Add irqfd support in platform device

Quoted 32 lines · Alex Williamson
Quoted 28 lines · Alexander Graf
Quoted 15 lines · Eric Auger

This patch aims at optimizing IRQ handling using irqfd framework. It brings significant performance improvement over "traditional" IRQ handling introduced in :
"vfio: Add initial IRQ support in platform device".

This new IRQ handling method depends on kernel KVM irqfd/GSI routing capability.

The IRQ handling method can be dynamically chosen (default is irqfd, if kernel supports it obviously). For example to disable irqfd handling, use:

-device vfio-platform,vfio_device="fff51000.ethernet",\
compat="calxeda/hb-xgmac",mmap-timeout-ms=110,irqfd=false\

Quoted 12 lines

Performances are improved for the following reasons:
- eventfds signalled by the VFIO platform driver are handled on kernel side by the KVM irqfd framework.
- the end of interrupt(EOI) is trapped at GIC level and not at MMIO region level. As a reminder, in traditional IRQ handling QEMU assumed the first guest access to a device MMIO region after IRQ hit was the IRQ status register reset. This trap was approximate and obliged to swap to slow path after IRQ hit. A mmap timer mechanism enabled to swap back to fast path after the mmap period introducing extra complexity. Now GIC detects the completion of the virtual IRQ and signals a resampler eventfd on maintenance IRQ. The corresponding handler re-enables the physical IRQ.

Quoted 3 lines

Ah, so if you're using irqfd you do unmask the interrupt on EOI. Why not without irqfd? And if the answer is "because it's too difficult" - why support VFIO without irqfd at all then?

Quoted 9 lines

Yes, it's too difficult, or at least it doesn't have sufficient ROI to try to plumb it through QEMU. What the hack in patch 5 enables is IRQ support that doesn't rely on a KVM irqchip. By not supporting it, we drop any hope of running tcg targets. I'll admit that a tcg target with an assigned device is mostly an unsupportable toy, but it's potentially useful for development and things like driver or even hardware debugging. IMHO, the EOI on device access is a nice, self contained solution, even if it ends up being rather sloppy with interrupts. Thanks,

Yeah, I didn't realize that this is the same mechanism we use for PCI. You convinced me :).

Alex