Skip to contentSkip to conversation

vfio-user server in QEMU

PATCH Superseded Jagannathan Raman 9 participants 99 messages Jan 19, 2022 – Feb 17, 2022 Arm/Amlogic Meson SoC support

v5 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v5 cover vfio-user server in QEMU

Hi,

Thank you for taking the time to provide a comprehensive feedback of the last series of patches. We have addressed all the comments.

We are posting this v5 of the series, which incorporates all the feedback. Kindly share your feedback for this latest series

We added the following patches to the series:
- [PATCH v5 03/18] pci: isolated address space for PCI bus
- [PATCH v5 04/18] pci: create and free isolated PCI buses
- [PATCH v5 05/18] qdev: unplug blocker for devices
- [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
- [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

We made the following changes to the existing patches:

[PATCH v5 09/18] vfio-user: define vfio-user-server object
- renamed object class member 'daemon' as 'auto_shutdown'
- set VfioUserServerProperties version to 6.3
- use SocketAddressType_str to compose error message
- refuse setting 'socket' and 'device' properties after server starts
- added VFU_OBJECT_ERROR macro to report error

[PATCH v5 10/18] vfio-user: instantiate vfio-user context
- set error variable to NULL after transferring ownership with error_propagate()

[PATCH v5 11/18] vfio-user: find and init PCI device
- block hot-unplug of PCI device when it is attached to the server object

[PATCH v5 12/18] vfio-user: run vfio-user context
- emit a hangup event to the monitor when the client disconnects
- reset vfu_poll_fd member and disable FD handler during finalize
- add a comment to explain that attach could block
- use VFU_OBJECT_ERROR instead of setting error_abort

[PATCH v5 14/18] vfio-user: handle DMA mappings
- use pci_address_space() to access device's root memory region
- given we're using one bus per device, mapped memory regions get destroyed automatically when device is unplugged

[PATCH v5 15/18] vfio-user: handle PCI BAR accesses
- use pci_isol_as_io() & pci_isol_as_mem() to access the device's PCI/CPU address space. This simultaneously fixes the AddressSpace issue noted in the last review cycle

[PATCH v5 16/18] vfio-user: handle device interrupts
- setting own IRQ handlers for each bus
- renamed vfu_object_dev_table to vfu_object_dev_to_ctx_table
- indexing into vfu_object_dev_to_ctx_table with device's address pointer instead of devfn
- not looking up before removing from table

[PATCH v5 17/18] vfio-user: register handlers to facilitate migration
- use VFU_OBJECT_ERROR instead of setting error_abort

We dropped the following patch from previous series:
- vfio-user: IOMMU support for remote device

Thank you very much!

Jagannathan Raman (18):
configure, meson: override C compiler for cmake tests/avocado: Specify target VM argument to helper routines
pci: isolated address space for PCI bus
pci: create and free isolated PCI buses
qdev: unplug blocker for devices
vfio-user: add HotplugHandler for remote machine
vfio-user: set qdev bus callbacks for remote machine
vfio-user: build library
vfio-user: define vfio-user-server object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
vfio-user: register handlers to facilitate migration
vfio-user: avocado tests for vfio-user

 configure                                  |   21 +-
 meson.build                                |   44 +-
 qapi/misc.json                             |   23 +
 qapi/qom.json                              |   20 +-
 include/hw/pci/pci.h                       |   12 +
 include/hw/pci/pci_bus.h                   |   17 +
 include/hw/qdev-core.h                     |   21 +
 include/migration/vmstate.h                |    2 +
 migration/savevm.h                         |    2 +
 hw/pci/msi.c                               |   13 +-
 hw/pci/msix.c                              |   12 +-
 hw/pci/pci.c                               |  186 ++++
 hw/pci/pci_bridge.c                        |    5 +
 hw/remote/machine.c                        |   86 ++
 hw/remote/vfio-user-obj.c                  | 1019 ++++++++++++++++++++
 migration/savevm.c                         |   73 ++
 migration/vmstate.c                        |   19 +
 softmmu/qdev-monitor.c                     |   74 +-
 .gitlab-ci.d/buildtest.yml                 |    2 +
 .gitmodules                                |    3 +
 Kconfig.host                               |    4 +
 MAINTAINERS                                |    3 +
 hw/remote/Kconfig                          |    4 +
 hw/remote/meson.build                      |    3 +
 hw/remote/trace-events                     |   11 +
 meson_options.txt                          |    2 +
 subprojects/libvfio-user                   |    1 +
 tests/avocado/avocado_qemu/__init__.py     |   14 +-
 tests/avocado/vfio-user.py                 |  225 +++++
 tests/docker/dockerfiles/centos8.docker    |    2 +
 tests/docker/dockerfiles/ubuntu2004.docker |    2 +
 31 files changed, 1912 insertions(+), 13 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
 create mode 160000 subprojects/libvfio-user
 create mode 100644 tests/avocado/vfio-user.py
Signature · 1 line
2.20.1
#1 lore raw

PATCH v5 1/18 configure, meson: override C compiler for cmake

The compiler path that cmake gets from meson is corrupted. It results in the following error:

| -- The C compiler identification is unknown
| CMake Error at CMakeLists.txt:35 (project):
| The CMAKE_C_COMPILER:
| /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16
| is not a full path to an existing compiler tool.

Explicitly specify the C compiler for cmake to avoid this error

---
 configure | 2 ++
 1 file changed, 2 insertions(+)
configure +2 -0
index e1a31fb332..6a865f8713 100755
@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then
   echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross
   echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross
   echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross
+  echo "[cmake]" >> $cross
+  echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   echo "[binaries]" >> $cross
   echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake

Quoted 8 lines · Jagannathan Raman

The compiler path that cmake gets from meson is corrupted. It results in the following error:

| -- The C compiler identification is unknown
| CMake Error at CMakeLists.txt:35 (project):
| The CMAKE_C_COMPILER:
| /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16
| is not a full path to an existing compiler tool.
Quoted 4 lines

Explicitly specify the C compiler for cmake to avoid this error

This should not be needed anymore, as the bug in Meson has been fixed.

Paolo

Quoted 16 lines
  configure | 2 ++
  1 file changed, 2 insertions(+)
configure +2 -0
index e1a31fb332..6a865f8713 100755
@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then
    echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross
    echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross
    echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross
+  echo "[cmake]" >> $cross
+  echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
    echo "[binaries]" >> $cross
    echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
    test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
author
#3 lore raw

Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake

Quoted 16 lines · Paolo Bonzini
Quoted 10 lines · Jagannathan Raman

The compiler path that cmake gets from meson is corrupted. It results in the following error:

| -- The C compiler identification is unknown
| CMake Error at CMakeLists.txt:35 (project):
| The CMAKE_C_COMPILER:
| /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16
| is not a full path to an existing compiler tool.

Explicitly specify the C compiler for cmake to avoid this error

Quoted 1 line

This should not be needed anymore, as the bug in Meson has been fixed.

OK, will drop this patch.

Thank you!

Quoted 19 lines

Paolo

Quoted 15 lines
 configure | 2 ++
 1 file changed, 2 insertions(+)
configure +2 -0
index e1a31fb332..6a865f8713 100755
@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then
   echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross
   echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross
   echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross
+  echo "[cmake]" >> $cross
+  echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   echo "[binaries]" >> $cross
   echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
author
#4 lore raw

Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake

In reply to Paolo Bonzini
Quoted 16 lines · Paolo Bonzini
Quoted 10 lines · Jagannathan Raman

The compiler path that cmake gets from meson is corrupted. It results in the following error:

| -- The C compiler identification is unknown
| CMake Error at CMakeLists.txt:35 (project):
| The CMAKE_C_COMPILER:
| /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16
| is not a full path to an existing compiler tool.

Explicitly specify the C compiler for cmake to avoid this error

Quoted 1 line

This should not be needed anymore, as the bug in Meson has been fixed.

Hi Paolo,

I’m able to see the bug with latest QEMU. The fix doesn’t appear to be available with meson version 0.59.3, which is what QEMU is presently using.

Thank you!
--
Jag

Quoted 19 lines

Paolo

Quoted 15 lines
 configure | 2 ++
 1 file changed, 2 insertions(+)
configure +2 -0
index e1a31fb332..6a865f8713 100755
@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then
   echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross
   echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross
   echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross
+  echo "[cmake]" >> $cross
+  echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   echo "[binaries]" >> $cross
   echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross
   test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
#5 lore raw

PATCH v5 2/18 tests/avocado: Specify target VM argument to helper routines

In reply to Jagannathan Raman

Specify target VM for exec_command and
exec_command_and_wait_for_pattern routines

---
 tests/avocado/avocado_qemu/__init__.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
tests/avocado/avocado_qemu/__init__.py +10 -4
index 75063c0c30..b3fbf77577 100644
@@ -198,7 +198,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None,
     """
     _console_interaction(test, success_message, failure_message, None, vm=vm)
 
-def exec_command(test, command):
+def exec_command(test, command, vm=None):
     """
     Send a command to a console (appending CRLF characters), while logging
     the content.
@@ -207,11 +207,14 @@ def exec_command(test, command):
     :type test: :class:`avocado_qemu.QemuSystemTest`
     :param command: the command to send
     :type command: str
+    :param vm: target vm
+    :type vm: :class:`qemu.machine.QEMUMachine`
     """
-    _console_interaction(test, None, None, command + '\r')
+    _console_interaction(test, None, None, command + '\r', vm=vm)
 
 def exec_command_and_wait_for_pattern(test, command,
-                                      success_message, failure_message=None):
+                                      success_message, failure_message=None,
+                                      vm=None):
     """
     Send a command to a console (appending CRLF characters), then wait
     for success_message to appear on the console, while logging the.
@@ -223,8 +226,11 @@ def exec_command_and_wait_for_pattern(test, command,
     :param command: the command to send
     :param success_message: if this message appears, test succeeds
     :param failure_message: if this message appears, test fails
+    :param vm: target vm
+    :type vm: :class:`qemu.machine.QEMUMachine`
     """
-    _console_interaction(test, success_message, failure_message, command + '\r')
+    _console_interaction(test, success_message, failure_message, command + '\r',
+                         vm=vm)
 
 class QemuBaseTest(avocado.Test):
     def _get_unique_tag_val(self, tag_name):
Signature · 1 line
2.20.1
Reviewed-by on v5 2/18
#6 lore raw

Re: [PATCH v5 02/18] tests/avocado: Specify target VM argument to helper routines

Quoted 11 lines · Jagannathan Raman

Specify target VM for exec_command and
exec_command_and_wait_for_pattern routines

---
 tests/avocado/avocado_qemu/__init__.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
#7 lore raw

PATCH v5 3/18 pci: isolated address space for PCI bus

In reply to Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

---
 include/hw/pci/pci.h     |  2 ++
 include/hw/pci/pci_bus.h | 17 +++++++++++++++++
 hw/pci/pci.c             | 17 +++++++++++++++++
 hw/pci/pci_bridge.c      |  5 +++++
 4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
  * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +17 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
     void *irq_opaque;
     PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
     PCIDevice *parent_dev;
+
     MemoryRegion *address_space_mem;
     MemoryRegion *address_space_io;
 
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io. This allows the
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients, as such it needs the PCI buses in the same machine
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
     QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
     QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
 
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
     bus->slot_reserved_mask = 0x0;
     bus->address_space_mem = address_space_mem;
     bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
     bus->flags |= PCI_BUS_IS_ROOT;
 
     /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
     return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
     PCIBus *bus = pci_get_bus(dev);
     PCIBus *iommu_bus = bus;
     uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
     if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
         return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
     }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
     return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
     sec_bus->address_space_io = &br->address_space_io;
     memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                        4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
     br->windows = pci_bridge_region_init(br);
     QLIST_INIT(&sec_bus->child);
     QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 7 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Quoted 29 lines
---
 include/hw/pci/pci.h     |  2 ++
 include/hw/pci/pci_bus.h | 17 +++++++++++++++++
 hw/pci/pci.c             | 17 +++++++++++++++++
 hw/pci/pci_bridge.c      |  5 +++++
 4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
  * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +0 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
     void *irq_opaque;

include/hw/pci/pci_bus.h struct PCIBus { · line 49

Quoted 10 lines
include/hw/pci/pci_bus.h +5 -0
     PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
     PCIDevice *parent_dev;
+
     MemoryRegion *address_space_mem;
     MemoryRegion *address_space_io;
 
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 6 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

Quoted 78 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
     QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
     QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
     bus->slot_reserved_mask = 0x0;
     bus->address_space_mem = address_space_mem;
     bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
     bus->flags |= PCI_BUS_IS_ROOT;
 
     /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
     return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
     PCIBus *bus = pci_get_bus(dev);
     PCIBus *iommu_bus = bus;
     uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
     if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
         return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
     }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
     return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
     sec_bus->address_space_io = &br->address_space_io;
     memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                        4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
     br->windows = pci_bridge_region_init(br);
     QLIST_INIT(&sec_bus->child);
     QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
author
#9 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.

Quoted 3 lines

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Could you please provide some more details on the design?

Quoted 36 lines
Quoted 36 lines

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
    void *irq_opaque;
    PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
    PCIDevice *parent_dev;
+
    MemoryRegion *address_space_mem;
    MemoryRegion *address_space_io;
 
Quoted 7 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 4 lines
include/hw/pci/pci_bus.h +4 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.

In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.

Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg

Quoted 9 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 7 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

It’s the client in the client - server model explained above.

Thank you!
--
Jag

Quoted 79 lines
Quoted 79 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
    QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
    QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
    bus->slot_reserved_mask = 0x0;
    bus->address_space_mem = address_space_mem;
    bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
    bus->flags |= PCI_BUS_IS_ROOT;
 
    /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
    return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
    DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
    PCIBus *bus = pci_get_bus(dev);
    PCIBus *iommu_bus = bus;
    uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
    if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
        return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
    }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
    return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
    sec_bus->address_space_io = &br->address_space_io;
    memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                       4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
    br->windows = pci_bridge_region_init(br);
    QLIST_INIT(&sec_bus->child);
    QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
#10 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 15 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Quoted 7 lines

Hi Michael!

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Dave

Quoted 163 lines

In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.

Quoted 3 lines

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Could you please provide some more details on the design?

Quoted 36 lines
Quoted 36 lines

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
    void *irq_opaque;
    PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
    PCIDevice *parent_dev;
+
    MemoryRegion *address_space_mem;
    MemoryRegion *address_space_io;
 
Quoted 7 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 4 lines
include/hw/pci/pci_bus.h +4 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.

In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.

Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg

Quoted 9 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 7 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

It’s the client in the client - server model explained above.

Thank you!
--
Jag

Quoted 79 lines
Quoted 79 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
    QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
    QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
    bus->slot_reserved_mask = 0x0;
    bus->address_space_mem = address_space_mem;
    bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
    bus->flags |= PCI_BUS_IS_ROOT;
 
    /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
    return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
    DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
    PCIBus *bus = pci_get_bus(dev);
    PCIBus *iommu_bus = bus;
    uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
    if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
        return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
    }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
    return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
    sec_bus->address_space_io = &br->address_space_io;
    memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                       4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
    br->windows = pci_bridge_region_init(br);
    QLIST_INIT(&sec_bus->child);
    QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#11 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

Thank you!
--
Jag

Quoted 169 lines

Dave

Quoted 163 lines

In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.

Quoted 3 lines

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Could you please provide some more details on the design?

Quoted 36 lines
Quoted 36 lines

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
   void *irq_opaque;
   PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
   PCIDevice *parent_dev;
+
   MemoryRegion *address_space_mem;
   MemoryRegion *address_space_io;
 
Quoted 7 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 4 lines
include/hw/pci/pci_bus.h +4 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.

In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.

Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg

Quoted 9 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 7 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

It’s the client in the client - server model explained above.

Thank you!
--
Jag

Quoted 79 lines
Quoted 79 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
   QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
   QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
   bus->slot_reserved_mask = 0x0;
   bus->address_space_mem = address_space_mem;
   bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
   bus->flags |= PCI_BUS_IS_ROOT;
 
   /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
   return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
   DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
   PCIBus *bus = pci_get_bus(dev);
   PCIBus *iommu_bus = bus;
   uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
   if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
       return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
   }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
   return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
   sec_bus->address_space_io = &br->address_space_io;
   memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                      4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
   br->windows = pci_bridge_region_init(br);
   QLIST_INIT(&sec_bus->child);
   QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#12 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

Stefan

#13 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Quoted 8 lines

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices? Or that the address space of a given PCIe bus is non-overlapping with one of the others?

Dave

Quoted 1 line

Stefan

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#14 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 64 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Quoted 8 lines

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

Quoted 3 lines

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Quoted 12 lines

Or that the
address space of a given PCIe bus is non-overlapping with one of the others?

Dave

Quoted 1 line

Stefan

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#15 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 62 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

Quoted 7 lines
Quoted 3 lines

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.

Stefan

#16 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 74 lines · Stefan Hajnoczi
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 62 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

Quoted 7 lines
Quoted 3 lines

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Quoted 7 lines

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.

Stefan

I was wondering about it here:
https://lore.kernel.org/r/20220119190742-mutt-send-email-mst%40kernel.org

Signature · 1 line
MST
#17 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Stefan Hajnoczi
Quoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Quoted 8 lines

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

Quoted 8 lines
Quoted 2 lines

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.

Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does? In fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

Alex

#18 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 83 lines · Alex Williamson
Quoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Quoted 8 lines

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

Quoted 8 lines
Quoted 2 lines

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.

Quoted 5 lines

Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.

Oops, thank you for pointing that out!

Stefan

#19 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Alex Williamson
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

Stefan

#20 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Alex

#21 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.

Stefan

#22 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Quoted 8 lines

What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.

This is getting a bit ridiculous, if vfio-user devices require this degree of manipulation of the VM topology into things that don't exist on bare metal, we've done something very wrong. Thanks,

Alex

author
#23 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 72 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.

Typically, the machines map the PCI address space into the system address space. For example, pc_pci_as_mapping_init() does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space.

Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for address_space_rw() to successfully vector BAR accesses into the PCI device.

Thank you!
--
Jag

Quoted 16 lines
Quoted 9 lines

What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.

This is getting a bit ridiculous, if vfio-user devices require this degree of manipulation of the VM topology into things that don't exist on bare metal, we've done something very wrong. Thanks,

Alex

#24 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 4 lines

The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.

These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from address_space_mem.

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Quoted 6 lines

Typically, the machines map the PCI address space into the system address space. For example, pc_pci_as_mapping_init() does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space.

"Typically" only if we're restricted to the "pc", ie. i440FX, machine type since it doesn't support a vIOMMU. There's no reason to focus on the identity map case versus the vIOMMU case.

Quoted 4 lines

Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for address_space_rw() to successfully vector BAR accesses into the PCI device.

But as far as device initiated transactions, there is only one address space for a given device, it's either address_space_mem or one provided by the vIOMMU and pci_device_iommu_address_space() tells us that address space. Furthermore, the device never operates on a "physical address", it only ever operates on an IOVA, ie. an offset within the address space assigned to the device. The IOVA should be considered arbitrary relative to mappings in any other address spaces.

Device initiated transactions operate on an IOVA within the (single) address space to which the device is assigned. Any attempt to do otherwise violates the isolation put in place by things like vIOMMUs and ought to be considered a security concern, especially for a device serviced by an external process. Thanks,

Alex

author
#25 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 109 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

Thank you!
--
Jag

Quoted 49 lines
Quoted 5 lines

The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.

These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from address_space_mem.

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Quoted 6 lines

Typically, the machines map the PCI address space into the system address space. For example, pc_pci_as_mapping_init() does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space.

"Typically" only if we're restricted to the "pc", ie. i440FX, machine type since it doesn't support a vIOMMU. There's no reason to focus on the identity map case versus the vIOMMU case.

Quoted 4 lines

Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for address_space_rw() to successfully vector BAR accesses into the PCI device.

But as far as device initiated transactions, there is only one address space for a given device, it's either address_space_mem or one provided by the vIOMMU and pci_device_iommu_address_space() tells us that address space. Furthermore, the device never operates on a "physical address", it only ever operates on an IOVA, ie. an offset within the address space assigned to the device. The IOVA should be considered arbitrary relative to mappings in any other address spaces.

Device initiated transactions operate on an IOVA within the (single) address space to which the device is assigned. Any attempt to do otherwise violates the isolation put in place by things like vIOMMUs and ought to be considered a security concern, especially for a device serviced by an external process. Thanks,

Alex

#26 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Alex

#27 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 124 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

Quoted 4 lines

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

The access comes via libvfio-user's vfu_region_access_cb_t callback, not via MemoryRegionOps. The callback is currently implemented by calling address_space_rw() on the pci_isol_as_mem/io() address space, depending on the BAR type. The code in "[PATCH v5 15/18] vfio-user: handle PCI BAR accesses".

It's possible to reimplement the patch to directly call memory_region_dispatch_read/write() on r->memory instead of address_space_rw() as you've described.

Quoted 14 lines
Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

Quoted 8 lines

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Good, I think nothing uses address_space_system/io when BAR dispatch is implemented with memory_region_dispatch_read/write() as you suggested.

It would be nice if there was a way to poison address_space_system/io to abort on dispatch - nothing should use them.

We now have the option of dropping pci_isol_as_mem/io() again and using ->iommu_fn() to return an isolated memory address space containing the vfio-user client's VFIO_USER_DMA_MAP regions.

Stefan

author
#28 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Alex Williamson
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Quoted 8 lines

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Thank you!
--
Jag

Quoted 3 lines

Alex

#29 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Quoted 8 lines

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU. I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 6 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

Quoted 7 lines

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

Quoted 7 lines

--
Jag

Quoted 3 lines

Alex

author
#30 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 187 lines · Michael S. Tsirkin
Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Quoted 8 lines

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

Thank you!
--
Jag

Quoted 21 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

Quoted 7 lines

--
Jag

Quoted 3 lines

Alex

#31 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 187 lines · Jag Raman
Quoted 187 lines · Michael S. Tsirkin
Quoted 181 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Quoted 8 lines

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 6 lines
Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU.

Hmm do you mean you analyzed how a physical device works? Or do you mean in QEMU?

Quoted 2 lines

The device is reading the BAR
address as if it was a DMA address.

I got that, my understanding of PCI was that a device can not be both a master and a target of a transaction at the same time though. Could not find this in the spec though, maybe I remember incorrectly.

Quoted 49 lines
Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

Thank you!
--
Jag

Quoted 21 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

Quoted 7 lines

--
Jag

Quoted 3 lines

Alex

author
#32 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 195 lines · Michael S. Tsirkin
Quoted 190 lines · Jag Raman
Quoted 189 lines · Michael S. Tsirkin
Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson

If the goal here is to restrict DMA between devices, ie. peer-to-peer
(p2p), why are we trying to re-invent what an IOMMU already does?

The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.

I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA
address space per BDF. Is the dynamic mapping overhead too much? What
physical hardware properties or specifications could we leverage to
restrict p2p mappings to a device? Should it be governed by machine
type to provide consistency between devices? Should each "isolated"
bus be in a separate root complex? Thanks,

There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.

Quoted 3 lines

I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.

This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.

Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.

Quoted 8 lines

In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.

If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,

Quoted 6 lines

Yes, BARs and IOVAs don't need to be unique across all devices.

The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.

Quoted 11 lines

Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.

Hi Alex,

I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).

In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.

Quoted 8 lines

The following gives an overview of issues concerning vectoring of BAR accesses.

The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.

Quoted 5 lines

The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.

Quoted 7 lines

Thank you for the explanation, Alex!

The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.

If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.

Quoted 8 lines

Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.

On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())

This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.

The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.

Quoted 8 lines

In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Quoted 6 lines
Quoted 3 lines

Hi Michael,

In LSI case, I did notice that it went to the IOMMU.

Hmm do you mean you analyzed how a physical device works? Or do you mean in QEMU?

I mean in QEMU, I did not analyze a physical device.

Quoted 8 lines
Quoted 3 lines

The device is reading the BAR
address as if it was a DMA address.

I got that, my understanding of PCI was that a device can not be both a master and a target of a transaction at the same time though. Could not find this in the spec though, maybe I remember incorrectly.

I see, OK. If this were to happen in a real device, PCI would raise an error because the master and target of a transaction can’t be the same. So you believe that this access is handled inside the device, and doesn’t go out.

Thanks!
--
Jag

Quoted 46 lines
Quoted 46 lines
Quoted 17 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

Thank you!
--
Jag

Quoted 18 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

Quoted 5 lines

--
Jag

Quoted 2 lines

Alex

#33 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Jag Raman
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture. From the processor perspective, each MMIO range is unique and possibly none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.

Quoted 11 lines
Quoted 11 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,

Alex

#34 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 55 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

Quoted 8 lines

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.

Quoted 1 line

From the processor perspective, each MMIO range is unique and possibly

none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have _TRA.

Quoted 7 lines

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Quoted 7 lines

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.

Quoted 3 lines

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.

You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

Quoted 6 lines

This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,

Alex

thanks!

Signature · 1 line
MST
#35 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.

Quoted 1 line

From the processor perspective, each MMIO range is unique and possibly

Quoted 8 lines
Quoted 2 lines

none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have _TRA.

How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used _TRA to provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.

Quoted 15 lines
Quoted 7 lines

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Quoted 7 lines

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

Quoted 3 lines

However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.

A shortcut that breaks when an IOMMU is involved.

Quoted 6 lines
Quoted 3 lines

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.

You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).

We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,

Alex

#36 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 74 lines · Alex Williamson
Quoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.

Quoted 1 line

From the processor perspective, each MMIO range is unique and possibly

Quoted 8 lines
Quoted 2 lines

none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have _TRA.

Quoted 4 lines

How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used _TRA to provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.

It's an interesting idea. To clarify what I said, I suspect some devices are broken in presence of translating bridges unless DMA
is also translated to match.

I agree it's a mess though, in that some devices when given their own BAR to DMA to will probably just satisfy the access from internal memory, while others will ignore that and send it up as DMA and both types are probably out there in the field.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

Quoted 5 lines
Quoted 3 lines

However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.

A shortcut that breaks when an IOMMU is involved.

Maybe. But if that's how hardware behaves, we have little choice but emulate it.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.

You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).

Quoted 6 lines

We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,

Alex

More importantly lots devices most likely don't support arbitrary configurations and break if you try to create something that matches the spec but never or even rarely occurs on bare metal.

Signature · 1 line
MST
author
#37 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 109 lines · Michael S. Tsirkin
Quoted 74 lines · Alex Williamson
Quoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.

Quoted 1 line

From the processor perspective, each MMIO range is unique and possibly

Quoted 8 lines
Quoted 2 lines

none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have _TRA.

Quoted 4 lines

How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used _TRA to provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.

It's an interesting idea. To clarify what I said, I suspect some devices are broken in presence of translating bridges unless DMA
is also translated to match.

I agree it's a mess though, in that some devices when given their own BAR to DMA to will probably just satisfy the access from internal memory, while others will ignore that and send it up as DMA and both types are probably out there in the field.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.

A shortcut that breaks when an IOMMU is involved.

Maybe. But if that's how hardware behaves, we have little choice but emulate it.

I was wondering if we could map the BARs into the IOVA for a limited set of devices - the ones which are designed before IOMMU such as lsi53c895a. This would ensure that we follow the spec to the best without breaking existing devices?

--
Jag

Quoted 22 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.

You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).

Quoted 6 lines

We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,

Alex

More importantly lots devices most likely don't support arbitrary configurations and break if you try to create something that matches the spec but never or even rarely occurs on bare metal.

Signature · 1 line
MST
author
#38 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Alex Williamson
Quoted 83 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman

Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.

We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.

Quoted 10 lines

Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.

Quoted 2 lines

So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.

Hi Michael,

In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.

Quoted 16 lines

I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.

Quoted 12 lines

Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.

I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thank you!

I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles

OK sounds good, I’ll create a mapping of the device BARs in the IOVA.

I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture. From the processor perspective, each MMIO range is unique and possibly none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.

So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.

Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.

Quoted 2 lines

It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this

It does look like the driver is actually programming the device to DMA to itself.

The driver first programs the DSP (DMA Scripts Pointer) register with the BAR address. It does so by performing a series of MMIO writes (lsi_mmio_write()) to offsets 0x2C - 0x2F. Immediately after programming this register, the device fetches some instructions located at the programmed address.

Thank you!
--
Jag

Quoted 21 lines

driver is actually compatible with an IOMMU on bare metal.

Quoted 11 lines
Quoted 11 lines

within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.

As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.

I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?

This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,

Alex

#39 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Alex Williamson
Quoted 8 lines · Alex Williamson
Quoted 4 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:

The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.

Quoted 4 lines

These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from address_space_mem.

Note that those fields in PCIBus are just whatever MemoryRegions the pci controller model passed in to the call to pci_root_bus_init() or equivalent. They may or may not be specifically the CPU's view of anything. (For instance on the versatilepb board, the PCI controller is visible to the CPU via several MMIO "windows" at known addresses, which let the CPU access into the PCI address space at a programmable offset. We model that by creating a couple of container MRs which we pass to pci_root_bus_init() to be the PCI memory and IO spaces, and then using alias MRs to provide the view into those at the guest-programmed offset. The CPU sees those windows, and doesn't have direct access to the whole PCIBus::address_space_mem.) I guess you could say they're the PCI controller's view of the PCI address space ?

We have a tendency to be a bit sloppy with use of AddressSpaces within QEMU where it happens that the view of the world that a DMA-capable device matches that of the CPU, but conceptually they can definitely be different, especially in the non-x86 world. (Linux also confuses matters here by preferring to program a 1:1 mapping even if the hardware is more flexible and can do other things. The model of the h/w in QEMU should support the other cases too, not just 1:1.)

Quoted 5 lines

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?

thanks
-- PMM

#40 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 6 lines · Peter Maydell
Quoted 5 lines

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Quoted 6 lines

Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?

thanks
-- PMM

Hmm, the only thing vaguely related in the spec that I know of is this:

PCI Express supports I/O Space for compatibility with legacy devices which require their use. Future revisions of this specification may deprecate the use of I/O Space.

Alex, what did you refer to?

Signature · 1 line
MST
#41 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 14 lines · On Wed, 2 Feb 2022 05:06:49 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 6 lines · Peter Maydell
Quoted 5 lines

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Quoted 6 lines

Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?

thanks
-- PMM

Quoted 6 lines

Hmm, the only thing vaguely related in the spec that I know of is this:

PCI Express supports I/O Space for compatibility with legacy devices which require their use. Future revisions of this specification may deprecate the use of I/O Space.

Alex, what did you refer to?

My evidence is largely by omission, but that might be that in practice it's not used rather than explicitly forbidden. I note that the bus master enable bit specifies:

Bus Master Enable - Controls the ability of a Function to issue
Memory and I/O Read/Write Requests, and the ability of a Port to forward Memory and I/O Read/Write Requests in the Upstream direction.

That would suggest it's possible, but for PCI device assignment, I'm not aware of any means through which we could support this. There is no support in the IOMMU core for mapping I/O port space, nor could we trap such device initiated transactions to emulate them. I can't spot any mention of I/O port space in the VT-d spec, however the AMD-Vi spec does include a field in the device table:

controlIoCtl: port I/O control. Specifies whether device-initiated port I/O space transactions are blocked, forwarded, or translated.

00b=Device-initiated port I/O is not allowed. The IOMMU target aborts the transaction if a port I/O space transaction is received. Translation requests are target aborted.

01b=Device-initiated port I/O space transactions are allowed. The IOMMU must pass port I/O accesses untranslated. Translation requests are target aborted.

10b=Transactions in the port I/O space address range are translated by the IOMMU page tables as memory transactions.

11b=Reserved.

I don't see this field among the macros used by the Linux driver in configuring these device entries, so I assume it's left to the default value, ie. zero, blocking device initiated I/O port transactions.

So yes, I suppose device initiated I/O port transactions are possible, but we have no support or reason to support them, so I'm going to go ahead and continue believing any I/O port address space from the device perspective is largely irrelevant ;) Thanks,

Alex

#42 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 39 lines · Alex Williamson
Quoted 1 line

Alex, what did you refer to?

My evidence is largely by omission, but that might be that in practice it's not used rather than explicitly forbidden. I note that the bus master enable bit specifies:

Bus Master Enable - Controls the ability of a Function to issue
Memory and I/O Read/Write Requests, and the ability of a Port to forward Memory and I/O Read/Write Requests in the Upstream direction.

That would suggest it's possible, but for PCI device assignment, I'm not aware of any means through which we could support this. There is no support in the IOMMU core for mapping I/O port space, nor could we trap such device initiated transactions to emulate them. I can't spot any mention of I/O port space in the VT-d spec, however the AMD-Vi spec does include a field in the device table:

controlIoCtl: port I/O control. Specifies whether device-initiated port I/O space transactions are blocked, forwarded, or translated.

00b=Device-initiated port I/O is not allowed. The IOMMU target aborts the transaction if a port I/O space transaction is received. Translation requests are target aborted.

01b=Device-initiated port I/O space transactions are allowed. The IOMMU must pass port I/O accesses untranslated. Translation requests are target aborted.

10b=Transactions in the port I/O space address range are translated by the IOMMU page tables as memory transactions.

11b=Reserved.

I don't see this field among the macros used by the Linux driver in configuring these device entries, so I assume it's left to the default value, ie. zero, blocking device initiated I/O port transactions.

Quoted 6 lines

So yes, I suppose device initiated I/O port transactions are possible, but we have no support or reason to support them, so I'm going to go ahead and continue believing any I/O port address space from the device perspective is largely irrelevant ;) Thanks,

Alex

Right, it would seem devices can initiate I/O space transactions but IOMMUs don't support virtualizing them and so neither does VFIO.

Signature · 1 line
MST
#43 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Peter Maydell
Quoted 14 lines · On Wed, 2 Feb 2022 09:30:42 +0000 Peter Maydell <[email protected]>…
Quoted 8 lines · Alex Williamson
Quoted 4 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:

The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.

Quoted 4 lines

These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from address_space_mem.

Quoted 13 lines

Note that those fields in PCIBus are just whatever MemoryRegions the pci controller model passed in to the call to pci_root_bus_init() or equivalent. They may or may not be specifically the CPU's view of anything. (For instance on the versatilepb board, the PCI controller is visible to the CPU via several MMIO "windows" at known addresses, which let the CPU access into the PCI address space at a programmable offset. We model that by creating a couple of container MRs which we pass to pci_root_bus_init() to be the PCI memory and IO spaces, and then using alias MRs to provide the view into those at the guest-programmed offset. The CPU sees those windows, and doesn't have direct access to the whole PCIBus::address_space_mem.) I guess you could say they're the PCI controller's view of the PCI address space ?

Sure, that's fair.

Quoted 8 lines

We have a tendency to be a bit sloppy with use of AddressSpaces within QEMU where it happens that the view of the world that a DMA-capable device matches that of the CPU, but conceptually they can definitely be different, especially in the non-x86 world. (Linux also confuses matters here by preferring to program a 1:1 mapping even if the hardware is more flexible and can do other things. The model of the h/w in QEMU should support the other cases too, not just 1:1.)

Right, this is why I prefer to look at the device address space as simply an IOVA. The IOVA might be a direct physical address or coincidental identity mapped physical address via an IOMMU, but none of that should be the concern of the device.

Quoted 9 lines
Quoted 5 lines

I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.

Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?

As realized in reply to MST, more the latter. Not used, no point to enabling, no means to enable depending on the physical IOMMU implementation. Thanks,

Alex

#44 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Alex Williamson
Quoted 83 lines · * Alex Williamson ([email protected]) wrote:
Quoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 8 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.

Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read

Quoted 8 lines

Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.

I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?

Quoted 8 lines
Quoted 2 lines

But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.

Michael: Are you saying just turn on vIOMMU? :)

Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.

Quoted 8 lines

Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.

If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?

That was what I was curious about - is it possible to get an IOMMU to do that, and how? (Not knowing much about IOMMUs).
In my DAX/virtiofs case, I want the device to be able to DMA to guest RAM but for other devices not to try to DMA to it and in particular for it not to have to DMA to other devices.

Quoted 7 lines

In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,

Dave

Quoted 2 lines

Alex

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#45 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Jag Raman
Quoted 32 lines · * Jag Raman ([email protected]) wrote:
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 8 lines
Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

Are all devices mappable like that?

Quoted 3 lines

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I didn't have one in mind; I was just hitting a similar problem on Virtiofs DAX.

Dave

Quoted 174 lines

Thank you!
--
Jag

Quoted 169 lines

Dave

Quoted 163 lines

In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.

Quoted 3 lines

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Could you please provide some more details on the design?

Quoted 36 lines
Quoted 36 lines

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
   void *irq_opaque;
   PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
   PCIDevice *parent_dev;
+
   MemoryRegion *address_space_mem;
   MemoryRegion *address_space_io;
 
Quoted 7 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 4 lines
include/hw/pci/pci_bus.h +4 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.

In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.

Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg

Quoted 9 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 7 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

It’s the client in the client - server model explained above.

Thank you!
--
Jag

Quoted 79 lines
Quoted 79 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
   QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
   QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
   bus->slot_reserved_mask = 0x0;
   bus->address_space_mem = address_space_mem;
   bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
   bus->flags |= PCI_BUS_IS_ROOT;
 
   /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
   return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
   DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
   PCIBus *bus = pci_get_bus(dev);
   PCIBus *iommu_bus = bus;
   uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
   if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
       return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
   }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
   return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
   sec_bus->address_space_io = &br->address_space_io;
   memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                      4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
   br->windows = pci_bridge_region_init(br);
   QLIST_INIT(&sec_bus->child);
   QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
author
#46 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 41 lines · Dr. David Alan Gilbert
Quoted 36 lines · * Jag Raman ([email protected]) wrote:
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

Quoted 6 lines
Quoted 4 lines

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

Fascinating, but I am not sure I understand. any examples?

Hi Michael!

Quoted 7 lines
Quoted 5 lines

multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.

Do you ever have the opposite problem? i.e. when an emulated PCI device

That’s an interesting question.

Quoted 3 lines

exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?

Quoted 6 lines
Quoted 4 lines

In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.

Are all devices mappable like that?

If there is an IOMMU that supports DMA-Remapping (DMAR), that would be possible - the kernel could configure the DMAR to facilitate such mapping.

If there is no DMAR, the kernel/cpu could buffer the write between devices.

--
Jag

Quoted 187 lines
Quoted 4 lines

I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?

I didn't have one in mind; I was just hitting a similar problem on Virtiofs DAX.

Dave

Quoted 174 lines

Thank you!
--
Jag

Quoted 169 lines

Dave

Quoted 163 lines

In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.

Quoted 3 lines

I also wonder whether this special type could be modelled like a special kind of iommu internally.

Could you please provide some more details on the design?

Quoted 36 lines
Quoted 36 lines

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
  void *irq_opaque;
  PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
  PCIDevice *parent_dev;
+
  MemoryRegion *address_space_mem;
  MemoryRegion *address_space_io;
 
Quoted 7 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 4 lines
include/hw/pci/pci_bus.h +4 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io.

Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?

On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.

In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.

Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg

Quoted 9 lines

include/hw/pci/pci_bus.h struct PCIBus {

Quoted 7 lines

This allows the

include/hw/pci/pci_bus.h +5 -0
+     * bus to be attached to CPUs from different machines. The
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients,

what are VM clients?

It’s the client in the client - server model explained above.

Thank you!
--
Jag

Quoted 79 lines
Quoted 79 lines

as such it needs the PCI buses in the same machine

include/hw/pci/pci_bus.h +7 -0
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
+
  QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
  QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
  bus->slot_reserved_mask = 0x0;
  bus->address_space_mem = address_space_mem;
  bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
  bus->flags |= PCI_BUS_IS_ROOT;
 
  /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
  return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
  DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
  PCIBus *bus = pci_get_bus(dev);
  PCIBus *iommu_bus = bus;
  uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
  if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
      return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
  }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
  return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
  sec_bus->address_space_io = &br->address_space_io;
  memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                     4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
  br->windows = pci_bridge_region_init(br);
  QLIST_INIT(&sec_bus->child);
  QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#47 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

In reply to Jagannathan Raman
Quoted 36 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

---
 include/hw/pci/pci.h     |  2 ++
 include/hw/pci/pci_bus.h | 17 +++++++++++++++++
 hw/pci/pci.c             | 17 +++++++++++++++++
 hw/pci/pci_bridge.c      |  5 +++++
 4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
  * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +0 -0
index 347440d42c..d78258e79e 100644

include/hw/pci/pci_bus.h struct PCIBus { · line 44

Quoted 6 lines
include/hw/pci/pci_bus.h +1 -0
@@ -39,9 +39,26 @@ struct PCIBus {
     void *irq_opaque;
     PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
     PCIDevice *parent_dev;
+
     MemoryRegion *address_space_mem;
     MemoryRegion *address_space_io;

This seems like a good point to rename address_space_mem, address_space_io, as well as PCIIORegion->address_space since they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr and PCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.

Quoted 6 lines
include/hw/pci/pci_bus.h +5 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io. This allows the
+     * bus to be attached to CPUs from different machines. The

include/hw/pci/pci_bus.h struct PCIBus { · line 60

Quoted 10 lines
include/hw/pci/pci_bus.h +10 -0
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients, as such it needs the PCI buses in the same machine
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;

Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.

isol_as_io isn't used by this patch?

Quoted 73 lines
include/hw/pci/pci_bus.h +1 -0
+
     QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
     QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
 
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
     bus->slot_reserved_mask = 0x0;
     bus->address_space_mem = address_space_mem;
     bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
     bus->flags |= PCI_BUS_IS_ROOT;
 
     /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
     return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
     PCIBus *bus = pci_get_bus(dev);
     PCIBus *iommu_bus = bus;
     uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
     if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
         return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
     }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
     return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
     sec_bus->address_space_io = &br->address_space_io;
     memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                        4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
     br->windows = pci_bridge_region_init(br);
     QLIST_INIT(&sec_bus->child);
     QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
author
#48 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 46 lines · Stefan Hajnoczi
Quoted 41 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
    void *irq_opaque;
    PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
    PCIDevice *parent_dev;
+
    MemoryRegion *address_space_mem;
Quoted 8 lines

include/hw/pci/pci_bus.h struct PCIBus { · line 44

Quoted 1 line
include/hw/pci/pci_bus.h +0 -0
    MemoryRegion *address_space_io;

This seems like a good point to rename address_space_mem, address_space_io, as well as PCIIORegion->address_space since they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr and PCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.

Are you referring to renaming address_space_mem, address_space_io and PCIIORegion->address_space alone? I’m asking because there are many other symbols in the code which are named similarly.

Quoted 18 lines
Quoted 7 lines
include/hw/pci/pci_bus.h +5 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io. This allows the
+     * bus to be attached to CPUs from different machines. The

include/hw/pci/pci_bus.h struct PCIBus { · line 60

Quoted 10 lines
include/hw/pci/pci_bus.h +10 -0
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients, as such it needs the PCI buses in the same machine
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
Quoted 3 lines

Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.

I did start off with using these pointers unconditionally - but adopted an optional isolated address space for the following reasons:
- There is a potential for regression
- CPU address space per bus is not a common scenario. In most case, all PCI buses are attached to CPU sharing the same address space. Therefore, an optional address space made sense for this special scenario

We can also set it unconditionally if you prefer, kindly confirm.

Quoted 2 lines

isol_as_io isn't used by this patch?

This patch introduces these variables, defines its getters and sets them to NULL in places where new PCI buses are presently created. The following patch creates a separate isolated address space:
[PATCH v5 04/18] pci: create and free isolated PCI buses

I could merge these patches if you prefer.

--
Jag

Quoted 74 lines
Quoted 74 lines
include/hw/pci/pci_bus.h +1 -0
+
    QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */
    QLIST_ENTRY(PCIBus) sibling;/* this will be replaced by qdev later */
 
hw/pci/pci.c +17 -0
index 5d30f9ca60..d5f1c6c421 100644
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent,
    bus->slot_reserved_mask = 0x0;
    bus->address_space_mem = address_space_mem;
    bus->address_space_io = address_space_io;
+    bus->isol_as_mem = NULL;
+    bus->isol_as_io = NULL;
    bus->flags |= PCI_BUS_IS_ROOT;
 
    /* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)
    return pci_get_bus(dev)->address_space_io;
 }
 
+AddressSpace *pci_isol_as_mem(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_mem;
+}
+
+AddressSpace *pci_isol_as_io(PCIDevice *dev)
+{
+    return pci_get_bus(dev)->isol_as_io;
+}
+
 static void pci_device_class_init(ObjectClass *klass, void *data)
 {
    DeviceClass *k = DEVICE_CLASS(klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
 
 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
 {
+    AddressSpace *iommu_as = NULL;
    PCIBus *bus = pci_get_bus(dev);
    PCIBus *iommu_bus = bus;
    uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)
    if (!pci_bus_bypass_iommu(bus) && iommu_bus && iommu_bus->iommu_fn) {
        return iommu_bus->iommu_fn(bus, iommu_bus->iommu_opaque, devfn);
    }
+    iommu_as = pci_isol_as_mem(dev);
+    if (iommu_as) {
+        return iommu_as;
+    }
    return &address_space_memory;
 }
 
hw/pci/pci_bridge.c +5 -0
index da34c8ebcd..98366768d2 100644
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename)
    sec_bus->address_space_io = &br->address_space_io;
    memory_region_init(&br->address_space_io, OBJECT(br), "pci_bridge_io",
                       4 * GiB);
+
+    /* This PCI bridge puts the sec_bus in its parent's address space */
+    sec_bus->isol_as_mem = pci_isol_as_mem(dev);
+    sec_bus->isol_as_io = pci_isol_as_io(dev);
+
    br->windows = pci_bridge_region_init(br);
    QLIST_INIT(&sec_bus->child);
    QLIST_INSERT_HEAD(&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
#49 lore raw

Re: [PATCH v5 03/18] pci: isolated address space for PCI bus

Quoted 55 lines · Jag Raman
Quoted 46 lines · Stefan Hajnoczi
Quoted 41 lines · Jagannathan Raman

Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.

TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.

---

include/hw/pci/pci.h     |  2 ++
include/hw/pci/pci_bus.h | 17 +++++++++++++++++
hw/pci/pci.c             | 17 +++++++++++++++++
hw/pci/pci_bridge.c      |  5 +++++
4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
index 023abc0f79..9bb4472abc 100644
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);
 int pci_device_load(PCIDevice *s, QEMUFile *f);
 MemoryRegion *pci_address_space(PCIDevice *dev);
 MemoryRegion *pci_address_space_io(PCIDevice *dev);
+AddressSpace *pci_isol_as_mem(PCIDevice *dev);
+AddressSpace *pci_isol_as_io(PCIDevice *dev);
 
 /*
 * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +1 -0
index 347440d42c..d78258e79e 100644
@@ -39,9 +39,26 @@ struct PCIBus {
    void *irq_opaque;
    PCIDevice *devices[PCI_SLOT_MAX * PCI_FUNC_MAX];
    PCIDevice *parent_dev;
+
    MemoryRegion *address_space_mem;
Quoted 8 lines

include/hw/pci/pci_bus.h struct PCIBus { · line 44

Quoted 1 line
include/hw/pci/pci_bus.h +0 -0
    MemoryRegion *address_space_io;

This seems like a good point to rename address_space_mem, address_space_io, as well as PCIIORegion->address_space since they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr and PCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.

Quoted 3 lines

Are you referring to renaming address_space_mem, address_space_io and PCIIORegion->address_space alone? I’m asking because there are many other symbols in the code which are named similarly.

I only see those symbols in hw/pci/pci.c. Which ones were you thinking about?

Quoted 22 lines
Quoted 18 lines
Quoted 7 lines
include/hw/pci/pci_bus.h +5 -0
+    /**
+     * Isolated address spaces - these allow the PCI bus to be part
+     * of an isolated address space as opposed to the global
+     * address_space_memory & address_space_io. This allows the
+     * bus to be attached to CPUs from different machines. The

include/hw/pci/pci_bus.h struct PCIBus { · line 60

Quoted 10 lines
include/hw/pci/pci_bus.h +10 -0
+     * following is not used used commonly.
+     *
+     * TYPE_REMOTE_MACHINE allows emulating devices from multiple
+     * VM clients, as such it needs the PCI buses in the same machine
+     * to be part of different CPU address spaces. The following is
+     * useful in that scenario.
+     *
+     */
+    AddressSpace *isol_as_mem;
+    AddressSpace *isol_as_io;
Quoted 3 lines

Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.

Quoted 8 lines

I did start off with using these pointers unconditionally - but adopted an optional isolated address space for the following reasons:
- There is a potential for regression
- CPU address space per bus is not a common scenario. In most case, all PCI buses are attached to CPU sharing the same address space. Therefore, an optional address space made sense for this special scenario

We can also set it unconditionally if you prefer, kindly confirm.

It's a matter of taste. I don't have a strong opinion on it but personally I would try to make it unconditional. I think the risk of regressions is low and the code complexity will be lower than making it a special case. If you wanted to keep it as is, that's fine.

Quoted 10 lines
Quoted 3 lines

isol_as_io isn't used by this patch?

This patch introduces these variables, defines its getters and sets them to NULL in places where new PCI buses are presently created. The following patch creates a separate isolated address space:
[PATCH v5 04/18] pci: create and free isolated PCI buses

I could merge these patches if you prefer.

The only place I saw that reads isol_as_io is "[PATCH v5 15/18]
vfio-user: handle PCI BAR accesses", but that's for PCI I/O Space accesses. Did I miss where I/O Space BARs are mapped into isol_as_io?

Stefan

#50 lore raw

PATCH v5 4/18 pci: create and free isolated PCI buses

In reply to Jagannathan Raman

Adds pci_isol_bus_new() and pci_isol_bus_free() functions to manage creation and destruction of isolated PCI buses. Also adds qdev_get_bus and qdev_put_bus callbacks to allow the choice of parent bus.

---
 include/hw/pci/pci.h   |   4 +
 include/hw/qdev-core.h |  16 ++++
 hw/pci/pci.c           | 169 +++++++++++++++++++++++++++++++++++++++++
 softmmu/qdev-monitor.c |  39 +++++++++-
 4 files changed, 225 insertions(+), 3 deletions(-)
include/hw/pci/pci.h +4 -0
index 9bb4472abc..8c18f10d9d 100644
@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,
 
 PCIDevice *pci_vga_init(PCIBus *bus);
 
+PCIBus *pci_isol_bus_new(BusState *parent_bus, const char *new_bus_type,
+                         Error **errp);
+bool pci_isol_bus_free(PCIBus *pci_bus, Error **errp);
+
 static inline PCIBus *pci_get_bus(const PCIDevice *dev)
 {
     return PCI_BUS(qdev_get_parent_bus(DEVICE(dev)));
include/hw/qdev-core.h +16 -0
index 92c3d65208..eed2983072 100644
@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
 void qdev_machine_creation_done(void);
 bool qdev_machine_modified(void);
 
+/**
+ * Find parent bus - these callbacks are used during device addition
+ * and deletion.
+ *
+ * During addition, if no parent bus is specified in the options,
+ * these callbacks provide a way to figure it out based on the
+ * bus type. If these callbacks are not defined, defaults to
+ * finding the parent bus starting from default system bus
+ */
+typedef bool (QDevGetBusFunc)(const char *type, BusState **bus, Error **errp);
+typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
+bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
+                      Error **errp);
+
 /**
  * GpioPolarity: Polarity of a GPIO line
  *
@@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);
 /*** BUS API. ***/
 
 DeviceState *qdev_find_recursive(BusState *bus, const char *id);
+BusState *qbus_find_recursive(BusState *bus, const char *name,
+                              const char *bus_typename);
 
 /* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */
 typedef int (qbus_walkerfn)(BusState *bus, void *opaque);
hw/pci/pci.c +169 -0

175 lines not on this page · raw message

softmmu/qdev-monitor.c +36 -3
index 01f3834db5..7306074019 100644
@@ -64,6 +64,9 @@ typedef struct QDevAlias
 #define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)
 #define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K)
 
+static QDevGetBusFunc *qdev_get_bus;
+static QDevPutBusFunc *qdev_put_bus;
+
 /* Please keep this table sorted by typename. */
 static const QDevAlias qdev_alias_table[] = {
     { "AC97", "ac97" }, /* -soundhw name */
@@ -450,7 +453,7 @@ static inline bool qbus_is_full(BusState *bus)
  * If more than one exists, prefer one that can take another device.
  * Return the bus if found, else %NULL.
  */
-static BusState *qbus_find_recursive(BusState *bus, const char *name,
+BusState *qbus_find_recursive(BusState *bus, const char *name,
                                      const char *bus_typename)
 {
     BusChild *kid;
@@ -608,6 +611,20 @@ const char *qdev_set_id(DeviceState *dev, char *id, Error **errp)
     return prop->name;
 }
 
+
+bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
+                      Error **errp)
+{
+    if (qdev_get_bus || qdev_put_bus) {
+        error_setg(errp, "callbacks already set");
+        return false;
+    }
+
+    qdev_get_bus = get_bus;
+    qdev_put_bus = put_bus;
+    return true;
+}
+
 DeviceState *qdev_device_add_from_qdict(const QDict *opts,
                                         bool from_json, Error **errp)
 {
@@ -642,7 +659,13 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts,
                        driver, object_get_typename(OBJECT(bus)));
             return NULL;
         }
-    } else if (dc->bus_type != NULL) {
+    } else if (dc->bus_type != NULL && qdev_get_bus != NULL) {
+        if (!qdev_get_bus(dc->bus_type, &bus, errp)) {
+            return NULL;
+        }
+    }
+
+    if (!bus && dc->bus_type != NULL) {
         bus = qbus_find_recursive(sysbus_get_default(), NULL, dc->bus_type);
         if (!bus || qbus_is_full(bus)) {
             error_setg(errp, "No '%s' bus found for device '%s'",
@@ -891,10 +914,12 @@ static DeviceState *find_device_state(const char *id, Error **errp)
 
 void qdev_unplug(DeviceState *dev, Error **errp)
 {
+    ERRP_GUARD();
     DeviceClass *dc = DEVICE_GET_CLASS(dev);
     HotplugHandler *hotplug_ctrl;
     HotplugHandlerClass *hdc;
     Error *local_err = NULL;
+    BusState *parent_bus = qdev_get_parent_bus(dev);
 
     if (dev->parent_bus && !qbus_is_hotpluggable(dev->parent_bus)) {
         error_setg(errp, QERR_BUS_NO_HOTPLUG, dev->parent_bus->name);
@@ -930,7 +955,15 @@ void qdev_unplug(DeviceState *dev, Error **errp)
             object_unparent(OBJECT(dev));
         }
     }
-    error_propagate(errp, local_err);
+
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    if (qdev_put_bus) {
+        qdev_put_bus(parent_bus, errp);
+    }
 }
 
 void qmp_device_del(const char *id, Error **errp)
Signature · 1 line
2.20.1
#51 lore raw

Re: [PATCH v5 04/18] pci: create and free isolated PCI buses

Quoted 40 lines · Jagannathan Raman

Adds pci_isol_bus_new() and pci_isol_bus_free() functions to manage creation and destruction of isolated PCI buses. Also adds qdev_get_bus and qdev_put_bus callbacks to allow the choice of parent bus.

---
 include/hw/pci/pci.h   |   4 +
 include/hw/qdev-core.h |  16 ++++
 hw/pci/pci.c           | 169 +++++++++++++++++++++++++++++++++++++++++
 softmmu/qdev-monitor.c |  39 +++++++++-
 4 files changed, 225 insertions(+), 3 deletions(-)
include/hw/pci/pci.h +4 -0
index 9bb4472abc..8c18f10d9d 100644
@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,
 
 PCIDevice *pci_vga_init(PCIBus *bus);
 
+PCIBus *pci_isol_bus_new(BusState *parent_bus, const char *new_bus_type,
+                         Error **errp);
+bool pci_isol_bus_free(PCIBus *pci_bus, Error **errp);
+
 static inline PCIBus *pci_get_bus(const PCIDevice *dev)
 {
     return PCI_BUS(qdev_get_parent_bus(DEVICE(dev)));
include/hw/qdev-core.h +3 -0
index 92c3d65208..eed2983072 100644
@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
 void qdev_machine_creation_done(void);
 bool qdev_machine_modified(void);
 
+/**
+ * Find parent bus - these callbacks are used during device addition
+ * and deletion.

include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 434

Quoted 10 lines
include/hw/qdev-core.h +10 -0
+ *
+ * During addition, if no parent bus is specified in the options,
+ * these callbacks provide a way to figure it out based on the
+ * bus type. If these callbacks are not defined, defaults to
+ * finding the parent bus starting from default system bus
+ */
+typedef bool (QDevGetBusFunc)(const char *type, BusState **bus, Error **errp);
+typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
+bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
+                      Error **errp);

Where is this used, it doesn't seem related to pci_isol_bus_new()?

Quoted 18 lines
include/hw/qdev-core.h +3 -0
+
 /**
  * GpioPolarity: Polarity of a GPIO line
  *
@@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);
 /*** BUS API. ***/
 
 DeviceState *qdev_find_recursive(BusState *bus, const char *id);
+BusState *qbus_find_recursive(BusState *bus, const char *name,
+                              const char *bus_typename);
 
 /* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */
 typedef int (qbus_walkerfn)(BusState *bus, void *opaque);
hw/pci/pci.c +0 -0
index d5f1c6c421..63ec1e47b5 100644

hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 499

Quoted 7 lines
hw/pci/pci.c +4 -0
@@ -493,6 +493,175 @@ void pci_root_bus_cleanup(PCIBus *bus)
     qbus_unrealize(BUS(bus));
 }
 
+static void pci_bus_free_isol_mem(PCIBus *pci_bus)
+{
+    if (pci_bus->address_space_mem) {
+        memory_region_unref(pci_bus->address_space_mem);

memory_region_unref() already does a NULL pointer check so the if statements in this function aren't needed.

Quoted 34 lines
hw/pci/pci.c +34 -0
+        pci_bus->address_space_mem = NULL;
+    }
+
+    if (pci_bus->isol_as_mem) {
+        address_space_destroy(pci_bus->isol_as_mem);
+        pci_bus->isol_as_mem = NULL;
+    }
+
+    if (pci_bus->address_space_io) {
+        memory_region_unref(pci_bus->address_space_io);
+        pci_bus->address_space_io = NULL;
+    }
+
+    if (pci_bus->isol_as_io) {
+        address_space_destroy(pci_bus->isol_as_io);
+        pci_bus->isol_as_io = NULL;
+    }
+}
+
+static void pci_bus_init_isol_mem(PCIBus *pci_bus, uint32_t unique_id)
+{
+    g_autofree char *mem_mr_name = NULL;
+    g_autofree char *mem_as_name = NULL;
+    g_autofree char *io_mr_name = NULL;
+    g_autofree char *io_as_name = NULL;
+
+    if (!pci_bus) {
+        return;
+    }
+
+    mem_mr_name = g_strdup_printf("mem-mr-%u", unique_id);
+    mem_as_name = g_strdup_printf("mem-as-%u", unique_id);
+    io_mr_name = g_strdup_printf("io-mr-%u", unique_id);
+    io_as_name = g_strdup_printf("io-as-%u", unique_id);

hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 543

Quoted 10 lines
hw/pci/pci.c +10 -0
+
+    pci_bus->address_space_mem = g_malloc0(sizeof(MemoryRegion));
+    pci_bus->isol_as_mem = g_malloc0(sizeof(AddressSpace));
+    memory_region_init(pci_bus->address_space_mem, NULL,
+                       mem_mr_name, UINT64_MAX);
+    address_space_init(pci_bus->isol_as_mem,
+                       pci_bus->address_space_mem, mem_as_name);
+
+    pci_bus->address_space_io = g_malloc0(sizeof(MemoryRegion));
+    pci_bus->isol_as_io = g_malloc0(sizeof(AddressSpace));

Where are address_space_mem, isol_as_mem, address_space_io, and isol_as_io freed? I think the unref calls won't free them because the objects were created with object_initialize() instead of object_new().

author
#52 lore raw

Re: [PATCH v5 04/18] pci: create and free isolated PCI buses

Quoted 42 lines · Stefan Hajnoczi
Quoted 37 lines · Jagannathan Raman

Adds pci_isol_bus_new() and pci_isol_bus_free() functions to manage creation and destruction of isolated PCI buses. Also adds qdev_get_bus and qdev_put_bus callbacks to allow the choice of parent bus.

---

include/hw/pci/pci.h   |   4 +
include/hw/qdev-core.h |  16 ++++
hw/pci/pci.c           | 169 +++++++++++++++++++++++++++++++++++++++++
softmmu/qdev-monitor.c |  39 +++++++++-
4 files changed, 225 insertions(+), 3 deletions(-)
include/hw/pci/pci.h +4 -0
index 9bb4472abc..8c18f10d9d 100644
@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,
 
 PCIDevice *pci_vga_init(PCIBus *bus);
 
+PCIBus *pci_isol_bus_new(BusState *parent_bus, const char *new_bus_type,
+                         Error **errp);
+bool pci_isol_bus_free(PCIBus *pci_bus, Error **errp);
+
 static inline PCIBus *pci_get_bus(const PCIDevice *dev)
 {
    return PCI_BUS(qdev_get_parent_bus(DEVICE(dev)));
include/hw/qdev-core.h +0 -0
index 92c3d65208..eed2983072 100644
@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,
 void qdev_machine_creation_done(void);
 bool qdev_machine_modified(void);
 
Quoted 15 lines

include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 434

Quoted 13 lines
include/hw/qdev-core.h +13 -0
+/**
+ * Find parent bus - these callbacks are used during device addition
+ * and deletion.
+ *
+ * During addition, if no parent bus is specified in the options,
+ * these callbacks provide a way to figure it out based on the
+ * bus type. If these callbacks are not defined, defaults to
+ * finding the parent bus starting from default system bus
+ */
+typedef bool (QDevGetBusFunc)(const char *type, BusState **bus, Error **errp);
+typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
+bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
+                      Error **errp);

Where is this used, it doesn't seem related to pci_isol_bus_new()?

Yes, this is no directly related to pci_isol_bus_new() - will move it to a separate patch.

Quoted 22 lines
Quoted 22 lines
include/hw/qdev-core.h +3 -0
+
 /**
 * GpioPolarity: Polarity of a GPIO line
 *
@@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);
 /*** BUS API. ***/
 
 DeviceState *qdev_find_recursive(BusState *bus, const char *id);
+BusState *qbus_find_recursive(BusState *bus, const char *name,
+                              const char *bus_typename);
 
 /* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */
 typedef int (qbus_walkerfn)(BusState *bus, void *opaque);
hw/pci/pci.c +0 -0
index d5f1c6c421..63ec1e47b5 100644
@@ -493,6 +493,175 @@ void pci_root_bus_cleanup(PCIBus *bus)
    qbus_unrealize(BUS(bus));
 }
 
Quoted 7 lines

hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 499

Quoted 4 lines
hw/pci/pci.c +4 -0
+static void pci_bus_free_isol_mem(PCIBus *pci_bus)
+{
+    if (pci_bus->address_space_mem) {
+        memory_region_unref(pci_bus->address_space_mem);

memory_region_unref() already does a NULL pointer check so the if statements in this function aren't needed.

Got it, thank you!

Quoted 41 lines
Quoted 41 lines
hw/pci/pci.c +40 -0
+        pci_bus->address_space_mem = NULL;
+    }
+
+    if (pci_bus->isol_as_mem) {
+        address_space_destroy(pci_bus->isol_as_mem);
+        pci_bus->isol_as_mem = NULL;
+    }
+
+    if (pci_bus->address_space_io) {
+        memory_region_unref(pci_bus->address_space_io);
+        pci_bus->address_space_io = NULL;
+    }
+
+    if (pci_bus->isol_as_io) {
+        address_space_destroy(pci_bus->isol_as_io);
+        pci_bus->isol_as_io = NULL;
+    }
+}
+
+static void pci_bus_init_isol_mem(PCIBus *pci_bus, uint32_t unique_id)
+{
+    g_autofree char *mem_mr_name = NULL;
+    g_autofree char *mem_as_name = NULL;
+    g_autofree char *io_mr_name = NULL;
+    g_autofree char *io_as_name = NULL;
+
+    if (!pci_bus) {
+        return;
+    }
+
+    mem_mr_name = g_strdup_printf("mem-mr-%u", unique_id);
+    mem_as_name = g_strdup_printf("mem-as-%u", unique_id);
+    io_mr_name = g_strdup_printf("io-mr-%u", unique_id);
+    io_as_name = g_strdup_printf("io-as-%u", unique_id);
+
+    pci_bus->address_space_mem = g_malloc0(sizeof(MemoryRegion));
+    pci_bus->isol_as_mem = g_malloc0(sizeof(AddressSpace));
+    memory_region_init(pci_bus->address_space_mem, NULL,
+                       mem_mr_name, UINT64_MAX);
+    address_space_init(pci_bus->isol_as_mem,
Quoted 8 lines

hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 543

Quoted 4 lines
hw/pci/pci.c +4 -0
+                       pci_bus->address_space_mem, mem_as_name);
+
+    pci_bus->address_space_io = g_malloc0(sizeof(MemoryRegion));
+    pci_bus->isol_as_io = g_malloc0(sizeof(AddressSpace));

Where are address_space_mem, isol_as_mem, address_space_io, and isol_as_io freed? I think the unref calls won't free them because the objects were created with object_initialize() instead of object_new().

Ah OK got it, thank you! Will fix it.

I think we could set the owner of the the memory regions to the PCI bus, as opposed to NULL. We could also add an ‘instance_finalize’ function to PCI bus which would free them.

--
Jag

#53 lore raw

PATCH v5 5/18 qdev: unplug blocker for devices

In reply to Jagannathan Raman
---
 include/hw/qdev-core.h |  5 +++++
 softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
index eed2983072..67df5e0081 100644
@@ -193,6 +193,7 @@ struct DeviceState {
     int instance_id_alias;
     int alias_required_for_version;
     ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
 bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
                       Error **errp);
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp);
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
  * GpioPolarity: Polarity of a GPIO line
  *
softmmu/qdev-monitor.c +35 -0
index 7306074019..1a169f89a2 100644
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)
             return;
         }
 
+        if (qdev_unplug_blocked(dev, errp)) {
+            return;
+        }
+
         qdev_unplug(dev, errp);
     }
 }
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (!migration_is_idle()) {
+        error_setg(errp, "migration is in progress");
+        return -EBUSY;
+    }
+
+    dev->unplug_blockers = g_slist_prepend(dev->unplug_blockers, reason);
+
+    return 0;
+}
+
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason)
+{
+    dev->unplug_blockers = g_slist_remove(dev->unplug_blockers, reason);
+}
+
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (dev->unplug_blockers) {
+        error_propagate(errp, error_copy(dev->unplug_blockers->data));
+        return true;
+    }
+
+    return false;
+}
+
 void hmp_device_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
Signature · 1 line
2.20.1
#54 lore raw

Re: [PATCH v5 05/18] qdev: unplug blocker for devices

Quoted 45 lines · Jagannathan Raman
---
 include/hw/qdev-core.h |  5 +++++
 softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
index eed2983072..67df5e0081 100644
@@ -193,6 +193,7 @@ struct DeviceState {
     int instance_id_alias;
     int alias_required_for_version;
     ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
 bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
                       Error **errp);
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp);
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
  * GpioPolarity: Polarity of a GPIO line
  *
softmmu/qdev-monitor.c +4 -0
index 7306074019..1a169f89a2 100644
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)
             return;
         }
 
+        if (qdev_unplug_blocked(dev, errp)) {
+            return;
+        }
+
         qdev_unplug(dev, errp);
     }

softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996

Quoted 10 lines
softmmu/qdev-monitor.c +8 -0
 }
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (!migration_is_idle()) {
+        error_setg(errp, "migration is in progress");
+        return -EBUSY;
+    }

Why can this function not be called during migration?

Quoted 29 lines
softmmu/qdev-monitor.c +23 -0
+
+    dev->unplug_blockers = g_slist_prepend(dev->unplug_blockers, reason);
+
+    return 0;
+}
+
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason)
+{
+    dev->unplug_blockers = g_slist_remove(dev->unplug_blockers, reason);
+}
+
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (dev->unplug_blockers) {
+        error_propagate(errp, error_copy(dev->unplug_blockers->data));
+        return true;
+    }
+
+    return false;
+}
+
 void hmp_device_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
Signature · 1 line
2.20.1
author
#55 lore raw

Re: [PATCH v5 05/18] qdev: unplug blocker for devices

Quoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman

---
include/hw/qdev-core.h | 5 +++++ softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++

2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
index eed2983072..67df5e0081 100644
@@ -193,6 +193,7 @@ struct DeviceState {
    int instance_id_alias;
    int alias_required_for_version;
    ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
 bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
                      Error **errp);
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp);
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
 * GpioPolarity: Polarity of a GPIO line
 *
softmmu/qdev-monitor.c +4 -0
index 7306074019..1a169f89a2 100644
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)
            return;
        }
 
+        if (qdev_unplug_blocked(dev, errp)) {
+            return;
+        }
+
        qdev_unplug(dev, errp);
    }
 }
 
Quoted 10 lines

softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996

Quoted 8 lines
softmmu/qdev-monitor.c +8 -0
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (!migration_is_idle()) {
+        error_setg(errp, "migration is in progress");
+        return -EBUSY;
+    }

Why can this function not be called during migration?

Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.

I did weigh the following reasons against adding this check:
- unplug_blockers is not migrated to the destination anyway, so it doesn’t matter if it changes after migration starts
- whichever code/object that needs to add the blocker could add it at the destination if needed

However, unlike qmp_device_add(), qmp_object_add() doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.

Would you prefer to throw a warning, or fully remove this test?

--
Jag

Quoted 30 lines
Quoted 30 lines
softmmu/qdev-monitor.c +23 -0
+
+    dev->unplug_blockers = g_slist_prepend(dev->unplug_blockers, reason);
+
+    return 0;
+}
+
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason)
+{
+    dev->unplug_blockers = g_slist_remove(dev->unplug_blockers, reason);
+}
+
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (dev->unplug_blockers) {
+        error_propagate(errp, error_copy(dev->unplug_blockers->data));
+        return true;
+    }
+
+    return false;
+}
+
 void hmp_device_add(Monitor *mon, const QDict *qdict)
 {
    Error *err = NULL;
Signature · 1 line
2.20.1
#56 lore raw

Re: [PATCH v5 05/18] qdev: unplug blocker for devices

Quoted 63 lines · Jag Raman
Quoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman

---
include/hw/qdev-core.h | 5 +++++ softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++

2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
index eed2983072..67df5e0081 100644
@@ -193,6 +193,7 @@ struct DeviceState {
    int instance_id_alias;
    int alias_required_for_version;
    ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
 bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
                      Error **errp);
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp);
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
 * GpioPolarity: Polarity of a GPIO line
 *
softmmu/qdev-monitor.c +4 -0
index 7306074019..1a169f89a2 100644
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)
            return;
        }
 
+        if (qdev_unplug_blocked(dev, errp)) {
+            return;
+        }
+
        qdev_unplug(dev, errp);
    }
 }
 
Quoted 10 lines

softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996

Quoted 8 lines
softmmu/qdev-monitor.c +8 -0
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (!migration_is_idle()) {
+        error_setg(errp, "migration is in progress");
+        return -EBUSY;
+    }

Why can this function not be called during migration?

Quoted 6 lines

Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.

I did weigh the following reasons against adding this check:
- unplug_blockers is not migrated to the destination anyway, so it doesn’t matter if it changes after migration starts

Yes.

Quoted 2 lines

- whichever code/object that needs to add the blocker could add it at the destination if needed

Yes.

Quoted 5 lines

However, unlike qmp_device_add(), qmp_object_add() doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.

Would you prefer to throw a warning, or fully remove this test?

Adding an unplug blocker during migration seems safe to me. I would remove this test.

Stefan

author
#57 lore raw

Re: [PATCH v5 05/18] qdev: unplug blocker for devices

Quoted 82 lines · Stefan Hajnoczi
Quoted 63 lines · Jag Raman
Quoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman

---
include/hw/qdev-core.h | 5 +++++ softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++

2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
index eed2983072..67df5e0081 100644
@@ -193,6 +193,7 @@ struct DeviceState {
   int instance_id_alias;
   int alias_required_for_version;
   ResettableState reset;
+    GSList *unplug_blockers;
 };
 
 struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);
 bool qdev_set_bus_cbs(QDevGetBusFunc *get_bus, QDevPutBusFunc *put_bus,
                     Error **errp);
 
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp);
+void qdev_del_unplug_blocker(DeviceState *dev, Error *reason);
+bool qdev_unplug_blocked(DeviceState *dev, Error **errp);
+
 /**
 * GpioPolarity: Polarity of a GPIO line
 *
softmmu/qdev-monitor.c +4 -0
index 7306074019..1a169f89a2 100644
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)
           return;
       }
 
+        if (qdev_unplug_blocked(dev, errp)) {
+            return;
+        }
+
       qdev_unplug(dev, errp);
   }
 }
 
Quoted 10 lines

softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996

Quoted 8 lines
softmmu/qdev-monitor.c +8 -0
+int qdev_add_unplug_blocker(DeviceState *dev, Error *reason, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (!migration_is_idle()) {
+        error_setg(errp, "migration is in progress");
+        return -EBUSY;
+    }

Why can this function not be called during migration?

Quoted 6 lines

Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.

I did weigh the following reasons against adding this check:
- unplug_blockers is not migrated to the destination anyway, so it doesn’t matter if it changes after migration starts

Yes.

Quoted 2 lines

- whichever code/object that needs to add the blocker could add it at the destination if needed

Yes.

Quoted 8 lines
Quoted 5 lines

However, unlike qmp_device_add(), qmp_object_add() doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.

Would you prefer to throw a warning, or fully remove this test?

Adding an unplug blocker during migration seems safe to me. I would remove this test.

OK, will do.

Thank you!
--
Jag

Quoted 2 lines

Stefan

#58 lore raw

PATCH v5 6/18 vfio-user: add HotplugHandler for remote machine

In reply to Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

---
 hw/remote/machine.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
hw/remote/machine.c +29 -0
index 952105eab5..220ff01aa9 100644
@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)
 
     pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+}
+
+static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
+                                       DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hotplug");
+    }
+}
+
+static void remote_machine_unplug_cb(HotplugHandler *hotplug_dev,
+                                     DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hot-unplug");
+        return;
+    }
+
+    qdev_unrealize(dev);
 }
 
 static void remote_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
     mc->init = remote_machine_init;
     mc->desc = "Experimental remote machine";
+
+    hc->pre_plug = remote_machine_pre_plug_cb;
+    hc->unplug = remote_machine_unplug_cb;
 }
 
 static const TypeInfo remote_machine = {
@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = {
     .parent = TYPE_MACHINE,
     .instance_size = sizeof(RemoteMachineState),
     .class_init = remote_machine_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
 };
 
 static void remote_machine_register_types(void)
Signature · 1 line
2.20.1
#59 lore raw

Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine

Quoted 8 lines · Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

---
 hw/remote/machine.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

Why is this code necessary? I expected the default hotplug behavior to pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.

Quoted 59 lines
hw/remote/machine.c +29 -0
index 952105eab5..220ff01aa9 100644
@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)
 
     pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+}
+
+static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
+                                       DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hotplug");
+    }
+}
+
+static void remote_machine_unplug_cb(HotplugHandler *hotplug_dev,
+                                     DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hot-unplug");
+        return;
+    }
+
+    qdev_unrealize(dev);
 }
 
 static void remote_machine_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
     mc->init = remote_machine_init;
     mc->desc = "Experimental remote machine";
+
+    hc->pre_plug = remote_machine_pre_plug_cb;
+    hc->unplug = remote_machine_unplug_cb;
 }
 
 static const TypeInfo remote_machine = {
@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = {
     .parent = TYPE_MACHINE,
     .instance_size = sizeof(RemoteMachineState),
     .class_init = remote_machine_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
 };
 
 static void remote_machine_register_types(void)
Signature · 1 line
2.20.1
author
#60 lore raw

Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine

Quoted 7 lines · Stefan Hajnoczi
Quoted 2 lines · Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

Quoted 8 lines
Quoted 6 lines

---
hw/remote/machine.c | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)

Why is this code necessary? I expected the default hotplug behavior to

I just discovered that TYPE_REMOTE_MACHINE wasn't setting up a hotplug handler for the root PCI bus.

Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.

We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.

Quoted 3 lines

pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.

OK, that makes sense. We don’t need to test the device type during plug and unplug.

Therefore, I don’t think we need a callback for the plug operation. We could set HotplugHandlerClass->unplug callback to the default qdev_simple_device_unplug_cb() callback.

—
Jag

Quoted 60 lines
Quoted 60 lines
hw/remote/machine.c +29 -0
index 952105eab5..220ff01aa9 100644
@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)
 
    pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+}
+
+static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
+                                       DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hotplug");
+    }
+}
+
+static void remote_machine_unplug_cb(HotplugHandler *hotplug_dev,
+                                     DeviceState *dev, Error **errp)
+{
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "Only allowing PCI hot-unplug");
+        return;
+    }
+
+    qdev_unrealize(dev);
 }
 
 static void remote_machine_class_init(ObjectClass *oc, void *data)
 {
    MachineClass *mc = MACHINE_CLASS(oc);
+    HotplugHandlerClass *hc = HOTPLUG_HANDLER_CLASS(oc);
 
    mc->init = remote_machine_init;
    mc->desc = "Experimental remote machine";
+
+    hc->pre_plug = remote_machine_pre_plug_cb;
+    hc->unplug = remote_machine_unplug_cb;
 }
 
 static const TypeInfo remote_machine = {
@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = {
    .parent = TYPE_MACHINE,
    .instance_size = sizeof(RemoteMachineState),
    .class_init = remote_machine_class_init,
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_HOTPLUG_HANDLER },
+        { }
+    }
 };
 
 static void remote_machine_register_types(void)
Signature · 1 line
2.20.1
#61 lore raw

Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine

Quoted 14 lines · Jag Raman
Quoted 14 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

---
hw/remote/machine.c | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)
Quoted 7 lines
Quoted 1 line

Why is this code necessary? I expected the default hotplug behavior to

I just discovered that TYPE_REMOTE_MACHINE wasn't setting up a hotplug handler for the root PCI bus.

Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.

Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.

Quoted 8 lines

We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.

Quoted 3 lines

pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.

Quoted 6 lines

OK, that makes sense. We don’t need to test the device type during plug and unplug.

Therefore, I don’t think we need a callback for the plug operation. We could set HotplugHandlerClass->unplug callback to the default qdev_simple_device_unplug_cb() callback.

Great!

Stefan

author
#62 lore raw

Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine

Quoted 21 lines · Stefan Hajnoczi
Quoted 16 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

---
hw/remote/machine.c | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)

Why is this code necessary? I expected the default hotplug behavior to

Quoted 8 lines
Quoted 5 lines

I just discovered that TYPE_REMOTE_MACHINE wasn't setting up a hotplug handler for the root PCI bus.

Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.

Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.

I was referring to the root PCI bus, other buses in Q35 probably support hotplug. Please see error message below:

QEMU 6.2.50 monitor - type 'help' for more information (qemu) device_add lsi53c895a,id=lsi2
Error: Bus 'pcie.0' does not support hotplugging

--
Jag

Quoted 19 lines
Quoted 9 lines

We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.

Quoted 3 lines

pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.

Quoted 6 lines

OK, that makes sense. We don’t need to test the device type during plug and unplug.

Therefore, I don’t think we need a callback for the plug operation. We could set HotplugHandlerClass->unplug callback to the default qdev_simple_device_unplug_cb() callback.

Great!

Stefan

#63 lore raw

Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine

Quoted 30 lines · Jag Raman
Quoted 21 lines · Stefan Hajnoczi
Quoted 16 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman

Allow hotplugging of PCI(e) devices to remote machine

---
hw/remote/machine.c | 29 +++++++++++++++++++++++++++++

1 file changed, 29 insertions(+)

Why is this code necessary? I expected the default hotplug behavior to

Quoted 8 lines
Quoted 5 lines

I just discovered that TYPE_REMOTE_MACHINE wasn't setting up a hotplug handler for the root PCI bus.

Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.

Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.

Quoted 6 lines

I was referring to the root PCI bus, other buses in Q35 probably support hotplug. Please see error message below:

QEMU 6.2.50 monitor - type 'help' for more information (qemu) device_add lsi53c895a,id=lsi2
Error: Bus 'pcie.0' does not support hotplugging

Yes, I think that's because it's PCIe and not PCI.

Stefan

#64 lore raw

PATCH v5 7/18 vfio-user: set qdev bus callbacks for remote machine

In reply to Jagannathan Raman
---
 hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
hw/remote/machine.c +57 -0
index 220ff01aa9..221a8430c1 100644
@@ -22,6 +22,60 @@
 #include "hw/pci/pci_host.h"
 #include "hw/remote/iohub.h"
 
+static bool remote_machine_get_bus(const char *type, BusState **bus,
+                                   Error **errp)
+{
+    ERRP_GUARD();
+    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
+    BusState *root_bus = NULL;
+    PCIBus *new_pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
+    if (!root_bus) {
+        error_setg(errp, "Unable to find root PCI device");
+        return false;
+    }
+
+    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
+    if (!new_pci_bus) {
+        return false;
+    }
+
+    *bus = BUS(new_pci_bus);
+
+    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    return true;
+}
+
+static bool remote_machine_put_bus(BusState *bus, Error **errp)
+{
+    PCIBus *pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    pci_bus = PCI_BUS(bus);
+
+    return pci_isol_bus_free(pci_bus, errp);
+}
+
 static void remote_machine_init(MachineState *machine)
 {
     MemoryRegion *system_memory, *system_io, *pci_memory;
@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 
     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+
+    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
+                     &error_fatal);
 }
 
 static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.20.1
#65 lore raw

Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

Quoted 40 lines · Jagannathan Raman
---
 hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
hw/remote/machine.c +25 -0
index 220ff01aa9..221a8430c1 100644
@@ -22,6 +22,60 @@
 #include "hw/pci/pci_host.h"
 #include "hw/remote/iohub.h"
 
+static bool remote_machine_get_bus(const char *type, BusState **bus,
+                                   Error **errp)
+{
+    ERRP_GUARD();
+    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
+    BusState *root_bus = NULL;
+    PCIBus *new_pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
+    if (!root_bus) {
+        error_setg(errp, "Unable to find root PCI device");
+        return false;
+    }
+
+    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
+    if (!new_pci_bus) {

hw/remote/machine.c line 59

Quoted 10 lines
hw/remote/machine.c +10 -0
+        return false;
+    }
+
+    *bus = BUS(new_pci_bus);
+
+    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    return true;
+}

Can the user create the same PCI bus via QMP commands? If so, then this is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP device_add?

I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.

Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev

Unplug would work in the reverse order.

It may be more convenient to automatically create a PCIBus when the PCIDevice is hotplugged, but this kind of magic also has drawbacks (hidden devices, naming collisions, etc).

Quoted 35 lines
hw/remote/machine.c +22 -0
+
+static bool remote_machine_put_bus(BusState *bus, Error **errp)
+{
+    PCIBus *pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    pci_bus = PCI_BUS(bus);
+
+    return pci_isol_bus_free(pci_bus, errp);
+}
+
 static void remote_machine_init(MachineState *machine)
 {
     MemoryRegion *system_memory, *system_io, *pci_memory;
@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 
     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+
+    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
+                     &error_fatal);
 }
 
 static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.20.1
author
#66 lore raw

Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

Quoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman

---
hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 57 insertions(+)
hw/remote/machine.c +29 -0
index 220ff01aa9..221a8430c1 100644
@@ -22,6 +22,60 @@
 #include "hw/pci/pci_host.h"
 #include "hw/remote/iohub.h"
 
+static bool remote_machine_get_bus(const char *type, BusState **bus,
+                                   Error **errp)
+{
+    ERRP_GUARD();
+    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
+    BusState *root_bus = NULL;
+    PCIBus *new_pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
+    if (!root_bus) {
+        error_setg(errp, "Unable to find root PCI device");
+        return false;
+    }
+
+    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
+    if (!new_pci_bus) {
+        return false;
+    }
+
+    *bus = BUS(new_pci_bus);
Quoted 8 lines

hw/remote/machine.c line 59

Quoted 6 lines
hw/remote/machine.c +6 -0
+
+    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    return true;
+}

Can the user create the same PCI bus via QMP commands? If so, then this

I think there is a way we could achieve it.

When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices like TYPE_PCI_BRIDGE which create a secondary PCI bus.

Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.

Quoted 9 lines

is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP device_add?

I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.

Quoted 6 lines

Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev

We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.

Quoted 6 lines

Unplug would work in the reverse order.

It may be more convenient to automatically create a PCIBus when the PCIDevice is hotplugged, but this kind of magic also has drawbacks (hidden devices, naming collisions, etc).

OK, makes sense.

--
Jag

Quoted 36 lines
Quoted 36 lines
hw/remote/machine.c +22 -0
+
+static bool remote_machine_put_bus(BusState *bus, Error **errp)
+{
+    PCIBus *pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    pci_bus = PCI_BUS(bus);
+
+    return pci_isol_bus_free(pci_bus, errp);
+}
+
 static void remote_machine_init(MachineState *machine)
 {
    MemoryRegion *system_memory, *system_io, *pci_memory;
@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 
    qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
+
+    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
+                     &error_fatal);
 }
 
 static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.20.1
#67 lore raw

Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

Quoted 60 lines · Jag Raman
Quoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman

---
hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 57 insertions(+)
hw/remote/machine.c +29 -0
index 220ff01aa9..221a8430c1 100644
@@ -22,6 +22,60 @@
 #include "hw/pci/pci_host.h"
 #include "hw/remote/iohub.h"
 
+static bool remote_machine_get_bus(const char *type, BusState **bus,
+                                   Error **errp)
+{
+    ERRP_GUARD();
+    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
+    BusState *root_bus = NULL;
+    PCIBus *new_pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
+    if (!root_bus) {
+        error_setg(errp, "Unable to find root PCI device");
+        return false;
+    }
+
+    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
+    if (!new_pci_bus) {
+        return false;
+    }
+
+    *bus = BUS(new_pci_bus);
Quoted 8 lines

hw/remote/machine.c line 59

Quoted 6 lines
hw/remote/machine.c +6 -0
+
+    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    return true;
+}

Can the user create the same PCI bus via QMP commands? If so, then this

I think there is a way we could achieve it.

Quoted 8 lines

When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices like TYPE_PCI_BRIDGE which create a secondary PCI bus.

Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.

Exactly. device_add instantiates DeviceStates, not busses, so there needs to be a parent device like a SCSI controller, a PCI bridge, etc that owns and creates the bus.

Quoted 16 lines
Quoted 9 lines

is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP device_add?

I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.

Quoted 6 lines

Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev

Quoted 2 lines

We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.

I wonder if a new device is needed or whether it's possible to add an isol_as=on|off (default: off) option to an existing PCI bridge/expander? Hopefully most of the code is already there.

Stefan

author
#68 lore raw

Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

Quoted 94 lines · Stefan Hajnoczi
Quoted 60 lines · Jag Raman
Quoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman

---
hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 57 insertions(+)
hw/remote/machine.c +29 -0
index 220ff01aa9..221a8430c1 100644
@@ -22,6 +22,60 @@
 #include "hw/pci/pci_host.h"
 #include "hw/remote/iohub.h"
 
+static bool remote_machine_get_bus(const char *type, BusState **bus,
+                                   Error **errp)
+{
+    ERRP_GUARD();
+    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
+    BusState *root_bus = NULL;
+    PCIBus *new_pci_bus = NULL;
+
+    if (!bus) {
+        error_setg(errp, "Invalid argument");
+        return false;
+    }
+
+    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
+        return true;
+    }
+
+    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
+    if (!root_bus) {
+        error_setg(errp, "Unable to find root PCI device");
+        return false;
+    }
+
+    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
+    if (!new_pci_bus) {
+        return false;
+    }
+
+    *bus = BUS(new_pci_bus);
Quoted 8 lines

hw/remote/machine.c line 59

Quoted 6 lines
hw/remote/machine.c +6 -0
+
+    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
+
+    return true;
+}

Can the user create the same PCI bus via QMP commands? If so, then this

I think there is a way we could achieve it.

Quoted 8 lines

When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices like TYPE_PCI_BRIDGE which create a secondary PCI bus.

Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.

Exactly. device_add instantiates DeviceStates, not busses, so there needs to be a parent device like a SCSI controller, a PCI bridge, etc that owns and creates the bus.

Quoted 16 lines
Quoted 15 lines

is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP device_add?

I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.

Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev

Quoted 6 lines
Quoted 2 lines

We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.

I wonder if a new device is needed or whether it's possible to add an isol_as=on|off (default: off) option to an existing PCI bridge/expander? Hopefully most of the code is already there.

OK, it makes sense to add isol_as=on|off (default: off) option to an existing PCI bridge/expander. Will shortly confirm with you the device that seems interesting.

--
Jag

Quoted 2 lines

Stefan

#69 lore raw

PATCH v5 8/18 vfio-user: build library

In reply to Jagannathan Raman

add the libvfio-user library as a submodule. build it as a cmake subproject.

---
 configure                                  | 19 +++++++++-
 meson.build                                | 44 +++++++++++++++++++++-
 .gitlab-ci.d/buildtest.yml                 |  2 +
 .gitmodules                                |  3 ++
 Kconfig.host                               |  4 ++
 MAINTAINERS                                |  1 +
 hw/remote/Kconfig                          |  4 ++
 hw/remote/meson.build                      |  2 +
 meson_options.txt                          |  2 +
 subprojects/libvfio-user                   |  1 +
 tests/docker/dockerfiles/centos8.docker    |  2 +
 tests/docker/dockerfiles/ubuntu2004.docker |  2 +
 12 files changed, 84 insertions(+), 2 deletions(-)
 create mode 160000 subprojects/libvfio-user
configure +18 -1
index 6a865f8713..c8035de952 100755
@@ -356,6 +356,7 @@ ninja=""
 gio="$default_feature"
 skip_meson=no
 slirp_smbd="$default_feature"
+vfio_user_server="disabled"
 
 # The following Meson options are handled manually (still they
 # are included in the automatically generated help message)
@@ -1172,6 +1173,10 @@ for opt do
   ;;
   --disable-blobs) meson_option_parse --disable-install-blobs ""
   ;;
+  --enable-vfio-user-server) vfio_user_server="enabled"
+  ;;
+  --disable-vfio-user-server) vfio_user_server="disabled"
+  ;;
   --enable-tcmalloc) meson_option_parse --enable-malloc=tcmalloc tcmalloc
   ;;
   --enable-jemalloc) meson_option_parse --enable-malloc=jemalloc jemalloc
@@ -1439,6 +1444,7 @@ cat << EOF
   rng-none        dummy RNG, avoid using /dev/(u)random and getrandom()
   gio             libgio support
   slirp-smbd      use smbd (at path --smbd=*) in slirp networking
+  vfio-user-server    vfio-user server support
 
 NOTE: The object files are built at the place where configure is launched
 EOF
@@ -3121,6 +3127,17 @@ but not implemented on your system"
     fi
 fi
 
+##########################################
+# check for vfio_user_server
+
+case "$vfio_user_server" in
+  auto | enabled )
+    if test "$git_submodules_action" != "ignore"; then
+      git_submodules="${git_submodules} subprojects/libvfio-user"
+    fi
+    ;;
+esac
+
 ##########################################
 # End of CC checks
 # After here, no more $cc or $ld runs
@@ -3811,7 +3828,7 @@ if test "$skip_meson" = no; then
         -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \
         -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \
         -Db_lto=$lto -Dcfi=$cfi -Dtcg=$tcg -Dxen=$xen \
-        -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp \
+        -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp -Dvfio_user_server=$vfio_user_server \
         $(test -n "${LIB_FUZZING_ENGINE+xxx}" && echo "-Dfuzzing_engine=$LIB_FUZZING_ENGINE") \
         $(if test "$default_feature" = no; then echo "-Dauto_features=disabled"; fi) \
         "$@" $cross_arg "$PWD" "$source_path"
meson.build +43 -1

68 lines not on this page · raw message

.gitlab-ci.d/buildtest.yml +2 -0
index 8f2a3c8f5b..07c36fb15d 100644
@@ -42,6 +42,7 @@ build-system-ubuntu:
   variables:
     IMAGE: ubuntu2004
     CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
+                    --enable-vfio-user-server
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -165,6 +166,7 @@ build-system-centos:
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
       --enable-modules --enable-trace-backends=dtrace --enable-docs
+      --enable-vfio-user-server
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
.gitmodules +3 -0
index 84425d87e2..4ae2a165d9 100644
@@ -67,3 +67,6 @@
 [submodule "tests/lcitool/libvirt-ci"]
 	path = tests/lcitool/libvirt-ci
 	url = http://gitlab.com/libvirt/libvirt-ci
+[submodule "subprojects/libvfio-user"]
+	path = subprojects/libvfio-user
+	url = https://github.com/nutanix/libvfio-user.git
Kconfig.host +4 -0
index 60b9c07b5e..f2da8bcf8a 100644
@@ -45,3 +45,7 @@ config MULTIPROCESS_ALLOWED
 config FUZZ
     bool
     select SPARSE_MEM
+
+config VFIO_USER_SERVER_ALLOWED
+    bool
+    imply VFIO_USER_SERVER
MAINTAINERS +1 -0
index 2fd74c4642..8d7bebc74a 100644
@@ -3486,6 +3486,7 @@ F: hw/remote/proxy-memory-listener.c
 F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
+F: subprojects/libvfio-user
 
 EBPF:
 M: Jason Wang <[email protected]>
hw/remote/Kconfig +4 -0
index 08c16e235f..2d6b4f4cf4 100644
@@ -2,3 +2,7 @@ config MULTIPROCESS
     bool
     depends on PCI && PCI_EXPRESS && KVM
     select REMOTE_PCIHOST
+
+config VFIO_USER_SERVER
+    bool
+    depends on MULTIPROCESS
hw/remote/meson.build +2 -0
index e6a5574242..dfea6b533b 100644
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
 
+remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser)
+
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c'))
 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
 
meson_options.txt +2 -0
index 921967eddb..7f02794d4b 100644
@@ -68,6 +68,8 @@ option('multiprocess', type: 'feature', value: 'auto',
        description: 'Out of process device emulation support')
 option('dbus_display', type: 'feature', value: 'auto',
        description: '-display dbus support')
+option('vfio_user_server', type: 'feature', value: 'auto',
+       description: 'vfio-user server support')
 
 option('attr', type : 'feature', value : 'auto',
        description: 'attr/xattr support')
subprojects/libvfio-user new file +1 -0
new file mode 160000
index 0000000000..7056525da5
@@ -0,0 +1 @@
+Subproject commit 7056525da5399d00831e90bed4aedb4b8442c9b2
tests/docker/dockerfiles/centos8.docker +2 -0
index cbb909d02b..f8dff989de 100644
@@ -23,6 +23,7 @@ RUN dnf update -y && \
         capstone-devel \
         ccache \
         clang \
+        cmake \
         ctags \
         cyrus-sasl-devel \
         daxctl-devel \
@@ -45,6 +46,7 @@ RUN dnf update -y && \
         gtk3-devel \
         hostname \
         jemalloc-devel \
+        json-c-devel \
         libaio-devel \
         libasan \
         libattr-devel \
tests/docker/dockerfiles/ubuntu2004.docker +2 -0
index 4e562dfdcd..d16a73dec8 100644
@@ -18,6 +18,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
             ca-certificates \
             ccache \
             clang \
+            cmake \
             dbus \
             debianutils \
             diffutils \
@@ -57,6 +58,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
             libiscsi-dev \
             libjemalloc-dev \
             libjpeg-turbo8-dev \
+            libjson-c-dev \
             liblttng-ust-dev \
             liblzo2-dev \
             libncursesw5-dev \
Signature · 1 line
2.20.1
#70 lore raw

PATCH v5 9/18 vfio-user: define vfio-user-server object

In reply to Jagannathan Raman

Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object

---
 qapi/qom.json             |  20 +++-
 hw/remote/vfio-user-obj.c | 194 ++++++++++++++++++++++++++++++++++++++
 MAINTAINERS               |   1 +
 hw/remote/meson.build     |   1 +
 hw/remote/trace-events    |   3 +
 5 files changed, 217 insertions(+), 2 deletions(-)
 create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
index eeb5395ff3..ff266e4732 100644
@@ -703,6 +703,20 @@
 { 'struct': 'RemoteObjectProperties',
   'data': { 'fd': 'str', 'devid': 'str' } }
 
+##
+# @VfioUserServerProperties:
+#
+# Properties for x-vfio-user-server objects.
+#
+# @socket: socket to be used by the libvfiouser library
+#
+# @device: the id of the device to be emulated at the server
+#
+# Since: 6.3
+##
+{ 'struct': 'VfioUserServerProperties',
+  'data': { 'socket': 'SocketAddress', 'device': 'str' } }
+
 ##
 # @RngProperties:
 #
@@ -842,7 +856,8 @@
     'tls-creds-psk',
     'tls-creds-x509',
     'tls-cipher-suites',
-    { 'name': 'x-remote-object', 'features': [ 'unstable' ] }
+    { 'name': 'x-remote-object', 'features': [ 'unstable' ] },
+    { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] }
   ] }
 
 ##
@@ -905,7 +920,8 @@
       'tls-creds-psk':              'TlsCredsPskProperties',
       'tls-creds-x509':             'TlsCredsX509Properties',
       'tls-cipher-suites':          'TlsCredsProperties',
-      'x-remote-object':            'RemoteObjectProperties'
+      'x-remote-object':            'RemoteObjectProperties',
+      'x-vfio-user-server':         'VfioUserServerProperties'
   } }
 
 ##
hw/remote/vfio-user-obj.c new file +194 -0

194 lines not on this page · raw message

MAINTAINERS +1 -0
index 8d7bebc74a..93bce3fa62 100644
@@ -3487,6 +3487,7 @@ F: include/hw/remote/proxy-memory-listener.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: subprojects/libvfio-user
+F: hw/remote/vfio-user-obj.c
 
 EBPF:
 M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
index dfea6b533b..534ac5df79 100644
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c'))
 remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c'))
+remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c'))
 
 remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser)
 
hw/remote/trace-events +3 -0
index 0b23974f90..7da12f0d96 100644
@@ -2,3 +2,6 @@
 
 mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process"
 mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process"
+
+# vfio-user-obj.c
+vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
2.20.1
Reviewed-by on v5 9/18
#71 lore raw

Re: [PATCH v5 09/18] vfio-user: define vfio-user-server object

Quoted 3 lines · Jagannathan Raman

+/**
+ * VFU_OBJECT_ERROR - reports an error message. If auto_shutdown
+ * is set, it abort the machine on error. Otherwise, it logs an

s/abort/aborts/

#72 lore raw

PATCH v5 10/18 vfio-user: instantiate vfio-user context

In reply to Jagannathan Raman

create a context with the vfio-user library to run a PCI device

---
 hw/remote/vfio-user-obj.c | 78 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
hw/remote/vfio-user-obj.c +78 -0
index 80757b0029..810a7c3943 100644
@@ -41,6 +41,9 @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qemu/notify.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -82,13 +85,23 @@ struct VfuObject {
     char *device;
 
     Error *err;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 
+static void vfu_object_init_ctx(VfuObject *o, Error **errp);
+
 static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
                                   void *opaque, Error **errp)
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        return;
+    }
+
     qapi_free_SocketAddress(o->socket);
 
     o->socket = NULL;
@@ -104,17 +117,68 @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
     }
 
     trace_vfu_prop("socket", o->socket->u.q_unix.path);
+
+    vfu_object_init_ctx(o, errp);
 }
 
 static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        return;
+    }
+
     g_free(o->device);
 
     o->device = g_strdup(str);
 
     trace_vfu_prop("device", str);
+
+    vfu_object_init_ctx(o, errp);
+}
+
+/*
+ * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
+ * properties. It also depends on devices instantiated in QEMU. These
+ * dependencies are not available during the instance_init phase of this
+ * object's life-cycle. As such, the server is initialized after the
+ * machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT
+ * when the machine is setup, and the dependencies are available.
+ */
+static void vfu_object_machine_done(Notifier *notifier, void *data)
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+    Error *err = NULL;
+
+    vfu_object_init_ctx(o, &err);
+
+    if (err) {
+        error_propagate(&error_abort, err);
+    }
+}
+
+static void vfu_object_init_ctx(VfuObject *o, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (o->vfu_ctx || !o->socket || !o->device ||
+            !phase_check(PHASE_MACHINE_READY)) {
+        return;
+    }
+
+    if (o->err) {
+        error_propagate(errp, o->err);
+        o->err = NULL;
+        return;
+    }
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
+        return;
+    }
 }
 
 static void vfu_object_init(Object *obj)
@@ -124,6 +188,11 @@ static void vfu_object_init(Object *obj)
 
     k->nr_devs++;
 
+    if (!phase_check(PHASE_MACHINE_READY)) {
+        o->machine_done.notify = vfu_object_machine_done;
+        qemu_add_machine_init_done_notifier(&o->machine_done);
+    }
+
     if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
         error_setg(&o->err, "vfu: %s only compatible with %s machine",
                    TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
@@ -142,6 +211,10 @@ static void vfu_object_finalize(Object *obj)
 
     o->socket = NULL;
 
+    if (o->vfu_ctx) {
+        vfu_destroy_ctx(o->vfu_ctx);
+    }
+
     g_free(o->device);
 
     o->device = NULL;
@@ -149,6 +222,11 @@ static void vfu_object_finalize(Object *obj)
     if (!k->nr_devs && k->auto_shutdown) {
         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
     }
+
+    if (o->machine_done.notify) {
+        qemu_remove_machine_init_done_notifier(&o->machine_done);
+        o->machine_done.notify = NULL;
+    }
 }
 
 static void vfu_object_class_init(ObjectClass *klass, void *data)
Signature · 1 line
2.20.1
#73 lore raw

Re: [PATCH v5 10/18] vfio-user: instantiate vfio-user context

Quoted 33 lines · Jagannathan Raman

create a context with the vfio-user library to run a PCI device

---
 hw/remote/vfio-user-obj.c | 78 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
hw/remote/vfio-user-obj.c +7 -0
index 80757b0029..810a7c3943 100644
@@ -41,6 +41,9 @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qemu/notify.h"
+#include "sysemu/sysemu.h"
+#include "libvfio-user.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -82,13 +85,23 @@ struct VfuObject {
     char *device;
 
     Error *err;
+
+    Notifier machine_done;
+
+    vfu_ctx_t *vfu_ctx;
 };
 

hw/remote/vfio-user-obj.c struct VfuObject { · line 103

Quoted 10 lines
hw/remote/vfio-user-obj.c +5 -0
+static void vfu_object_init_ctx(VfuObject *o, Error **errp);
+
 static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
                                   void *opaque, Error **errp)
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        return;
+    }

No error?

Quoted 9 lines
hw/remote/vfio-user-obj.c +2 -0
+
     qapi_free_SocketAddress(o->socket);
 
     o->socket = NULL;
@@ -104,17 +117,68 @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
     }
 
     trace_vfu_prop("socket", o->socket->u.q_unix.path);
+

hw/remote/vfio-user-obj.c static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name, · line 130

Quoted 10 lines
hw/remote/vfio-user-obj.c +4 -0
+    vfu_object_init_ctx(o, errp);
 }
 
 static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
 {
     VfuObject *o = VFU_OBJECT(obj);
 
+    if (o->vfu_ctx) {
+        return;
+    }

No error?

Quoted 92 lines
hw/remote/vfio-user-obj.c +60 -0
+
     g_free(o->device);
 
     o->device = g_strdup(str);
 
     trace_vfu_prop("device", str);
+
+    vfu_object_init_ctx(o, errp);
+}
+
+/*
+ * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
+ * properties. It also depends on devices instantiated in QEMU. These
+ * dependencies are not available during the instance_init phase of this
+ * object's life-cycle. As such, the server is initialized after the
+ * machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT
+ * when the machine is setup, and the dependencies are available.
+ */
+static void vfu_object_machine_done(Notifier *notifier, void *data)
+{
+    VfuObject *o = container_of(notifier, VfuObject, machine_done);
+    Error *err = NULL;
+
+    vfu_object_init_ctx(o, &err);
+
+    if (err) {
+        error_propagate(&error_abort, err);
+    }
+}
+
+static void vfu_object_init_ctx(VfuObject *o, Error **errp)
+{
+    ERRP_GUARD();
+
+    if (o->vfu_ctx || !o->socket || !o->device ||
+            !phase_check(PHASE_MACHINE_READY)) {
+        return;
+    }
+
+    if (o->err) {
+        error_propagate(errp, o->err);
+        o->err = NULL;
+        return;
+    }
+
+    o->vfu_ctx = vfu_create_ctx(VFU_TRANS_SOCK, o->socket->u.q_unix.path, 0,
+                                o, VFU_DEV_TYPE_PCI);
+    if (o->vfu_ctx == NULL) {
+        error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
+        return;
+    }
 }
 
 static void vfu_object_init(Object *obj)
@@ -124,6 +188,11 @@ static void vfu_object_init(Object *obj)
 
     k->nr_devs++;
 
+    if (!phase_check(PHASE_MACHINE_READY)) {
+        o->machine_done.notify = vfu_object_machine_done;
+        qemu_add_machine_init_done_notifier(&o->machine_done);
+    }
+
     if (!object_dynamic_cast(OBJECT(current_machine), TYPE_REMOTE_MACHINE)) {
         error_setg(&o->err, "vfu: %s only compatible with %s machine",
                    TYPE_VFU_OBJECT, TYPE_REMOTE_MACHINE);
@@ -142,6 +211,10 @@ static void vfu_object_finalize(Object *obj)
 
     o->socket = NULL;
 
+    if (o->vfu_ctx) {
+        vfu_destroy_ctx(o->vfu_ctx);
+    }
+
     g_free(o->device);
 
     o->device = NULL;
@@ -149,6 +222,11 @@ static void vfu_object_finalize(Object *obj)
     if (!k->nr_devs && k->auto_shutdown) {
         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
     }
+
+    if (o->machine_done.notify) {
+        qemu_remove_machine_init_done_notifier(&o->machine_done);
+        o->machine_done.notify = NULL;
+    }
 }
 
 static void vfu_object_class_init(ObjectClass *klass, void *data)
Signature · 1 line
2.20.1
#74 lore raw

PATCH v5 11/18 vfio-user: find and init PCI device

In reply to Jagannathan Raman

Find the PCI device with specified id. Initialize the device context with the QEMU PCI device

---
 hw/remote/vfio-user-obj.c | 60 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +60 -0
index 810a7c3943..10db78eb8d 100644
@@ -44,6 +44,8 @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -89,6 +91,10 @@ struct VfuObject {
     Notifier machine_done;
 
     vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
+
+    Error *unplug_blocker;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 {
     ERRP_GUARD();
+    DeviceState *dev = NULL;
+    vfu_pci_type_t pci_type = VFU_PCI_TYPE_CONVENTIONAL;
+    int ret;
 
     if (o->vfu_ctx || !o->socket || !o->device ||
             !phase_check(PHASE_MACHINE_READY)) {
@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->device);
+    if (dev == NULL) {
+        error_setg(errp, "vfu: Device %s not found", o->device);
+        goto fail;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "vfu: %s not a PCI device", o->device);
+        goto fail;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    if (pci_is_express(o->pci_dev)) {
+        pci_type = VFU_PCI_TYPE_EXPRESS;
+    }
+
+    ret = vfu_pci_init(o->vfu_ctx, pci_type, PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to attach PCI device %s to context - %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
+    error_setg(&o->unplug_blocker, "%s is in use", o->device);
+    qdev_add_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker, errp);
+    if (*errp) {
+        goto fail;
+    }
+
+    return;
+
+fail:
+    vfu_destroy_ctx(o->vfu_ctx);
+    if (o->unplug_blocker && o->pci_dev) {
+        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
+        error_free(o->unplug_blocker);
+        o->unplug_blocker = NULL;
+    }
+    o->vfu_ctx = NULL;
+    o->pci_dev = NULL;
 }
 
 static void vfu_object_init(Object *obj)
@@ -219,6 +271,14 @@ static void vfu_object_finalize(Object *obj)
 
     o->device = NULL;
 
+    if (o->unplug_blocker && o->pci_dev) {
+        qdev_del_unplug_blocker(DEVICE(o->pci_dev), o->unplug_blocker);
+        error_free(o->unplug_blocker);
+        o->unplug_blocker = NULL;
+    }
+
+    o->pci_dev = NULL;
+
     if (!k->nr_devs && k->auto_shutdown) {
         qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
     }
Signature · 1 line
2.20.1
Reviewed-by on v5 11/18
#75 lore raw

Re: [PATCH v5 11/18] vfio-user: find and init PCI device

Quoted 65 lines · Jagannathan Raman

Find the PCI device with specified id. Initialize the device context with the QEMU PCI device

---
 hw/remote/vfio-user-obj.c | 60 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +26 -0
index 810a7c3943..10db78eb8d 100644
@@ -44,6 +44,8 @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -89,6 +91,10 @@ struct VfuObject {
     Notifier machine_done;
 
     vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
+
+    Error *unplug_blocker;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 {
     ERRP_GUARD();
+    DeviceState *dev = NULL;
+    vfu_pci_type_t pci_type = VFU_PCI_TYPE_CONVENTIONAL;
+    int ret;
 
     if (o->vfu_ctx || !o->socket || !o->device ||
             !phase_check(PHASE_MACHINE_READY)) {
@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
         return;
     }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->device);
+    if (dev == NULL) {
+        error_setg(errp, "vfu: Device %s not found", o->device);
+        goto fail;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "vfu: %s not a PCI device", o->device);
+        goto fail;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    if (pci_is_express(o->pci_dev)) {
+        pci_type = VFU_PCI_TYPE_EXPRESS;
+    }

hw/remote/vfio-user-obj.c static void vfu_object_init_ctx(VfuObject *o, Error **errp) · line 217

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+
+    ret = vfu_pci_init(o->vfu_ctx, pci_type, PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to attach PCI device %s to context - %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
+    error_setg(&o->unplug_blocker, "%s is in use", o->device);

More detailed error message:
"x-vfio-user-server for %s must be deleted before unplugging"

author
#76 lore raw

Re: [PATCH v5 11/18] vfio-user: find and init PCI device

Quoted 75 lines · Stefan Hajnoczi
Quoted 70 lines · Jagannathan Raman

Find the PCI device with specified id. Initialize the device context with the QEMU PCI device

---
hw/remote/vfio-user-obj.c | 60 +++++++++++++++++++++++++++++++++++++++

1 file changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +31 -0
index 810a7c3943..10db78eb8d 100644
@@ -44,6 +44,8 @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
+#include "hw/qdev-core.h"
+#include "hw/pci/pci.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -89,6 +91,10 @@ struct VfuObject {
    Notifier machine_done;
 
    vfu_ctx_t *vfu_ctx;
+
+    PCIDevice *pci_dev;
+
+    Error *unplug_blocker;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)
 static void vfu_object_init_ctx(VfuObject *o, Error **errp)
 {
    ERRP_GUARD();
+    DeviceState *dev = NULL;
+    vfu_pci_type_t pci_type = VFU_PCI_TYPE_CONVENTIONAL;
+    int ret;
 
    if (o->vfu_ctx || !o->socket || !o->device ||
            !phase_check(PHASE_MACHINE_READY)) {
@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
        error_setg(errp, "vfu: Failed to create context - %s", strerror(errno));
        return;
    }
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->device);
+    if (dev == NULL) {
+        error_setg(errp, "vfu: Device %s not found", o->device);
+        goto fail;
+    }
+
+    if (!object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_setg(errp, "vfu: %s not a PCI device", o->device);
+        goto fail;
+    }
+
+    o->pci_dev = PCI_DEVICE(dev);
+
+    if (pci_is_express(o->pci_dev)) {
+        pci_type = VFU_PCI_TYPE_EXPRESS;
+    }
+
+    ret = vfu_pci_init(o->vfu_ctx, pci_type, PCI_HEADER_TYPE_NORMAL, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to attach PCI device %s to context - %s",
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_init_ctx(VfuObject *o, Error **errp) · line 217

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
+    error_setg(&o->unplug_blocker, "%s is in use", o->device);

More detailed error message:
"x-vfio-user-server for %s must be deleted before unplugging"

Got it, thank you!

--
Jag

Quoted 2 lines
#77 lore raw

PATCH v5 12/18 vfio-user: run vfio-user context

In reply to Jagannathan Raman

Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it

---
 qapi/misc.json            | 23 ++++++++++
 hw/remote/vfio-user-obj.c | 90 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 112 insertions(+), 1 deletion(-)
qapi/misc.json +23 -0
index e8054f415b..f0791d3311 100644
@@ -527,3 +527,26 @@
  'data': { '*option': 'str' },
  'returns': ['CommandLineOptionInfo'],
  'allow-preconfig': true }
+
+##
+# @VFU_CLIENT_HANGUP:
+#
+# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the
+# communication channel
+#
+# @device: ID of attached PCI device
+#
+# @path: path of the socket
+#
+# Since: 6.3
+#
+# Example:
+#
+# <- { "event": "VFU_CLIENT_HANGUP",
+#      "data": { "device": "lsi1",
+#                "path": "/tmp/vfu1-sock" },
+#      "timestamp": { "seconds": 1265044230, "microseconds": 450486 } }
+#
+##
+{ 'event': 'VFU_CLIENT_HANGUP',
+  'data': { 'device': 'str', 'path': 'str' } }
hw/remote/vfio-user-obj.c +89 -1

137 lines not on this page · raw message

Signature · 1 line
2.20.1
#78 lore raw

Re: [PATCH v5 12/18] vfio-user: run vfio-user context

Quoted 20 lines · Jagannathan Raman

Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it

---
 qapi/misc.json            | 23 ++++++++++
 hw/remote/vfio-user-obj.c | 90 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 112 insertions(+), 1 deletion(-)
qapi/misc.json +0 -0
index e8054f415b..f0791d3311 100644
@@ -527,3 +527,26 @@
  'data': { '*option': 'str' },
  'returns': ['CommandLineOptionInfo'],
  'allow-preconfig': true }

qapi/misc.json line 539

Quoted 10 lines
qapi/misc.json +10 -0
+
+##
+# @VFU_CLIENT_HANGUP:
+#
+# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the
+# communication channel
+#
+# @device: ID of attached PCI device
+#
+# @path: path of the socket

This assumes a UNIX domain socket path was given. It doesn't work well with file descriptor passing. The x-vfio-user-server is an object with a unique QEMU Object Model path (the last path component is its id). You can get the id like this:

object_get_canonical_path_component(OBJECT(o))

I suggest dropping @path and including the server object's id instead.

Quoted 59 lines
qapi/misc.json +13 -0
+#
+# Since: 6.3
+#
+# Example:
+#
+# <- { "event": "VFU_CLIENT_HANGUP",
+#      "data": { "device": "lsi1",
+#                "path": "/tmp/vfu1-sock" },
+#      "timestamp": { "seconds": 1265044230, "microseconds": 450486 } }
+#
+##
+{ 'event': 'VFU_CLIENT_HANGUP',
+  'data': { 'device': 'str', 'path': 'str' } }
hw/remote/vfio-user-obj.c +12 -0
index 10db78eb8d..91d49a221f 100644
@@ -27,6 +27,9 @@
  *
  * device - id of a device on the server, a required option. PCI devices
  *          alone are supported presently.
+ *
+ * notes - x-vfio-user-server could block IO and monitor during the
+ *         initialization phase.
  */
 
 #include "qemu/osdep.h"
@@ -41,11 +44,14 @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qapi/qapi-events-misc.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "qemu/timer.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -95,6 +101,8 @@ struct VfuObject {
     PCIDevice *pci_dev;
 
     Error *unplug_blocker;
+
+    int vfu_poll_fd;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -144,6 +152,68 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
     vfu_object_init_ctx(o, errp);
 }
 
+static void vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret = -1;

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 168

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+
+    while (ret != 0) {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
+                continue;
+            } else if (errno == ENOTCONN) {
+                qapi_event_send_vfu_client_hangup(o->device,
+                                                  o->socket->u.q_unix.path);
+                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);

Do we also stop monitoring o->vfu_poll_fd when object-del is used to delete the x-vfio-user-server object?

author
#79 lore raw

Re: [PATCH v5 12/18] vfio-user: run vfio-user context

Quoted 41 lines · Stefan Hajnoczi
Quoted 20 lines · Jagannathan Raman

Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it

---
qapi/misc.json | 23 ++++++++++
hw/remote/vfio-user-obj.c | 90 ++++++++++++++++++++++++++++++++++++++-

2 files changed, 112 insertions(+), 1 deletion(-)
qapi/misc.json +0 -0
index e8054f415b..f0791d3311 100644
@@ -527,3 +527,26 @@
 'data': { '*option': 'str' },
 'returns': ['CommandLineOptionInfo'],
 'allow-preconfig': true }

qapi/misc.json line 539

Quoted 10 lines
qapi/misc.json +10 -0
+
+##
+# @VFU_CLIENT_HANGUP:
+#
+# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the
+# communication channel
+#
+# @device: ID of attached PCI device
+#
+# @path: path of the socket

This assumes a UNIX domain socket path was given. It doesn't work well with file descriptor passing. The x-vfio-user-server is an object with a unique QEMU Object Model path (the last path component is its id). You can get the id like this:

Quoted 1 line

object_get_canonical_path_component(OBJECT(o))

I was also wondering how to get the object ID. Thank you for the pointer!

Quoted 2 lines

I suggest dropping @path and including the server object's id instead.

OK, will do.

Quoted 65 lines
Quoted 65 lines
qapi/misc.json +13 -0
+#
+# Since: 6.3
+#
+# Example:
+#
+# <- { "event": "VFU_CLIENT_HANGUP",
+#      "data": { "device": "lsi1",
+#                "path": "/tmp/vfu1-sock" },
+#      "timestamp": { "seconds": 1265044230, "microseconds": 450486 } }
+#
+##
+{ 'event': 'VFU_CLIENT_HANGUP',
+  'data': { 'device': 'str', 'path': 'str' } }
hw/remote/vfio-user-obj.c +17 -0
index 10db78eb8d..91d49a221f 100644
@@ -27,6 +27,9 @@
 *
 * device - id of a device on the server, a required option. PCI devices
 *          alone are supported presently.
+ *
+ * notes - x-vfio-user-server could block IO and monitor during the
+ *         initialization phase.
 */
 
 #include "qemu/osdep.h"
@@ -41,11 +44,14 @@
 #include "hw/remote/machine.h"
 #include "qapi/error.h"
 #include "qapi/qapi-visit-sockets.h"
+#include "qapi/qapi-events-misc.h"
 #include "qemu/notify.h"
+#include "qemu/thread.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "qemu/timer.h"
 
 #define TYPE_VFU_OBJECT "x-vfio-user-server"
 OBJECT_DECLARE_TYPE(VfuObject, VfuObjectClass, VFU_OBJECT)
@@ -95,6 +101,8 @@ struct VfuObject {
    PCIDevice *pci_dev;
 
    Error *unplug_blocker;
+
+    int vfu_poll_fd;
 };
 
 static void vfu_object_init_ctx(VfuObject *o, Error **errp);
@@ -144,6 +152,68 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)
    vfu_object_init_ctx(o, errp);
 }
 
+static void vfu_object_ctx_run(void *opaque)
+{
+    VfuObject *o = opaque;
+    int ret = -1;
+
+    while (ret != 0) {
+        ret = vfu_run_ctx(o->vfu_ctx);
+        if (ret < 0) {
+            if (errno == EINTR) {
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 168

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+                continue;
+            } else if (errno == ENOTCONN) {
+                qapi_event_send_vfu_client_hangup(o->device,
+                                                  o->socket->u.q_unix.path);
+                qemu_set_fd_handler(o->vfu_poll_fd, NULL, NULL, NULL);

Do we also stop monitoring o->vfu_poll_fd when object-del is used to delete the x-vfio-user-server object?

Yes, we should to stop monitoring the o->vfu_poll_fd during object-del. Will do so.

--
Jag

#80 lore raw

PATCH v5 13/18 vfio-user: handle PCI config space accesses

In reply to Jagannathan Raman

Define and register handlers for PCI config space accesses

---
 hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 47 insertions(+)
hw/remote/vfio-user-obj.c +45 -0
index 91d49a221f..8951617545 100644
@@ -47,6 +47,7 @@
 #include "qapi/qapi-events-misc.h"
 #include "qemu/notify.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 #include "sysemu/sysemu.h"
 #include "libvfio-user.h"
 #include "hw/qdev-core.h"
@@ -214,6 +215,39 @@ retry_attach:
     qemu_set_fd_handler(o->vfu_poll_fd, vfu_object_ctx_run, NULL, o);
 }
 
+static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
+                                     size_t count, loff_t offset,
+                                     const bool is_write)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    uint32_t pci_access_width = sizeof(uint32_t);
+    size_t bytes = count;
+    uint32_t val = 0;
+    char *ptr = buf;
+    int len;
+
+    while (bytes > 0) {
+        len = (bytes > pci_access_width) ? pci_access_width : bytes;
+        if (is_write) {
+            memcpy(&val, ptr, len);
+            pci_host_config_write_common(o->pci_dev, offset,
+                                         pci_config_size(o->pci_dev),
+                                         val, len);
+            trace_vfu_cfg_write(offset, val);
+        } else {
+            val = pci_host_config_read_common(o->pci_dev, offset,
+                                              pci_config_size(o->pci_dev), len);
+            memcpy(ptr, &val, len);
+            trace_vfu_cfg_read(offset, val);
+        }
+        offset += len;
+        ptr += len;
+        bytes -= len;
+    }
+
+    return count;
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -291,6 +325,17 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    ret = vfu_setup_region(o->vfu_ctx, VFU_PCI_DEV_CFG_REGION_IDX,
+                           pci_config_size(o->pci_dev), &vfu_object_cfg_access,
+                           VFU_REGION_FLAG_RW | VFU_REGION_FLAG_ALWAYS_CB,
+                           NULL, 0, -1, 0);
+    if (ret < 0) {
+        error_setg(errp,
+                   "vfu: Failed to setup config space handlers for %s- %s",
+                   o->device, strerror(errno));
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
index 7da12f0d96..2ef7884346 100644
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 
 # vfio-user-obj.c
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
+vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
+vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
2.20.1
Reviewed-by on v5 13/18
#81 lore raw

Re: [PATCH v5 13/18] vfio-user: handle PCI config space accesses

Quoted 9 lines · Jagannathan Raman

Define and register handlers for PCI config space accesses

---
 hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 47 insertions(+)
#82 lore raw

PATCH v5 14/18 vfio-user: handle DMA mappings

In reply to Jagannathan Raman

Define and register callbacks to manage the RAM regions used for device DMA

---
 hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  2 ++
 2 files changed, 52 insertions(+)
hw/remote/vfio-user-obj.c +50 -0
index 8951617545..e690f1eaae 100644
@@ -248,6 +248,49 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,
     return count;
 }
 
+static void dma_register(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    MemoryRegion *subregion = NULL;
+    g_autofree char *name = NULL;
+    struct iovec *iov = &info->iova;
+
+    if (!info->vaddr) {
+        return;
+    }
+
+    name = g_strdup_printf("mem-%s-%"PRIx64"", o->device,
+                           (uint64_t)info->vaddr);
+
+    subregion = g_new0(MemoryRegion, 1);
+
+    memory_region_init_ram_ptr(subregion, NULL, name,
+                               iov->iov_len, info->vaddr);
+
+    memory_region_add_subregion(pci_address_space(o->pci_dev),
+                                (hwaddr)iov->iov_base, subregion);
+
+    trace_vfu_dma_register((uint64_t)iov->iov_base, iov->iov_len);
+}
+
+static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    MemoryRegion *mr = NULL;
+    ram_addr_t offset;
+
+    mr = memory_region_from_host(info->vaddr, &offset);
+    if (!mr) {
+        return;
+    }
+
+    memory_region_del_subregion(pci_address_space(o->pci_dev), mr);
+
+    object_unparent((OBJECT(mr)));
+
+    trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -336,6 +379,13 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    ret = vfu_setup_device_dma(o->vfu_ctx, &dma_register, &dma_unregister);
+    if (ret < 0) {
+        error_setg(errp, "vfu: Failed to setup DMA handlers for %s",
+                   o->device);
+        goto fail;
+    }
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
index 2ef7884346..f945c7e33b 100644
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d,
 vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
 vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
+vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
+vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
2.20.1
#83 lore raw

PATCH v5 15/18 vfio-user: handle PCI BAR accesses

In reply to Jagannathan Raman

Determine the BARs used by the PCI device and register handlers to manage the access to the same.

---
 hw/remote/vfio-user-obj.c | 92 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  3 ++
 2 files changed, 95 insertions(+)
hw/remote/vfio-user-obj.c +92 -0
index e690f1eaae..bf88eac8f1 100644
@@ -291,6 +291,96 @@ static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)
     trace_vfu_dma_unregister((uint64_t)info->iova.iov_base);
 }
 
+static ssize_t vfu_object_bar_rw(PCIDevice *pci_dev, hwaddr addr, size_t count,
+                                 char * const buf, const bool is_write,
+                                 bool is_io)
+{
+    AddressSpace *as = NULL;
+    MemTxResult res;
+
+    if (is_io) {
+        as = pci_isol_as_io(pci_dev);
+        as = as ? as : &address_space_io;
+    } else {
+        as = pci_isol_as_mem(pci_dev);
+        as = as ? as : &address_space_memory;
+    }
+
+    trace_vfu_bar_rw_enter(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    res = address_space_rw(as, addr, MEMTXATTRS_UNSPECIFIED, (void *)buf,
+                           (hwaddr)count, is_write);
+    if (res != MEMTX_OK) {
+        warn_report("vfu: failed to %s 0x%"PRIx64"",
+                    is_write ? "write to" : "read from",
+                    addr);
+        return -1;
+    }
+
+    trace_vfu_bar_rw_exit(is_write ? "Write" : "Read", (uint64_t)addr);
+
+    return count;
+}
+
+/**
+ * VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs.
+ *
+ * To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would
+ * define vfu_object_bar2_handler
+ */
+#define VFU_OBJECT_BAR_HANDLER(BAR_NO)                                         \
+    static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx,        \
+                                        char * const buf, size_t count,        \
+                                        loff_t offset, const bool is_write)    \
+    {                                                                          \
+        VfuObject *o = vfu_get_private(vfu_ctx);                               \
+        PCIDevice *pci_dev = o->pci_dev;                                       \
+        hwaddr addr = (hwaddr)(pci_get_bar_addr(pci_dev, BAR_NO) + offset);    \
+        bool is_io = !!(pci_dev->io_regions[BAR_NO].type &                     \
+                        PCI_BASE_ADDRESS_SPACE);                               \
+                                                                               \
+        return vfu_object_bar_rw(pci_dev, addr, count, buf, is_write, is_io);  \
+    }                                                                          \
+
+VFU_OBJECT_BAR_HANDLER(0)
+VFU_OBJECT_BAR_HANDLER(1)
+VFU_OBJECT_BAR_HANDLER(2)
+VFU_OBJECT_BAR_HANDLER(3)
+VFU_OBJECT_BAR_HANDLER(4)
+VFU_OBJECT_BAR_HANDLER(5)
+
+static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS] = {
+    &vfu_object_bar0_handler,
+    &vfu_object_bar1_handler,
+    &vfu_object_bar2_handler,
+    &vfu_object_bar3_handler,
+    &vfu_object_bar4_handler,
+    &vfu_object_bar5_handler,
+};
+
+/**
+ * vfu_object_register_bars - Identify active BAR regions of pdev and setup
+ *                            callbacks to handle read/write accesses
+ */
+static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev)
+{
+    int i;
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        if (!pdev->io_regions[i].size) {
+            continue;
+        }
+
+        vfu_setup_region(vfu_ctx, VFU_PCI_DEV_BAR0_REGION_IDX + i,
+                         (size_t)pdev->io_regions[i].size,
+                         vfu_object_bar_handlers[i],
+                         VFU_REGION_FLAG_RW, NULL, 0, -1, 0);
+
+        trace_vfu_bar_register(i, pdev->io_regions[i].addr,
+                               pdev->io_regions[i].size);
+    }
+}
+
 /*
  * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device'
  * properties. It also depends on devices instantiated in QEMU. These
@@ -386,6 +476,8 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)
         goto fail;
     }
 
+    vfu_object_register_bars(o->vfu_ctx, o->pci_dev);
+
     ret = vfu_realize_ctx(o->vfu_ctx);
     if (ret < 0) {
         error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +3 -0
index f945c7e33b..847d50d88f 100644
@@ -9,3 +9,6 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x"
 vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
 vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes"
 vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
+vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
+vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
+vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
2.20.1
#84 lore raw

PATCH v5 16/18 vfio-user: handle device interrupts

In reply to Jagannathan Raman

Forward remote device's interrupts to the guest

---
 include/hw/pci/pci.h      |  6 +++
 hw/pci/msi.c              | 13 +++++-
 hw/pci/msix.c             | 12 +++++-
 hw/remote/vfio-user-obj.c | 89 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  1 +
 5 files changed, 119 insertions(+), 2 deletions(-)
include/hw/pci/pci.h +6 -0
index 8c18f10d9d..092334d2af 100644
@@ -128,6 +128,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,
 typedef void PCIMapIORegionFunc(PCIDevice *pci_dev, int region_num,
                                 pcibus_t addr, pcibus_t size, int type);
 typedef void PCIUnregisterFunc(PCIDevice *pci_dev);
+typedef void PCIMSINotify(PCIDevice *pci_dev, unsigned vector);
+typedef void PCIMSIxNotify(PCIDevice *pci_dev, unsigned vector);
 
 typedef struct PCIIORegion {
     pcibus_t addr; /* current PCI mapping address. -1 means not mapped */
@@ -322,6 +324,10 @@ struct PCIDevice {
     /* Space to store MSIX table & pending bit array */
     uint8_t *msix_table;
     uint8_t *msix_pba;
+
+    PCIMSINotify *msi_notify;
+    PCIMSIxNotify *msix_notify;
+
     /* MemoryRegion container for msix exclusive BAR setup */
     MemoryRegion msix_exclusive_bar;
     /* Memory Regions for MSIX table and pending bit entries. */
hw/pci/msi.c +12 -1
index 47d2b0f33c..93f5e400cc 100644
@@ -51,6 +51,8 @@
  */
 bool msi_nonbroken;
 
+static void pci_msi_notify(PCIDevice *dev, unsigned int vector);
+
 /* If we get rid of cap allocator, we won't need this. */
 static inline uint8_t msi_cap_sizeof(uint16_t flags)
 {
@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset,
     dev->msi_cap = config_offset;
     dev->cap_present |= QEMU_PCI_CAP_MSI;
 
+    dev->msi_notify = pci_msi_notify;
+
     pci_set_word(dev->config + msi_flags_off(dev), flags);
     pci_set_word(dev->wmask + msi_flags_off(dev),
                  PCI_MSI_FLAGS_QSIZE | PCI_MSI_FLAGS_ENABLE);
@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)
     return mask & (1U << vector);
 }
 
-void msi_notify(PCIDevice *dev, unsigned int vector)
+static void pci_msi_notify(PCIDevice *dev, unsigned int vector)
 {
     uint16_t flags = pci_get_word(dev->config + msi_flags_off(dev));
     bool msi64bit = flags & PCI_MSI_FLAGS_64BIT;
@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)
     msi_send_message(dev, msg);
 }
 
+void msi_notify(PCIDevice *dev, unsigned int vector)
+{
+    if (dev->msi_notify) {
+        dev->msi_notify(dev, vector);
+    }
+}
+
 void msi_send_message(PCIDevice *dev, MSIMessage msg)
 {
     MemTxAttrs attrs = {};
hw/pci/msix.c +11 -1
index ae9331cd0b..1c71e67f53 100644
@@ -31,6 +31,8 @@
 #define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)
 #define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8)
 
+static void pci_msix_notify(PCIDevice *dev, unsigned vector);
+
 MSIMessage msix_get_message(PCIDevice *dev, unsigned vector)
 {
     uint8_t *table_entry = dev->msix_table + vector * PCI_MSIX_ENTRY_SIZE;
@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries,
     dev->msix_table = g_malloc0(table_size);
     dev->msix_pba = g_malloc0(pba_size);
     dev->msix_entry_used = g_malloc0(nentries * sizeof *dev->msix_entry_used);
+    dev->msix_notify = pci_msix_notify;
 
     msix_mask_all(dev, nentries);
 
@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev)
 }
 
 /* Send an MSI-X message */
-void msix_notify(PCIDevice *dev, unsigned vector)
+static void pci_msix_notify(PCIDevice *dev, unsigned vector)
 {
     MSIMessage msg;
 
@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)
     msi_send_message(dev, msg);
 }
 
+void msix_notify(PCIDevice *dev, unsigned vector)
+{
+    if (dev->msix_notify) {
+        dev->msix_notify(dev, vector);
+    }
+}
+
 void msix_reset(PCIDevice *dev)
 {
     if (!msix_present(dev)) {
hw/remote/vfio-user-obj.c +89 -0

134 lines not on this page · raw message

hw/remote/trace-events +1 -0
index 847d50d88f..c167b3c7a5 100644
@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
 vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""
 vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64""
 vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
Signature · 1 line
2.20.1
Reviewed-by on v5 16/18
#85 lore raw

Re: [PATCH v5 16/18] vfio-user: handle device interrupts

Quoted 5 lines · Jagannathan Raman

Forward remote device's interrupts to the guest

Quoted 7 lines
---
 include/hw/pci/pci.h      |  6 +++
 hw/pci/msi.c              | 13 +++++-
 hw/pci/msix.c             | 12 +++++-
 hw/remote/vfio-user-obj.c | 89 +++++++++++++++++++++++++++++++++++++++
 hw/remote/trace-events    |  1 +
 5 files changed, 119 insertions(+), 2 deletions(-)
#86 lore raw

PATCH v5 17/18 vfio-user: register handlers to facilitate migration

In reply to Jagannathan Raman

Store and load the device's state during migration. use libvfio-user's handlers for this purpose

---
 include/migration/vmstate.h |   2 +
 migration/savevm.h          |   2 +
 hw/remote/vfio-user-obj.c   | 323 ++++++++++++++++++++++++++++++++++++
 migration/savevm.c          |  73 ++++++++
 migration/vmstate.c         |  19 +++
 5 files changed, 419 insertions(+)
include/migration/vmstate.h +2 -0
index 017c03675c..68bea576ea 100644
@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;
 #define VMSTATE_END_OF_LIST()                                         \
     {}
 
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev);
+
 int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                        void *opaque, int version_id);
 int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
migration/savevm.h +2 -0
index 6461342cb4..8007064ff2 100644
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);
 int qemu_load_device_state(QEMUFile *f);
 int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f,
         bool in_postcopy, bool inactivate_disks);
+int qemu_remote_savevm(QEMUFile *f, DeviceState *dev);
+int qemu_remote_loadvm(QEMUFile *f);
 
 #endif
hw/remote/vfio-user-obj.c +323 -0

359 lines not on this page · raw message

migration/savevm.c +73 -0

85 lines not on this page · raw message

migration/vmstate.c +19 -0
index 05f87cdddc..83f8562792 100644
@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)
     return size;
 }
 
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
+    return size;
+}
+
 static void vmstate_handle_alloc(void *ptr, const VMStateField *field,
                                  void *opaque)
 {
Signature · 1 line
2.20.1
#87 lore raw

Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 1 line · Jagannathan Raman

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 18 lines
hw/remote/vfio-user-obj.c +18 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Quoted 7 lines
hw/remote/vfio-user-obj.c +7 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 10 lines
hw/remote/vfio-user-obj.c +10 -0
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Quoted 8 lines
migration/vmstate.c +8 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;

migration/vmstate.c static int vmstate_size(void *opaque, const VMStateField *field) · line 83

Quoted 10 lines
migration/vmstate.c +10 -0
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

author
#88 lore raw

Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 23 lines
hw/remote/vfio-user-obj.c +23 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.

Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 2 lines
hw/remote/vfio-user-obj.c +2 -0
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.

Will add a comment a comment to explain that.

Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.

Quoted 6 lines

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.

I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:

* Does this mean that partial reads are not allowed?

Thanos or John,

Could you please clarify this?

Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:

https://lore.kernel.org/all/[email protected]/

Thank you very much!
--
Jag

#89 lore raw

Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 23 lines
hw/remote/vfio-user-obj.c +23 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.

Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 2 lines
hw/remote/vfio-user-obj.c +2 -0
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.

Will add a comment a comment to explain that.

Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.

Quoted 6 lines

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.

I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:

* Does this mean that partial reads are not allowed?

Thanos or John,

Could you please clarify this?

Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:

Quoted 6 lines

https://lore.kernel.org/all/[email protected]/

libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?

The answer didn't make sense to me:

"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.

In the case of the callbacks, the library translates the BAR access to callbacks."

The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <linux/vfio.h>:

* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Read pending_bytes, indicating the start of a new iteration to get device

  *    data. Repeated read on pending_bytes at this stage should have no side
  *    effects.
  *    If pending_bytes == 0, the user application should not iterate to get data
  *    for that device.
  *    If pending_bytes > 0, perform the following steps.
  * b. Read data_offset, indicating that the vendor driver should make data
  *    available through the data section. The vendor driver should return this
  *    read operation only after data is available from (region + data_offset)
  *    to (region + data_offset + data_size).
  * c. Read data_size, which is the amount of data in bytes available through
  *    the migration region.
  *    Read on data_offset and data_size should return the offset and size of
  *    the current buffer if the user application reads data_offset and
  *    data_size more than once here.

* d. Read data_size bytes of data from (region + data_offset) from the
* migration region.
* e. Process the data.
* f. Read pending_bytes, which indicates that the data from the previous
* iteration has been read. If pending_bytes > 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in _SAVING
* (stop-and-copy) until pending_bytes is 0.

This means you can report pending_bytes > 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.

Stefan

#90 lore raw

RE: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 19 lines

-----Original Message-----
From: Stefan Hajnoczi <[email protected]>
Sent: 28 January 2022 08:29
To: Jag Raman <[email protected]>
Cc: John Levon <[email protected]>; Thanos Makatos <[email protected]>; qemu-devel <[email protected]>; Marc-André Lureau <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Paolo Bonzini <[email protected]>; Beraldo Leal <[email protected]>; Daniel P. Berrangé <[email protected]>; [email protected]; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Juan Quintela <[email protected]>; Dr . David Alan Gilbert <[email protected]>; Elena Ufimtseva <[email protected]>; John Johnson <[email protected]>
Subject: Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 76 lines
Quoted 3 lines · Jag Raman
Quoted 3 lines

On Jan 25, 2022, at 10:48 AM, Stefan Hajnoczi <[email protected]>

Quoted 71 lines · wrote:
Quoted 29 lines · Jagannathan Raman
Quoted 1 line

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 23 lines
hw/remote/vfio-user-obj.c +23 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.

Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 2 lines
hw/remote/vfio-user-obj.c +2 -0
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.

Will add a comment a comment to explain that.

libvfio-user does not automatically increment offset by size each time, since the vfio-user client can re-read the migration data multiple times. In libvfio-user API we state:

Function that is called to read migration data. offset and size can be any subrange on the offset and size previously returned by prepare_data.

Reading the pending_bytes register is what marks the end of the iteration, and this is where you need to decrement vfu_mig_buf_pending.

I'll add more unit tests to libvfio-user to validate this behavior.

Quoted 41 lines
Quoted 41 lines
Quoted 18 lines
Quoted 18 lines
migration/vmstate.c +16 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.

Quoted 6 lines

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.

Quoted 8 lines
Quoted 8 lines

I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:

* Does this mean that partial reads are not allowed?

Thanos or John,

Could you please clarify this?

libvfio-user does support streaming of migration data, this comment is based on the VFIO documentation:

d. Read data_size bytes of data from (region + data_offset) from the
migration region.

It's not clear to me whether streaming should be allowed, I'd be surprised if it didn't.

Quoted 61 lines
Quoted 6 lines

Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:

https://lore.kernel.org/all/F48606B1-15A4-4DD2-9D71-

[email protected]/

Quoted 4 lines

libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?

The answer didn't make sense to me:

"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.

In the case of the callbacks, the library translates the BAR access to callbacks."

The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <linux/vfio.h>:

* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Read pending_bytes, indicating the start of a new iteration to get device

  *    data. Repeated read on pending_bytes at this stage should have no side
  *    effects.
  *    If pending_bytes == 0, the user application should not iterate to get data
  *    for that device.
  *    If pending_bytes > 0, perform the following steps.
  * b. Read data_offset, indicating that the vendor driver should make data
  *    available through the data section. The vendor driver should return this
  *    read operation only after data is available from (region + data_offset)
  *    to (region + data_offset + data_size).
  * c. Read data_size, which is the amount of data in bytes available through
  *    the migration region.
  *    Read on data_offset and data_size should return the offset and size of
  *    the current buffer if the user application reads data_offset and
  *    data_size more than once here.

* d. Read data_size bytes of data from (region + data_offset) from the
* migration region.
* e. Process the data.
* f. Read pending_bytes, which indicates that the data from the previous
* iteration has been read. If pending_bytes > 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in _SAVING
* (stop-and-copy) until pending_bytes is 0.

This means you can report pending_bytes > 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.

Stefan

author
#91 lore raw

Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

In reply to Stefan Hajnoczi
Quoted 182 lines · Stefan Hajnoczi
Quoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 23 lines
hw/remote/vfio-user-obj.c +23 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.

Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 2 lines
hw/remote/vfio-user-obj.c +2 -0
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.

Will add a comment a comment to explain that.

Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.

Quoted 6 lines

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.

I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:

* Does this mean that partial reads are not allowed?

Thanos or John,

Could you please clarify this?

Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:

Quoted 6 lines

https://lore.kernel.org/all/[email protected]/

libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?

The answer didn't make sense to me:

"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.

In the case of the callbacks, the library translates the BAR access to callbacks."

The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <linux/vfio.h>:

* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Read pending_bytes, indicating the start of a new iteration to get device

 *    data. Repeated read on pending_bytes at this stage should have no side
 *    effects.
 *    If pending_bytes == 0, the user application should not iterate to get data
 *    for that device.
 *    If pending_bytes > 0, perform the following steps.
 * b. Read data_offset, indicating that the vendor driver should make data
 *    available through the data section. The vendor driver should return this
 *    read operation only after data is available from (region + data_offset)
 *    to (region + data_offset + data_size).
 * c. Read data_size, which is the amount of data in bytes available through
 *    the migration region.
 *    Read on data_offset and data_size should return the offset and size of
 *    the current buffer if the user application reads data_offset and
 *    data_size more than once here.

* d. Read data_size bytes of data from (region + data_offset) from the
* migration region.
* e. Process the data.
* f. Read pending_bytes, which indicates that the data from the previous
* iteration has been read. If pending_bytes > 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in _SAVING
* (stop-and-copy) until pending_bytes is 0.

Quoted 4 lines

This means you can report pending_bytes > 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.

Thank you for the pointer to generic VFIO migration, Stefan! Makes sense.

So I understand that the VFIO migration region carves out a section to stream/shuttle device data between the app (QEMU client in this case) and the driver (QEMU server). This section starts at data_offset within the region and spans data_size bytes.

We could change the server to stream the data as outlined above. Do you have a preference for the section size? Does qemu_target_page_size() work? I just tested and am able to stream with a fixed BAR size such as qemu_target_page_size().

Thank you!
--
Jag

Quoted 2 lines

Stefan

#92 lore raw

Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration

Quoted 189 lines · Jag Raman
Quoted 182 lines · Stefan Hajnoczi
Quoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman

+ * The client subsequetly asks the remote server for any data that

subsequently

Quoted 23 lines
hw/remote/vfio-user-obj.c +23 -0
+static void vfu_mig_state_running(vfu_ctx_t *vfu_ctx)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+    VfuObjectClass *k = VFU_OBJECT_GET_CLASS(OBJECT(o));
+    static int migrated_devs;
+    Error *local_err = NULL;
+    int ret;
+
+    /**
+     * TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the
+     * VMSD data from source is not available at RESUME state.
+     * Working on a fix for this.
+     */
+    if (!o->vfu_mig_file) {
+        o->vfu_mig_file = qemu_fopen_ops(o, &vfu_mig_fops_load, false);
+    }
+
+    ret = qemu_remote_loadvm(o->vfu_mig_file);
+    if (ret) {
+        VFU_OBJECT_ERROR(o, "vfu: failed to restore device state");
+        return;
+    }
+
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318

Quoted 5 lines
hw/remote/vfio-user-obj.c +5 -0
+    qemu_file_shutdown(o->vfu_mig_file);
+    o->vfu_mig_file = NULL;
+
+    /* VFU_MIGR_STATE_RUNNING begins here */
+    if (++migrated_devs == k->nr_devs) {

When is this counter reset so migration can be tried again if it fails/cancels?

Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.

Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15 -0
+static ssize_t vfu_mig_read_data(vfu_ctx_t *vfu_ctx, void *buf,
+                                 uint64_t size, uint64_t offset)
+{
+    VfuObject *o = vfu_get_private(vfu_ctx);
+
+    if (offset > o->vfu_mig_buf_size) {
+        return -1;
+    }
+
+    if ((offset + size) > o->vfu_mig_buf_size) {
+        warn_report("vfu: buffer overflow - check pending_bytes");
+        size = o->vfu_mig_buf_size - offset;
+    }
+
+    memcpy(buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines

hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424

Quoted 2 lines
hw/remote/vfio-user-obj.c +2 -0
+
+    o->vfu_mig_buf_pending -= size;

This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.

Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.

Will add a comment a comment to explain that.

Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16 -0
+uint64_t vmstate_vmsd_size(PCIDevice *pci_dev)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(DEVICE(pci_dev));
+    const VMStateField *field = NULL;
+    uint64_t size = 0;
+
+    if (!dc->vmsd) {
+        return 0;
+    }
+
+    field = dc->vmsd->fields;
+    while (field && field->name) {
+        size += vmstate_size(pci_dev, field);
+        field++;
+    }
+
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+    return size;
+}

This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.

I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.

Quoted 6 lines

I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.

The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.

I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:

* Does this mean that partial reads are not allowed?

Thanos or John,

Could you please clarify this?

Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:

Quoted 6 lines

https://lore.kernel.org/all/[email protected]/

libvfio-user has the vfu_migration_callbacks_t interface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?

The answer didn't make sense to me:

"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.

In the case of the callbacks, the library translates the BAR access to callbacks."

The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <linux/vfio.h>:

* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Read pending_bytes, indicating the start of a new iteration to get device

 *    data. Repeated read on pending_bytes at this stage should have no side
 *    effects.
 *    If pending_bytes == 0, the user application should not iterate to get data
 *    for that device.
 *    If pending_bytes > 0, perform the following steps.
 * b. Read data_offset, indicating that the vendor driver should make data
 *    available through the data section. The vendor driver should return this
 *    read operation only after data is available from (region + data_offset)
 *    to (region + data_offset + data_size).
 * c. Read data_size, which is the amount of data in bytes available through
 *    the migration region.
 *    Read on data_offset and data_size should return the offset and size of
 *    the current buffer if the user application reads data_offset and
 *    data_size more than once here.

* d. Read data_size bytes of data from (region + data_offset) from the
* migration region.
* e. Process the data.
* f. Read pending_bytes, which indicates that the data from the previous
* iteration has been read. If pending_bytes > 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in _SAVING
* (stop-and-copy) until pending_bytes is 0.

Quoted 4 lines

This means you can report pending_bytes > 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.

Thank you for the pointer to generic VFIO migration, Stefan! Makes sense.

Quoted 8 lines

So I understand that the VFIO migration region carves out a section to stream/shuttle device data between the app (QEMU client in this case) and the driver (QEMU server). This section starts at data_offset within the region and spans data_size bytes.

We could change the server to stream the data as outlined above. Do you have a preference for the section size? Does qemu_target_page_size() work? I just tested and am able to stream with a fixed BAR size such as qemu_target_page_size().

The VFIO migration API requires that data is written in the same chunk sizes as it was read, so there is no way to merge or split chunks for performance reasons once they have been read.

4KB may result in lots of chunks and that means more network traffic and read()/write() calls. I think it's too small.

Something large like 1MB might create issues with responsiveness because a 1MB chunk hogs the migration stream and read()/write() latency could hog the event loop.

I'd go for 64KB. Dave and Juan might also have a suggestion for the size.

Stefan

#93 lore raw

PATCH v5 18/18 vfio-user: avocado tests for vfio-user

In reply to Jagannathan Raman

Avocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object

---
 MAINTAINERS                |   1 +
 tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++
 2 files changed, 226 insertions(+)
 create mode 100644 tests/avocado/vfio-user.py
MAINTAINERS +1 -0
index 93bce3fa62..9ef9e1f75a 100644
@@ -3488,6 +3488,7 @@ F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: subprojects/libvfio-user
 F: hw/remote/vfio-user-obj.c
+F: tests/avocado/vfio-user.py
 
 EBPF:
 M: Jason Wang <[email protected]>
tests/avocado/vfio-user.py new file +225 -0

225 lines not on this page · raw message

Signature · 1 line
2.20.1
#94 lore raw

Re: [PATCH v5 18/18] vfio-user: avocado tests for vfio-user

Hi Jagannathan,

Quoted 14 lines · Jagannathan Raman

Avocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object

---
  MAINTAINERS                |   1 +
  tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++
  2 files changed, 226 insertions(+)

create mode 100644 tests/avocado/vfio-user.py

tests/avocado/vfio-user.py new file +2 -0
+class VfioUser(QemuSystemTest):
+    """

tests/avocado/vfio-user.py line 30

Quoted 10 lines
tests/avocado/vfio-user.py new file +10 -0
+    :avocado: tags=vfiouser
+    """
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    timeout = 20
+
+    @staticmethod
+    def migration_finished(vm):
+        res = vm.command('query-migrate')
+        if 'status' in res:
+            return res['status'] in ('completed', 'failed')

Do we need to check for failed migration in do_test_migrate()?

Quoted 2 lines
tests/avocado/vfio-user.py new file +2 -0
+        else:
+            return False

[...]

tests/avocado/vfio-user.py line 67

Quoted 8 lines
tests/avocado/vfio-user.py new file +8 -0
+    def launch_server_hotplug(self, socket):
+        server_vm = self.get_vm()
+        server_vm.add_args('-machine', 'x-remote')
+        server_vm.add_args('-nodefaults')
+        server_vm.add_args('-device', 'lsi53c895a,id=lsi1')
+        server_vm.launch()
+        server_vm.command('human-monitor-command',
+                          command_line='object_add x-vfio-user-server,'

Why not use qmp('object-add', ...) directly?

tests/avocado/vfio-user.py line 70

Quoted 3 lines
tests/avocado/vfio-user.py new file +3 -0
+                                       'id=vfioobj,socket.type=unix,'
+                                       'socket.path='+socket+',device=lsi1')
+        return server_vm

Otherwise LGTM.

author
#95 lore raw

Re: [PATCH v5 18/18] vfio-user: avocado tests for vfio-user

Quoted 18 lines · Philippe Mathieu-Daudé

Hi Jagannathan,

Quoted 10 lines · Jagannathan Raman

Avocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object

---
 MAINTAINERS                |   1 +
 tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++
 2 files changed, 226 insertions(+)
 create mode 100644 tests/avocado/vfio-user.py
Quoted 14 lines

tests/avocado/vfio-user.py line 30

Quoted 12 lines
tests/avocado/vfio-user.py new file +12 -0
+class VfioUser(QemuSystemTest):
+    """
+    :avocado: tags=vfiouser
+    """
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    timeout = 20
+
+    @staticmethod
+    def migration_finished(vm):
+        res = vm.command('query-migrate')
+        if 'status' in res:
+            return res['status'] in ('completed', 'failed')

Do we need to check for failed migration in do_test_migrate()?

OK, will do.

Quoted 15 lines
Quoted 3 lines
tests/avocado/vfio-user.py new file +2 -0
+        else:
+            return False

[...]

tests/avocado/vfio-user.py line 67

Quoted 8 lines
tests/avocado/vfio-user.py new file +8 -0
+    def launch_server_hotplug(self, socket):
+        server_vm = self.get_vm()
+        server_vm.add_args('-machine', 'x-remote')
+        server_vm.add_args('-nodefaults')
+        server_vm.add_args('-device', 'lsi53c895a,id=lsi1')
+        server_vm.launch()
+        server_vm.command('human-monitor-command',
+                          command_line='object_add x-vfio-user-server,'
Quoted 1 line

Why not use qmp('object-add', ...) directly?

OK, will use qmp directly.

Thank you!
--
Jag

Quoted 6 lines

tests/avocado/vfio-user.py line 70

Quoted 4 lines
tests/avocado/vfio-user.py new file +3 -0
+                                       'id=vfioobj,socket.type=unix,'
+                                       'socket.path='+socket+',device=lsi1')
+        return server_vm

Otherwise LGTM.

#96 lore raw
In reply to Jagannathan Raman

Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.

This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.

The current approach adds special code to hw/pci/pci.c so that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.

Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying hw/pci/pci.c with a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely on qdev_find_recursive() and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).

I wanted to raise this to see if Peter, Kevin, Michael, and others are happy with the current approach or have ideas for a clean solution.

Stefan

author
#97 lore raw
Quoted 7 lines · Stefan Hajnoczi

Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.

Thank you for taking the time to review the patches, Stefan!

Quoted 7 lines

This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.

Quoted 8 lines

The current approach adds special code to hw/pci/pci.c so that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.

We are discussing the automatic creation part you just mentioned in the following email:
[PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

I agree that automatic creation of a parent bus is not ideal - we could specify the parent bus as a separate option in the command-line or QMP. This change would avoid modification to hw/pci/pci.c - the new PCI bus could be created inplace during device creation/hotplug.

The following image gives an idea of the bus/device topology in the remote machine, as implemented in the current series. Each secondary bus and its children have isolated memory and IO spaces.
https://gitlab.com/jraman/qemu/-/commit/2e2ebf004894075ad8044739b0b16ce875114c4c

Quoted 10 lines

Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying hw/pci/pci.c with a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely on qdev_find_recursive() and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).

Based on further discussion with Stefan, I got some clarity. We could consider one more option as well - somewhere in-between multiple root SysBuses and the topology discussed above (with secondary PCI buses). We could implement a TYPE_SYS_BUS_DEVICE that creates a root PCI bus with isolated memory ranges. Something along the lines in the following diagram:
https://gitlab.com/jraman/qemu/-/commit/81f6a998278a2a795be0db7acdeb1caa2d6744fb

An example set of QMP commands to attach PCI devices would be:
device_add pci-root-bus,id=rb1
device_add <driver>,id=mydev,bus=rb1
object-add x-vfio-user-server,device=mydev

where ‘pci-root-bus’ is a TYPE_SYS_BUS_DEVICE that creates its own root PCI bus.

I’m very sorry if the above two web links disturb your review workflow.

Quoted 3 lines

I wanted to raise this to see if Peter, Kevin, Michael, and others are happy with the current approach or have ideas for a clean solution.

Looking forward to your comments.

Thank you!
--
Jag

Quoted 2 lines

Stefan

#98 lore raw
Quoted 35 lines · Jag Raman
Quoted 7 lines · Stefan Hajnoczi

Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.

Thank you for taking the time to review the patches, Stefan!

Quoted 7 lines

This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.

Quoted 8 lines

The current approach adds special code to hw/pci/pci.c so that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.

We are discussing the automatic creation part you just mentioned in the following email:
[PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine

I agree that automatic creation of a parent bus is not ideal - we could specify the parent bus as a separate option in the command-line or QMP. This change would avoid modification to hw/pci/pci.c - the new PCI bus could be created inplace during device creation/hotplug.

Quoted 4 lines

The following image gives an idea of the bus/device topology in the remote machine, as implemented in the current series. Each secondary bus and its children have isolated memory and IO spaces.
https://gitlab.com/jraman/qemu/-/commit/2e2ebf004894075ad8044739b0b16ce875114c4c

Do isolated PCI busses have any relationship with their parent at all? I think the parent plays a useful role in DMA/IOMMU, interrupts, or PCI addressing. That leaves me wondering if a parent/child relationship is an appropriate way to model this.

That said, this approach solves two problems:
1. There must be some parent for the new PCI bus.
2. qdev_find_recursive() and friends must be able to find the PCIDevice on the isolated bus.

Quoted 17 lines
Quoted 9 lines

Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying hw/pci/pci.c with a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely on qdev_find_recursive() and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).

Based on further discussion with Stefan, I got some clarity. We could consider one more option as well - somewhere in-between multiple root SysBuses and the topology discussed above (with secondary PCI buses). We could implement a TYPE_SYS_BUS_DEVICE that creates a root PCI bus with isolated memory ranges. Something along the lines in the following diagram:
https://gitlab.com/jraman/qemu/-/commit/81f6a998278a2a795be0db7acdeb1caa2d6744fb

Quoted 6 lines

An example set of QMP commands to attach PCI devices would be:
device_add pci-root-bus,id=rb1
device_add <driver>,id=mydev,bus=rb1
object-add x-vfio-user-server,device=mydev

where ‘pci-root-bus’ is a TYPE_SYS_BUS_DEVICE that creates its own root PCI bus.

If it's less code then that's an advantage but it still places unrelated DMA/interrupt spaces onto the same SysBus and therefore requires isolation. I think this alternative doesn't fundamentally fix the design.

If multiple roots are possible then isolation doesn't need to be implemented explicitly, it comes for free as part of the regular qdev/qbus hierarchy. The devices would be isolated by the fact that they live on different roots :).

I have never tried the multi-root approach though, so I'm not sure how much work it is.

Stefan