vfio-user server in QEMU
v5 cover vfio-user server in QEMU
Hi,
Thank you for taking the time to provide a comprehensive feedback of the last series of patches. We have addressed all the comments.
We are posting this v5 of the series, which incorporates all the feedback. Kindly share your feedback for this latest series
We added the following patches to the series:
- [PATCH v5 03/18] pci: isolated address space for PCI bus
- [PATCH v5 04/18] pci: create and free isolated PCI buses
- [PATCH v5 05/18] qdev: unplug blocker for devices
- [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
- [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
We made the following changes to the existing patches:
[PATCH v5 09/18] vfio-user: define vfio-user-server object
- renamed object class member 'daemon' as 'auto_shutdown'
- set VfioUserServerProperties version to 6.3
- use SocketAddressType_str to compose error message
- refuse setting 'socket' and 'device' properties after server starts
- added VFU_OBJECT_ERROR macro to report error
[PATCH v5 10/18] vfio-user: instantiate vfio-user context
- set error variable to NULL after transferring ownership with error_propagate()
[PATCH v5 11/18] vfio-user: find and init PCI device
- block hot-unplug of PCI device when it is attached to the server object
[PATCH v5 12/18] vfio-user: run vfio-user context
- emit a hangup event to the monitor when the client disconnects
- reset vfu_poll_fd member and disable FD handler during finalize
- add a comment to explain that attach could block
- use VFU_OBJECT_ERROR instead of setting error_abort
[PATCH v5 14/18] vfio-user: handle DMA mappings
- use pci_address_space() to access device's root memory region
- given we're using one bus per device, mapped memory regions get destroyed automatically when device is unplugged
[PATCH v5 15/18] vfio-user: handle PCI BAR accesses
- use pci_isol_as_io() & pci_isol_as_mem() to access the device's PCI/CPU address space. This simultaneously fixes the AddressSpace issue noted in the last review cycle
[PATCH v5 16/18] vfio-user: handle device interrupts
- setting own IRQ handlers for each bus
- renamed vfu_object_dev_table to vfu_object_dev_to_ctx_table
- indexing into vfu_object_dev_to_ctx_table with device's address pointer instead of devfn
- not looking up before removing from table
[PATCH v5 17/18] vfio-user: register handlers to facilitate migration
- use VFU_OBJECT_ERROR instead of setting error_abort
We dropped the following patch from previous series:
- vfio-user: IOMMU support for remote device
Thank you very much!
Jagannathan Raman (18):
configure, meson: override C compiler for cmake tests/avocado: Specify target VM argument to helper routines
pci: isolated address space for PCI bus
pci: create and free isolated PCI buses
qdev: unplug blocker for devices
vfio-user: add HotplugHandler for remote machine
vfio-user: set qdev bus callbacks for remote machine
vfio-user: build library
vfio-user: define vfio-user-server object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
vfio-user: register handlers to facilitate migration
vfio-user: avocado tests for vfio-user
configure | 21 +- meson.build | 44 +- qapi/misc.json | 23 + qapi/qom.json | 20 +- include/hw/pci/pci.h | 12 + include/hw/pci/pci_bus.h | 17 + include/hw/qdev-core.h | 21 + include/migration/vmstate.h | 2 + migration/savevm.h | 2 + hw/pci/msi.c | 13 +- hw/pci/msix.c | 12 +- hw/pci/pci.c | 186 ++++ hw/pci/pci_bridge.c | 5 + hw/remote/machine.c | 86 ++ hw/remote/vfio-user-obj.c | 1019 ++++++++++++++++++++ migration/savevm.c | 73 ++ migration/vmstate.c | 19 + softmmu/qdev-monitor.c | 74 +- .gitlab-ci.d/buildtest.yml | 2 + .gitmodules | 3 + Kconfig.host | 4 + MAINTAINERS | 3 + hw/remote/Kconfig | 4 + hw/remote/meson.build | 3 + hw/remote/trace-events | 11 + meson_options.txt | 2 + subprojects/libvfio-user | 1 + tests/avocado/avocado_qemu/__init__.py | 14 +- tests/avocado/vfio-user.py | 225 +++++ tests/docker/dockerfiles/centos8.docker | 2 + tests/docker/dockerfiles/ubuntu2004.docker | 2 + 31 files changed, 1912 insertions(+), 13 deletions(-) create mode 100644 hw/remote/vfio-user-obj.c create mode 160000 subprojects/libvfio-user create mode 100644 tests/avocado/vfio-user.py
Signature · 1 line
2.20.1
PATCH v5 2/18 tests/avocado: Specify target VM argument to helper routines
Specify target VM for exec_command andexec_command_and_wait_for_pattern routines
--- tests/avocado/avocado_qemu/__init__.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
tests/avocado/avocado_qemu/__init__.py +10 -4
@@ -198,7 +198,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None, """ _console_interaction(test, success_message, failure_message, None, vm=vm) -def exec_command(test, command): +def exec_command(test, command, vm=None): """ Send a command to a console (appending CRLF characters), while logging the content.
@@ -207,11 +207,14 @@ def exec_command(test, command): :type test: :class:`avocado_qemu.QemuSystemTest` :param command: the command to send :type command: str + :param vm: target vm + :type vm: :class:`qemu.machine.QEMUMachine` """ - _console_interaction(test, None, None, command + '\r') + _console_interaction(test, None, None, command + '\r', vm=vm) def exec_command_and_wait_for_pattern(test, command, - success_message, failure_message=None): + success_message, failure_message=None, + vm=None): """ Send a command to a console (appending CRLF characters), then wait for success_message to appear on the console, while logging the.
@@ -223,8 +226,11 @@ def exec_command_and_wait_for_pattern(test, command, :param command: the command to send :param success_message: if this message appears, test succeeds :param failure_message: if this message appears, test fails + :param vm: target vm + :type vm: :class:`qemu.machine.QEMUMachine` """ - _console_interaction(test, success_message, failure_message, command + '\r') + _console_interaction(test, success_message, failure_message, command + '\r', + vm=vm) class QemuBaseTest(avocado.Test): def _get_unique_tag_val(self, tag_name):
Signature · 1 line
2.20.1
PATCH v5 5/18 qdev: unplug blocker for devices
In reply to Jagannathan Raman--- include/hw/qdev-core.h | 5 +++++ softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
include/hw/qdev-core.h +5 -0
@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {
@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus,Error **errp); +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp); +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line *
softmmu/qdev-monitor.c +35 -0
@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } } +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp) +{ +ERRP_GUARD (); + +if (!migration_is_idle ()) { +error_setg (errp,"migration is in progress" ); +return -EBUSY ; + } + + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); + +return 0 ; +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;
Signature · 1 line
2.20.1
PATCH v5 3/18 pci: isolated address space for PCI bus
In reply to Jagannathan RamanAllow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
TYPE_REMOTE_MACHINE allows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.
--- include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)
include/hw/pci/pci.h +2 -0
@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target
include/hw/pci/pci_bus.h +17 -0
@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io; +/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. This allows the +* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, as such it needs the PCI buses in the same machine +* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */
hw/pci/pci.c +17 -0
@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */
@@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);
@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;
@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }
hw/pci/pci_bridge.c +5 -0
@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);
Signature · 1 line
2.20.1
PATCH v5 8/18 vfio-user: build library
In reply to Jagannathan Ramanadd the libvfio-user library as a submodule. build it as a cmake subproject.
--- configure | 19 +++++++++- meson.build | 44 +++++++++++++++++++++- .gitlab-ci.d/buildtest.yml | 2 + .gitmodules | 3 ++ Kconfig.host | 4 ++ MAINTAINERS | 1 + hw/remote/Kconfig | 4 ++ hw/remote/meson.build | 2 + meson_options.txt | 2 + subprojects/libvfio-user | 1 + tests/docker/dockerfiles/centos8.docker | 2 + tests/docker/dockerfiles/ubuntu2004.docker | 2 + 12 files changed, 84 insertions(+), 2 deletions(-) create mode 160000 subprojects/libvfio-user
configure +18 -1
@@ -356,6 +356,7 @@ ninja="" gio="$default_feature" skip_meson=no slirp_smbd="$default_feature" +vfio_user_server="disabled" # The following Meson options are handled manually (still they # are included in the automatically generated help message)
@@ -1172,6 +1173,10 @@ for opt do ;; --disable-blobs) meson_option_parse --disable-install-blobs "" ;; + --enable-vfio-user-server) vfio_user_server="enabled" + ;; + --disable-vfio-user-server) vfio_user_server="disabled" + ;; --enable-tcmalloc) meson_option_parse --enable-malloc=tcmalloc tcmalloc ;; --enable-jemalloc) meson_option_parse --enable-malloc=jemalloc jemalloc
@@ -1439,6 +1444,7 @@ cat << EOF rng-none dummy RNG, avoid using /dev/(u)random and getrandom() gio libgio support slirp-smbd use smbd (at path --smbd=*) in slirp networking + vfio-user-server vfio-user server support NOTE: The object files are built at the place where configure is launched EOF
@@ -3121,6 +3127,17 @@ but not implemented on your system" fi fi +########################################## +# check for vfio_user_server + +case "$vfio_user_server" in + auto | enabled ) + if test "$git_submodules_action" != "ignore"; then + git_submodules="${git_submodules} subprojects/libvfio-user" + fi + ;; +esac + ########################################## # End of CC checks # After here, no more $cc or $ld runs
@@ -3811,7 +3828,7 @@ if test "$skip_meson" = no; then -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \ -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \ -Db_lto=$lto -Dcfi=$cfi -Dtcg=$tcg -Dxen=$xen \ - -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp \ + -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp -Dvfio_user_server=$vfio_user_server \ $(test -n "${LIB_FUZZING_ENGINE+xxx}" && echo "-Dfuzzing_engine=$LIB_FUZZING_ENGINE") \ $(if test "$default_feature" = no; then echo "-Dauto_features=disabled"; fi) \ "$@" $cross_arg "$PWD" "$source_path"
meson.build +43 -1
68 lines not on this page · raw message
.gitlab-ci.d/buildtest.yml +2 -0
@@ -42,6 +42,7 @@ build-system-ubuntu: variables: IMAGE: ubuntu2004 CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system + --enable-vfio-user-server TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu microblazeel-softmmu mips64el-softmmu MAKE_CHECK_ARGS: check-build
@@ -165,6 +166,7 @@ build-system-centos: IMAGE: centos8 CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system --enable-modules --enable-trace-backends=dtrace --enable-docs + --enable-vfio-user-server TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu MAKE_CHECK_ARGS: check-build
.gitmodules +3 -0
@@ -67,3 +67,6 @@ [submodule "tests/lcitool/libvirt-ci"] path = tests/lcitool/libvirt-ci url = http://gitlab.com/libvirt/libvirt-ci +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.git
Kconfig.host +4 -0
@@ -45,3 +45,7 @@ config MULTIPROCESS_ALLOWED config FUZZ bool select SPARSE_MEM + +config VFIO_USER_SERVER_ALLOWED + bool + imply VFIO_USER_SERVER
MAINTAINERS +1 -0
@@ -3486,6 +3486,7 @@ F: hw/remote/proxy-memory-listener.c F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h +F: subprojects/libvfio-user EBPF: M: Jason Wang <[email protected]>
hw/remote/Kconfig +4 -0
@@ -2,3 +2,7 @@ config MULTIPROCESS bool depends on PCI && PCI_EXPRESS && KVM select REMOTE_PCIHOST + +config VFIO_USER_SERVER + bool + depends on MULTIPROCESS
hw/remote/meson.build +2 -0
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser) + specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
meson_options.txt +2 -0
@@ -68,6 +68,8 @@ option('multiprocess', type: 'feature', value: 'auto', description: 'Out of process device emulation support') option('dbus_display', type: 'feature', value: 'auto', description: '-display dbus support') +option('vfio_user_server', type: 'feature', value: 'auto', + description: 'vfio-user server support') option('attr', type : 'feature', value : 'auto', description: 'attr/xattr support')
subprojects/libvfio-user new file +1 -0
@@ -0,0 +1 @@ +Subproject commit 7056525da5399d00831e90bed4aedb4b8442c9b2
tests/docker/dockerfiles/centos8.docker +2 -0
@@ -23,6 +23,7 @@ RUN dnf update -y && \ capstone-devel \ ccache \ clang \ + cmake \ ctags \ cyrus-sasl-devel \ daxctl-devel \
@@ -45,6 +46,7 @@ RUN dnf update -y && \ gtk3-devel \ hostname \ jemalloc-devel \ + json-c-devel \ libaio-devel \ libasan \ libattr-devel \
tests/docker/dockerfiles/ubuntu2004.docker +2 -0
@@ -18,6 +18,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \ ca-certificates \ ccache \ clang \ + cmake \ dbus \ debianutils \ diffutils \
@@ -57,6 +58,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \ libiscsi-dev \ libjemalloc-dev \ libjpeg-turbo8-dev \ + libjson-c-dev \ liblttng-ust-dev \ liblzo2-dev \ libncursesw5-dev \
Signature · 1 line
2.20.1
PATCH v5 9/18 vfio-user: define vfio-user-server object
In reply to Jagannathan RamanDefine vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
--- qapi/qom.json | 20 +++- hw/remote/vfio-user-obj.c | 194 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 217 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
@@ -703,6 +703,20 @@ { 'struct': 'RemoteObjectProperties', 'data': { 'fd': 'str', 'devid': 'str' } } +## +# @VfioUserServerProperties: +# +# Properties for x-vfio-user-server objects. +# +# @socket: socket to be used by the libvfiouser library +# +# @device: the id of the device to be emulated at the server +# +# Since: 6.3 +## +{ 'struct': 'VfioUserServerProperties', + 'data': { 'socket': 'SocketAddress', 'device': 'str' } } + ## # @RngProperties: #
@@ -842,7 +856,8 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', - { 'name': 'x-remote-object', 'features': [ 'unstable' ] } + { 'name': 'x-remote-object', 'features': [ 'unstable' ] }, + { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] } ] } ##
@@ -905,7 +920,8 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', - 'x-remote-object': 'RemoteObjectProperties' + 'x-remote-object': 'RemoteObjectProperties', + 'x-vfio-user-server': 'VfioUserServerProperties' } } ##
hw/remote/vfio-user-obj.c new file +194 -0
194 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -3487,6 +3487,7 @@ F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user +F: hw/remote/vfio-user-obj.c EBPF: M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c')) remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser)
hw/remote/trace-events +3 -0
@@ -2,3 +2,6 @@ mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process" mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process" + +# vfio-user-obj.c +vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
2.20.1
PATCH v5 7/18 vfio-user: set qdev bus callbacks for remote machine
In reply to Jagannathan Raman--- hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
hw/remote/machine.c +57 -0
@@ -22,6 +22,60 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +static bool remote_machine_get_bus (const char *type,BusState **bus, +Error **errp) +{ +ERRP_GUARD (); +RemoteMachineState *s =REMOTE_MACHINE (current_machine); +BusState *root_bus =NULL ; +PCIBus *new_pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (strcmp (type,TYPE_PCI_BUS ) &&strcmp (type,TYPE_PCI_BUS )) { +return true; + } + + root_bus =qbus_find_recursive (sysbus_get_default (),NULL ,TYPE_PCIE_BUS ); +if (!root_bus) { +error_setg (errp,"Unable to find root PCI device" ); +return false; + } + + new_pci_bus =pci_isol_bus_new (root_bus, type, errp); +if (!new_pci_bus) { +return false; + } + + *bus =BUS (new_pci_bus); + +pci_bus_irqs (new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +return true; +} + +static bool remote_machine_put_bus (BusState *bus,Error **errp) +{ +PCIBus *pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (!object_dynamic_cast (OBJECT (bus),TYPE_PCI_BUS )) { +return true; + } + + pci_bus =PCI_BUS (bus); + +return pci_isol_bus_free (pci_bus, errp); +} +static void remote_machine_init (MachineState *machine) {MemoryRegion *system_memory, *system_io, *pci_memory;
@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine) &s->iohub, REMOTE_IOHUB_NB_PIRQS);qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); + +qdev_set_bus_cbs (remote_machine_get_bus, remote_machine_put_bus, + &error_fatal); } static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
Signature · 1 line
2.20.1
PATCH v5 1/18 configure, meson: override C compiler for cmake
In reply to Jagannathan RamanThe compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Explicitly specify the C compiler for cmake to avoid this error
--- configure | 2 ++ 1 file changed, 2 insertions(+)
configure +2 -0
@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross + echo "[cmake]" >> $cross + echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross echo "[binaries]" >> $cross echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
Signature · 1 line
2.20.1
PATCH v5 13/18 vfio-user: handle PCI config space accesses
In reply to Jagannathan RamanDefine and register handlers for PCI config space accesses
--- hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 47 insertions(+)
hw/remote/vfio-user-obj.c +45 -0
@@ -47,6 +47,7 @@#include "qapi/qapi-events-misc.h" #include "qemu/notify.h" #include "qemu/thread.h" +#include "qemu/main-loop.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" #include "hw/qdev-core.h"
@@ -214,6 +215,39 @@ retry_attach:qemu_set_fd_handler (o->vfu_poll_fd, vfu_object_ctx_run,NULL , o); } +static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) { + len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_host_config_write_common (o->pci_dev, offset, +pci_config_size (o->pci_dev), + val, len); +trace_vfu_cfg_write (offset, val); + }else { + val =pci_host_config_read_common (o->pci_dev, offset, +pci_config_size (o->pci_dev), len); +memcpy (ptr, &val, len); +trace_vfu_cfg_read (offset, val); + } + offset += len; + ptr += len; + bytes -= len; + } + +return count; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These
@@ -291,6 +325,17 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_CFG_REGION_IDX , +pci_config_size (o->pci_dev), &vfu_object_cfg_access, +VFU_REGION_FLAG_RW |VFU_REGION_FLAG_ALWAYS_CB , +NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (errp, +"vfu: Failed to setup config space handlers for %s- %s" , + o->device,strerror (errno)); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, # vfio-user-obj.c vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" +vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" +vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
2.20.1
PATCH v5 10/18 vfio-user: instantiate vfio-user context
In reply to Jagannathan Ramancreate a context with the vfio-user library to run a PCI device
--- hw/remote/vfio-user-obj.c | 78 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+)
hw/remote/vfio-user-obj.c +78 -0
@@ -41,6 +41,9 @@#include "hw/remote/machine.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qemu/notify.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -82,13 +85,23 @@ struct VfuObject {char *device;Error *err; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; }; +static void vfu_object_init_ctx (VfuObject *o,Error **errp); +static void vfu_object_set_socket (Object *obj,Visitor *v,const char *name,void *opaque,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +return ; + } +qapi_free_SocketAddress (o->socket); o->socket =NULL ;
@@ -104,17 +117,68 @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name, }trace_vfu_prop ("socket" , o->socket->u.q_unix.path); + +vfu_object_init_ctx (o, errp); }static void vfu_object_set_device (Object *obj,const char *str,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +return ; + } +g_free (o->device); o->device =g_strdup (str);trace_vfu_prop ("device" , str); + +vfu_object_init_ctx (o, errp); +} + +/* +* TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' +* properties. It also depends on devices instantiated in QEMU. These +* dependencies are not available during the instance_init phase of this +* object's life-cycle. As such, the server is initialized after the +* machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT +* when the machine is setup, and the dependencies are available. +*/ +static void vfu_object_machine_done (Notifier *notifier,void *data) +{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); +Error *err =NULL ; + +vfu_object_init_ctx (o, &err); + +if (err) { +error_propagate (&error_abort, err); + } +} + +static void vfu_object_init_ctx (VfuObject *o,Error **errp) +{ +ERRP_GUARD (); + +if (o->vfu_ctx || !o->socket || !o->device || + !phase_check (PHASE_MACHINE_READY )) { +return ; + } + +if (o->err) { +error_propagate (errp, o->err); + o->err =NULL ; +return ; + } + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket->u.q_unix.path,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno)); +return ; + } }static void vfu_object_init (Object *obj)
@@ -124,6 +188,11 @@ static void vfu_object_init(Object *obj) k->nr_devs++; +if (!phase_check (PHASE_MACHINE_READY )) { + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); + } +if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) {error_setg (&o->err,"vfu: %s only compatible with %s machine" ,TYPE_VFU_OBJECT ,TYPE_REMOTE_MACHINE );
@@ -142,6 +211,10 @@ static void vfu_object_finalize(Object *obj) o->socket =NULL ; +if (o->vfu_ctx) { +vfu_destroy_ctx (o->vfu_ctx); + } +g_free (o->device); o->device =NULL ;
@@ -149,6 +222,11 @@ static void vfu_object_finalize(Object *obj)if (!k->nr_devs && k->auto_shutdown) {qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); } + +if (o->machine_done.notify) { +qemu_remove_machine_init_done_notifier (&o->machine_done); + o->machine_done.notify =NULL ; + } }static void vfu_object_class_init (ObjectClass *klass,void *data)
Signature · 1 line
2.20.1
PATCH v5 11/18 vfio-user: find and init PCI device
In reply to Jagannathan RamanFind the PCI device with specified id. Initialize the device context with the QEMU PCI device
--- hw/remote/vfio-user-obj.c | 60 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
hw/remote/vfio-user-obj.c +60 -0
@@ -44,6 +44,8 @@#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" +#include "hw/qdev-core.h" +#include "hw/pci/pci.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -89,6 +91,10 @@ struct VfuObject {Notifier machine_done;vfu_ctx_t *vfu_ctx; + +PCIDevice *pci_dev; + +Error *unplug_blocker; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);
@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)static void vfu_object_init_ctx (VfuObject *o,Error **errp) {ERRP_GUARD (); +DeviceState *dev =NULL ; +vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +int ret;if (o->vfu_ctx || !o->socket || !o->device || !phase_check (PHASE_MACHINE_READY )) {
@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->device); +if (dev ==NULL ) { +error_setg (errp,"vfu: Device %s not found" , o->device); + goto fail; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"vfu: %s not a PCI device" , o->device); + goto fail; + } + + o->pci_dev =PCI_DEVICE (dev); + +if (pci_is_express (o->pci_dev)) { + pci_type =VFU_PCI_TYPE_EXPRESS ; + } + + ret =vfu_pci_init (o->vfu_ctx, pci_type,PCI_HEADER_TYPE_NORMAL ,0 ); +if (ret <0 ) { +error_setg (errp, +"vfu: Failed to attach PCI device %s to context - %s" , + o->device,strerror (errno)); + goto fail; + } + +error_setg (&o->unplug_blocker,"%s is in use" , o->device); +qdev_add_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker, errp); +if (*errp) { + goto fail; + } + +return ; + +fail: +vfu_destroy_ctx (o->vfu_ctx); +if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + o->vfu_ctx =NULL ; + o->pci_dev =NULL ; }static void vfu_object_init (Object *obj)
@@ -219,6 +271,14 @@ static void vfu_object_finalize(Object *obj) o->device =NULL ; +if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + + o->pci_dev =NULL ; +if (!k->nr_devs && k->auto_shutdown) {qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); }
Signature · 1 line
2.20.1
PATCH v5 4/18 pci: create and free isolated PCI buses
In reply to Jagannathan RamanAdds pci_isol_bus_new() and pci_isol_bus_free() functions to manage creation and destruction of isolated PCI buses. Also adds qdev_get_bus and qdev_put_bus callbacks to allow the choice of parent bus.
--- include/hw/pci/pci.h | 4 + include/hw/qdev-core.h | 16 ++++ hw/pci/pci.c | 169 +++++++++++++++++++++++++++++++++++++++++ softmmu/qdev-monitor.c | 39 +++++++++- 4 files changed, 225 insertions(+), 3 deletions(-)
include/hw/pci/pci.h +4 -0
@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,PCIDevice *pci_vga_init (PCIBus *bus); +PCIBus *pci_isol_bus_new (BusState *parent_bus,const char *new_bus_type, +Error **errp); +bool pci_isol_bus_free (PCIBus *pci_bus,Error **errp); +static inline PCIBus *pci_get_bus (const PCIDevice *dev) {return PCI_BUS (qdev_get_parent_bus (DEVICE (dev)));
include/hw/qdev-core.h +16 -0
@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void ); +/** +* Find parent bus - these callbacks are used during device addition +* and deletion. +* +* During addition, if no parent bus is specified in the options, +* these callbacks provide a way to figure it out based on the +* bus type. If these callbacks are not defined, defaults to +* finding the parent bus starting from default system bus +*/ +typedef bool (QDevGetBusFunc )(const char *type,BusState **bus,Error **errp); +typedef bool (QDevPutBusFunc )(BusState *bus,Error **errp); +bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus, +Error **errp); +/** * GpioPolarity: Polarity of a GPIO line *
@@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);/*** BUS API. ***/ DeviceState *qdev_find_recursive (BusState *bus,const char *id); +BusState *qbus_find_recursive (BusState *bus,const char *name, +const char *bus_typename);/* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */ typedef int (qbus_walkerfn )(BusState *bus,void *opaque);
hw/pci/pci.c +169 -0
175 lines not on this page · raw message
softmmu/qdev-monitor.c +36 -3
@@ -64,6 +64,9 @@ typedef struct QDevAlias#define QEMU_ARCH_VIRTIO_CCW (QEMU_ARCH_S390X)#define QEMU_ARCH_VIRTIO_MMIO (QEMU_ARCH_M68K) +static QDevGetBusFunc *qdev_get_bus; +static QDevPutBusFunc *qdev_put_bus; +/* Please keep this table sorted by typename. */ static const QDevAlias qdev_alias_table[] = { {"AC97" ,"ac97" },/* -soundhw name */
@@ -450,7 +453,7 @@ static inline bool qbus_is_full(BusState *bus)* If more than one exists, prefer one that can take another device. * Return the bus if found, else %NULL. */ -static BusState *qbus_find_recursive (BusState *bus,const char *name, +BusState *qbus_find_recursive (BusState *bus,const char *name,const char *bus_typename) {BusChild *kid;
@@ -608,6 +611,20 @@ const char *qdev_set_id(DeviceState *dev, char *id, Error **errp)return prop->name; } + +bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus, +Error **errp) +{ +if (qdev_get_bus || qdev_put_bus) { +error_setg (errp,"callbacks already set" ); +return false; + } + + qdev_get_bus = get_bus; + qdev_put_bus = put_bus; +return true; +} +DeviceState *qdev_device_add_from_qdict (const QDict *opts,bool from_json,Error **errp) {
@@ -642,7 +659,13 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts, driver, object_get_typename(OBJECT(bus)));return NULL ; } - }else if (dc->bus_type !=NULL ) { + }else if (dc->bus_type !=NULL && qdev_get_bus !=NULL ) { +if (!qdev_get_bus (dc->bus_type, &bus, errp)) { +return NULL ; + } + } + +if (!bus && dc->bus_type !=NULL ) { bus =qbus_find_recursive (sysbus_get_default (),NULL , dc->bus_type);if (!bus ||qbus_is_full (bus)) { error_setg(errp, "No '%s' bus found for device '%s'",
@@ -891,10 +914,12 @@ static DeviceState *find_device_state(const char *id, Error **errp)void qdev_unplug (DeviceState *dev,Error **errp) { +ERRP_GUARD ();DeviceClass *dc =DEVICE_GET_CLASS (dev);HotplugHandler *hotplug_ctrl;HotplugHandlerClass *hdc;Error *local_err =NULL ; +BusState *parent_bus =qdev_get_parent_bus (dev);if (dev->parent_bus && !qbus_is_hotpluggable (dev->parent_bus)) {error_setg (errp,QERR_BUS_NO_HOTPLUG , dev->parent_bus->name);
@@ -930,7 +955,15 @@ void qdev_unplug(DeviceState *dev, Error **errp)object_unparent (OBJECT (dev)); } } -error_propagate (errp, local_err); + +if (local_err) { +error_propagate (errp, local_err); +return ; + } + +if (qdev_put_bus) { +qdev_put_bus (parent_bus, errp); + } }void qmp_device_del (const char *id,Error **errp)
Signature · 1 line
2.20.1
PATCH v5 12/18 vfio-user: run vfio-user context
In reply to Jagannathan RamanSetup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
--- qapi/misc.json | 23 ++++++++++ hw/remote/vfio-user-obj.c | 90 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 112 insertions(+), 1 deletion(-)
qapi/misc.json +23 -0
@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true } + +## +# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel +# +# @device: ID of attached PCI device +# +# @path: path of the socket +# +# Since: 6.3 +# +# Example: +# +# <- { "event": "VFU_CLIENT_HANGUP", +# "data": { "device": "lsi1", +# "path": "/tmp/vfu1-sock" }, +# "timestamp": { "seconds": 1265044230, "microseconds": 450486 } } +# +## +{ 'event': 'VFU_CLIENT_HANGUP', + 'data': { 'device': 'str', 'path': 'str' } }
hw/remote/vfio-user-obj.c +89 -1
137 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v5 14/18 vfio-user: handle DMA mappings
In reply to Jagannathan RamanDefine and register callbacks to manage the RAM regions used for device DMA
--- hw/remote/vfio-user-obj.c | 50 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 52 insertions(+)
hw/remote/vfio-user-obj.c +50 -0
@@ -248,6 +248,49 @@ static ssize_t vfu_object_cfg_access(vfu_ctx_t *vfu_ctx, char * const buf,return count; } +static void dma_register (vfu_ctx_t *vfu_ctx,vfu_dma_info_t *info) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +MemoryRegion *subregion =NULL ; + g_autofree char *name = NULL; +struct iovec *iov = &info->iova; + +if (!info->vaddr) { +return ; + } + + name =g_strdup_printf ("mem-%s-%" PRIx64"" , o->device, + (uint64_t )info->vaddr); + + subregion =g_new0 (MemoryRegion,1 ); + +memory_region_init_ram_ptr (subregion,NULL , name, + iov->iov_len, info->vaddr); + +memory_region_add_subregion (pci_address_space (o->pci_dev), + (hwaddr )iov->iov_base, subregion); + +trace_vfu_dma_register ((uint64_t )iov->iov_base, iov->iov_len); +} + +static void dma_unregister (vfu_ctx_t *vfu_ctx,vfu_dma_info_t *info) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +MemoryRegion *mr =NULL ; +ram_addr_t offset; + + mr =memory_region_from_host (info->vaddr, &offset); +if (!mr) { +return ; + } + +memory_region_del_subregion (pci_address_space (o->pci_dev), mr); + +object_unparent ((OBJECT (mr))); + +trace_vfu_dma_unregister ((uint64_t )info->iova.iov_base); +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These
@@ -336,6 +379,13 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + ret =vfu_setup_device_dma (o->vfu_ctx, &dma_register, &dma_unregister); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup DMA handlers for %s" , + o->device); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" +vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" +vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
2.20.1
PATCH v5 17/18 vfio-user: register handlers to facilitate migration
In reply to Jagannathan RamanStore and load the device's state during migration. use libvfio-user's handlers for this purpose
--- include/migration/vmstate.h | 2 + migration/savevm.h | 2 + hw/remote/vfio-user-obj.c | 323 ++++++++++++++++++++++++++++++++++++ migration/savevm.c | 73 ++++++++ migration/vmstate.c | 19 +++ 5 files changed, 419 insertions(+)
include/migration/vmstate.h +2 -0
@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;#define VMSTATE_END_OF_LIST () \ {} +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev); +int vmstate_load_state (QEMUFile *f,const VMStateDescription *vmsd,void *opaque,int version_id); int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
migration/savevm.h +2 -0
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif
hw/remote/vfio-user-obj.c +323 -0
359 lines not on this page · raw message
migration/savevm.c +73 -0
85 lines not on this page · raw message
migration/vmstate.c +19 -0
@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)return size; } +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } + +return size; +} +static void vmstate_handle_alloc (void *ptr,const VMStateField *field,void *opaque) {
Signature · 1 line
2.20.1
PATCH v5 6/18 vfio-user: add HotplugHandler for remote machine
In reply to Jagannathan RamanAllow hotplugging of PCI(e) devices to remote machine
--- hw/remote/machine.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
hw/remote/machine.c +29 -0
@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); +} + +static void remote_machine_pre_plug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hotplug" ); + } +} + +static void remote_machine_unplug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hot-unplug" ); +return ; + } + +qdev_unrealize (dev); }static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); mc->init = remote_machine_init; mc->desc ="Experimental remote machine" ; + + hc->pre_plug = remote_machine_pre_plug_cb; + hc->unplug = remote_machine_unplug_cb; }static const TypeInfo remote_machine = {
@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = { .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), .class_init = remote_machine_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } };static void remote_machine_register_types (void )
Signature · 1 line
2.20.1
PATCH v5 15/18 vfio-user: handle PCI BAR accesses
In reply to Jagannathan RamanDetermine the BARs used by the PCI device and register handlers to manage the access to the same.
--- hw/remote/vfio-user-obj.c | 92 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 3 ++ 2 files changed, 95 insertions(+)
hw/remote/vfio-user-obj.c +92 -0
@@ -291,6 +291,96 @@ static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)trace_vfu_dma_unregister ((uint64_t )info->iova.iov_base); } +static ssize_t vfu_object_bar_rw (PCIDevice *pci_dev,hwaddr addr,size_t count, +char *const buf,const bool is_write, +bool is_io) +{ +AddressSpace *as =NULL ; +MemTxResult res; + +if (is_io) { + as =pci_isol_as_io (pci_dev); + as = as ? as : &address_space_io; + }else { + as =pci_isol_as_mem (pci_dev); + as = as ? as : &address_space_memory; + } + +trace_vfu_bar_rw_enter (is_write ?"Write" :"Read" , (uint64_t )addr); + + res =address_space_rw (as, addr,MEMTXATTRS_UNSPECIFIED , (void *)buf, + (hwaddr )count, is_write); +if (res !=MEMTX_OK ) { +warn_report ("vfu: failed to %s 0x%" PRIx64"" , + is_write ?"write to" :"read from" , + addr); +return -1 ; + } + +trace_vfu_bar_rw_exit (is_write ?"Write" :"Read" , (uint64_t )addr); + +return count; +} + +/** +* VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs. +* +* To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would +* define vfu_object_bar2_handler +*/ +#define VFU_OBJECT_BAR_HANDLER (BAR_NO ) \ + static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx, \ + char * const buf, size_t count, \ + loff_t offset, const bool is_write) \ + { \ + VfuObject *o = vfu_get_private(vfu_ctx); \ + PCIDevice *pci_dev = o->pci_dev; \ + hwaddr addr = (hwaddr)(pci_get_bar_addr(pci_dev, BAR_NO) + offset); \ + bool is_io = !!(pci_dev->io_regions[BAR_NO].type & \ + PCI_BASE_ADDRESS_SPACE); \ + \ + return vfu_object_bar_rw(pci_dev, addr, count, buf, is_write, is_io); \ + } \ + +VFU_OBJECT_BAR_HANDLER(0) +VFU_OBJECT_BAR_HANDLER(1) +VFU_OBJECT_BAR_HANDLER(2) +VFU_OBJECT_BAR_HANDLER(3) +VFU_OBJECT_BAR_HANDLER(4) +VFU_OBJECT_BAR_HANDLER(5) + +static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS ] = { + &vfu_object_bar0_handler, + &vfu_object_bar1_handler, + &vfu_object_bar2_handler, + &vfu_object_bar3_handler, + &vfu_object_bar4_handler, + &vfu_object_bar5_handler, +}; + +/** +* vfu_object_register_bars - Identify active BAR regions of pdev and setup +* callbacks to handle read/write accesses +*/ +static void vfu_object_register_bars (vfu_ctx_t *vfu_ctx,PCIDevice *pdev) +{ +int i; + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { +if (!pdev->io_regions[i].size) { +continue ; + } + +vfu_setup_region (vfu_ctx,VFU_PCI_DEV_BAR0_REGION_IDX + i, + (size_t )pdev->io_regions[i].size, + vfu_object_bar_handlers[i], +VFU_REGION_FLAG_RW ,NULL ,0 ,-1 ,0 ); + +trace_vfu_bar_register (i, pdev->io_regions[i].addr, + pdev->io_regions[i].size); + } +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These
@@ -386,6 +476,8 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } +vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +3 -0
@@ -9,3 +9,6 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" +vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" +vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" +vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
2.20.1
PATCH v5 16/18 vfio-user: handle device interrupts
In reply to Jagannathan RamanForward remote device's interrupts to the guest
--- include/hw/pci/pci.h | 6 +++ hw/pci/msi.c | 13 +++++- hw/pci/msix.c | 12 +++++- hw/remote/vfio-user-obj.c | 89 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 1 + 5 files changed, 119 insertions(+), 2 deletions(-)
include/hw/pci/pci.h +6 -0
@@ -128,6 +128,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */
@@ -322,6 +324,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */
hw/pci/msi.c +12 -1
@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {
@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ; + dev->msi_notify = pci_msi_notify; +pci_set_word (dev->config +msi_flags_off (dev), flags);pci_set_word (dev->wmask +msi_flags_off (dev),PCI_MSI_FLAGS_QSIZE |PCI_MSI_FLAGS_ENABLE );
@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)return mask & (1U << vector); } -void msi_notify (PCIDevice *dev,unsigned int vector) +static void pci_msi_notify (PCIDevice *dev,unsigned int vector) {uint16_t flags =pci_get_word (dev->config +msi_flags_off (dev));bool msi64bit = flags &PCI_MSI_FLAGS_64BIT ;
@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); } +void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) { + dev->msi_notify (dev, vector); + } +} +void msi_send_message (PCIDevice *dev,MSIMessage msg) {MemTxAttrs attrs = {};
hw/pci/msix.c +11 -1
@@ -31,6 +31,8 @@#define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)#define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8) +static void pci_msix_notify (PCIDevice *dev,unsigned vector); +MSIMessage msix_get_message (PCIDevice *dev,unsigned vector) {uint8_t *table_entry = dev->msix_table + vector *PCI_MSIX_ENTRY_SIZE ;
@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries, dev->msix_table =g_malloc0 (table_size); dev->msix_pba =g_malloc0 (pba_size); dev->msix_entry_used =g_malloc0 (nentries *sizeof *dev->msix_entry_used); + dev->msix_notify = pci_msix_notify;msix_mask_all (dev, nentries);
@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev) }/* Send an MSI-X message */ -void msix_notify (PCIDevice *dev,unsigned vector) +static void pci_msix_notify (PCIDevice *dev,unsigned vector) {MSIMessage msg;
@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)msi_send_message (dev, msg); } +void msix_notify (PCIDevice *dev,unsigned vector) +{ +if (dev->msix_notify) { + dev->msix_notify (dev, vector); + } +} +void msix_reset (PCIDevice *dev) {if (!msix_present (dev)) {
hw/remote/vfio-user-obj.c +89 -0
134 lines not on this page · raw message
hw/remote/trace-events +1 -0
@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64"" +vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
Signature · 1 line
2.20.1
PATCH v5 18/18 vfio-user: avocado tests for vfio-user
In reply to Jagannathan RamanAvocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object
--- MAINTAINERS | 1 + tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++ 2 files changed, 226 insertions(+) create mode 100644 tests/avocado/vfio-user.py
MAINTAINERS +1 -0
@@ -3488,6 +3488,7 @@ F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c +F: tests/avocado/vfio-user.py EBPF: M: Jason Wang <[email protected]>
tests/avocado/vfio-user.py new file +225 -0
225 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jagannathan RamanQuoted 7 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.
Fascinating, but I am not sure I understand. any examples?
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Quoted 29 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +0
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;
include/hw/pci/pci_bus.h struct PCIBus { · line 49
Quoted 10 lines
include/hw/pci/pci_bus.h +5
-0PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io; +/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io.
Are you sure address_space_memory & address_space_io are always global? even in the case of an iommu?
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 6 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients,
what are VM clients?
Quoted 78 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1
Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake
In reply to Jagannathan RamanQuoted 8 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Quoted 4 lines
Explicitly specify the C compiler for cmake to avoid this error
Signed-off-by Jagannathan Raman <[email protected]>Acked-by Paolo Bonzini <[email protected]>
This should not be needed anymore, as the bug in Meson has been fixed.
Paolo
Quoted 16 lines
configure | 2 ++ 1 file changed, 2 insertions(+)configure +2
-0@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross + echo "[cmake]" >> $cross + echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross echo "[binaries]" >> $cross echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake
Quoted 16 lines · Paolo Bonzini
Quoted 10 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.Explicitly specify the C compiler for cmake to avoid this error
Signed-off-by Jagannathan Raman <[email protected]>Acked-by Paolo Bonzini <[email protected]>
Quoted 1 line
This should not be needed anymore, as the bug in Meson has been fixed.
OK, will drop this patch.
Thank you!
Quoted 19 lines
Paolo
Quoted 15 lines
configure | 2 ++ 1 file changed, 2 insertions(+)configure +2
-0@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross + echo "[cmake]" >> $cross + echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross echo "[binaries]" >> $cross echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Michael S. TsirkinQuoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.
Quoted 3 lines
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Could you please provide some more details on the design?
Quoted 36 lines
Quoted 36 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;
Quoted 7 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 4 lines
include/hw/pci/pci_bus.h +4
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. Are you sure
address_space_memory&address_space_ioare always global? even in the case of an iommu?
On the CPU side of the Root Complex, I believe address_space_memory & address_space_io are global.
In the vfio-user case, devices on the same machine (TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.
Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svg
Quoted 9 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 7 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, what are VM clients?
It’s the client in the client - server model explained above.
Thank you!
--
Jag
Quoted 79 lines
Quoted 79 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1
Re: [PATCH v5 02/18] tests/avocado: Specify target VM argument to helper routines
In reply to Jagannathan RamanQuoted 11 lines · Jagannathan Raman
Specify target VM for
exec_commandandexec_command_and_wait_for_patternroutinesSigned-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Reviewed-by Philippe Mathieu-Daudé <[email protected]>Reviewed-by Beraldo Leal <[email protected]>--- tests/avocado/avocado_qemu/__init__.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jagannathan RamanQuoted 36 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +0
-0
include/hw/pci/pci_bus.h struct PCIBus { · line 44
Quoted 6 lines
include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;
This seems like a good point to rename address_space_mem, address_space_io, as well as PCIIORegion->address_space since they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr and PCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.
Quoted 6 lines
include/hw/pci/pci_bus.h +5
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. This allows the +* bus to be attached to CPUs from different machines. The
include/hw/pci/pci_bus.h struct PCIBus { · line 60
Quoted 10 lines
include/hw/pci/pci_bus.h +10
-0+* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, as such it needs the PCI buses in the same machine +* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io;
Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.
isol_as_io isn't used by this patch?
Quoted 73 lines
include/hw/pci/pci_bus.h +1
-0+ QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1
Re: [PATCH v5 04/18] pci: create and free isolated PCI buses
In reply to Jagannathan RamanQuoted 40 lines · Jagannathan Raman
Adds
pci_isol_bus_new()andpci_isol_bus_free()functions to manage creation and destruction of isolated PCI buses. Also addsqdev_get_busandqdev_put_buscallbacks to allow the choice of parent bus.Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/pci/pci.h | 4 + include/hw/qdev-core.h | 16 ++++ hw/pci/pci.c | 169 +++++++++++++++++++++++++++++++++++++++++ softmmu/qdev-monitor.c | 39 +++++++++- 4 files changed, 225 insertions(+), 3 deletions(-)include/hw/pci/pci.h +4
-0@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,PCIDevice *pci_vga_init (PCIBus *bus); +PCIBus *pci_isol_bus_new (BusState *parent_bus,const char *new_bus_type, +Error **errp); +bool pci_isol_bus_free (PCIBus *pci_bus,Error **errp); +static inline PCIBus *pci_get_bus (const PCIDevice *dev) {return PCI_BUS (qdev_get_parent_bus (DEVICE (dev)));include/hw/qdev-core.h +3
-0@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void ); +/** +* Find parent bus - these callbacks are used during device addition +* and deletion.
include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 434
Quoted 10 lines
include/hw/qdev-core.h +10
-0+* +* During addition, if no parent bus is specified in the options, +* these callbacks provide a way to figure it out based on the +* bus type. If these callbacks are not defined, defaults to +* finding the parent bus starting from default system bus +*/ +typedef bool (QDevGetBusFunc )(const char *type,BusState **bus,Error **errp); +typedef bool (QDevPutBusFunc )(BusState *bus,Error **errp); +bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus, +Error **errp);
Where is this used, it doesn't seem related to pci_isol_bus_new()?
Quoted 18 lines
include/hw/qdev-core.h +3
-0+/** * GpioPolarity: Polarity of a GPIO line * @@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);/*** BUS API. ***/ DeviceState *qdev_find_recursive (BusState *bus,const char *id); +BusState *qbus_find_recursive (BusState *bus,const char *name, +const char *bus_typename);/* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */ typedef int (qbus_walkerfn )(BusState *bus,void *opaque);hw/pci/pci.c +0
-0
hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 499
Quoted 7 lines
hw/pci/pci.c +4
-0@@ -493,6 +493,175 @@ void pci_root_bus_cleanup(PCIBus *bus)qbus_unrealize (BUS (bus)); } +static void pci_bus_free_isol_mem (PCIBus *pci_bus) +{ +if (pci_bus->address_space_mem) { +memory_region_unref (pci_bus->address_space_mem);
memory_region_unref() already does a NULL pointer check so the if statements in this function aren't needed.
Quoted 34 lines
hw/pci/pci.c +34
-0+ pci_bus->address_space_mem =NULL ; + } + +if (pci_bus->isol_as_mem) { +address_space_destroy (pci_bus->isol_as_mem); + pci_bus->isol_as_mem =NULL ; + } + +if (pci_bus->address_space_io) { +memory_region_unref (pci_bus->address_space_io); + pci_bus->address_space_io =NULL ; + } + +if (pci_bus->isol_as_io) { +address_space_destroy (pci_bus->isol_as_io); + pci_bus->isol_as_io =NULL ; + } +} + +static void pci_bus_init_isol_mem (PCIBus *pci_bus,uint32_t unique_id) +{ + g_autofree char *mem_mr_name = NULL; + g_autofree char *mem_as_name = NULL; + g_autofree char *io_mr_name = NULL; + g_autofree char *io_as_name = NULL; + +if (!pci_bus) { +return ; + } + + mem_mr_name =g_strdup_printf ("mem-mr-%u" , unique_id); + mem_as_name =g_strdup_printf ("mem-as-%u" , unique_id); + io_mr_name =g_strdup_printf ("io-mr-%u" , unique_id); + io_as_name =g_strdup_printf ("io-as-%u" , unique_id);
hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 543
Quoted 10 lines
hw/pci/pci.c +10
-0+ + pci_bus->address_space_mem =g_malloc0 (sizeof (MemoryRegion)); + pci_bus->isol_as_mem =g_malloc0 (sizeof (AddressSpace)); +memory_region_init (pci_bus->address_space_mem,NULL , + mem_mr_name,UINT64_MAX ); +address_space_init (pci_bus->isol_as_mem, + pci_bus->address_space_mem, mem_as_name); + + pci_bus->address_space_io =g_malloc0 (sizeof (MemoryRegion)); + pci_bus->isol_as_io =g_malloc0 (sizeof (AddressSpace));
Where are address_space_mem, isol_as_mem, address_space_io, and isol_as_io freed? I think the unref calls won't free them because the objects were created with object_initialize() instead of object_new().
Re: [PATCH v5 05/18] qdev: unplug blocker for devices
In reply to Jagannathan RamanQuoted 45 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/qdev-core.h | 5 +++++ softmmu/qdev-monitor.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)include/hw/qdev-core.h +5
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus,Error **errp); +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp); +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +4
-0@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); }
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996
Quoted 10 lines
softmmu/qdev-monitor.c +8
-0} +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp) +{ +ERRP_GUARD (); + +if (!migration_is_idle ()) { +error_setg (errp,"migration is in progress" ); +return -EBUSY ; + }
Why can this function not be called during migration?
Quoted 29 lines
softmmu/qdev-monitor.c +23
-0+ + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); + +return 0 ; +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;Signature · 1 line
2.20.1
Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
In reply to Jagannathan RamanQuoted 8 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/machine.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
Why is this code necessary? I expected the default hotplug behavior to pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.
Quoted 59 lines
hw/remote/machine.c +29
-0@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); +} + +static void remote_machine_pre_plug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hotplug" ); + } +} + +static void remote_machine_unplug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hot-unplug" ); +return ; + } + +qdev_unrealize (dev); }static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); mc->init = remote_machine_init; mc->desc ="Experimental remote machine" ; + + hc->pre_plug = remote_machine_pre_plug_cb; + hc->unplug = remote_machine_unplug_cb; }static const TypeInfo remote_machine = {@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = { .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), .class_init = remote_machine_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } };static void remote_machine_register_types (void )Signature · 1 line
2.20.1
Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
In reply to Jagannathan RamanQuoted 40 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)hw/remote/machine.c +25
-0@@ -22,6 +22,60 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +static bool remote_machine_get_bus (const char *type,BusState **bus, +Error **errp) +{ +ERRP_GUARD (); +RemoteMachineState *s =REMOTE_MACHINE (current_machine); +BusState *root_bus =NULL ; +PCIBus *new_pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (strcmp (type,TYPE_PCI_BUS ) &&strcmp (type,TYPE_PCI_BUS )) { +return true; + } + + root_bus =qbus_find_recursive (sysbus_get_default (),NULL ,TYPE_PCIE_BUS ); +if (!root_bus) { +error_setg (errp,"Unable to find root PCI device" ); +return false; + } + + new_pci_bus =pci_isol_bus_new (root_bus, type, errp); +if (!new_pci_bus) {
hw/remote/machine.c line 59
Quoted 10 lines
hw/remote/machine.c +10
-0+return false; + } + + *bus =BUS (new_pci_bus); + +pci_bus_irqs (new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +return true; +}
Can the user create the same PCI bus via QMP commands? If so, then this is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP device_add?
I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.
Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev
Unplug would work in the reverse order.
It may be more convenient to automatically create a PCIBus when the PCIDevice is hotplugged, but this kind of magic also has drawbacks (hidden devices, naming collisions, etc).
Quoted 35 lines
hw/remote/machine.c +22
-0+ +static bool remote_machine_put_bus (BusState *bus,Error **errp) +{ +PCIBus *pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (!object_dynamic_cast (OBJECT (bus),TYPE_PCI_BUS )) { +return true; + } + + pci_bus =PCI_BUS (bus); + +return pci_isol_bus_free (pci_bus, errp); +} +static void remote_machine_init (MachineState *machine) {MemoryRegion *system_memory, *system_io, *pci_memory;@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine) &s->iohub, REMOTE_IOHUB_NB_PIRQS);qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); + +qdev_set_bus_cbs (remote_machine_get_bus, remote_machine_put_bus, + &error_fatal); } static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,Signature · 1 line
2.20.1
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Stefan HajnocziQuoted 46 lines · Stefan Hajnoczi
Quoted 41 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;
Quoted 8 lines
include/hw/pci/pci_bus.h struct PCIBus { · line 44
Quoted 1 line
include/hw/pci/pci_bus.h +0
-0MemoryRegion *address_space_io;This seems like a good point to rename
address_space_mem,address_space_io, as well asPCIIORegion->address_spacesince they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr andPCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.
Are you referring to renaming address_space_mem, address_space_io and PCIIORegion->address_space alone? I’m asking because there are many other symbols in the code which are named similarly.
Quoted 18 lines
Quoted 7 lines
include/hw/pci/pci_bus.h +5
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. This allows the +* bus to be attached to CPUs from different machines. The include/hw/pci/pci_bus.h struct PCIBus { · line 60
Quoted 10 lines
include/hw/pci/pci_bus.h +10
-0+* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, as such it needs the PCI buses in the same machine +* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io;
Quoted 3 lines
Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.
I did start off with using these pointers unconditionally - but adopted an optional isolated address space for the following reasons:
- There is a potential for regression
- CPU address space per bus is not a common scenario. In most case, all PCI buses are attached to CPU sharing the same address space. Therefore, an optional address space made sense for this special scenario
We can also set it unconditionally if you prefer, kindly confirm.
Quoted 2 lines
isol_as_ioisn't used by this patch?
This patch introduces these variables, defines its getters and sets them to NULL in places where new PCI buses are presently created. The following patch creates a separate isolated address space:
[PATCH v5 04/18] pci: create and free isolated PCI buses
I could merge these patches if you prefer.
--
Jag
Quoted 74 lines
Quoted 74 lines
include/hw/pci/pci_bus.h +1
-0+ QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 55 lines · Jag Raman
Quoted 46 lines · Stefan Hajnoczi
Quoted 41 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;Quoted 8 lines
include/hw/pci/pci_bus.h struct PCIBus { · line 44
Quoted 1 line
include/hw/pci/pci_bus.h +0
-0MemoryRegion *address_space_io;This seems like a good point to rename
address_space_mem,address_space_io, as well asPCIIORegion->address_spacesince they are all MemoryRegions and not AddressSpaces. Names could be mem_space_mr/io_space_mr andPCIIORegion->container_mr. This avoids confusion with the actual AddressSpaces that are introduced in this patch.
Quoted 3 lines
Are you referring to renaming
address_space_mem,address_space_ioandPCIIORegion->address_spacealone? I’m asking because there are many other symbols in the code which are named similarly.
I only see those symbols in hw/pci/pci.c. Which ones were you thinking about?
Quoted 22 lines
Quoted 18 lines
Quoted 7 lines
include/hw/pci/pci_bus.h +5
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. This allows the +* bus to be attached to CPUs from different machines. The include/hw/pci/pci_bus.h struct PCIBus { · line 60
Quoted 10 lines
include/hw/pci/pci_bus.h +10
-0+* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, as such it needs the PCI buses in the same machine +* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io;Quoted 3 lines
Or use the pointers unconditionally and initialize them to the global address_space_memory/address_space_io? That might simplify the code so isolated address spaces is no longer a special case.
Quoted 8 lines
I did start off with using these pointers unconditionally - but adopted an optional isolated address space for the following reasons:
- There is a potential for regression
- CPU address space per bus is not a common scenario. In most case, all PCI buses are attached to CPU sharing the same address space. Therefore, an optional address space made sense for this special scenarioWe can also set it unconditionally if you prefer, kindly confirm.
It's a matter of taste. I don't have a strong opinion on it but personally I would try to make it unconditional. I think the risk of regressions is low and the code complexity will be lower than making it a special case. If you wanted to keep it as is, that's fine.
Quoted 10 lines
Quoted 3 lines
isol_as_ioisn't used by this patch?This patch introduces these variables, defines its getters and sets them to NULL in places where new PCI buses are presently created. The following patch creates a separate isolated address space:
[PATCH v5 04/18] pci: create and free isolated PCI busesI could merge these patches if you prefer.
The only place I saw that reads isol_as_io is "[PATCH v5 15/18]
vfio-user: handle PCI BAR accesses", but that's for PCI I/O Space accesses. Did I miss where I/O Space BARs are mapped into isol_as_io?
Stefan
Re: [PATCH v5 13/18] vfio-user: handle PCI config space accesses
In reply to Jagannathan RamanQuoted 9 lines · Jagannathan Raman
Define and register handlers for PCI config space accesses
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 47 insertions(+)
Re: [PATCH v5 09/18] vfio-user: define vfio-user-server object
In reply to Jagannathan RamanQuoted 3 lines · Jagannathan Raman
+/**
+ *VFU_OBJECT_ERROR- reports an error message. Ifauto_shutdown
+ * is set, it abort the machine on error. Otherwise, it logs an
s/abort/aborts/
Re: [PATCH v5 16/18] vfio-user: handle device interrupts
In reply to Jagannathan RamanQuoted 5 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>
Quoted 7 lines
--- include/hw/pci/pci.h | 6 +++ hw/pci/msi.c | 13 +++++- hw/pci/msix.c | 12 +++++- hw/remote/vfio-user-obj.c | 89 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 1 + 5 files changed, 119 insertions(+), 2 deletions(-)
Re: [PATCH v5 04/18] pci: create and free isolated PCI buses
In reply to Stefan HajnocziQuoted 42 lines · Stefan Hajnoczi
Quoted 37 lines · Jagannathan Raman
Adds
pci_isol_bus_new()andpci_isol_bus_free()functions to manage creation and destruction of isolated PCI buses. Also addsqdev_get_busandqdev_put_buscallbacks to allow the choice of parent bus.Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 4 + include/hw/qdev-core.h | 16 ++++ hw/pci/pci.c | 169 +++++++++++++++++++++++++++++++++++++++++ softmmu/qdev-monitor.c | 39 +++++++++- 4 files changed, 225 insertions(+), 3 deletions(-)include/hw/pci/pci.h +4
-0@@ -452,6 +452,10 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,PCIDevice *pci_vga_init (PCIBus *bus); +PCIBus *pci_isol_bus_new (BusState *parent_bus,const char *new_bus_type, +Error **errp); +bool pci_isol_bus_free (PCIBus *pci_bus,Error **errp); +static inline PCIBus *pci_get_bus (const PCIDevice *dev) {return PCI_BUS (qdev_get_parent_bus (DEVICE (dev)));include/hw/qdev-core.h +0
-0@@ -419,6 +419,20 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void );
Quoted 15 lines
include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 434
Quoted 13 lines
include/hw/qdev-core.h +13
-0+/** +* Find parent bus - these callbacks are used during device addition +* and deletion. +* +* During addition, if no parent bus is specified in the options, +* these callbacks provide a way to figure it out based on the +* bus type. If these callbacks are not defined, defaults to +* finding the parent bus starting from default system bus +*/ +typedef bool (QDevGetBusFunc )(const char *type,BusState **bus,Error **errp); +typedef bool (QDevPutBusFunc )(BusState *bus,Error **errp); +bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus, +Error **errp);Where is this used, it doesn't seem related to
pci_isol_bus_new()?
Yes, this is no directly related to pci_isol_bus_new() - will move it to a separate patch.
Quoted 22 lines
Quoted 22 lines
include/hw/qdev-core.h +3
-0+/** * GpioPolarity: Polarity of a GPIO line * @@ -691,6 +705,8 @@ BusState *qdev_get_parent_bus(DeviceState *dev);/*** BUS API. ***/ DeviceState *qdev_find_recursive (BusState *bus,const char *id); +BusState *qbus_find_recursive (BusState *bus,const char *name, +const char *bus_typename);/* Returns 0 to walk children, > 0 to skip walk, < 0 to terminate walk. */ typedef int (qbus_walkerfn )(BusState *bus,void *opaque);hw/pci/pci.c +0
-0@@ -493,6 +493,175 @@ void pci_root_bus_cleanup(PCIBus *bus)qbus_unrealize (BUS (bus)); }
Quoted 7 lines
hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 499
Quoted 4 lines
hw/pci/pci.c +4
-0+static void pci_bus_free_isol_mem (PCIBus *pci_bus) +{ +if (pci_bus->address_space_mem) { +memory_region_unref (pci_bus->address_space_mem);
memory_region_unref()already does a NULL pointer check so the if statements in this function aren't needed.
Got it, thank you!
Quoted 41 lines
Quoted 41 lines
hw/pci/pci.c +40
-0+ pci_bus->address_space_mem =NULL ; + } + +if (pci_bus->isol_as_mem) { +address_space_destroy (pci_bus->isol_as_mem); + pci_bus->isol_as_mem =NULL ; + } + +if (pci_bus->address_space_io) { +memory_region_unref (pci_bus->address_space_io); + pci_bus->address_space_io =NULL ; + } + +if (pci_bus->isol_as_io) { +address_space_destroy (pci_bus->isol_as_io); + pci_bus->isol_as_io =NULL ; + } +} + +static void pci_bus_init_isol_mem (PCIBus *pci_bus,uint32_t unique_id) +{ + g_autofree char *mem_mr_name = NULL; + g_autofree char *mem_as_name = NULL; + g_autofree char *io_mr_name = NULL; + g_autofree char *io_as_name = NULL; + +if (!pci_bus) { +return ; + } + + mem_mr_name =g_strdup_printf ("mem-mr-%u" , unique_id); + mem_as_name =g_strdup_printf ("mem-as-%u" , unique_id); + io_mr_name =g_strdup_printf ("io-mr-%u" , unique_id); + io_as_name =g_strdup_printf ("io-as-%u" , unique_id); + + pci_bus->address_space_mem =g_malloc0 (sizeof (MemoryRegion)); + pci_bus->isol_as_mem =g_malloc0 (sizeof (AddressSpace)); +memory_region_init (pci_bus->address_space_mem,NULL , + mem_mr_name,UINT64_MAX ); + address_space_init(pci_bus->isol_as_mem,
Quoted 8 lines
hw/pci/pci.c void pci_root_bus_cleanup(PCIBus *bus) · line 543
Quoted 4 lines
hw/pci/pci.c +4
-0+ pci_bus->address_space_mem, mem_as_name); + + pci_bus->address_space_io =g_malloc0 (sizeof (MemoryRegion)); + pci_bus->isol_as_io =g_malloc0 (sizeof (AddressSpace));Where are
address_space_mem,isol_as_mem,address_space_io, andisol_as_iofreed? I think the unref calls won't free them because the objects were created withobject_initialize()instead ofobject_new().
Ah OK got it, thank you! Will fix it.
I think we could set the owner of the the memory regions to the PCI bus, as opposed to NULL. We could also add an ‘instance_finalize’ function to PCI bus which would free them.
--
Jag
Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Jagannathan RamanQuoted 1 line · Jagannathan Raman
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 18 lines
hw/remote/vfio-user-obj.c +18
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file);
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } + +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {
When is this counter reset so migration can be tried again if it fails/cancels?
Quoted 7 lines
hw/remote/vfio-user-obj.c +7
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ;
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size); + + o->vfu_mig_buf_pending -= size;
This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely update vfu_mig_buf_pending when offset jumps around arbitrarily between calls.
Quoted 8 lines
migration/vmstate.c +8
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ;
migration/vmstate.c static int vmstate_size(void *opaque, const VMStateField *field) · line 83
Quoted 10 lines
migration/vmstate.c +10
-0+ } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } + +return size; +}
This function looks incorrect because it ignores subsections as well as runtime behavior during save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile at save() time so there's no way of knowing the size ahead of time.
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
Re: [PATCH v5 05/18] qdev: unplug blocker for devices
In reply to Stefan HajnocziQuoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/qdev-core.h| 5 +++++softmmu/qdev-monitor.c| 35 +++++++++++++++++++++++++++++++++++2 files changed, 40 insertions(+)include/hw/qdev-core.h +5
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus,Error **errp); +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp); +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +4
-0@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } }
Quoted 10 lines
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996
Quoted 8 lines
softmmu/qdev-monitor.c +8
-0+int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp) +{ +ERRP_GUARD (); + +if (!migration_is_idle ()) { +error_setg (errp,"migration is in progress" ); +return -EBUSY ; + }Why can this function not be called during migration?
Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.
I did weigh the following reasons against adding this check:
- unplug_blockers is not migrated to the destination anyway, so it doesn’t matter if it changes after migration starts
- whichever code/object that needs to add the blocker could add it at the destination if needed
However, unlike qmp_device_add(), qmp_object_add() doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.
Would you prefer to throw a warning, or fully remove this test?
--
Jag
Quoted 30 lines
Quoted 30 lines
softmmu/qdev-monitor.c +23
-0+ + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); + +return 0 ; +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;Signature · 1 line
2.20.1
Re: [PATCH v5 10/18] vfio-user: instantiate vfio-user context
In reply to Jagannathan RamanQuoted 33 lines · Jagannathan Raman
create a context with the vfio-user library to run a PCI device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 78 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+)hw/remote/vfio-user-obj.c +7
-0@@ -41,6 +41,9 @@#include "hw/remote/machine.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qemu/notify.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -82,13 +85,23 @@ struct VfuObject {char *device;Error *err; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; };
hw/remote/vfio-user-obj.c struct VfuObject { · line 103
Quoted 10 lines
hw/remote/vfio-user-obj.c +5
-0+static void vfu_object_init_ctx (VfuObject *o,Error **errp); +static void vfu_object_set_socket (Object *obj,Visitor *v,const char *name,void *opaque,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +return ; + }
No error?
Quoted 9 lines
hw/remote/vfio-user-obj.c +2
-0+qapi_free_SocketAddress (o->socket); o->socket =NULL ;@@ -104,17 +117,68 @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name, }trace_vfu_prop ("socket" , o->socket->u.q_unix.path); +
hw/remote/vfio-user-obj.c static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name, · line 130
Quoted 10 lines
hw/remote/vfio-user-obj.c +4
-0+vfu_object_init_ctx (o, errp); }static void vfu_object_set_device (Object *obj,const char *str,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +return ; + }
No error?
Quoted 92 lines
hw/remote/vfio-user-obj.c +60
-0+g_free (o->device); o->device =g_strdup (str);trace_vfu_prop ("device" , str); + +vfu_object_init_ctx (o, errp); +} + +/* +* TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' +* properties. It also depends on devices instantiated in QEMU. These +* dependencies are not available during the instance_init phase of this +* object's life-cycle. As such, the server is initialized after the +* machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT +* when the machine is setup, and the dependencies are available. +*/ +static void vfu_object_machine_done (Notifier *notifier,void *data) +{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); +Error *err =NULL ; + +vfu_object_init_ctx (o, &err); + +if (err) { +error_propagate (&error_abort, err); + } +} + +static void vfu_object_init_ctx (VfuObject *o,Error **errp) +{ +ERRP_GUARD (); + +if (o->vfu_ctx || !o->socket || !o->device || + !phase_check (PHASE_MACHINE_READY )) { +return ; + } + +if (o->err) { +error_propagate (errp, o->err); + o->err =NULL ; +return ; + } + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket->u.q_unix.path,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno)); +return ; + } }static void vfu_object_init (Object *obj)@@ -124,6 +188,11 @@ static void vfu_object_init(Object *obj) k->nr_devs++; +if (!phase_check (PHASE_MACHINE_READY )) { + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); + } +if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) {error_setg (&o->err,"vfu: %s only compatible with %s machine" ,TYPE_VFU_OBJECT ,TYPE_REMOTE_MACHINE );@@ -142,6 +211,10 @@ static void vfu_object_finalize(Object *obj) o->socket =NULL ; +if (o->vfu_ctx) { +vfu_destroy_ctx (o->vfu_ctx); + } +g_free (o->device); o->device =NULL ;@@ -149,6 +222,11 @@ static void vfu_object_finalize(Object *obj)if (!k->nr_devs && k->auto_shutdown) {qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); } + +if (o->machine_done.notify) { +qemu_remove_machine_init_done_notifier (&o->machine_done); + o->machine_done.notify =NULL ; + } }static void vfu_object_class_init (ObjectClass *klass,void *data)Signature · 1 line
2.20.1
Re: [PATCH v5 11/18] vfio-user: find and init PCI device
In reply to Jagannathan RamanQuoted 65 lines · Jagannathan Raman
Find the PCI device with specified id. Initialize the device context with the QEMU PCI device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 60 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)hw/remote/vfio-user-obj.c +26
-0@@ -44,6 +44,8 @@#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" +#include "hw/qdev-core.h" +#include "hw/pci/pci.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -89,6 +91,10 @@ struct VfuObject {Notifier machine_done;vfu_ctx_t *vfu_ctx; + +PCIDevice *pci_dev; + +Error *unplug_blocker; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)static void vfu_object_init_ctx (VfuObject *o,Error **errp) {ERRP_GUARD (); +DeviceState *dev =NULL ; +vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +int ret;if (o->vfu_ctx || !o->socket || !o->device || !phase_check (PHASE_MACHINE_READY )) {@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->device); +if (dev ==NULL ) { +error_setg (errp,"vfu: Device %s not found" , o->device); + goto fail; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"vfu: %s not a PCI device" , o->device); + goto fail; + } + + o->pci_dev =PCI_DEVICE (dev); + +if (pci_is_express (o->pci_dev)) { + pci_type =VFU_PCI_TYPE_EXPRESS ; + }
hw/remote/vfio-user-obj.c static void vfu_object_init_ctx(VfuObject *o, Error **errp) · line 217
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ + ret =vfu_pci_init (o->vfu_ctx, pci_type,PCI_HEADER_TYPE_NORMAL ,0 ); +if (ret <0 ) { +error_setg (errp, +"vfu: Failed to attach PCI device %s to context - %s" , + o->device,strerror (errno)); + goto fail; + } + +error_setg (&o->unplug_blocker,"%s is in use" , o->device);
More detailed error message:
"x-vfio-user-server for %s must be deleted before unplugging"
Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.
This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.
The current approach adds special code to hw/pci/pci.c so that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.
Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying hw/pci/pci.c with a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely on qdev_find_recursive() and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).
I wanted to raise this to see if Peter, Kevin, Michael, and others are happy with the current approach or have ideas for a clean solution.
Stefan
Re: [PATCH v5 12/18] vfio-user: run vfio-user context
In reply to Jagannathan RamanQuoted 20 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- qapi/misc.json | 23 ++++++++++ hw/remote/vfio-user-obj.c | 90 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 112 insertions(+), 1 deletion(-)qapi/misc.json +0
-0@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true }
qapi/misc.json line 539
Quoted 10 lines
qapi/misc.json +10
-0+ +## +# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel +# +# @device: ID of attached PCI device +# +# @path: path of the socket
This assumes a UNIX domain socket path was given. It doesn't work well with file descriptor passing. The x-vfio-user-server is an object with a unique QEMU Object Model path (the last path component is its id). You can get the id like this:
object_get_canonical_path_component(OBJECT(o))
I suggest dropping @path and including the server object's id instead.
Quoted 59 lines
qapi/misc.json +13
-0+# +# Since: 6.3 +# +# Example: +# +# <- { "event": "VFU_CLIENT_HANGUP", +# "data": { "device": "lsi1", +# "path": "/tmp/vfu1-sock" }, +# "timestamp": { "seconds": 1265044230, "microseconds": 450486 } } +# +## +{ 'event': 'VFU_CLIENT_HANGUP', + 'data': { 'device': 'str', 'path': 'str' } }hw/remote/vfio-user-obj.c +12
-0@@ -27,6 +27,9 @@* * device - id of a device on the server, a required option. PCI devices * alone are supported presently. +* +* notes - x-vfio-user-server could block IO and monitor during the +* initialization phase. */ #include "qemu/osdep.h" @@ -41,11 +44,14 @@#include "hw/remote/machine.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qapi/qapi-events-misc.h" #include "qemu/notify.h" +#include "qemu/thread.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" #include "hw/qdev-core.h" #include "hw/pci/pci.h" +#include "qemu/timer.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -95,6 +101,8 @@ struct VfuObject {PCIDevice *pci_dev;Error *unplug_blocker; + +int vfu_poll_fd; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);@@ -144,6 +152,68 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)vfu_object_init_ctx (o, errp); } +static void vfu_object_ctx_run (void *opaque) +{ +VfuObject *o = opaque; +int ret =-1 ;
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 168
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+ +while (ret !=0 ) { + ret =vfu_run_ctx (o->vfu_ctx); +if (ret <0 ) { +if (errno ==EINTR ) { +continue ; + }else if (errno ==ENOTCONN ) { +qapi_event_send_vfu_client_hangup (o->device, + o->socket->u.q_unix.path); +qemu_set_fd_handler (o->vfu_poll_fd,NULL ,NULL ,NULL );
Do we also stop monitoring o->vfu_poll_fd when object-del is used to delete the x-vfio-user-server object?
Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
In reply to Stefan HajnocziQuoted 7 lines · Stefan Hajnoczi
Quoted 2 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Quoted 8 lines
Quoted 6 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 29 +++++++++++++++++++++++++++++1 file changed, 29 insertions(+)Why is this code necessary? I expected the default hotplug behavior to
I just discovered that TYPE_REMOTE_MACHINE wasn't setting up a hotplug handler for the root PCI bus.
Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.
We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.
Quoted 3 lines
pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.
OK, that makes sense. We don’t need to test the device type during plug and unplug.
Therefore, I don’t think we need a callback for the plug operation. We could set HotplugHandlerClass->unplug callback to the default qdev_simple_device_unplug_cb() callback.
—
Jag
Quoted 60 lines
Quoted 60 lines
hw/remote/machine.c +29
-0@@ -54,14 +54,39 @@ static void remote_machine_init(MachineState *machine)pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); +} + +static void remote_machine_pre_plug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hotplug" ); + } +} + +static void remote_machine_unplug_cb (HotplugHandler *hotplug_dev, +DeviceState *dev,Error **errp) +{ +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"Only allowing PCI hot-unplug" ); +return ; + } + +qdev_unrealize (dev); }static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); mc->init = remote_machine_init; mc->desc ="Experimental remote machine" ; + + hc->pre_plug = remote_machine_pre_plug_cb; + hc->unplug = remote_machine_unplug_cb; }static const TypeInfo remote_machine = {@@ -69,6 +94,10 @@ static const TypeInfo remote_machine = { .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), .class_init = remote_machine_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } };static void remote_machine_register_types (void )Signature · 1 line
2.20.1
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jag RamanQuoted 15 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Quoted 7 lines
Hi Michael!
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Dave
Quoted 163 lines
In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.
Quoted 3 lines
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Could you please provide some more details on the design?
Quoted 36 lines
Quoted 36 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;Quoted 7 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 4 lines
include/hw/pci/pci_bus.h +4
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. Are you sure
address_space_memory&address_space_ioare always global? even in the case of an iommu?On the CPU side of the Root Complex, I believe
address_space_memory&address_space_ioare global.In the vfio-user case, devices on the same machine (
TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svgQuoted 9 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 7 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, what are VM clients?
It’s the client in the client - server model explained above.
Thank you!
--
JagQuoted 79 lines
Quoted 79 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
In reply to Stefan HajnocziQuoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 57 +++++++++++++++++++++++++++++++++++++++++++++1 file changed, 57 insertions(+)hw/remote/machine.c +29
-0@@ -22,6 +22,60 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +static bool remote_machine_get_bus (const char *type,BusState **bus, +Error **errp) +{ +ERRP_GUARD (); +RemoteMachineState *s =REMOTE_MACHINE (current_machine); +BusState *root_bus =NULL ; +PCIBus *new_pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (strcmp (type,TYPE_PCI_BUS ) &&strcmp (type,TYPE_PCI_BUS )) { +return true; + } + + root_bus =qbus_find_recursive (sysbus_get_default (),NULL ,TYPE_PCIE_BUS ); +if (!root_bus) { +error_setg (errp,"Unable to find root PCI device" ); +return false; + } + + new_pci_bus =pci_isol_bus_new (root_bus, type, errp); +if (!new_pci_bus) { +return false; + } + + *bus =BUS (new_pci_bus);
Quoted 8 lines
hw/remote/machine.c line 59
Quoted 6 lines
hw/remote/machine.c +6
-0+ +pci_bus_irqs (new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +return true; +}Can the user create the same PCI bus via QMP commands? If so, then this
I think there is a way we could achieve it.
When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices like TYPE_PCI_BRIDGE which create a secondary PCI bus.
Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.
Quoted 9 lines
is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP
device_add?I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.
Quoted 6 lines
Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1.device_add...some PCI bus stuff here...,id=isol-pci-bus0
2.device_add...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev
We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.
Quoted 6 lines
Unplug would work in the reverse order.
It may be more convenient to automatically create a PCIBus when the PCIDevice is hotplugged, but this kind of magic also has drawbacks (hidden devices, naming collisions, etc).
OK, makes sense.
--
Jag
Quoted 36 lines
Quoted 36 lines
hw/remote/machine.c +22
-0+ +static bool remote_machine_put_bus (BusState *bus,Error **errp) +{ +PCIBus *pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (!object_dynamic_cast (OBJECT (bus),TYPE_PCI_BUS )) { +return true; + } + + pci_bus =PCI_BUS (bus); + +return pci_isol_bus_free (pci_bus, errp); +} +static void remote_machine_init (MachineState *machine) {MemoryRegion *system_memory, *system_io, *pci_memory;@@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine) &s->iohub, REMOTE_IOHUB_NB_PIRQS);qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); + +qdev_set_bus_cbs (remote_machine_get_bus, remote_machine_put_bus, + &error_fatal); } static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,Signature · 1 line
2.20.1
Re: [PATCH v5 11/18] vfio-user: find and init PCI device
In reply to Stefan HajnocziQuoted 75 lines · Stefan Hajnoczi
Quoted 70 lines · Jagannathan Raman
Find the PCI device with specified id. Initialize the device context with the QEMU PCI device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/vfio-user-obj.c| 60 +++++++++++++++++++++++++++++++++++++++1 file changed, 60 insertions(+)hw/remote/vfio-user-obj.c +31
-0@@ -44,6 +44,8 @@#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" +#include "hw/qdev-core.h" +#include "hw/pci/pci.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -89,6 +91,10 @@ struct VfuObject {Notifier machine_done;vfu_ctx_t *vfu_ctx; + +PCIDevice *pci_dev; + +Error *unplug_blocker; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);@@ -161,6 +167,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)static void vfu_object_init_ctx (VfuObject *o,Error **errp) {ERRP_GUARD (); +DeviceState *dev =NULL ; +vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +int ret;if (o->vfu_ctx || !o->socket || !o->device || !phase_check (PHASE_MACHINE_READY )) {@@ -179,6 +188,49 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->device); +if (dev ==NULL ) { +error_setg (errp,"vfu: Device %s not found" , o->device); + goto fail; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"vfu: %s not a PCI device" , o->device); + goto fail; + } + + o->pci_dev =PCI_DEVICE (dev); + +if (pci_is_express (o->pci_dev)) { + pci_type =VFU_PCI_TYPE_EXPRESS ; + } + + ret =vfu_pci_init (o->vfu_ctx, pci_type,PCI_HEADER_TYPE_NORMAL ,0 ); +if (ret <0 ) { + error_setg(errp, + "vfu: Failed to attach PCI device %s to context - %s",
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_init_ctx(VfuObject *o, Error **errp) · line 217
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+ o->device, strerror(errno)); + goto fail; + } + +error_setg (&o->unplug_blocker,"%s is in use" , o->device);More detailed error message:
"x-vfio-user-server for %s must be deleted before unplugging"
Got it, thank you!
--
Jag
Quoted 2 lines
Reviewed-by Stefan Hajnoczi <[email protected]>
Re: [PATCH v5 12/18] vfio-user: run vfio-user context
In reply to Stefan HajnocziQuoted 41 lines · Stefan Hajnoczi
Quoted 20 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
qapi/misc.json| 23 ++++++++++hw/remote/vfio-user-obj.c| 90 ++++++++++++++++++++++++++++++++++++++-2 files changed, 112 insertions(+), 1 deletion(-)qapi/misc.json +0
-0@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true }qapi/misc.json line 539
Quoted 10 lines
qapi/misc.json +10
-0+ +## +# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel +# +# @device: ID of attached PCI device +# +# @path: path of the socketThis assumes a UNIX domain socket path was given. It doesn't work well with file descriptor passing. The x-vfio-user-server is an object with a unique QEMU Object Model path (the last path component is its id). You can get the id like this:
Quoted 1 line
object_get_canonical_path_component(OBJECT(o))
I was also wondering how to get the object ID. Thank you for the pointer!
Quoted 2 lines
I suggest dropping @path and including the server object's id instead.
OK, will do.
Quoted 65 lines
Quoted 65 lines
qapi/misc.json +13
-0+# +# Since: 6.3 +# +# Example: +# +# <- { "event": "VFU_CLIENT_HANGUP", +# "data": { "device": "lsi1", +# "path": "/tmp/vfu1-sock" }, +# "timestamp": { "seconds": 1265044230, "microseconds": 450486 } } +# +## +{ 'event': 'VFU_CLIENT_HANGUP', + 'data': { 'device': 'str', 'path': 'str' } }hw/remote/vfio-user-obj.c +17
-0@@ -27,6 +27,9 @@* * device - id of a device on the server, a required option. PCI devices * alone are supported presently. +* +* notes - x-vfio-user-server could block IO and monitor during the +* initialization phase. */ #include "qemu/osdep.h" @@ -41,11 +44,14 @@#include "hw/remote/machine.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qapi/qapi-events-misc.h" #include "qemu/notify.h" +#include "qemu/thread.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" #include "hw/qdev-core.h" #include "hw/pci/pci.h" +#include "qemu/timer.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -95,6 +101,8 @@ struct VfuObject {PCIDevice *pci_dev;Error *unplug_blocker; + +int vfu_poll_fd; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);@@ -144,6 +152,68 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)vfu_object_init_ctx (o, errp); } +static void vfu_object_ctx_run (void *opaque) +{ +VfuObject *o = opaque; +int ret =-1 ; + +while (ret !=0 ) { + ret =vfu_run_ctx (o->vfu_ctx); +if (ret <0 ) { +if (errno ==EINTR ) {
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 168
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+continue ; + }else if (errno ==ENOTCONN ) { +qapi_event_send_vfu_client_hangup (o->device, + o->socket->u.q_unix.path); +qemu_set_fd_handler (o->vfu_poll_fd,NULL ,NULL ,NULL );Do we also stop monitoring
o->vfu_poll_fdwhen object-del is used to delete the x-vfio-user-server object?
Yes, we should to stop monitoring the o->vfu_poll_fd during object-del. Will do so.
--
Jag
Re: [PATCH v5 18/18] vfio-user: avocado tests for vfio-user
In reply to Jagannathan RamanHi Jagannathan,
Quoted 14 lines · Jagannathan Raman
Avocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- MAINTAINERS | 1 + tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++ 2 files changed, 226 insertions(+)create mode 100644
tests/avocado/vfio-user.pytests/avocado/vfio-user.py new file +2
-0+class VfioUser(QemuSystemTest): + """
tests/avocado/vfio-user.py line 30
Quoted 10 lines
tests/avocado/vfio-user.py new file +10
-0+ :avocado: tags=vfiouser + """ + KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' + timeout = 20 + + @staticmethod + def migration_finished(vm): + res = vm.command('query-migrate') + if 'status' in res: + return res['status'] in ('completed', 'failed')
Do we need to check for failed migration in do_test_migrate()?
Quoted 2 lines
tests/avocado/vfio-user.py new file +2
-0+ else: + return False
[...]
tests/avocado/vfio-user.py line 67
Quoted 8 lines
tests/avocado/vfio-user.py new file +8
-0+ def launch_server_hotplug(self, socket): + server_vm = self.get_vm() + server_vm.add_args('-machine', 'x-remote') + server_vm.add_args('-nodefaults') + server_vm.add_args('-device', 'lsi53c895a,id=lsi1') + server_vm.launch() + server_vm.command('human-monitor-command', + command_line='object_add x-vfio-user-server,'
Why not use qmp('object-add', ...) directly?
tests/avocado/vfio-user.py line 70
Quoted 3 lines
tests/avocado/vfio-user.py new file +3
-0+ 'id=vfioobj,socket.type=unix,' + 'socket.path='+socket+',device=lsi1') + return server_vm
Otherwise LGTM.
Quoted 7 lines · Stefan Hajnoczi
Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.
Thank you for taking the time to review the patches, Stefan!
Quoted 7 lines
This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.
Quoted 8 lines
The current approach adds special code to
hw/pci/pci.cso that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.
We are discussing the automatic creation part you just mentioned in the following email:
[PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
I agree that automatic creation of a parent bus is not ideal - we could specify the parent bus as a separate option in the command-line or QMP. This change would avoid modification to hw/pci/pci.c - the new PCI bus could be created inplace during device creation/hotplug.
The following image gives an idea of the bus/device topology in the remote machine, as implemented in the current series. Each secondary bus and its children have isolated memory and IO spaces.
https://gitlab.com/jraman/qemu/-/commit/2e2ebf004894075ad8044739b0b16ce875114c4c
Quoted 10 lines
Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying
hw/pci/pci.cwith a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely onqdev_find_recursive()and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).
Based on further discussion with Stefan, I got some clarity. We could consider one more option as well - somewhere in-between multiple root SysBuses and the topology discussed above (with secondary PCI buses). We could implement a TYPE_SYS_BUS_DEVICE that creates a root PCI bus with isolated memory ranges. Something along the lines in the following diagram:
https://gitlab.com/jraman/qemu/-/commit/81f6a998278a2a795be0db7acdeb1caa2d6744fb
An example set of QMP commands to attach PCI devices would be:device_add pci-root-bus,id=rb1device_add <driver>,id=mydev,bus=rb1
object-add x-vfio-user-server,device=mydev
where ‘pci-root-bus’ is a TYPE_SYS_BUS_DEVICE that creates its own root PCI bus.
I’m very sorry if the above two web links disturb your review workflow.
Quoted 3 lines
I wanted to raise this to see if Peter, Kevin, Michael, and others are happy with the current approach or have ideas for a clean solution.
Looking forward to your comments.
Thank you!
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Dr. David Alan GilbertQuoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
Thank you!
--
Jag
Quoted 169 lines
Dave
Quoted 163 lines
In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.
Quoted 3 lines
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Could you please provide some more details on the design?
Quoted 36 lines
Quoted 36 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;Quoted 7 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 4 lines
include/hw/pci/pci_bus.h +4
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. Are you sure
address_space_memory&address_space_ioare always global? even in the case of an iommu?On the CPU side of the Root Complex, I believe
address_space_memory&address_space_ioare global.In the vfio-user case, devices on the same machine (
TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svgQuoted 9 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 7 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, what are VM clients?
It’s the client in the client - server model explained above.
Thank you!
--
JagQuoted 79 lines
Quoted 79 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 05/18] qdev: unplug blocker for devices
In reply to Jag RamanQuoted 63 lines · Jag Raman
Quoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/qdev-core.h| 5 +++++softmmu/qdev-monitor.c| 35 +++++++++++++++++++++++++++++++++++2 files changed, 40 insertions(+)include/hw/qdev-core.h +5
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus,Error **errp); +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp); +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +4
-0@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } }Quoted 10 lines
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996
Quoted 8 lines
softmmu/qdev-monitor.c +8
-0+int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp) +{ +ERRP_GUARD (); + +if (!migration_is_idle ()) { +error_setg (errp,"migration is in progress" ); +return -EBUSY ; + }Why can this function not be called during migration?
Quoted 6 lines
Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.
I did weigh the following reasons against adding this check:
-unplug_blockersis not migrated to the destination anyway, so it doesn’t matter if it changes after migration starts
Yes.
Quoted 2 lines
- whichever code/object that needs to add the blocker could add it at the destination if needed
Yes.
Quoted 5 lines
However, unlike
qmp_device_add(),qmp_object_add()doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.Would you prefer to throw a warning, or fully remove this test?
Adding an unplug blocker during migration seems safe to me. I would remove this test.
Stefan
Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
In reply to Jag RamanQuoted 14 lines · Jag Raman
Quoted 14 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 29 +++++++++++++++++++++++++++++1 file changed, 29 insertions(+)
Quoted 7 lines
Quoted 1 line
Why is this code necessary? I expected the default hotplug behavior to
I just discovered that
TYPE_REMOTE_MACHINEwasn't setting up a hotplug handler for the root PCI bus.Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.
Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.
Quoted 8 lines
We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.
Quoted 3 lines
pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.
Quoted 6 lines
OK, that makes sense. We don’t need to test the device type during plug and unplug.
Therefore, I don’t think we need a callback for the plug operation. We could set
HotplugHandlerClass->unplugcallback to the defaultqdev_simple_device_unplug_cb()callback.
Great!
Stefan
Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
In reply to Jag RamanQuoted 60 lines · Jag Raman
Quoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 57 +++++++++++++++++++++++++++++++++++++++++++++1 file changed, 57 insertions(+)hw/remote/machine.c +29
-0@@ -22,6 +22,60 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +static bool remote_machine_get_bus (const char *type,BusState **bus, +Error **errp) +{ +ERRP_GUARD (); +RemoteMachineState *s =REMOTE_MACHINE (current_machine); +BusState *root_bus =NULL ; +PCIBus *new_pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (strcmp (type,TYPE_PCI_BUS ) &&strcmp (type,TYPE_PCI_BUS )) { +return true; + } + + root_bus =qbus_find_recursive (sysbus_get_default (),NULL ,TYPE_PCIE_BUS ); +if (!root_bus) { +error_setg (errp,"Unable to find root PCI device" ); +return false; + } + + new_pci_bus =pci_isol_bus_new (root_bus, type, errp); +if (!new_pci_bus) { +return false; + } + + *bus =BUS (new_pci_bus);Quoted 8 lines
hw/remote/machine.c line 59
Quoted 6 lines
hw/remote/machine.c +6
-0+ +pci_bus_irqs (new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +return true; +}Can the user create the same PCI bus via QMP commands? If so, then this
I think there is a way we could achieve it.
Quoted 8 lines
When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the
TYPE_LSI53C895Adevice creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices likeTYPE_PCI_BRIDGEwhich create a secondary PCI bus.Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.
Exactly. device_add instantiates DeviceStates, not busses, so there needs to be a parent device like a SCSI controller, a PCI bridge, etc that owns and creates the bus.
Quoted 16 lines
Quoted 9 lines
is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP
device_add?I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.Quoted 6 lines
Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1.device_add...some PCI bus stuff here...,id=isol-pci-bus0
2.device_add...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev
Quoted 2 lines
We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.
I wonder if a new device is needed or whether it's possible to add an isol_as=on|off (default: off) option to an existing PCI bridge/expander? Hopefully most of the code is already there.
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jag RamanQuoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.
Stefan
Quoted 35 lines · Jag Raman
Quoted 7 lines · Stefan Hajnoczi
Hi Jag,
Thanks for this latest revision. The biggest outstanding question I have is about the isolated address spaces design.Thank you for taking the time to review the patches, Stefan!
Quoted 7 lines
This patch series needs a PCIBus with its own Memory Space, I/O Space, and interrupts. That way a single QEMU process can host vfio-user servers that different VMs connect to. They all need isolated address spaces so that mapping a BAR in Device A does not conflict with mapping a BAR in Device B.
Quoted 8 lines
The current approach adds special code to
hw/pci/pci.cso that custom AddressSpace can be set up. The isolated PCIBus is an automatically created PCIe root port that's a child of the machine's main PCI bus. On one hand it's neat because QEMU's assumption that there is only one root SysBus isn't violated. On the other hand it seems like a special case hack for PCI and I'm not sure in what sense these PCIBusses are really children of the machine's main PCI bus since they don't share or interact in any way.We are discussing the automatic creation part you just mentioned in the following email:
[PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machineI agree that automatic creation of a parent bus is not ideal - we could specify the parent bus as a separate option in the command-line or QMP. This change would avoid modification to
hw/pci/pci.c- the new PCI bus could be created inplace during device creation/hotplug.
Quoted 4 lines
The following image gives an idea of the bus/device topology in the remote machine, as implemented in the current series. Each secondary bus and its children have isolated memory and IO spaces.
https://gitlab.com/jraman/qemu/-/commit/2e2ebf004894075ad8044739b0b16ce875114c4c
Do isolated PCI busses have any relationship with their parent at all? I think the parent plays a useful role in DMA/IOMMU, interrupts, or PCI addressing. That leaves me wondering if a parent/child relationship is an appropriate way to model this.
That said, this approach solves two problems:
1. There must be some parent for the new PCI bus.
2. qdev_find_recursive() and friends must be able to find the PCIDevice on the isolated bus.
Quoted 17 lines
Quoted 9 lines
Another approach that came to mind is to allow multiple root SysBusses. Each vfio-user server would need its own SysBus and put a regular PCI host onto that isolated SysBus without modifying
hw/pci/pci.cwith a special case. The downside to this is that violating the single SysBus assumption probably breaks monitor commands that rely onqdev_find_recursive()and friends. It seems cleaner than adding isolated address spaces to PCI specifically, but also raises the question if multiple machine instances are needed (which would raise even more questions).Based on further discussion with Stefan, I got some clarity. We could consider one more option as well - somewhere in-between multiple root SysBuses and the topology discussed above (with secondary PCI buses). We could implement a
TYPE_SYS_BUS_DEVICEthat creates a root PCI bus with isolated memory ranges. Something along the lines in the following diagram:
https://gitlab.com/jraman/qemu/-/commit/81f6a998278a2a795be0db7acdeb1caa2d6744fb
Quoted 6 lines
An example set of QMP commands to attach PCI devices would be:
device_addpci-root-bus,id=rb1device_add<driver>,id=mydev,bus=rb1
object-add x-vfio-user-server,device=mydevwhere ‘pci-root-bus’ is a
TYPE_SYS_BUS_DEVICEthat creates its own root PCI bus.
If it's less code then that's an advantage but it still places unrelated DMA/interrupt spaces onto the same SysBus and therefore requires isolation. I think this alternative doesn't fundamentally fix the design.
If multiple roots are possible then isolation doesn't need to be implemented explicitly, it comes for free as part of the regular qdev/qbus hierarchy. The devices would be isolated by the fact that they live on different roots :).
I have never tried the multi-root approach though, so I'm not sure how much work it is.
Stefan
Re: [PATCH v5 18/18] vfio-user: avocado tests for vfio-user
In reply to Philippe Mathieu-Daudé viaQuoted 18 lines · Philippe Mathieu-Daudé
Hi Jagannathan,
Quoted 10 lines · Jagannathan Raman
Avocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- MAINTAINERS | 1 + tests/avocado/vfio-user.py | 225 +++++++++++++++++++++++++++++++++++++ 2 files changed, 226 insertions(+) create mode 100644 tests/avocado/vfio-user.py
Quoted 14 lines
tests/avocado/vfio-user.py line 30
Quoted 12 lines
tests/avocado/vfio-user.py new file +12
-0+class VfioUser(QemuSystemTest): + """ + :avocado: tags=vfiouser + """ + KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' + timeout = 20 + + @staticmethod + def migration_finished(vm): + res = vm.command('query-migrate') + if 'status' in res: + return res['status'] in ('completed', 'failed')Do we need to check for failed migration in
do_test_migrate()?
OK, will do.
Quoted 15 lines
Quoted 3 lines
tests/avocado/vfio-user.py new file +2
-0+ else: + return False[...]
tests/avocado/vfio-user.py line 67
Quoted 8 lines
tests/avocado/vfio-user.py new file +8
-0+ def launch_server_hotplug(self, socket): + server_vm = self.get_vm() + server_vm.add_args('-machine', 'x-remote') + server_vm.add_args('-nodefaults') + server_vm.add_args('-device', 'lsi53c895a,id=lsi1') + server_vm.launch() + server_vm.command('human-monitor-command', + command_line='object_add x-vfio-user-server,'
Quoted 1 line
Why not use
qmp('object-add', ...)directly?
OK, will use qmp directly.
Thank you!
--
Jag
Quoted 6 lines
tests/avocado/vfio-user.py line 70
Quoted 4 lines
tests/avocado/vfio-user.py new file +3
-0+ 'id=vfioobj,socket.type=unix,' + 'socket.path='+socket+',device=lsi1') + return server_vmOtherwise LGTM.
Re: [PATCH v5 05/18] qdev: unplug blocker for devices
In reply to Stefan HajnocziQuoted 82 lines · Stefan Hajnoczi
Quoted 63 lines · Jag Raman
Quoted 52 lines · Stefan Hajnoczi
Quoted 47 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/qdev-core.h| 5 +++++softmmu/qdev-monitor.c| 35 +++++++++++++++++++++++++++++++++++2 files changed, 40 insertions(+)include/hw/qdev-core.h +5
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {@@ -433,6 +434,10 @@ typedef bool (QDevPutBusFunc)(BusState *bus, Error **errp);bool qdev_set_bus_cbs (QDevGetBusFunc *get_bus,QDevPutBusFunc *put_bus,Error **errp); +int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp); +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +4
-0@@ -978,10 +978,45 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } }Quoted 10 lines
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 996
Quoted 8 lines
softmmu/qdev-monitor.c +8
-0+int qdev_add_unplug_blocker (DeviceState *dev,Error *reason,Error **errp) +{ +ERRP_GUARD (); + +if (!migration_is_idle ()) { +error_setg (errp,"migration is in progress" ); +return -EBUSY ; + }Why can this function not be called during migration?
Quoted 6 lines
Since ‘unplug_blockers' is a member of the device, I thought it wouldn’t be correct to allow changes to the device's state during migration.
I did weigh the following reasons against adding this check:
-unplug_blockersis not migrated to the destination anyway, so it doesn’t matter if it changes after migration startsYes.
Quoted 2 lines
- whichever code/object that needs to add the blocker could add it at the destination if needed
Yes.
Quoted 8 lines
Quoted 5 lines
However, unlike
qmp_device_add(),qmp_object_add()doesn’t reject during migration. As such, an object could add a blocker for the device when migration is in progress.Would you prefer to throw a warning, or fully remove this test?
Adding an unplug blocker during migration seems safe to me. I would remove this test.
OK, will do.
Thank you!
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
In reply to Stefan HajnocziQuoted 21 lines · Stefan Hajnoczi
Quoted 16 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 29 +++++++++++++++++++++++++++++1 file changed, 29 insertions(+)Why is this code necessary? I expected the default hotplug behavior to
Quoted 8 lines
Quoted 5 lines
I just discovered that
TYPE_REMOTE_MACHINEwasn't setting up a hotplug handler for the root PCI bus.Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.
Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.
I was referring to the root PCI bus, other buses in Q35 probably support hotplug. Please see error message below:
QEMU 6.2.50 monitor - type 'help' for more information (qemu) device_add lsi53c895a,id=lsi2
Error: Bus 'pcie.0' does not support hotplugging
--
Jag
Quoted 19 lines
Quoted 9 lines
We didn’t check hotplug in multiprocess-qemu previously because it was limited to one device per process, and the use cases attached the devices via command line.
Quoted 3 lines
pretty much handle this case - hotplugging device types that the bus doesn't support should fail and unplug should already unparent/unrealize the device.
Quoted 6 lines
OK, that makes sense. We don’t need to test the device type during plug and unplug.
Therefore, I don’t think we need a callback for the plug operation. We could set
HotplugHandlerClass->unplugcallback to the defaultqdev_simple_device_unplug_cb()callback.Great!
Stefan
Re: [PATCH v5 06/18] vfio-user: add HotplugHandler for remote machine
Quoted 30 lines · Jag Raman
Quoted 21 lines · Stefan Hajnoczi
Quoted 16 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 29 +++++++++++++++++++++++++++++1 file changed, 29 insertions(+)Why is this code necessary? I expected the default hotplug behavior to
Quoted 8 lines
Quoted 5 lines
I just discovered that
TYPE_REMOTE_MACHINEwasn't setting up a hotplug handler for the root PCI bus.Looks like, some of the machines don’t support hotplugging PCI devices. I see that the ‘pc’ machine does support hotplug, whereas ‘q35’ does not.
Hotplug is definitely possible with q35. I'm not familiar with the hotplug code though so I don't know how exactly that works for q35.
Quoted 6 lines
I was referring to the root PCI bus, other buses in Q35 probably support hotplug. Please see error message below:
QEMU 6.2.50 monitor - type 'help' for more information (qemu)
device_addlsi53c895a,id=lsi2
Error: Bus 'pcie.0' does not support hotplugging
Yes, I think that's because it's PCIe and not PCI.
Stefan
Re: [PATCH v5 07/18] vfio-user: set qdev bus callbacks for remote machine
In reply to Stefan HajnocziQuoted 94 lines · Stefan Hajnoczi
Quoted 60 lines · Jag Raman
Quoted 49 lines · Stefan Hajnoczi
Quoted 44 lines · Jagannathan Raman
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/machine.c| 57 +++++++++++++++++++++++++++++++++++++++++++++1 file changed, 57 insertions(+)hw/remote/machine.c +29
-0@@ -22,6 +22,60 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +static bool remote_machine_get_bus (const char *type,BusState **bus, +Error **errp) +{ +ERRP_GUARD (); +RemoteMachineState *s =REMOTE_MACHINE (current_machine); +BusState *root_bus =NULL ; +PCIBus *new_pci_bus =NULL ; + +if (!bus) { +error_setg (errp,"Invalid argument" ); +return false; + } + +if (strcmp (type,TYPE_PCI_BUS ) &&strcmp (type,TYPE_PCI_BUS )) { +return true; + } + + root_bus =qbus_find_recursive (sysbus_get_default (),NULL ,TYPE_PCIE_BUS ); +if (!root_bus) { +error_setg (errp,"Unable to find root PCI device" ); +return false; + } + + new_pci_bus =pci_isol_bus_new (root_bus, type, errp); +if (!new_pci_bus) { +return false; + } + + *bus =BUS (new_pci_bus);Quoted 8 lines
hw/remote/machine.c line 59
Quoted 6 lines
hw/remote/machine.c +6
-0+ +pci_bus_irqs (new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +return true; +}Can the user create the same PCI bus via QMP commands? If so, then this
I think there is a way we could achieve it.
Quoted 8 lines
When I looked around, both the command line and the QMP didn’t have a direct way to create a bus. However, there are some indirect ways. For example, the
TYPE_LSI53C895Adevice creates a SCSI bus to attach SCSI devices. Similarly, there are some special PCI devices likeTYPE_PCI_BRIDGEwhich create a secondary PCI bus.Similarly, we could implement a PCI device that creates a PCI bus with isolated address spaces.
Exactly.
device_addinstantiates DeviceStates, not busses, so there needs to be a parent device like a SCSI controller, a PCI bridge, etc that owns and creates the bus.Quoted 16 lines
Quoted 15 lines
is just a convenience that saves the extra step. Or is there some magic that cannot be done via QMP
device_add?I'm asking because there are 3 objects involved and I'd like to understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.Users can already create the PCIDevice via hotplug and the vfio-user server via object-add. So if there's no magic they could also create the PCI bus:
1.device_add...some PCI bus stuff here...,id=isol-pci-bus0
2.device_add...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev
Quoted 6 lines
Quoted 2 lines
We are able to do 2 & 3 already. We could introduce a PCI device that creates an isolated PCI bus. That would cover step 1 outlined above.
I wonder if a new device is needed or whether it's possible to add an isol_as=on|off (default: off) option to an existing PCI bridge/expander? Hopefully most of the code is already there.
OK, it makes sense to add isol_as=on|off (default: off) option to an existing PCI bridge/expander. Will shortly confirm with you the device that seems interesting.
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jag RamanQuoted 32 lines · * Jag Raman ([email protected]) wrote:
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 8 lines
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
Are all devices mappable like that?
Quoted 3 lines
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I didn't have one in mind; I was just hitting a similar problem on Virtiofs DAX.
Dave
Quoted 174 lines
Thank you!
--
JagQuoted 169 lines
Dave
Quoted 163 lines
In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.
Quoted 3 lines
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Could you please provide some more details on the design?
Quoted 36 lines
Quoted 36 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;Quoted 7 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 4 lines
include/hw/pci/pci_bus.h +4
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. Are you sure
address_space_memory&address_space_ioare always global? even in the case of an iommu?On the CPU side of the Root Complex, I believe
address_space_memory&address_space_ioare global.In the vfio-user case, devices on the same machine (
TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svgQuoted 9 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 7 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, what are VM clients?
It’s the client in the client - server model explained above.
Thank you!
--
JagQuoted 79 lines
Quoted 79 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Stefan HajnocziQuoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-read
Quoted 8 lines
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.
I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices? Or that the address space of a given PCIe bus is non-overlapping with one of the others?
Dave
Quoted 1 line
Stefan
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readQuoted 8 lines
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.
Quoted 3 lines
I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Quoted 12 lines
Or that the
address space of a given PCIe bus is non-overlapping with one of the others?Dave
Quoted 1 line
Stefan
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 62 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readHere is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.
Quoted 7 lines
Quoted 3 lines
I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Michael: Are you saying just turn on vIOMMU? :)
Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 74 lines · Stefan Hajnoczi
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Dr. David Alan Gilbert
Quoted 62 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readHere is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.Quoted 7 lines
Quoted 3 lines
I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Quoted 7 lines
Michael: Are you saying just turn on vIOMMU? :)
Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.
Stefan
I was wondering about it here:
https://lore.kernel.org/r/20220119190742-mutt-send-email-mst%40kernel.org
Signature · 1 line
MST
Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Stefan HajnocziQuoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 23 lines
hw/remote/vfio-user-obj.c +23
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } +
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {When is this counter reset so migration can be tried again if it fails/cancels?
Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.
Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size);
Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 2 lines
hw/remote/vfio-user-obj.c +2
-0+ + o->vfu_mig_buf_pending -= size;This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on
vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely updatevfu_mig_buf_pendingwhen offset jumps around arbitrarily between calls.
Going by the definition of vfu_migration_callbacks_t in the library, I assumed that read_data advances the offset by size bytes.
Will add a comment a comment to explain that.
Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } +
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return size; +}This function looks incorrect because it ignores subsections as well as runtime behavior during
save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile atsave()time so there's no way of knowing the size ahead of time.
I see your point, it would be a problem for any field which has the (VMS_BUFFER | VMS_ALLOC) flags set.
Quoted 6 lines
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using vfu_migration_callbacks_t callbacks.
I don’t believe the library supports streaming vmstate/migration-data - see the following comment in migration_region_access() defined in the library:
* Does this mean that partial reads are not allowed?
Thanos or John,
Could you please clarify this?
Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:
https://lore.kernel.org/all/[email protected]/
Thank you very much!
--
Jag
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Dr. David Alan GilbertQuoted 41 lines · Dr. David Alan Gilbert
Quoted 36 lines · * Jag Raman ([email protected]) wrote:
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 6 lines
Quoted 4 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
Are all devices mappable like that?
If there is an IOMMU that supports DMA-Remapping (DMAR), that would be possible - the kernel could configure the DMAR to facilitate such mapping.
If there is no DMAR, the kernel/cpu could buffer the write between devices.
--
Jag
Quoted 187 lines
Quoted 4 lines
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I didn't have one in mind; I was just hitting a similar problem on Virtiofs DAX.
Dave
Quoted 174 lines
Thank you!
--
JagQuoted 169 lines
Dave
Quoted 163 lines
In the case where multiple clients attach devices that are running on the same server, we need to ensure that each devices has isolated memory ranges. This ensures that the memory space of one device is not visible to other devices in the same server.
Quoted 3 lines
I also wonder whether this special type could be modelled like a special kind of iommu internally.
Could you please provide some more details on the design?
Quoted 36 lines
Quoted 36 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 2 ++ include/hw/pci/pci_bus.h | 17 +++++++++++++++++ hw/pci/pci.c | 17 +++++++++++++++++ hw/pci/pci_bridge.c | 5 +++++ 4 files changed, 41 insertions(+)include/hw/pci/pci.h +2
-0@@ -387,6 +387,8 @@ void pci_device_save(PCIDevice *s, QEMUFile *f);int pci_device_load (PCIDevice *s,QEMUFile *f);MemoryRegion *pci_address_space (PCIDevice *dev);MemoryRegion *pci_address_space_io (PCIDevice *dev); +AddressSpace *pci_isol_as_mem (PCIDevice *dev); +AddressSpace *pci_isol_as_io (PCIDevice *dev);/* * Should not normally be used by devices. For use by sPAPR target include/hw/pci/pci_bus.h +1
-0@@ -39,9 +39,26 @@ struct PCIBus {void *irq_opaque;PCIDevice *devices[PCI_SLOT_MAX *PCI_FUNC_MAX ];PCIDevice *parent_dev; +MemoryRegion *address_space_mem;MemoryRegion *address_space_io;Quoted 7 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 4 lines
include/hw/pci/pci_bus.h +4
-0+/** +* Isolated address spaces - these allow the PCI bus to be part +* of an isolated address space as opposed to the global +* address_space_memory & address_space_io. Are you sure
address_space_memory&address_space_ioare always global? even in the case of an iommu?On the CPU side of the Root Complex, I believe
address_space_memory&address_space_ioare global.In the vfio-user case, devices on the same machine (
TYPE_REMOTE_MACHINE) could be attached to different clients VMs. Each client would have their own address space for their CPUs. With isolated address spaces, we ensure that the devices see the address space of the CPUs they’re attached to.Not sure if it’s OK to share weblinks in this mailing list, please let me know if that’s not preferred. But I’m referring to the terminology used in the following block diagram:
https://en.wikipedia.org/wiki/Root_complex#/media/File:Example_PCI_Express_Topology.svgQuoted 9 lines
include/hw/pci/pci_bus.h struct PCIBus {
Quoted 7 lines
This allows the
include/hw/pci/pci_bus.h +5
-0+* bus to be attached to CPUs from different machines. The +* following is not used used commonly. +* +* TYPE_REMOTE_MACHINE allows emulating devices from multiple +* VM clients, what are VM clients?
It’s the client in the client - server model explained above.
Thank you!
--
JagQuoted 79 lines
Quoted 79 lines
as such it needs the PCI buses in the same machine
include/hw/pci/pci_bus.h +7
-0+* to be part of different CPU address spaces. The following is +* useful in that scenario. +* +*/ +AddressSpace *isol_as_mem; +AddressSpace *isol_as_io; + QLIST_HEAD(, PCIBus) child; /* this will be replaced by qdev later */QLIST_ENTRY (PCIBus) sibling;/* this will be replaced by qdev later */ hw/pci/pci.c +17
-0@@ -442,6 +442,8 @@ static void pci_root_bus_internal_init(PCIBus *bus, DeviceState *parent, bus->slot_reserved_mask =0x0 ; bus->address_space_mem = address_space_mem; bus->address_space_io = address_space_io; + bus->isol_as_mem =NULL ; + bus->isol_as_io =NULL ; bus->flags |=PCI_BUS_IS_ROOT ;/* host bridge */ @@ -2676,6 +2678,16 @@ MemoryRegion *pci_address_space_io(PCIDevice *dev)return pci_get_bus (dev)->address_space_io; } +AddressSpace *pci_isol_as_mem (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_mem; +} + +AddressSpace *pci_isol_as_io (PCIDevice *dev) +{ +return pci_get_bus (dev)->isol_as_io; +} +static void pci_device_class_init (ObjectClass *klass,void *data) {DeviceClass *k =DEVICE_CLASS (klass);@@ -2699,6 +2711,7 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)AddressSpace *pci_device_iommu_address_space (PCIDevice *dev) { +AddressSpace *iommu_as =NULL ;PCIBus *bus =pci_get_bus (dev);PCIBus *iommu_bus = bus;uint8_t devfn = dev->devfn;@@ -2745,6 +2758,10 @@ AddressSpace *pci_device_iommu_address_space(PCIDevice *dev)if (!pci_bus_bypass_iommu (bus) && iommu_bus && iommu_bus->iommu_fn) {return iommu_bus->iommu_fn (bus, iommu_bus->iommu_opaque, devfn); } + iommu_as =pci_isol_as_mem (dev); +if (iommu_as) { +return iommu_as; + }return &address_space_memory; }hw/pci/pci_bridge.c +5
-0@@ -383,6 +383,11 @@ void pci_bridge_initfn(PCIDevice *dev, const char *typename) sec_bus->address_space_io = &br->address_space_io;memory_region_init (&br->address_space_io,OBJECT (br),"pci_bridge_io" ,4 * GiB); + +/* This PCI bridge puts the sec_bus in its parent's address space */ + sec_bus->isol_as_mem =pci_isol_as_mem (dev); + sec_bus->isol_as_io =pci_isol_as_io (dev); + br->windows =pci_bridge_region_init (br);QLIST_INIT (&sec_bus->child);QLIST_INSERT_HEAD (&parent->child, sec_bus, sibling);Signature · 1 line
2.20.1Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UKSignature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Stefan HajnocziQuoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readQuoted 8 lines
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
Quoted 8 lines
Quoted 2 lines
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Michael: Are you saying just turn on vIOMMU? :)
Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.
Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does? In fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 83 lines · Alex Williamson
Quoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readQuoted 8 lines
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
Quoted 8 lines
Quoted 2 lines
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Michael: Are you saying just turn on vIOMMU? :)
Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.
Quoted 5 lines
Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.
Oops, thank you for pointing that out!
Stefan
Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Jag RamanQuoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 23 lines
hw/remote/vfio-user-obj.c +23
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } +Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {When is this counter reset so migration can be tried again if it fails/cancels?
Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.
Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size);Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 2 lines
hw/remote/vfio-user-obj.c +2
-0+ + o->vfu_mig_buf_pending -= size;This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on
vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely updatevfu_mig_buf_pendingwhen offset jumps around arbitrarily between calls.Going by the definition of
vfu_migration_callbacks_tin the library, I assumed thatread_dataadvances the offset by size bytes.Will add a comment a comment to explain that.
Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } +Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return size; +}This function looks incorrect because it ignores subsections as well as runtime behavior during
save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile atsave()time so there's no way of knowing the size ahead of time.I see your point, it would be a problem for any field which has the (
VMS_BUFFER|VMS_ALLOC) flags set.Quoted 6 lines
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using
vfu_migration_callbacks_tcallbacks.I don’t believe the library supports streaming vmstate/migration-data - see the following comment in
migration_region_access()defined in the library:* Does this mean that partial reads are not allowed?
Thanos or John,
Could you please clarify this?
Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:
Quoted 6 lines
https://lore.kernel.org/all/[email protected]/
libvfio-user has the
vfu_migration_callbacks_tinterface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?
The answer didn't make sense to me:
"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.
In the case of the callbacks, the library translates the BAR access to callbacks."
The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <linux/vfio.h>:
* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Read pending_bytes, indicating the start of a new iteration to get device
* data. Repeated read on pending_bytes at this stage should have no side * effects. * If pending_bytes == 0, the user application should not iterate to get data * for that device. * If pending_bytes > 0, perform the following steps. * b. Read data_offset, indicating that the vendor driver should make data * available through the data section. The vendor driver should return this * read operation only after data is available from (region + data_offset) * to (region + data_offset + data_size). * c. Read data_size, which is the amount of data in bytes available through * the migration region. * Read on data_offset and data_size should return the offset and size of * the current buffer if the user application reads data_offset and * data_size more than once here.
* d. Read data_size bytes of data from (region + data_offset) from the
* migration region.
* e. Process the data.
* f. Read pending_bytes, which indicates that the data from the previous
* iteration has been read. If pending_bytes > 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the _SAVING (stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in _SAVING
* (stop-and-copy) until pending_bytes is 0.
This means you can report pending_bytes > 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,
There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
Stefan
RE: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Stefan HajnocziQuoted 19 lines
-----Original Message-----
From: Stefan Hajnoczi <[email protected]>
Sent: 28 January 2022 08:29
To: Jag Raman <[email protected]>
Cc: John Levon <[email protected]>; Thanos Makatos <[email protected]>; qemu-devel <[email protected]>; Marc-André Lureau <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Paolo Bonzini <[email protected]>; Beraldo Leal <[email protected]>; Daniel P. Berrangé <[email protected]>; [email protected]; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Juan Quintela <[email protected]>; Dr . David Alan Gilbert <[email protected]>; Elena Ufimtseva <[email protected]>; John Johnson <[email protected]>
Subject: Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
Quoted 76 lines
Quoted 3 lines · Jag Raman
Quoted 3 lines
On Jan 25, 2022, at 10:48 AM, Stefan Hajnoczi <[email protected]>
Quoted 71 lines · wrote:
Quoted 29 lines · Jagannathan Raman
Quoted 1 line
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 23 lines
hw/remote/vfio-user-obj.c +23
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } +Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {When is this counter reset so migration can be tried again if it fails/cancels?
Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.
Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size);Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 2 lines
hw/remote/vfio-user-obj.c +2
-0+ + o->vfu_mig_buf_pending -= size;This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on
vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely updatevfu_mig_buf_pendingwhen offset jumps around arbitrarily between calls.Going by the definition of
vfu_migration_callbacks_tin the library, I assumed thatread_dataadvances the offset by size bytes.Will add a comment a comment to explain that.
libvfio-user does not automatically increment offset by size each time, since the vfio-user client can re-read the migration data multiple times. In libvfio-user API we state:
Function that is called to read migration data. offset and size can be any subrange on the offset and size previously returned by prepare_data.
Reading the pending_bytes register is what marks the end of the iteration, and this is where you need to decrement vfu_mig_buf_pending.
I'll add more unit tests to libvfio-user to validate this behavior.
Quoted 41 lines
Quoted 41 lines
Quoted 18 lines
Quoted 18 lines
migration/vmstate.c +16
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } +Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return size; +}This function looks incorrect because it ignores subsections as well as runtime behavior during
save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile atsave()time so there's no way of knowing the size ahead of time.I see your point, it would be a problem for any field which has the (
VMS_BUFFER|VMS_ALLOC) flags set.Quoted 6 lines
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using
vfu_migration_callbacks_tcallbacks.
Quoted 8 lines
Quoted 8 lines
I don’t believe the library supports streaming vmstate/migration-data - see the following comment in
migration_region_access()defined in the library:* Does this mean that partial reads are not allowed?
Thanos or John,
Could you please clarify this?
libvfio-user does support streaming of migration data, this comment is based on the VFIO documentation:
d. Read data_size bytes of data from (region + data_offset) from the
migration region.
It's not clear to me whether streaming should be allowed, I'd be surprised if it didn't.
Quoted 61 lines
Quoted 6 lines
Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:Quoted 4 lines
libvfio-user has the
vfu_migration_callbacks_tinterface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?The answer didn't make sense to me:
"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.
In the case of the callbacks, the library translates the BAR access to callbacks."
The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <
linux/vfio.h>:* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Readpending_bytes, indicating the start of a new iteration to get device* data. Repeated read on pending_bytes at this stage should have no side * effects. * If pending_bytes == 0, the user application should not iterate to get data * for that device. * If pending_bytes > 0, perform the following steps. * b. Read data_offset, indicating that the vendor driver should make data * available through the data section. The vendor driver should return this * read operation only after data is available from (region + data_offset) * to (region + data_offset + data_size). * c. Read data_size, which is the amount of data in bytes available through * the migration region. * Read on data_offset and data_size should return the offset and size of * the current buffer if the user application reads data_offset and * data_size more than once here.* d. Read
data_sizebytes of data from (region +data_offset) from the
* migration region.
* e. Process the data.
* f. Readpending_bytes, which indicates that the data from the previous
* iteration has been read. Ifpending_bytes> 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the_SAVING(stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in_SAVING
* (stop-and-copy) untilpending_bytesis 0.This means you can report
pending_bytes> 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Stefan HajnocziQuoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Alex
Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Stefan HajnocziQuoted 182 lines · Stefan Hajnoczi
Quoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 23 lines
hw/remote/vfio-user-obj.c +23
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } +Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {When is this counter reset so migration can be tried again if it fails/cancels?
Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.
Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size);Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 2 lines
hw/remote/vfio-user-obj.c +2
-0+ + o->vfu_mig_buf_pending -= size;This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on
vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely updatevfu_mig_buf_pendingwhen offset jumps around arbitrarily between calls.Going by the definition of
vfu_migration_callbacks_tin the library, I assumed thatread_dataadvances the offset by size bytes.Will add a comment a comment to explain that.
Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } +Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return size; +}This function looks incorrect because it ignores subsections as well as runtime behavior during
save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile atsave()time so there's no way of knowing the size ahead of time.I see your point, it would be a problem for any field which has the (
VMS_BUFFER|VMS_ALLOC) flags set.Quoted 6 lines
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using
vfu_migration_callbacks_tcallbacks.I don’t believe the library supports streaming vmstate/migration-data - see the following comment in
migration_region_access()defined in the library:* Does this mean that partial reads are not allowed?
Thanos or John,
Could you please clarify this?
Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:Quoted 6 lines
https://lore.kernel.org/all/[email protected]/
libvfio-user has the
vfu_migration_callbacks_tinterface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?The answer didn't make sense to me:
"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.
In the case of the callbacks, the library translates the BAR access to callbacks."
The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <
linux/vfio.h>:* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Readpending_bytes, indicating the start of a new iteration to get device* data. Repeated read on pending_bytes at this stage should have no side * effects. * If pending_bytes == 0, the user application should not iterate to get data * for that device. * If pending_bytes > 0, perform the following steps. * b. Read data_offset, indicating that the vendor driver should make data * available through the data section. The vendor driver should return this * read operation only after data is available from (region + data_offset) * to (region + data_offset + data_size). * c. Read data_size, which is the amount of data in bytes available through * the migration region. * Read on data_offset and data_size should return the offset and size of * the current buffer if the user application reads data_offset and * data_size more than once here.* d. Read
data_sizebytes of data from (region +data_offset) from the
* migration region.
* e. Process the data.
* f. Readpending_bytes, which indicates that the data from the previous
* iteration has been read. Ifpending_bytes> 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the_SAVING(stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in_SAVING
* (stop-and-copy) untilpending_bytesis 0.
Quoted 4 lines
This means you can report
pending_bytes> 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.
Thank you for the pointer to generic VFIO migration, Stefan! Makes sense.
So I understand that the VFIO migration region carves out a section to stream/shuttle device data between the app (QEMU client in this case) and the driver (QEMU server). This section starts at data_offset within the region and spans data_size bytes.
We could change the server to stream the data as outlined above. Do you have a preference for the section size? Does qemu_target_page_size() work? I just tested and am able to stream with a fixed BAR size such as qemu_target_page_size().
Thank you!
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.
Stefan
Re: [PATCH v5 17/18] vfio-user: register handlers to facilitate migration
In reply to Jag RamanQuoted 189 lines · Jag Raman
Quoted 182 lines · Stefan Hajnoczi
Quoted 128 lines · Jag Raman
Quoted 32 lines · Stefan Hajnoczi
Quoted 1 line · Jagannathan Raman
+ * The client subsequetly asks the remote server for any data that
subsequently
Quoted 23 lines
hw/remote/vfio-user-obj.c +23
-0+static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +VfuObjectClass *k =VFU_OBJECT_GET_CLASS (OBJECT (o)); +static int migrated_devs; +Error *local_err =NULL ; +int ret; + +/** +* TODO: move to VFU_MIGR_STATE_RESUME handler. Presently, the +* VMSD data from source is not available at RESUME state. +* Working on a fix for this. +*/ +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } +Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 318
Quoted 5 lines
hw/remote/vfio-user-obj.c +5
-0+qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +/* VFU_MIGR_STATE_RUNNING begins here */ +if (++migrated_devs == k->nr_devs) {When is this counter reset so migration can be tried again if it fails/cancels?
Detecting cancellation is a pending item. We will address it in the next rev. Will check with you if we get stuck during the process of implementing it.
Quoted 16 lines
Quoted 16 lines
hw/remote/vfio-user-obj.c +15
-0+static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (offset > o->vfu_mig_buf_size) { +return -1 ; + } + +if ((offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - check pending_bytes" ); + size = o->vfu_mig_buf_size - offset; + } + +memcpy (buf, (o->vfu_mig_buf + offset), size);Quoted 8 lines
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 424
Quoted 2 lines
hw/remote/vfio-user-obj.c +2
-0+ + o->vfu_mig_buf_pending -= size;This assumes that the caller increments offset by size each time. If that assumption is okay, then we can just trust offset and don't need to do arithmetic on
vfu_mig_buf_pending. If that assumption is not correct, then the code needs to be extended to safely updatevfu_mig_buf_pendingwhen offset jumps around arbitrarily between calls.Going by the definition of
vfu_migration_callbacks_tin the library, I assumed thatread_dataadvances the offset by size bytes.Will add a comment a comment to explain that.
Quoted 17 lines
Quoted 17 lines
migration/vmstate.c +16
-0+uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } +Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return size; +}This function looks incorrect because it ignores subsections as well as runtime behavior during
save(). Although VMStateDescription is partially declarative, there is still a bunch of imperative code that can write to the QEMUFile atsave()time so there's no way of knowing the size ahead of time.I see your point, it would be a problem for any field which has the (
VMS_BUFFER|VMS_ALLOC) flags set.Quoted 6 lines
I asked this in a previous revision of this series but I'm not sure if it was answered: is it really necessary to know the size of the vmstate? I thought the VFIO migration interface is designed to support streaming reads/writes. We could choose a fixed size like 64KB and stream the vmstate in 64KB chunks.
The library exposes the migration data to the client as a device BAR with fixed size - the size of which is fixed at boot time, even when using
vfu_migration_callbacks_tcallbacks.I don’t believe the library supports streaming vmstate/migration-data - see the following comment in
migration_region_access()defined in the library:* Does this mean that partial reads are not allowed?
Thanos or John,
Could you please clarify this?
Stefan,
We attempted to answer the migration cancellation and vmstate size questions previously also, in the following email:Quoted 6 lines
https://lore.kernel.org/all/[email protected]/
libvfio-user has the
vfu_migration_callbacks_tinterface that allows the device to save/load more data regardless of the size of the migration region. I don't see the issue here since the region doesn't need to be sized to fit the savevm data?The answer didn't make sense to me:
"In both scenarios at the server end - whether using the migration BAR or using callbacks, the migration data is transported to the other end using the BAR. As such we need to specify the BAR’s size during initialization.
In the case of the callbacks, the library translates the BAR access to callbacks."
The BAR and the migration region within it need a size but my understanding is that VFIO migration is designed to stream the device state, allowing it to be broken up into multiple reads/writes with knowing the device state's size upfront. Here is the description from <
linux/vfio.h>:* The sequence to be followed while in pre-copy state and stop-and-copy state
* is as follows:
* a. Readpending_bytes, indicating the start of a new iteration to get device* data. Repeated read on pending_bytes at this stage should have no side * effects. * If pending_bytes == 0, the user application should not iterate to get data * for that device. * If pending_bytes > 0, perform the following steps. * b. Read data_offset, indicating that the vendor driver should make data * available through the data section. The vendor driver should return this * read operation only after data is available from (region + data_offset) * to (region + data_offset + data_size). * c. Read data_size, which is the amount of data in bytes available through * the migration region. * Read on data_offset and data_size should return the offset and size of * the current buffer if the user application reads data_offset and * data_size more than once here.* d. Read
data_sizebytes of data from (region +data_offset) from the
* migration region.
* e. Process the data.
* f. Readpending_bytes, which indicates that the data from the previous
* iteration has been read. Ifpending_bytes> 0, go to step b.
*
* The user application can transition from the _SAVING|_RUNNING
* (pre-copy state) to the_SAVING(stop-and-copy) state regardless of the
* number of pending bytes. The user application should iterate in_SAVING
* (stop-and-copy) untilpending_bytesis 0.Quoted 4 lines
This means you can report
pending_bytes> 0 until the entire vmstate has been read and can pick a fixed chunk size like 64KB for the migration region. There's no need to size the migration region to fit the entire vmstate.Thank you for the pointer to generic VFIO migration, Stefan! Makes sense.
Quoted 8 lines
So I understand that the VFIO migration region carves out a section to stream/shuttle device data between the app (QEMU client in this case) and the driver (QEMU server). This section starts at
data_offsetwithin the region and spansdata_sizebytes.We could change the server to stream the data as outlined above. Do you have a preference for the section size? Does
qemu_target_page_size()work? I just tested and am able to stream with a fixed BAR size such asqemu_target_page_size().
The VFIO migration API requires that data is written in the same chunk sizes as it was read, so there is no way to merge or split chunks for performance reasons once they have been read.
4KB may result in lots of chunks and that means more network traffic and read()/write() calls. I think it's too small.
Something large like 1MB might create issues with responsiveness because a 1MB chunk hogs the migration stream and read()/write() latency could hog the event loop.
I'd go for 64KB. Dave and Juan might also have a suggestion for the size.
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 83 lines · * Alex Williamson ([email protected]) wrote:
Quoted 71 lines · On Thu, 27 Jan 2022 08:30:13 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 70 lines · Michael S. Tsirkin
Quoted 67 lines · Dr. David Alan Gilbert
Quoted 54 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 36 lines · Jag Raman
Quoted 22 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Jag Raman ([email protected]) wrote:
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Jagannathan Raman
Allow PCI buses to be part of isolated CPU address spaces. This has a niche usage.
Quoted 6 lines
Quoted 4 lines
TYPE_REMOTE_MACHINEallows multiple VMs to house their PCI devices in the same machine/server. This would cause address space collision as well as be a security vulnerability. Having separate address spaces for each PCI bus would solve this problem.Fascinating, but I am not sure I understand. any examples?
Hi Michael!
Quoted 7 lines
Quoted 5 lines
multiprocess QEMU and vfio-user implement a client-server model to allow out-of-process emulation of devices. The client QEMU, which makes ioctls to the kernel and runs VCPUs, could attach devices running in a server QEMU. The server QEMU needs access to parts of the client’s RAM to perform DMA.
Do you ever have the opposite problem? i.e. when an emulated PCI device
That’s an interesting question.
Quoted 3 lines
exposes a chunk of RAM-like space (frame buffer, or maybe a mapped file) that the client can see. What happens if two emulated devices need to access each others emulated address space?
Quoted 8 lines
In this case, the kernel driver would map the destination’s chunk of internal RAM into the DMA space of the source device. Then the source device could write to that mapped address range, and the IOMMU should direct those writes to the destination device.
I would like to take a closer look at the IOMMU implementation on how to achieve this, and get back to you. I think the IOMMU would handle this. Could you please point me to the IOMMU implementation you have in mind?
I don't know if the current vfio-user client/server patches already implement device-to-device DMA, but the functionality is supported by the vfio-user protocol.
Basically: if the DMA regions lookup inside the vfio-user server fails, fall back to VFIO_USER_DMA_READ/WRITE messages instead.
https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst#vfio-user-dma-readQuoted 8 lines
Here is the flow:
1. The vfio-user server with device A sends a DMA read to QEMU.
2. QEMU finds the MemoryRegion associated with the DMA address and sees it's a device.
a. If it's emulated inside the QEMU process then the normal
device emulation code kicks in.
b. If it's another vfio-user PCI device then the vfio-user PCI proxy
device forwards the DMA to the second vfio-user server's device B.I'm starting to be curious if there's a way to persuade the guest kernel to do it for us; in general is there a way to say to PCI devices that they can only DMA to the host and not other PCI devices?
Quoted 8 lines
Quoted 2 lines
But of course - this is how e.g. VFIO protects host PCI devices from each other when one of them is passed through to a VM.
Michael: Are you saying just turn on vIOMMU? :)
Devices in different VFIO groups have their own IOMMU context, so their IOVA space is isolated. Just don't map other devices into the IOVA space and those other devices will be inaccessible.
Quoted 8 lines
Devices in different VFIO *containers* have their own IOMMU context. Based on the group attachment to a container, groups can either have shared or isolated IOVA space. That determination is made by looking at the address space of the bus, which is governed by the presence of a vIOMMU.
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
That was what I was curious about - is it possible to get an IOMMU to do that, and how? (Not knowing much about IOMMUs).
In my DAX/virtiofs case, I want the device to be able to DMA to guest RAM but for other devices not to try to DMA to it and in particular for it not to have to DMA to other devices.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,
Dave
Quoted 2 lines
Alex
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Stefan HajnocziQuoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Quoted 8 lines
What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.
This is getting a bit ridiculous, if vfio-user devices require this degree of manipulation of the VM topology into things that don't exist on bare metal, we've done something very wrong. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jag RamanQuoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.
Quoted 4 lines
The PCIBus data structure already has
address_space_memandaddress_space_ioto contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.
These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from address_space_mem.
I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Quoted 6 lines
Typically, the machines map the PCI address space into the system address space. For example,
pc_pci_as_mapping_init()does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space.
"Typically" only if we're restricted to the "pc", ie. i440FX, machine type since it doesn't support a vIOMMU. There's no reason to focus on the identity map case versus the vIOMMU case.
Quoted 4 lines
Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for
address_space_rw()to successfully vector BAR accesses into the PCI device.
But as far as device initiated transactions, there is only one address space for a given device, it's either address_space_mem or one provided by the vIOMMU and pci_device_iommu_address_space() tells us that address space. Furthermore, the device never operates on a "physical address", it only ever operates on an IOVA, ie. an offset within the address space assigned to the device. The IOVA should be considered arbitrary relative to mappings in any other address spaces.
Device initiated transactions operate on an IOVA within the (single) address space to which the device is assigned. Any attempt to do otherwise violates the isolation put in place by things like vIOMMUs and ought to be considered a security concern, especially for a device serviced by an external process. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 72 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses address_space_rw() which vectors the physical address access to the device BAR region handlers.
The PCIBus data structure already has address_space_mem and address_space_io to contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.
Typically, the machines map the PCI address space into the system address space. For example, pc_pci_as_mapping_init() does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space.
Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for address_space_rw() to successfully vector BAR accesses into the PCI device.
Thank you!
--
Jag
Quoted 16 lines
Quoted 9 lines
What I find strange about this approach is that exported PCI devices are on PCI root ports that are connected to the machine's main PCI bus. The PCI devices don't interact with the main bus's IOVA space, guest physical memory space, or interrupts. It seems hacky to graft isolated devices onto a parent bus that provides nothing to its children. I wonder if it would be cleaner for every vfio-user server to have its own PCIHost. Then it may be possible to drop the new pci_isol_as_mem/io() API.
This is getting a bit ridiculous, if vfio-user devices require this degree of manipulation of the VM topology into things that don't exist on bare metal, we've done something very wrong. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 109 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.
Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into address_space_mem? (pci_default_write_config() -> pci_update_mappings())
Thank you!
--
Jag
Quoted 49 lines
Quoted 5 lines
The PCIBus data structure already has
address_space_memandaddress_space_ioto contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from
address_space_mem.I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Quoted 6 lines
Typically, the machines map the PCI address space into the system address space. For example,
pc_pci_as_mapping_init()does this for ‘pc' machine types. As such, there is a 1:1 mapping between system address space and PCI address space of the root bus. Since all the PCI devices in the machine are assigned to the same VM, we could map the PCI address space of all PCI buses to the same system address space."Typically" only if we're restricted to the "pc", ie. i440FX, machine type since it doesn't support a vIOMMU. There's no reason to focus on the identity map case versus the vIOMMU case.
Quoted 4 lines
Whereas in the case of vfio-user, the devices running in the server could belong to different VMs. Therefore, along with the physical address, we would need to know the address space that the device belongs for
address_space_rw()to successfully vector BAR accesses into the PCI device.But as far as device initiated transactions, there is only one address space for a given device, it's either
address_space_memor one provided by the vIOMMU andpci_device_iommu_address_space()tells us that address space. Furthermore, the device never operates on a "physical address", it only ever operates on an IOVA, ie. an offset within the address space assigned to the device. The IOVA should be considered arbitrary relative to mappings in any other address spaces.Device initiated transactions operate on an IOVA within the (single) address space to which the device is assigned. Any attempt to do otherwise violates the isolation put in place by things like vIOMMUs and ought to be considered a security concern, especially for a device serviced by an external process. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.
Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())
This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually system_memory, which is used to initialize address_space_system.
The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 124 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
Quoted 4 lines
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
The access comes via libvfio-user's vfu_region_access_cb_t callback, not via MemoryRegionOps. The callback is currently implemented by calling address_space_rw() on the pci_isol_as_mem/io() address space, depending on the BAR type. The code in "[PATCH v5 15/18] vfio-user: handle PCI BAR accesses".
It's possible to reimplement the patch to directly call memory_region_dispatch_read/write() on r->memory instead of address_space_rw() as you've described.
Quoted 14 lines
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.
Quoted 8 lines
The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Good, I think nothing uses address_space_system/io when BAR dispatch is implemented with memory_region_dispatch_read/write() as you suggested.
It would be nice if there was a way to poison address_space_system/io to abort on dispatch - nothing should use them.
We now have the option of dropping pci_isol_as_mem/io() again and using ->iommu_fn() to return an isolated memory address space containing the vfio-user client's VFIO_USER_DMA_MAP regions.
Stefan
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 8 lines · Alex Williamson
Quoted 4 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
The PCIBus data structure already has
address_space_memandaddress_space_ioto contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.
Quoted 4 lines
These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from
address_space_mem.
Note that those fields in PCIBus are just whatever MemoryRegions the pci controller model passed in to the call to pci_root_bus_init() or equivalent. They may or may not be specifically the CPU's view of anything. (For instance on the versatilepb board, the PCI controller is visible to the CPU via several MMIO "windows" at known addresses, which let the CPU access into the PCI address space at a programmable offset. We model that by creating a couple of container MRs which we pass to pci_root_bus_init() to be the PCI memory and IO spaces, and then using alias MRs to provide the view into those at the guest-programmed offset. The CPU sees those windows, and doesn't have direct access to the whole PCIBus::address_space_mem.) I guess you could say they're the PCI controller's view of the PCI address space ?
We have a tendency to be a bit sloppy with use of AddressSpaces within QEMU where it happens that the view of the world that a DMA-capable device matches that of the CPU, but conceptually they can definitely be different, especially in the non-x86 world. (Linux also confuses matters here by preferring to program a 1:1 mapping even if the hardware is more flexible and can do other things. The model of the h/w in QEMU should support the other cases too, not just 1:1.)
Quoted 5 lines
I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?
thanks
-- PMM
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 6 lines · Peter Maydell
Quoted 5 lines
I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Quoted 6 lines
Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?
thanks
-- PMM
Hmm, the only thing vaguely related in the spec that I know of is this:
PCI Express supports I/O Space for compatibility with legacy devices which require their use. Future revisions of this specification may deprecate the use of I/O Space.
Alex, what did you refer to?
Signature · 1 line
MST
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 14 lines · On Wed, 2 Feb 2022 05:06:49 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 6 lines · Peter Maydell
Quoted 5 lines
I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Quoted 6 lines
Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?
thanks
-- PMM
Quoted 6 lines
Hmm, the only thing vaguely related in the spec that I know of is this:
PCI Express supports I/O Space for compatibility with legacy devices which require their use. Future revisions of this specification may deprecate the use of I/O Space.
Alex, what did you refer to?
My evidence is largely by omission, but that might be that in practice it's not used rather than explicitly forbidden. I note that the bus master enable bit specifies:
Bus Master Enable - Controls the ability of a Function to issue
Memory and I/O Read/Write Requests, and the ability of a Port to forward Memory and I/O Read/Write Requests in the Upstream direction.
That would suggest it's possible, but for PCI device assignment, I'm not aware of any means through which we could support this. There is no support in the IOMMU core for mapping I/O port space, nor could we trap such device initiated transactions to emulate them. I can't spot any mention of I/O port space in the VT-d spec, however the AMD-Vi spec does include a field in the device table:
controlIoCtl: port I/O control. Specifies whether device-initiated port I/O space transactions are blocked, forwarded, or translated.
00b=Device-initiated port I/O is not allowed. The IOMMU target aborts the transaction if a port I/O space transaction is received. Translation requests are target aborted.
01b=Device-initiated port I/O space transactions are allowed. The IOMMU must pass port I/O accesses untranslated. Translation requests are target aborted.
10b=Transactions in the port I/O space address range are translated by the IOMMU page tables as memory transactions.
11b=Reserved.
I don't see this field among the macros used by the Linux driver in configuring these device entries, so I assume it's left to the default value, ie. zero, blocking device initiated I/O port transactions.
So yes, I suppose device initiated I/O port transactions are possible, but we have no support or reason to support them, so I'm going to go ahead and continue believing any I/O port address space from the device perspective is largely irrelevant ;) Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 39 lines · Alex Williamson
Quoted 1 line
Alex, what did you refer to?
My evidence is largely by omission, but that might be that in practice it's not used rather than explicitly forbidden. I note that the bus master enable bit specifies:
Bus Master Enable - Controls the ability of a Function to issue
Memory and I/O Read/Write Requests, and the ability of a Port to forward Memory and I/O Read/Write Requests in the Upstream direction.That would suggest it's possible, but for PCI device assignment, I'm not aware of any means through which we could support this. There is no support in the IOMMU core for mapping I/O port space, nor could we trap such device initiated transactions to emulate them. I can't spot any mention of I/O port space in the VT-d spec, however the AMD-Vi spec does include a field in the device table:
controlIoCtl: port I/O control. Specifies whether device-initiated port I/O space transactions are blocked, forwarded, or translated.
00b=Device-initiated port I/O is not allowed. The IOMMU target aborts the transaction if a port I/O space transaction is received. Translation requests are target aborted.
01b=Device-initiated port I/O space transactions are allowed. The IOMMU must pass port I/O accesses untranslated. Translation requests are target aborted.
10b=Transactions in the port I/O space address range are translated by the IOMMU page tables as memory transactions.
11b=Reserved.
I don't see this field among the macros used by the Linux driver in configuring these device entries, so I assume it's left to the default value, ie. zero, blocking device initiated I/O port transactions.
Quoted 6 lines
So yes, I suppose device initiated I/O port transactions are possible, but we have no support or reason to support them, so I'm going to go ahead and continue believing any I/O port address space from the device perspective is largely irrelevant ;) Thanks,
Alex
Right, it would seem devices can initiate I/O space transactions but IOMMUs don't support virtualizing them and so neither does VFIO.
Signature · 1 line
MST
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Peter MaydellQuoted 14 lines · On Wed, 2 Feb 2022 09:30:42 +0000 Peter Maydell <[email protected]>…
Quoted 8 lines · Alex Williamson
Quoted 4 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
The PCIBus data structure already has
address_space_memandaddress_space_ioto contain the BAR regions of devices attached to it. I understand that these two PCIBus members form the PCI address space.Quoted 4 lines
These are the CPU address spaces. When there's no IOMMU, the PCI bus is identity mapped to the CPU address space. When there is an IOMMU, the device address space is determined by the granularity of the IOMMU and may be entirely separate from
address_space_mem.
Quoted 13 lines
Note that those fields in PCIBus are just whatever MemoryRegions the pci controller model passed in to the call to
pci_root_bus_init()or equivalent. They may or may not be specifically the CPU's view of anything. (For instance on the versatilepb board, the PCI controller is visible to the CPU via several MMIO "windows" at known addresses, which let the CPU access into the PCI address space at a programmable offset. We model that by creating a couple of container MRs which we pass topci_root_bus_init()to be the PCI memory and IO spaces, and then using alias MRs to provide the view into those at the guest-programmed offset. The CPU sees those windows, and doesn't have direct access to the wholePCIBus::address_space_mem.) I guess you could say they're the PCI controller's view of the PCI address space ?
Sure, that's fair.
Quoted 8 lines
We have a tendency to be a bit sloppy with use of AddressSpaces within QEMU where it happens that the view of the world that a DMA-capable device matches that of the CPU, but conceptually they can definitely be different, especially in the non-x86 world. (Linux also confuses matters here by preferring to program a 1:1 mapping even if the hardware is more flexible and can do other things. The model of the h/w in QEMU should support the other cases too, not just 1:1.)
Right, this is why I prefer to look at the device address space as simply an IOVA. The IOVA might be a direct physical address or coincidental identity mapped physical address via an IOMMU, but none of that should be the concern of the device.
Quoted 9 lines
Quoted 5 lines
I/O port space is always the identity mapped CPU address space unless sparse translations are used to create multiple I/O port spaces (not implemented). I/O port space is only accessed by the CPU, there are no device initiated I/O port transactions, so the address space relative to the device is irrelevant.
Does the PCI spec actually forbid any master except the CPU from issuing I/O port transactions, or is it just that in practice nobody makes a PCI device that does weird stuff like that ?
As realized in reply to MST, more the latter. Not used, no point to enabling, no means to enable depending on the physical IOMMU implementation. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Quoted 8 lines
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script() -> read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Thank you!
--
Jag
Quoted 3 lines
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Quoted 8 lines
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU. I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 6 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
Quoted 7 lines
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
Quoted 7 lines
--
JagQuoted 3 lines
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 187 lines · Michael S. Tsirkin
Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Quoted 8 lines
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.
Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
Thank you!
--
Jag
Quoted 21 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
Quoted 7 lines
--
JagQuoted 3 lines
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 187 lines · Jag Raman
Quoted 187 lines · Michael S. Tsirkin
Quoted 181 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer (p2p), why are we trying to re-invent what an IOMMU already does?
The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA address space per BDF. Is the dynamic mapping overhead too much? What physical hardware properties or specifications could we leverage to restrict p2p mappings to a device? Should it be governed by machine type to provide consistency between devices? Should each "isolated" bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Quoted 8 lines
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.
Quoted 6 lines
Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU.
Hmm do you mean you analyzed how a physical device works? Or do you mean in QEMU?
Quoted 2 lines
The device is reading the BAR
address as if it was a DMA address.
I got that, my understanding of PCI was that a device can not be both a master and a target of a transaction at the same time though. Could not find this in the spec though, maybe I remember incorrectly.
Quoted 49 lines
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
Thank you!
--
JagQuoted 21 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
Quoted 7 lines
--
JagQuoted 3 lines
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Jag RamanQuoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
I don't think this is correct. Look for instance at ACPI _TRA support where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture. From the processor perspective, each MMIO range is unique and possibly none of those devices have a zero _TRA, there could be system memory at the equivalent flat memory address.
So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.
Quoted 11 lines
Quoted 11 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 55 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
Quoted 8 lines
I don't think this is correct. Look for instance at ACPI
_TRAsupport where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.Quoted 1 line
From the processor perspective, each MMIO range is unique and possibly
none of those devices have a zero
_TRA, there could be system memory at the equivalent flat memory address.
I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have _TRA.
Quoted 7 lines
So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Quoted 7 lines
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.
Quoted 3 lines
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.
You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
Quoted 6 lines
This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,
Alex
thanks!
Signature · 1 line
MST
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Michael S. TsirkinQuoted 195 lines · Michael S. Tsirkin
Quoted 190 lines · Jag Raman
Quoted 189 lines · Michael S. Tsirkin
Quoted 171 lines · Jag Raman
Quoted 156 lines · Alex Williamson
Quoted 113 lines · On Wed, 2 Feb 2022 01:13:22 +0000 Jag Raman <[email protected]> wrote:
Quoted 107 lines · Alex Williamson
Quoted 93 lines · On Tue, 1 Feb 2022 21:24:08 +0000 Jag Raman <[email protected]> wrote:
Quoted 70 lines · Alex Williamson
Quoted 58 lines · On Tue, 1 Feb 2022 09:30:35 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 48 lines · Alex Williamson
Quoted 31 lines · On Fri, 28 Jan 2022 09:18:08 +0000 Stefan Hajnoczi <[email protected]> wrote:
Quoted 2 lines · Alex Williamson
If the goal here is to restrict DMA between devices, ie. peer-to-peer
(p2p), why are we trying to re-invent what an IOMMU already does?The issue Dave raised is that vfio-user servers run in separate processses from QEMU with shared memory access to RAM but no direct access to non-RAM MemoryRegions. The virtiofs DAX Window BAR is one example of a non-RAM MemoryRegion that can be the source/target of DMA requests.
I don't think IOMMUs solve this problem but luckily the vfio-user protocol already has messages that vfio-user servers can use as a fallback when DMA cannot be completed through the shared memory RAM accesses.
Quoted 7 lines
In
fact, it seems like an IOMMU does this better in providing an IOVA
address space per BDF. Is the dynamic mapping overhead too much? What
physical hardware properties or specifications could we leverage to
restrict p2p mappings to a device? Should it be governed by machine
type to provide consistency between devices? Should each "isolated"
bus be in a separate root complex? Thanks,There is a separate issue in this patch series regarding isolating the address space where BAR accesses are made (i.e. the global address_space_memory/io). When one process hosts multiple vfio-user server instances (e.g. a software-defined network switch with multiple ethernet devices) then each instance needs isolated memory and io address spaces so that vfio-user clients don't cause collisions when they map BARs to the same address.
Quoted 3 lines
I think the the separate root complex idea is a good solution. This patch series takes a different approach by adding the concept of isolated address spaces into hw/pci/.
This all still seems pretty sketchy, BARs cannot overlap within the same vCPU address space, perhaps with the exception of when they're being sized, but DMA should be disabled during sizing.
Devices within the same VM context with identical BARs would need to operate in different address spaces. For example a translation offset in the vCPU address space would allow unique addressing to the devices, perhaps using the translation offset bits to address a root complex and masking those bits for downstream transactions.
Quoted 8 lines
In general, the device simply operates in an address space, ie. an IOVA. When a mapping is made within that address space, we perform a translation as necessary to generate a guest physical address. The IOVA itself is only meaningful within the context of the address space, there is no requirement or expectation for it to be globally unique.
If the vfio-user server is making some sort of requirement that IOVAs are unique across all devices, that seems very, very wrong. Thanks,
Quoted 6 lines
Yes, BARs and IOVAs don't need to be unique across all devices.
The issue is that there can be as many guest physical address spaces as there are vfio-user clients connected, so per-client isolated address spaces are required. This patch series has a solution to that problem with the new pci_isol_as_mem/io() API.
Quoted 11 lines
Sorry, this still doesn't follow for me. A server that hosts multiple devices across many VMs (I'm not sure if you're referring to the device or the VM as a client) needs to deal with different address spaces per device. The server needs to be able to uniquely identify every DMA, which must be part of the interface protocol. But I don't see how that imposes a requirement of an isolated address space. If we want the device isolated because we don't trust the server, that's where an IOMMU provides per device isolation. What is the restriction of the per-client isolated address space and why do we need it? The server needing to support multiple clients is not a sufficient answer to impose new PCI bus types with an implicit restriction on the VM.
Hi Alex,
I believe there are two separate problems with running PCI devices in the vfio-user server. The first one is concerning memory isolation and second one is vectoring of BAR accesses (as explained below).
In our previous patches (v3), we used an IOMMU to isolate memory spaces. But we still had trouble with the vectoring. So we implemented separate address spaces for each PCIBus to tackle both problems simultaneously, based on the feedback we got.
Quoted 8 lines
The following gives an overview of issues concerning vectoring of BAR accesses.
The device’s BAR regions are mapped into the guest physical address space. The guest writes the guest PA of each BAR into the device’s BAR registers. To access the BAR regions of the device, QEMU uses
address_space_rw()which vectors the physical address access to the device BAR region handlers.Quoted 5 lines
The guest physical address written to the BAR is irrelevant from the device perspective, this only serves to assign the BAR an offset within the
address_space_mem, which is used by the vCPU (and possibly other devices depending on their address space). There is no reason for the device itself to care about this address.Quoted 7 lines
Thank you for the explanation, Alex!
The confusion at my part is whether we are inside the device already when the server receives a request to access BAR region of a device. Based on your explanation, I get that your view is the BAR access request has propagated into the device already, whereas I was under the impression that the request is still on the CPU side of the PCI root complex.
If you are getting an access through your MemoryRegionOps, all the translations have been made, you simply need to use the hwaddr as the offset into the MemoryRegion for the access. Perform the read/write to your device, no further translations required.
Quoted 8 lines
Your view makes sense to me - once the BAR access request reaches the client (on the other side), we could consider that the request has reached the device.
On a separate note, if devices don’t care about the values in BAR registers, why do the default PCI config handlers intercept and map the BAR region into
address_space_mem? (pci_default_write_config()->pci_update_mappings())This is the part that's actually placing the BAR MemoryRegion as a sub-region into the vCPU address space. I think if you track it, you'll see PCIDevice.io_regions[i].address_space is actually
system_memory, which is used to initializeaddress_space_system.The machine assembles PCI devices onto buses as instructed by the command line or hot plug operations. It's the responsibility of the guest firmware and guest OS to probe those devices, size the BARs, and place the BARs into the memory hierarchy of the PCI bus, ie. system memory. The BARs are necessarily in the "guest physical memory" for vCPU access, but it's essentially only coincidental that PCI devices might be in an address space that provides a mapping to their own BAR. There's no reason to ever use it.
Quoted 8 lines
In the vIOMMU case, we can't know that the device address space includes those BAR mappings or if they do, that they're identity mapped to the physical address. Devices really need to not infer anything about an address. Think about real hardware, a device is told by driver programming to perform a DMA operation. The device doesn't know the target of that operation, it's the guest driver's responsibility to make sure the IOVA within the device address space is valid and maps to the desired target. Thanks,
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Quoted 6 lines
Quoted 3 lines
Hi Michael,
In LSI case, I did notice that it went to the IOMMU.
Hmm do you mean you analyzed how a physical device works? Or do you mean in QEMU?
I mean in QEMU, I did not analyze a physical device.
Quoted 8 lines
Quoted 3 lines
The device is reading the BAR
address as if it was a DMA address.I got that, my understanding of PCI was that a device can not be both a master and a target of a transaction at the same time though. Could not find this in the spec though, maybe I remember incorrectly.
I see, OK. If this were to happen in a real device, PCI would raise an error because the master and target of a transaction can’t be the same. So you believe that this access is handled inside the device, and doesn’t go out.
Thanks!
--
Jag
Quoted 46 lines
Quoted 46 lines
Quoted 17 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
Thank you!
--
JagQuoted 18 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
Quoted 5 lines
--
JagQuoted 2 lines
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Michael S. TsirkinQuoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
I don't think this is correct. Look for instance at ACPI
_TRAsupport where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.Quoted 1 line
From the processor perspective, each MMIO range is unique and possibly
Quoted 8 lines
Quoted 2 lines
none of those devices have a zero
_TRA, there could be system memory at the equivalent flat memory address.I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have
_TRA.
How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used _TRA to provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.
Quoted 15 lines
Quoted 7 lines
So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Quoted 7 lines
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
Quoted 3 lines
However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.
A shortcut that breaks when an IOMMU is involved.
Quoted 6 lines
Quoted 3 lines
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.
You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).
We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 83 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
I don't think this is correct. Look for instance at ACPI
_TRAsupport where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture. From the processor perspective, each MMIO range is unique and possibly none of those devices have a zero_TRA, there could be system memory at the equivalent flat memory address.So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
Quoted 2 lines
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this
It does look like the driver is actually programming the device to DMA to itself.
The driver first programs the DSP (DMA Scripts Pointer) register with the BAR address. It does so by performing a series of MMIO writes (lsi_mmio_write()) to offsets 0x2C - 0x2F. Immediately after programming this register, the device fetches some instructions located at the programmed address.
Thank you!
--
Jag
Quoted 21 lines
driver is actually compatible with an IOMMU on bare metal.
Quoted 11 lines
Quoted 11 lines
within their own BAR range, and PCIe is software-compatible with PCI. So devices won't be able to access this IOVA even if it was programmed in the IOMMU.
As was mentioned elsewhere on this thread, devices accessing each other's BAR is a different matter.
I do not remember which rules apply to multiple functions of a multi-function device though. I think in a traditional PCI they will never go out on the bus, but with e.g. SRIOV they would probably do go out? Alex, any idea?
This falls under implementation specific behavior in the spec, IIRC. This is actually why IOMMU grouping requires ACS support on multi-function devices to clarify the behavior of p2p between functions in the same slot. Thanks,
Alex
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
In reply to Alex WilliamsonQuoted 74 lines · Alex Williamson
Quoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
I don't think this is correct. Look for instance at ACPI
_TRAsupport where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.Quoted 1 line
From the processor perspective, each MMIO range is unique and possibly
Quoted 8 lines
Quoted 2 lines
none of those devices have a zero
_TRA, there could be system memory at the equivalent flat memory address.I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have
_TRA.
Quoted 4 lines
How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used
_TRAto provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.
It's an interesting idea. To clarify what I said, I suspect some devices are broken in presence of translating bridges unless DMA
is also translated to match.
I agree it's a mess though, in that some devices when given their own BAR to DMA to will probably just satisfy the access from internal memory, while others will ignore that and send it up as DMA and both types are probably out there in the field.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
Quoted 5 lines
Quoted 3 lines
However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.
A shortcut that breaks when an IOMMU is involved.
Maybe. But if that's how hardware behaves, we have little choice but emulate it.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.
You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).
Quoted 6 lines
We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,
Alex
More importantly lots devices most likely don't support arbitrary configurations and break if you try to create something that matches the spec but never or even rarely occurs on bare metal.
Signature · 1 line
MST
Re: [PATCH v5 03/18] pci: isolated address space for PCI bus
Quoted 109 lines · Michael S. Tsirkin
Quoted 74 lines · Alex Williamson
Quoted 62 lines · On Thu, 10 Feb 2022 18:28:56 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 61 lines · Alex Williamson
Quoted 45 lines · On Thu, 10 Feb 2022 22:23:01 +0000 Jag Raman <[email protected]> wrote:
Quoted 21 lines · Michael S. Tsirkin
Quoted 7 lines · Jag Raman
Thanks for the explanation, Alex. Thanks to everyone else in the thread who helped to clarify this problem.
We have implemented the memory isolation based on the discussion in the thread. We will send the patches out shortly.
Quoted 10 lines
Devices such as “name" and “e1000” worked fine. But I’d like to note that the LSI device (
TYPE_LSI53C895A) had some problems - it doesn’t seem to be IOMMU aware. In LSI’s case, the kernel driver is asking the device to read instructions from the CPU VA (lsi_execute_script()->read_dword()), which is forbidden when IOMMU is enabled. Specifically, the driver is asking the device to access other BAR regions by using the BAR address programmed in the PCI config space. This happens even without vfio-user patches. For example, we could enable IOMMU using “-device intel-iommu” QEMU option and also adding the following to the kernel command-line: “intel_iommu=on iommu=nopt”. In this case, we could see an IOMMU fault.Quoted 2 lines
So, device accessing its own BAR is different. Basically, these transactions never go on the bus at all, never mind get to the IOMMU.
Hi Michael,
In LSI case, I did notice that it went to the IOMMU. The device is reading the BAR address as if it was a DMA address.
Quoted 16 lines
I think it's just used as a handle to address internal device memory. This kind of trick is not universal, but not terribly unusual.
Quoted 12 lines
Unfortunately, we started off our project with the LSI device. So that lead to all the confusion about what is expected at the server end in-terms of vectoring/address-translation. It gave an impression as if the request was still on the CPU side of the PCI root complex, but the actual problem was with the device driver itself.
I’m wondering how to deal with this problem. Would it be OK if we mapped the device’s BAR into the IOVA, at the same CPU VA programmed in the BAR registers? This would help devices such as LSI to circumvent this problem. One problem with this approach is that it has the potential to collide with another legitimate IOVA address. Kindly share your thought on this.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thank you!
I am not 100% sure what do you plan to do but it sounds fine since even if it collides, with traditional PCI device must never initiate cycles
OK sounds good, I’ll create a mapping of the device BARs in the IOVA.
I don't think this is correct. Look for instance at ACPI
_TRAsupport where a system can specify a translation offset such that, for example, a CPU access to a device is required to add the provided offset to the bus address of the device. A system using this could have multiple root bridges, where each is given the same, overlapping MMIO aperture.Quoted 1 line
From the processor perspective, each MMIO range is unique and possibly
Quoted 8 lines
Quoted 2 lines
none of those devices have a zero
_TRA, there could be system memory at the equivalent flat memory address.I am guessing there are reasons to have these in acpi besides firmware vendors wanting to find corner cases in device implementations though :). E.g. it's possible something else is tweaking DMA in similar ways. I can't say for sure and I wonder why do we care as long as QEMU does not have
_TRA.Quoted 4 lines
How many complaints do we get about running out of I/O port space on q35 because we allow an arbitrary number of root ports? What if we used
_TRAto provide the full I/O port range per root port? 32-bit MMIO could be duplicated as well.It's an interesting idea. To clarify what I said, I suspect some devices are broken in presence of translating bridges unless DMA
is also translated to match.I agree it's a mess though, in that some devices when given their own BAR to DMA to will probably just satisfy the access from internal memory, while others will ignore that and send it up as DMA and both types are probably out there in the field.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
So if the transaction actually hits this bus, which I think is what making use of the device AddressSpace implies, I don't think it can assume that it's simply reflected back at itself. Conventional PCI and PCI Express may be software compatible, but there's a reason we don't see IOMMUs that provide both translation and isolation in conventional topologies.
Is this more a bug in the LSI device emulation model? For instance in vfio-pci, if I want to access an offset into a BAR from within QEMU, I don't care what address is programmed into that BAR, I perform an access relative to the vfio file descriptor region representing that BAR space. I'd expect that any viable device emulation model does the same, an access to device memory uses an offset from an internal resource, irrespective of the BAR address.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
However, using BAR seems like a reasonable shortcut allowing device to use the same 64 bit address to refer to system and device RAM interchangeably.
A shortcut that breaks when an IOMMU is involved.
Maybe. But if that's how hardware behaves, we have little choice but emulate it.
I was wondering if we could map the BARs into the IOVA for a limited set of devices - the ones which are designed before IOMMU such as lsi53c895a. This would ensure that we follow the spec to the best without breaking existing devices?
--
Jag
Quoted 22 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
It would seem strange if the driver is actually programming the device to DMA to itself and if that's actually happening, I'd wonder if this driver is actually compatible with an IOMMU on bare metal.
You know, it's hardware after all. As long as things work vendors happily ship the device. They don't really worry about theoretical issues ;).
Quoted 6 lines
We're talking about a 32-bit conventional PCI device from the previous century. IOMMUs are no longer theoretical, but not all drivers have kept up. It's maybe not the best choice as the de facto standard device, imo. Thanks,
Alex
More importantly lots devices most likely don't support arbitrary configurations and break if you try to create something that matches the spec but never or even rarely occurs on bare metal.
Signature · 1 line
MST
Re: [PATCH v5 01/18] configure, meson: override C compiler for cmake
In reply to Paolo BonziniQuoted 16 lines · Paolo Bonzini
Quoted 10 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.Explicitly specify the C compiler for cmake to avoid this error
Signed-off-by Jagannathan Raman <[email protected]>Acked-by Paolo Bonzini <[email protected]>
Quoted 1 line
This should not be needed anymore, as the bug in Meson has been fixed.
Hi Paolo,
I’m able to see the bug with latest QEMU. The fix doesn’t appear to be available with meson version 0.59.3, which is what QEMU is presently using.
Thank you!
--
Jag
Quoted 19 lines
Paolo
Quoted 15 lines
configure | 2 ++ 1 file changed, 2 insertions(+)configure +2
-0@@ -3747,6 +3747,8 @@ if test "$skip_meson" = no; then echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross + echo "[cmake]" >> $cross + echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross echo "[binaries]" >> $cross echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross