vfio-user server in QEMU
Subsystems (2)
v6 cover vfio-user server in QEMU
Hi,
This is v6 of the server side changes to enable vfio-user in QEMU.
Thank you very much for your feedback for the last revision which helped to streamline the overall design. We've made the following changes to this revision:
[PATCH v6 03/19] qdev: unplug blocker for devices
- removed test which prevented an unplug blocker from getting added if migration was in progress
- added comments to function
[PATCH v6 04/19] remote/machine: add HotplugHandler for remote machine
- changed commit message prefix from vfio-user to "remote/machine"
[PATCH v6 05/19] remote/machine: add vfio-user property
- new in this series
[PATCH v6 07/19] vfio-user: define vfio-user-server object
- fixed typo noted in the review
- moved error message before setting "o->socket = NULL" in vfu_object_set_socket()
- added "vfio-user=on" to the usage comment at the top of file
[PATCH v6 08/19] vfio-user: instantiate vfio-user context
- added error message to the object set property message when server is already running
[PATCH v6 09/19] vfio-user: find and init PCI device
- added more detailed error message for device unplug blocker
[PATCH v6 10/19] vfio-user: run vfio-user context
- send ID of device in VFU_CLIENT_HANGUP instead of path
- disable FD handler in object finalize
[PATCH v6 12/19] vfio-user: IOMMU support for remote device
- new in this series
[PATCH v6 13/19] vfio-user: handle DMA mappings
- Setup IOMMU for remote machine if vfio-user is enabled
- Map/Unmap the DMA regions in the IOMMU address space in dma_register()/dma_unregister() using pci_device_iommu_address_space() function
[PATCH v6 14/19] vfio-user: handle PCI BAR accesses
- vfu_object_bar_rw() - directly access the bar region instead of accessing via address_space_rw()
- register handler for PCI ROM region
- set read only flags for read only MemoryRegions with vfu_setup_region()
[PATCH v6 15/19] vfio-user: handle device interrupts
- setup separate PCI bus map_irq and set_irq for vfio-user during remote machine init
- index hash table using PCI bud device function numbers
[PATCH v6 16/19] vfio-user: handle device interrupts
- new in this series
[PATCH v6 17/19] vfio-user: register handlers to facilitate migration
- enable streaming for migration data instead pre-determining the migration data size at boot
- dropped migrated_devs static variable to track the number of devices migrated
- added helper functions to independently start stop block and network devices
- updated qemu_remote_savevm() to migrate data of all the devices under the target device
[PATCH v6 18/19] vfio-user: handle reset of remote device
- new in this series
[PATCH v6 19/19] vfio-user: avocado tests for vfio-user
- use QMP command for hotplug instead of HMP command
- confirm the state of source and destination VMs after migration
- testing megasas device instead of lsi53c895a as lsi53c895a doesn't seem to support IOMMU, which is enabled by default on the server
We dropped the following patches from the previous revision:
- pci: isolated address space for PCI bus
- pci: create and free isolated PCI buses
- vfio-user: set qdev bus callbacks for remote machine
We are looking forward to your comments.
Thank you very much!
Jagannathan Raman (19):
configure, meson: override C compiler for cmake tests/avocado: Specify target VM argument to helper routines
qdev: unplug blocker for devices
remote/machine: add HotplugHandler for remote machine remote/machine: add vfio-user property
vfio-user: build library
vfio-user: define vfio-user-server object
vfio-user: instantiate vfio-user context
vfio-user: find and init PCI device
vfio-user: run vfio-user context
vfio-user: handle PCI config space accesses
vfio-user: IOMMU support for remote device
vfio-user: handle DMA mappings
vfio-user: handle PCI BAR accesses
vfio-user: handle device interrupts
softmmu/vl: defer backend init
vfio-user: register handlers to facilitate migration
vfio-user: handle reset of remote device
vfio-user: avocado tests for vfio-user
configure | 21 +- meson.build | 44 +- qapi/misc.json | 23 + qapi/qom.json | 20 +- include/block/block.h | 1 + include/exec/memory.h | 3 + include/hw/pci/pci.h | 6 + include/hw/qdev-core.h | 35 + include/hw/remote/iommu.h | 18 + include/hw/remote/machine.h | 2 + include/hw/remote/vfio-user-obj.h | 6 + include/migration/vmstate.h | 2 + include/sysemu/sysemu.h | 4 + migration/savevm.h | 2 + block.c | 5 + block/block-backend.c | 3 +- blockdev.c | 2 +- hw/pci/msi.c | 13 +- hw/pci/msix.c | 12 +- hw/remote/iommu.c | 78 ++ hw/remote/machine.c | 54 +- hw/remote/vfio-user-obj.c | 1286 ++++++++++++++++++++ migration/savevm.c | 89 ++ migration/vmstate.c | 19 + softmmu/physmem.c | 4 +- softmmu/qdev-monitor.c | 26 + softmmu/vl.c | 17 + stubs/defer-backend-init.c | 7 + stubs/vfio-user-obj.c | 6 + tests/qtest/fuzz/generic_fuzz.c | 9 +- .gitlab-ci.d/buildtest.yml | 2 + .gitmodules | 3 + Kconfig.host | 4 + MAINTAINERS | 7 + hw/remote/Kconfig | 4 + hw/remote/meson.build | 4 + hw/remote/trace-events | 11 + meson_options.txt | 2 + stubs/meson.build | 2 + subprojects/libvfio-user | 1 + tests/avocado/avocado_qemu/__init__.py | 14 +- tests/avocado/vfio-user.py | 234 ++++ tests/docker/dockerfiles/centos8.docker | 2 + tests/docker/dockerfiles/ubuntu2004.docker | 2 + 44 files changed, 2088 insertions(+), 21 deletions(-) create mode 100644 include/hw/remote/iommu.h create mode 100644 include/hw/remote/vfio-user-obj.h create mode 100644 hw/remote/iommu.c create mode 100644 hw/remote/vfio-user-obj.c create mode 100644 stubs/defer-backend-init.c create mode 100644 stubs/vfio-user-obj.c create mode 160000 subprojects/libvfio-user create mode 100644 tests/avocado/vfio-user.py
Signature · 1 line
2.20.1
PATCH v6 1/19 configure, meson: override C compiler for cmake
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Explicitly specify the C compiler for cmake to avoid this error
--- configure | 2 ++ 1 file changed, 2 insertions(+)
configure +2 -0
@@ -3726,6 +3726,8 @@ if test "$skip_meson" = no; then echo "cpp_args = [$(meson_quote $CXXFLAGS $EXTRA_CXXFLAGS)]" >> $cross echo "c_link_args = [$(meson_quote $CFLAGS $LDFLAGS $EXTRA_CFLAGS $EXTRA_LDFLAGS)]" >> $cross echo "cpp_link_args = [$(meson_quote $CXXFLAGS $LDFLAGS $EXTRA_CXXFLAGS $EXTRA_LDFLAGS)]" >> $cross + echo "[cmake]" >> $cross + echo "CMAKE_C_COMPILER = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross echo "[binaries]" >> $cross echo "c = [$(meson_quote $cc $CPU_CFLAGS)]" >> $cross test -n "$cxx" && echo "cpp = [$(meson_quote $cxx $CPU_CFLAGS)]" >> $cross
Signature · 1 line
2.20.1
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 10 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
thanks
-- PMM
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 14 lines · Peter Maydell
Quoted 9 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Quoted 4 lines
Quoted 1 line
Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
Hi Peter,
I’ll try to locate the bug report and the meson version which has the fix, and get back to you.
Thank you!
--
Jag
Quoted 3 lines
thanks
-- PMM
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
In reply to Peter MaydellQuoted 14 lines · Peter Maydell
Quoted 9 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Quoted 4 lines
Quoted 1 line
Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
Hi Peter,
This issue reproduces with the latest meson [1] also.
I noticed the following about the “binaries” section [2]. The manual says meson could pass the values in this section to find_program [3]. As such I’m wondering if it’s OK to set compiler flags in this section because find_program doesn’t seem to accept any compiler flags.
The compiler flags could be set in the “built-in options” section using options such as “c_args”, “cpp_args” and “objc_args” [4]. When I moved CPU_CFLAGS from the binaries section to the built-in-options section in “configure", I don’t see the issue anymore.
[1]: https://github.com/mesonbuild/meson.git
[2]: https://mesonbuild.com/Machine-files.html#binaries
[3]: https://cmake.org/cmake/help/latest/command/find_program.html
[4]: https://github.com/mesonbuild/meson/blob/master/docs/markdown/Reference-tables.md (section “Language arguments parameter names")
Thank you!
--
Jag
Quoted 3 lines
thanks
-- PMM
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 14 lines · Jag Raman
Quoted 14 lines · Peter Maydell
Quoted 9 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
Hi Peter,
This issue reproduces with the latest meson [1] also.
0.60.0 or more recent versions should have a fix, which would do exactly what this patch does: do not define CMAKE_C_COMPILER_LAUNCHER, and place the whole binaries.c variable in CMAKE_C_COMPILER. What are the contents of the genrated CMakeMesonToolchainFile.cmake and CMakeCache.txt files, without and with your patch?
Quoted 4 lines
I noticed the following about the “binaries” section [2]. The manual says meson could pass the values in this section to
find_program[3]. As such I’m wondering if it’s OK to set compiler flags in this section becausefind_programdoesn’t seem to accept any compiler flags.
The full quote of the manual is "These can be used internally by Meson, or by the find_program function", and the C compiler variable "c" is in the former category.
There is an important difference between the flags in "binaries" and those in "built-in options". What is in "binaries" is used when requesting e.g. the compiler search path, while what is in "built-in options" is not. So options like "-m32" are definitely part of "binaries", not "built-in options":
$ gcc --print-multi-os-directory
../lib64
$ gcc -m32 --print-multi-os-directory
../libPaolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 17 lines · Paolo Bonzini
Quoted 12 lines · Jag Raman
Quoted 12 lines · Peter Maydell
Quoted 9 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
Hi Peter,
This issue reproduces with the latest meson [1] also.0.60.0 or more recent versions should have a fix, which would do exactly what this patch does: do not define
CMAKE_C_COMPILER_LAUNCHER, and place the wholebinaries.cvariable inCMAKE_C_COMPILER. What are the contents of the genrated CMakeMesonToolchainFile.cmake and CMakeCache.txt files, without and with your patch?
I’ll checkout what’s going on at my end. But the issue reproduces with meson 0.61 from what I can tell:
# ../configure --target-list=x86_64-softmmu --enable-debug --enable-vfio-user-server; The Meson build system Version: 0.61.2 … … | /opt/rh/devtoolset-9/root/usr/bin/cc;-m64;-mcx16
| is not a full path to an existing compiler tool.
Concerning the generated files, I see the following in CMakeMesonToolchainFile.cmake:
Without patch: set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc" "-m64" "-mcx16”) With patch: set(CMAKE_C_COMPILER "cc" "-m64" "-mcx16")
Quoted 6 lines
Quoted 5 lines
I noticed the following about the “binaries” section [2]. The manual says meson could pass the values in this section to
find_program[3]. As such I’m wondering if it’s OK to set compiler flags in this section becausefind_programdoesn’t seem to accept any compiler flags.
Quoted 8 lines
The full quote of the manual is "These can be used internally by Meson, or by the
find_programfunction", and the C compiler variable "c" is in the former category.There is an important difference between the flags in "binaries" and those in "built-in options". What is in "binaries" is used when requesting e.g. the compiler search path, while what is in "built-in options" is not. So options like "-m32" are definitely part of "binaries", not "built-in options":
$ gcc --print-multi-os-directory ../lib64 $ gcc -m32 --print-multi-os-directory ../lib
Do you know if the “host_machine” section in cross build definition file [1] would be any help here?
[1]: https://mesonbuild.com/Cross-compilation.html#machine-entries
--
Jag
Quoted 2 lines
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 39 lines · Jag Raman
Quoted 18 lines · Paolo Bonzini
Quoted 12 lines · Jag Raman
Quoted 12 lines · Peter Maydell
Quoted 9 lines · Jagannathan Raman
The compiler path that cmake gets from meson is corrupted. It results in the following error:
| -- The C compiler identification is unknown | CMake Error at CMakeLists.txt:35 (project): | The CMAKE_C_COMPILER: | /opt/rh/devtoolset-9/root/bin/cc;-m64;-mcx16 | is not a full path to an existing compiler tool.Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Explicitly specify the C compiler for cmake to avoid this error
This sounds like a bug in Meson. Is there a Meson bug report we can reference in the commit message here ?
Hi Peter,
This issue reproduces with the latest meson [1] also.0.60.0 or more recent versions should have a fix, which would do exactly what this patch does: do not define
CMAKE_C_COMPILER_LAUNCHER, and place the wholebinaries.cvariable inCMAKE_C_COMPILER. What are the contents of the genrated CMakeMesonToolchainFile.cmake and CMakeCache.txt files, without and with your patch?I’ll checkout what’s going on at my end. But the issue reproduces with meson 0.61 from what I can tell:
# ../configure --target-list=x86_64-softmmu --enable-debug --enable-vfio-user-server; The Meson build system Version: 0.61.2 … … | /opt/rh/devtoolset-9/root/usr/bin/cc;-m64;-mcx16
Quoted 6 lines
| is not a full path to an existing compiler tool.
Concerning the generated files, I see the following in CMakeMesonToolchainFile.cmake:
Without patch:set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc" "-m64" "-mcx16”)With patch:set(CMAKE_C_COMPILER "cc" "-m64" "-mcx16")
I’m not sure if you’re interested in the contents of the whole file. But they’re here:
Without patch: https://pastebin.com/sbwtvHy0 (also has error log at the end) With patch: https://pastebin.com/buRYSp2R
Thank you!
--
Jag
Quoted 26 lines
Quoted 7 lines
Quoted 6 lines
I noticed the following about the “binaries” section [2]. The manual says meson could pass the values in this section to
find_program[3]. As such I’m wondering if it’s OK to set compiler flags in this section becausefind_programdoesn’t seem to accept any compiler flags.Quoted 8 lines
The full quote of the manual is "These can be used internally by Meson, or by the
find_programfunction", and the C compiler variable "c" is in the former category.There is an important difference between the flags in "binaries" and those in "built-in options". What is in "binaries" is used when requesting e.g. the compiler search path, while what is in "built-in options" is not. So options like "-m32" are definitely part of "binaries", not "built-in options":
$ gcc --print-multi-os-directory ../lib64 $ gcc -m32 --print-multi-os-directory ../libDo you know if the “host_machine” section in cross build definition file [1] would be any help here?
[1]: https://mesonbuild.com/Cross-compilation.html#machine-entries
--
JagQuoted 2 lines
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
In reply to Jag RamanQuoted 4 lines · Jag Raman
Concerning the generated files, I see the following in CMakeMesonToolchainFile.cmake:
Without patch:set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc" "-m64" "-mcx16”)With patch:set(CMAKE_C_COMPILER "cc" "-m64" "-mcx16")
I don't understand why it works at all with the latter, but the right solution could be
set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc") set(CMAKE_C_COMPILER_ARG1 "-m64") set(CMAKE_C_COMPILER_ARG2 "-mcx16")
Perhaps you can try the following patch to meson (patch it in qemu's build directory and make sure to use --meson=internal):
mesonbuild/cmake/toolchain.py +5 -2
@@ -191,11 +191,14 @@ class CMakeToolchain: continue if len(exe_list) >= 2 and not self.is_cmdline_option(comp_obj, exe_list[1]): - defaults[prefix + 'COMPILER_LAUNCHER'] = [make_abs(exe_list[0])] + defaults[f'{prefix}COMPILER_LAUNCHER'] = [make_abs(exe_list[0])] exe_list = exe_list[1:] exe_list[0] = make_abs(exe_list[0]) - defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()
Thanks,
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 6 lines · Paolo Bonzini
Quoted 4 lines · Jag Raman
Concerning the generated files, I see the following in CMakeMesonToolchainFile.cmake:
Without patch:set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc" "-m64" "-mcx16”)With patch:set(CMAKE_C_COMPILER "cc" "-m64" "-mcx16")
Quoted 6 lines
I don't understand why it works at all with the latter, but the right solution could be
set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc")set(CMAKE_C_COMPILER_ARG1 "-m64")set(CMAKE_C_COMPILER_ARG2 "-mcx16")
Anyhow it seems to be a cmake bug, because what QEMU/Meson are doing is exactly what is in the manual at https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html#variable:CMAKE_%3CLANG%3E_COMPILER:
Note: Options that are required to make the compiler work correctly can be included as items in a list; they can not be changed.
#set within user supplied toolchain file set(CMAKE_C_COMPILER /full/path/to/qcc --arg1 --arg2)
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
In reply to Paolo BonziniQuoted 27 lines · Paolo Bonzini
Quoted 3 lines · Jag Raman
Concerning the generated files, I see the following in CMakeMesonToolchainFile.cmake:
Without patch:set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc" "-m64" "-mcx16”)With patch:set(CMAKE_C_COMPILER "cc" "-m64" "-mcx16")I don't understand why it works at all with the latter, but the right solution could be
set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc")set(CMAKE_C_COMPILER_ARG1 "-m64")set(CMAKE_C_COMPILER_ARG2 "-mcx16")Perhaps you can try the following patch to meson (patch it in qemu's build directory and make sure to use --meson=internal):
mesonbuild/cmake/toolchain.py +0
-1@@ -191,11 +191,14 @@ class CMakeToolchain: continue if len(exe_list) >= 2 and not self.is_cmdline_option(comp_obj, exe_list[1]): - defaults[prefix + 'COMPILER_LAUNCHER'] = [make_abs(exe_list[0])]
mesonbuild/cmake/toolchain.py class CMakeToolchain: · line 201
Quoted 10 lines
mesonbuild/cmake/toolchain.py +5
-1+ defaults[f'{prefix}COMPILER_LAUNCHER'] = [make_abs(exe_list[0])] exe_list = exe_list[1:] exe_list[0] = make_abs(exe_list[0]) - defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()
This fix works at my end.
Thank you!
--
Jag
Quoted 5 lines
mesonbuild/cmake/toolchain.py +0
-0Thanks,Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 8 lines · Jag Raman
mesonbuild/cmake/toolchain.py class CMakeToolchain: · line 203
Quoted 7 lines
mesonbuild/cmake/toolchain.py +4
-1- defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()This fix works at my end.
Would you please check that -m64 and -mcx16 are passed indeed to the compiler?
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 12 lines · Paolo Bonzini
Quoted 7 lines · Jag Raman
Quoted 7 lines
mesonbuild/cmake/toolchain.py +4
-1- defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()
Quoted 3 lines
Quoted 1 line
This fix works at my end.
Would you please check that -m64 and -mcx16 are passed indeed to the compiler?
Hi Paolo,
Yes, I’m able to see that -m64 and -mcx16 are passed to the compiler.
# cat ./subprojects/libvfio-user/__CMake_build/CMakeMesonToolchainFile.cmake … set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc") set(CMAKE_C_COMPILER_ARG1 "-m64") set(CMAKE_C_COMPILER_ARG2 "-mcx16") …
Full log here: https://pastebin.com/PEwNSWMn
Thank you!
--
Jag
Quoted 2 lines
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 18 lines · Jag Raman
Quoted 12 lines · Paolo Bonzini
Quoted 7 lines · Jag Raman
Quoted 7 lines
mesonbuild/cmake/toolchain.py +4
-1- defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()Quoted 3 lines
Quoted 1 line
This fix works at my end.
Would you please check that -m64 and -mcx16 are passed indeed to the compiler?
Hi Paolo,
Quoted 8 lines
Yes, I’m able to see that -m64 and -mcx16 are passed to the compiler.
# cat ./subprojects/libvfio-user/__CMake_build/CMakeMesonToolchainFile.cmake … set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc") set(CMAKE_C_COMPILER_ARG1 "-m64") set(CMAKE_C_COMPILER_ARG2 "-mcx16") …
I reproduced this with CMake 3.17.x and it's fixed by 3.19.x. I'll send the fix to Meson, but for now I recommend requiring a newer CMake and just dropping this patch.
Paolo
Re: [PATCH v6 01/19] configure, meson: override C compiler for cmake
Quoted 27 lines · Paolo Bonzini
Quoted 13 lines · Jag Raman
Quoted 10 lines · Paolo Bonzini
Quoted 7 lines · Jag Raman
Quoted 7 lines
mesonbuild/cmake/toolchain.py +4
-1- defaults[prefix + 'COMPILER'] = exe_list + defaults[f'{prefix}COMPILER'] = [exe_list[0]] + for i in range(1, len(exe_list)): + defaults[f'{prefix}COMPILER_ARG{i}'] = [exe_list[i]] + if comp_obj.get_id() == 'clang-cl': defaults['CMAKE_LINKER'] = comp_obj.get_linker_exelist()Quoted 3 lines
Quoted 1 line
This fix works at my end.
Would you please check that -m64 and -mcx16 are passed indeed to the compiler?
Quoted 8 lines
Hi Paolo,
Yes, I’m able to see that -m64 and -mcx16 are passed to the compiler.# cat ./subprojects/libvfio-user/__CMake_build/CMakeMesonToolchainFile.cmake … set(CMAKE_C_COMPILER "/opt/rh/devtoolset-9/root/usr/bin/cc") set(CMAKE_C_COMPILER_ARG1 "-m64") set(CMAKE_C_COMPILER_ARG2 "-mcx16") …
Quoted 1 line
I reproduced this with CMake 3.17.x and it's fixed by 3.19.x. I'll send the fix to Meson, but for now I recommend requiring a newer CMake and just dropping this patch.
OK, got it.
Thank you!
--
Jag
Quoted 2 lines
Paolo
PATCH v6 2/19 tests/avocado: Specify target VM argument to helper routines
In reply to Jagannathan RamanSpecify target VM for exec_command andexec_command_and_wait_for_pattern routines
--- tests/avocado/avocado_qemu/__init__.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
tests/avocado/avocado_qemu/__init__.py +10 -4
@@ -198,7 +198,7 @@ def wait_for_console_pattern(test, success_message, failure_message=None, """ _console_interaction(test, success_message, failure_message, None, vm=vm) -def exec_command(test, command): +def exec_command(test, command, vm=None): """ Send a command to a console (appending CRLF characters), while logging the content.
@@ -207,11 +207,14 @@ def exec_command(test, command): :type test: :class:`avocado_qemu.QemuSystemTest` :param command: the command to send :type command: str + :param vm: target vm + :type vm: :class:`qemu.machine.QEMUMachine` """ - _console_interaction(test, None, None, command + '\r') + _console_interaction(test, None, None, command + '\r', vm=vm) def exec_command_and_wait_for_pattern(test, command, - success_message, failure_message=None): + success_message, failure_message=None, + vm=None): """ Send a command to a console (appending CRLF characters), then wait for success_message to appear on the console, while logging the.
@@ -223,8 +226,11 @@ def exec_command_and_wait_for_pattern(test, command, :param command: the command to send :param success_message: if this message appears, test succeeds :param failure_message: if this message appears, test fails + :param vm: target vm + :type vm: :class:`qemu.machine.QEMUMachine` """ - _console_interaction(test, success_message, failure_message, command + '\r') + _console_interaction(test, success_message, failure_message, command + '\r', + vm=vm) class QemuBaseTest(avocado.Test): def _get_unique_tag_val(self, tag_name):
Signature · 1 line
2.20.1
PATCH v6 3/19 qdev: unplug blocker for devices
In reply to Jagannathan RamanAdd blocker to prevent hot-unplug of devices
--- include/hw/qdev-core.h | 35 +++++++++++++++++++++++++++++++++++ softmmu/qdev-monitor.c | 26 ++++++++++++++++++++++++++ 2 files changed, 61 insertions(+)
include/hw/qdev-core.h +35 -0
@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {
@@ -419,6 +420,40 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void ); +/** +* Device Unplug blocker: prevents a device from being unplugged. It could +* be used to indicate that another object depends on the device. +* +* qdev_add_unplug_blocker: Adds an unplug blocker to a device +* +* @dev: Device to be blocked from unplug +* @reason: Reason for blocking +* +*/ +void qdev_add_unplug_blocker (DeviceState *dev,Error *reason); + +/** +* qdev_del_unplug_blocker: Removes an unplug blocker from a device +* +* @dev: Device to be unblocked +* @reason: Pointer to the Error used with qdev_add_unplug_blocker. +* Used as a handle to lookup the blocker for deletion. +* +*/ +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); + +/** +* qdev_unplug_blocked: Confirms if a device is blocked from unplug +* +* @dev: Device to be tested +* @reason: Returns one of the reasons why the device is blocked, +* if any +* +* Returns: true if device is blocked from unplug, false otherwise +* +*/ +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line *
softmmu/qdev-monitor.c +26 -0
@@ -945,10 +945,36 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } } +void qdev_add_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;
Signature · 1 line
2.20.1
Re: [PATCH v6 03/19] qdev: unplug blocker for devices
Quoted 23 lines · Jagannathan Raman
Add blocker to prevent hot-unplug of devices
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/qdev-core.h | 35 +++++++++++++++++++++++++++++++++++ softmmu/qdev-monitor.c | 26 ++++++++++++++++++++++++++ 2 files changed, 61 insertions(+)include/hw/qdev-core.h +1
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {
include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 424
Quoted 5 lines
include/hw/qdev-core.h +2
-0@@ -419,6 +420,40 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void ); +/** +* Device Unplug blocker: prevents a device from being unplugged. It could
^^^^^^^^^^^^^^^^^^^^^
This looks strange. gtkdoc will probably treat it as the doc comment for qdev_add_unplug_blocker(), which is actually defined below. I suggest not trying to define a new section in the documentation and instead just focussing on doc comments for qdev_add_unplug_block() and other functions.
The gtkdoc way of defining sections is covered here but it's almost never used in QEMU:
https://developer-old.gnome.org/gtk-doc-manual/stable/documenting_sections.html.en
include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 433
Quoted 9 lines
include/hw/qdev-core.h +9
-0+* be used to indicate that another object depends on the device. +* +* qdev_add_unplug_blocker: Adds an unplug blocker to a device +* +* @dev: Device to be blocked from unplug +* @reason: Reason for blocking +* +*/ +void qdev_add_unplug_blocker (DeviceState *dev,Error *reason);
Does the caller have to call qdev_del_unplug_blocker() later?
An assert(!dev->unplug_blockers) would be nice when DeviceState is destroyed. That way leaks will be caught.
Quoted 33 lines
include/hw/qdev-core.h +23
-0+ +/** +* qdev_del_unplug_blocker: Removes an unplug blocker from a device +* +* @dev: Device to be unblocked +* @reason: Pointer to the Error used with qdev_add_unplug_blocker. +* Used as a handle to lookup the blocker for deletion. +* +*/ +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); + +/** +* qdev_unplug_blocked: Confirms if a device is blocked from unplug +* +* @dev: Device to be tested +* @reason: Returns one of the reasons why the device is blocked, +* if any +* +* Returns: true if device is blocked from unplug, false otherwise +* +*/ +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +0
-0@@ -945,10 +945,36 @@ void qmp_device_del(const char *id, Error **errp)return ; }
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 956
Quoted 10 lines
softmmu/qdev-monitor.c +5
-0+if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } } +void qdev_add_unplug_blocker (DeviceState *dev,Error *reason)
These functions belong in hw/core/qdev.c because they are part of the DeviceState API, not qdev monitor commands?
Quoted 27 lines
softmmu/qdev-monitor.c +21
-0+{ + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;Signature · 1 line
2.20.1
Re: [PATCH v6 03/19] qdev: unplug blocker for devices
Quoted 35 lines · Stefan Hajnoczi
Quoted 23 lines · Jagannathan Raman
Add blocker to prevent hot-unplug of devices
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/qdev-core.h| 35 +++++++++++++++++++++++++++++++++++softmmu/qdev-monitor.c| 26 ++++++++++++++++++++++++++2 files changed, 61 insertions(+)include/hw/qdev-core.h +1
-0@@ -193,6 +193,7 @@ struct DeviceState {int instance_id_alias;int alias_required_for_version;ResettableState reset; +GSList *unplug_blockers; };struct DeviceListener {include/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 424
Quoted 5 lines
include/hw/qdev-core.h +2
-0@@ -419,6 +420,40 @@ void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev,void qdev_machine_creation_done (void );bool qdev_machine_modified (void ); +/** +* Device Unplug blocker: prevents a device from being unplugged. It could ^^^^^^^^^^^^^^^^^^^^^
Quoted 5 lines
This looks strange. gtkdoc will probably treat it as the doc comment for
qdev_add_unplug_blocker(), which is actually defined below. I suggest not trying to define a new section in the documentation and instead just focussing on doc comments forqdev_add_unplug_block()and other functions.
Sorry I assumed that we needed an extra ‘*’ at the beginning of the comment. I got this idea when checking out block.c and blockdev.c while working on the migration patch.
I’ll follow the “Comment style” section in “docs/devel/style.rst"
Quoted 15 lines
The gtkdoc way of defining sections is covered here but it's almost never used in QEMU:
https://developer-old.gnome.org/gtk-doc-manual/stable/documenting_sections.html.eninclude/hw/qdev-core.h void qdev_simple_device_unplug_cb(HotplugHandler *hotplug_dev, · line 433
Quoted 9 lines
include/hw/qdev-core.h +9
-0+* be used to indicate that another object depends on the device. +* +* qdev_add_unplug_blocker: Adds an unplug blocker to a device +* +* @dev: Device to be blocked from unplug +* @reason: Reason for blocking +* +*/ +void qdev_add_unplug_blocker (DeviceState *dev,Error *reason);
Quoted 4 lines
Does the caller have to call
qdev_del_unplug_blocker()later?An
assert(!dev->unplug_blockers)would be nice when DeviceState is destroyed. That way leaks will be caught.
Makes sense, will add.
Quoted 43 lines
Quoted 43 lines
include/hw/qdev-core.h +23
-0+ +/** +* qdev_del_unplug_blocker: Removes an unplug blocker from a device +* +* @dev: Device to be unblocked +* @reason: Pointer to the Error used with qdev_add_unplug_blocker. +* Used as a handle to lookup the blocker for deletion. +* +*/ +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason); + +/** +* qdev_unplug_blocked: Confirms if a device is blocked from unplug +* +* @dev: Device to be tested +* @reason: Returns one of the reasons why the device is blocked, +* if any +* +* Returns: true if device is blocked from unplug, false otherwise +* +*/ +bool qdev_unplug_blocked (DeviceState *dev,Error **errp); +/** * GpioPolarity: Polarity of a GPIO line * softmmu/qdev-monitor.c +4
-0@@ -945,10 +945,36 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp); } }
Quoted 4 lines
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 956
Quoted 1 line
softmmu/qdev-monitor.c +1
-0+void qdev_add_unplug_blocker (DeviceState *dev,Error *reason)These functions belong in
hw/core/qdev.cbecause they are part of the DeviceState API, not qdev monitor commands?
Both hw/core/qdev.c and softmmu/qdev-monitor.c seem to manage the DeviceState.
softmmu/qdev-monitor.c seems to manage device addition and removal using qdev_device_add() and qdev_unplug(). I noticed that some functions in this file change DeviceState. For example, qdev_device_add() sets DeviceState->opts, qdev_set_id() sets DeviceState->id. Given the above two reasons, I thought it the unplug blockers could be better place here.
Since hw/core/qdev.c makes the majority of changes to the DeviceState, moving the unplug blockers over there makes sense to me.
Thank you!
--
Jag
Quoted 28 lines
Quoted 28 lines
softmmu/qdev-monitor.c +21
-0+{ + dev->unplug_blockers =g_slist_prepend (dev->unplug_blockers, reason); +} + +void qdev_del_unplug_blocker (DeviceState *dev,Error *reason) +{ + dev->unplug_blockers =g_slist_remove (dev->unplug_blockers, reason); +} + +bool qdev_unplug_blocked (DeviceState *dev,Error **errp) +{ +ERRP_GUARD (); + +if (dev->unplug_blockers) { +error_propagate (errp,error_copy (dev->unplug_blockers->data)); +return true; + } + +return false; +} +void hmp_device_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ;Signature · 1 line
2.20.1
Re: [PATCH v6 03/19] qdev: unplug blocker for devices
In reply to Jagannathan Ramansoftmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 952
Quoted 13 lines · Jagannathan Raman
softmmu/qdev-monitor.c +4
-0@@ -945,10 +945,36 @@ void qmp_device_del(const char *id, Error **errp)return ; } +if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp);
Can qdev_unplug() check this internally?
Re: [PATCH v6 03/19] qdev: unplug blocker for devices
Quoted 13 lines · Stefan Hajnoczi
Quoted 8 lines · Jagannathan Raman
softmmu/qdev-monitor.c +0
-0@@ -945,10 +945,36 @@ void qmp_device_del(const char *id, Error **errp)return ; }
Quoted 7 lines
softmmu/qdev-monitor.c void qmp_device_del(const char *id, Error **errp) · line 952
Quoted 5 lines
softmmu/qdev-monitor.c +4
-0+if (qdev_unplug_blocked (dev, errp)) { +return ; + } +qdev_unplug (dev, errp);Can
qdev_unplug()check this internally?
Yes, I think qdev_unplug() could check this internally. Will move it there.
Thank you!
--
Jag
PATCH v6 4/19 remote/machine: add HotplugHandler for remote machine
In reply to Jagannathan RamanAllow hotplugging of PCI(e) devices to remote machine
--- hw/remote/machine.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
hw/remote/machine.c +10 -0
@@ -21,6 +21,7 @@#include "qapi/error.h" #include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" +#include "hw/qdev-core.h" static void remote_machine_init (MachineState *machine) {
@@ -54,14 +55,19 @@ static void remote_machine_init(MachineState *machine)pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + +qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); }static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc); +HotplugHandlerClass *hc =HOTPLUG_HANDLER_CLASS (oc); mc->init = remote_machine_init; mc->desc ="Experimental remote machine" ; + + hc->unplug = qdev_simple_device_unplug_cb; }static const TypeInfo remote_machine = {
@@ -69,6 +75,10 @@ static const TypeInfo remote_machine = { .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), .class_init = remote_machine_class_init, + .interfaces = (InterfaceInfo []) { + {TYPE_HOTPLUG_HANDLER }, + { } + } };static void remote_machine_register_types (void )
Signature · 1 line
2.20.1
Re: [PATCH v6 04/19] remote/machine: add HotplugHandler for remote machine
Quoted 8 lines · Jagannathan Raman
Allow hotplugging of PCI(e) devices to remote machine
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/machine.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
PATCH v6 5/19 remote/machine: add vfio-user property
In reply to Jagannathan RamanAdd vfio-user to x-remote machine. It is a boolean, which indicates if the machine supports vfio-user protocol. The machine configures the bus differently vfio-user and multiprocess protocols, so this property informs it on how to configure the bus.
This property should be short lived. Once vfio-user fully replaces multiprocess, this property could be removed.
--- include/hw/remote/machine.h | 2 ++ hw/remote/machine.c | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+)
include/hw/remote/machine.h +2 -0
@@ -22,6 +22,8 @@ struct RemoteMachineState {RemotePCIHost *host;RemoteIOHubState iohub; + +bool vfio_user; };/* Used to pass to co-routine device and ioc. */
hw/remote/machine.c +23 -0
@@ -59,6 +59,25 @@ static void remote_machine_init(MachineState *machine)qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); } +static bool remote_machine_get_vfio_user (Object *obj,Error **errp) +{ +RemoteMachineState *s =REMOTE_MACHINE (obj); + +return s->vfio_user; +} + +static void remote_machine_set_vfio_user (Object *obj,bool value,Error **errp) +{ +RemoteMachineState *s =REMOTE_MACHINE (obj); + +if (phase_check (PHASE_MACHINE_CREATED )) { +error_setg (errp,"Error enabling vfio-user - machine already created" ); +return ; + } + + s->vfio_user = value; +} +static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -68,6 +87,10 @@ static void remote_machine_class_init(ObjectClass *oc, void *data) mc->desc ="Experimental remote machine" ; hc->unplug = qdev_simple_device_unplug_cb; + +object_class_property_add_bool (oc,"vfio-user" , + remote_machine_get_vfio_user, + remote_machine_set_vfio_user); }static const TypeInfo remote_machine = {
Signature · 1 line
2.20.1
Re: [PATCH v6 05/19] remote/machine: add vfio-user property
Quoted 8 lines · Jagannathan Raman
Add vfio-user to x-remote machine. It is a boolean, which indicates if the machine supports vfio-user protocol. The machine configures the bus differently vfio-user and multiprocess protocols, so this property informs it on how to configure the bus.
This property should be short lived. Once vfio-user fully replaces multiprocess, this property could be removed.
Quoted 7 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/remote/machine.h | 2 ++ hw/remote/machine.c | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+)
PATCH v6 6/19 vfio-user: build library
In reply to Jagannathan Ramanadd the libvfio-user library as a submodule. build it as a cmake subproject.
--- configure | 19 +++++++++- meson.build | 44 +++++++++++++++++++++- .gitlab-ci.d/buildtest.yml | 2 + .gitmodules | 3 ++ Kconfig.host | 4 ++ MAINTAINERS | 1 + hw/remote/Kconfig | 4 ++ hw/remote/meson.build | 2 + meson_options.txt | 2 + subprojects/libvfio-user | 1 + tests/docker/dockerfiles/centos8.docker | 2 + tests/docker/dockerfiles/ubuntu2004.docker | 2 + 12 files changed, 84 insertions(+), 2 deletions(-) create mode 160000 subprojects/libvfio-user
configure +18 -1
@@ -356,6 +356,7 @@ ninja="" gio="$default_feature" skip_meson=no slirp_smbd="$default_feature" +vfio_user_server="disabled" # The following Meson options are handled manually (still they # are included in the automatically generated help message)
@@ -1172,6 +1173,10 @@ for opt do ;; --disable-blobs) meson_option_parse --disable-install-blobs "" ;; + --enable-vfio-user-server) vfio_user_server="enabled" + ;; + --disable-vfio-user-server) vfio_user_server="disabled" + ;; --enable-tcmalloc) meson_option_parse --enable-malloc=tcmalloc tcmalloc ;; --enable-jemalloc) meson_option_parse --enable-malloc=jemalloc jemalloc
@@ -1425,6 +1430,7 @@ cat << EOF rng-none dummy RNG, avoid using /dev/(u)random and getrandom() gio libgio support slirp-smbd use smbd (at path --smbd=*) in slirp networking + vfio-user-server vfio-user server support NOTE: The object files are built at the place where configure is launched EOF
@@ -3100,6 +3106,17 @@ but not implemented on your system" fi fi +########################################## +# check for vfio_user_server + +case "$vfio_user_server" in + auto | enabled ) + if test "$git_submodules_action" != "ignore"; then + git_submodules="${git_submodules} subprojects/libvfio-user" + fi + ;; +esac + ########################################## # End of CC checks # After here, no more $cc or $ld runs
@@ -3790,7 +3807,7 @@ if test "$skip_meson" = no; then -Db_pie=$(if test "$pie" = yes; then echo true; else echo false; fi) \ -Db_coverage=$(if test "$gcov" = yes; then echo true; else echo false; fi) \ -Db_lto=$lto -Dcfi=$cfi -Dtcg=$tcg -Dxen=$xen \ - -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp \ + -Dcapstone=$capstone -Dfdt=$fdt -Dslirp=$slirp -Dvfio_user_server=$vfio_user_server \ $(test -n "${LIB_FUZZING_ENGINE+xxx}" && echo "-Dfuzzing_engine=$LIB_FUZZING_ENGINE") \ $(if test "$default_feature" = no; then echo "-Dauto_features=disabled"; fi) \ "$@" $cross_arg "$PWD" "$source_path"
meson.build +43 -1
68 lines not on this page · raw message
.gitlab-ci.d/buildtest.yml +2 -0
@@ -42,6 +42,7 @@ build-system-ubuntu: variables: IMAGE: ubuntu2004 CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system + --enable-vfio-user-server TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu microblazeel-softmmu mips64el-softmmu MAKE_CHECK_ARGS: check-build
@@ -165,6 +166,7 @@ build-system-centos: IMAGE: centos8 CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system --enable-modules --enable-trace-backends=dtrace --enable-docs + --enable-vfio-user-server TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu MAKE_CHECK_ARGS: check-build
.gitmodules +3 -0
@@ -67,3 +67,6 @@ [submodule "tests/lcitool/libvirt-ci"] path = tests/lcitool/libvirt-ci url = https://gitlab.com/libvirt/libvirt-ci.git +[submodule "subprojects/libvfio-user"] + path = subprojects/libvfio-user + url = https://github.com/nutanix/libvfio-user.git
Kconfig.host +4 -0
@@ -45,3 +45,7 @@ config MULTIPROCESS_ALLOWED config FUZZ bool select SPARSE_MEM + +config VFIO_USER_SERVER_ALLOWED + bool + imply VFIO_USER_SERVER
MAINTAINERS +1 -0
@@ -3567,6 +3567,7 @@ F: hw/remote/proxy-memory-listener.c F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h +F: subprojects/libvfio-user EBPF: M: Jason Wang <[email protected]>
hw/remote/Kconfig +4 -0
@@ -2,3 +2,7 @@ config MULTIPROCESS bool depends on PCI && PCI_EXPRESS && KVM select REMOTE_PCIHOST + +config VFIO_USER_SERVER + bool + depends on MULTIPROCESS
hw/remote/meson.build +2 -0
@@ -7,6 +7,8 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser) + specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-listener.c'))
meson_options.txt +2 -0
@@ -68,6 +68,8 @@ option('multiprocess', type: 'feature', value: 'auto', description: 'Out of process device emulation support') option('dbus_display', type: 'feature', value: 'auto', description: '-display dbus support') +option('vfio_user_server', type: 'feature', value: 'auto', + description: 'vfio-user server support') option('attr', type : 'feature', value : 'auto', description: 'attr/xattr support')
subprojects/libvfio-user new file +1 -0
@@ -0,0 +1 @@ +Subproject commit 7056525da5399d00831e90bed4aedb4b8442c9b2
tests/docker/dockerfiles/centos8.docker +2 -0
@@ -23,6 +23,7 @@ RUN dnf update -y && \ capstone-devel \ ccache \ clang \ + cmake \ ctags \ cyrus-sasl-devel \ daxctl-devel \
@@ -45,6 +46,7 @@ RUN dnf update -y && \ gtk3-devel \ hostname \ jemalloc-devel \ + json-c-devel \ libaio-devel \ libasan \ libattr-devel \
tests/docker/dockerfiles/ubuntu2004.docker +2 -0
@@ -18,6 +18,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \ ca-certificates \ ccache \ clang \ + cmake \ dbus \ debianutils \ diffutils \
@@ -58,6 +59,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \ libiscsi-dev \ libjemalloc-dev \ libjpeg-turbo8-dev \ + libjson-c-dev \ liblttng-ust-dev \ liblzo2-dev \ libncursesw5-dev \
Signature · 1 line
2.20.1
PATCH v6 7/19 vfio-user: define vfio-user-server object
In reply to Jagannathan RamanDefine vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
--- qapi/qom.json | 20 +++- hw/remote/vfio-user-obj.c | 194 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 217 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.c
qapi/qom.json +18 -2
@@ -703,6 +703,20 @@ { 'struct': 'RemoteObjectProperties', 'data': { 'fd': 'str', 'devid': 'str' } } +## +# @VfioUserServerProperties: +# +# Properties for x-vfio-user-server objects. +# +# @socket: socket to be used by the libvfiouser library +# +# @device: the id of the device to be emulated at the server +# +# Since: 6.3 +## +{ 'struct': 'VfioUserServerProperties', + 'data': { 'socket': 'SocketAddress', 'device': 'str' } } + ## # @RngProperties: #
@@ -842,7 +856,8 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', - { 'name': 'x-remote-object', 'features': [ 'unstable' ] } + { 'name': 'x-remote-object', 'features': [ 'unstable' ] }, + { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] } ] } ##
@@ -905,7 +920,8 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', - 'x-remote-object': 'RemoteObjectProperties' + 'x-remote-object': 'RemoteObjectProperties', + 'x-vfio-user-server': 'VfioUserServerProperties' } } ##
hw/remote/vfio-user-obj.c new file +194 -0
194 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -3568,6 +3568,7 @@ F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user +F: hw/remote/vfio-user-obj.c EBPF: M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c')) remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser)
hw/remote/trace-events +3 -0
@@ -2,3 +2,6 @@ mpqemu_send_io_error(int cmd, int size, int nfds) "send command %d size %d, %d file descriptors to remote process" mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, %d file descriptors to remote process" + +# vfio-user-obj.c +vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s"
Signature · 1 line
2.20.1
Re: [PATCH v6 07/19] vfio-user: define vfio-user-server object
hw/remote/vfio-user-obj.c line 73
Quoted 10 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c new file +10
-0+struct VfuObjectClass { +ObjectClass parent_class; + +unsigned int nr_devs; + +/* +* Can be set to shutdown automatically when all server object +* instances are destroyed +*/ +bool auto_shutdown;
This field is introduced but it is hardcoded to true. Is there a way to set it to false?
Re: [PATCH v6 07/19] vfio-user: define vfio-user-server object
Quoted 18 lines · Stefan Hajnoczi
hw/remote/vfio-user-obj.c line 73
Quoted 10 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c new file +10
-0+struct VfuObjectClass { +ObjectClass parent_class; + +unsigned int nr_devs; + +/* +* Can be set to shutdown automatically when all server object +* instances are destroyed +*/ +bool auto_shutdown;This field is introduced but it is hardcoded to true. Is there a way to set it to false?
We could add a property to ’TYPE_REMOTE_MACHINE’ which indicates if it would run as a server/daemon.
Thank you!
--
Jag
Re: [PATCH v6 07/19] vfio-user: define vfio-user-server object
Quoted 16 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
hw/remote/vfio-user-obj.c line 73
Quoted 10 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c new file +10
-0+struct VfuObjectClass { +ObjectClass parent_class; + +unsigned int nr_devs; + +/* +* Can be set to shutdown automatically when all server object +* instances are destroyed +*/ +bool auto_shutdown;This field is introduced but it is hardcoded to true. Is there a way to set it to false?
Quoted 2 lines
We could add a property to ’TYPE_REMOTE_MACHINE’ which indicates if it would run as a server/daemon.
Yes.
An alternative is to add a per-instance property to --object x-vfio-user-server. In practice there is not much benefit since users are unlikely to mix auto-shutdown instances with non-auto-shutdown instances, but the code might be a little simpler and cleaner.
Stefan
Re: [PATCH v6 07/19] vfio-user: define vfio-user-server object
In reply to Jagannathan RamanQuoted 25 lines · Jagannathan Raman
Define vfio-user object which is remote process server for QEMU. Setup object initialization functions and properties necessary to instantiate the object
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>--- qapi/qom.json | 20 +++- hw/remote/vfio-user-obj.c | 194 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + hw/remote/meson.build | 1 + hw/remote/trace-events | 3 + 5 files changed, 217 insertions(+), 2 deletions(-) create mode 100644 hw/remote/vfio-user-obj.cqapi/qom.json +0
-0@@ -703,6 +703,20 @@ { 'struct': 'RemoteObjectProperties', 'data': { 'fd': 'str', 'devid': 'str' } }
qapi/qom.json line 715
Quoted 10 lines
qapi/qom.json +10
-0+## +# @VfioUserServerProperties: +# +# Properties for x-vfio-user-server objects. +# +# @socket: socket to be used by the libvfiouser library +# +# @device: the id of the device to be emulated at the server +# +# Since: 6.3
The next release is 7.0, not 6.3.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v6 8/19 vfio-user: instantiate vfio-user context
In reply to Jagannathan Ramancreate a context with the vfio-user library to run a PCI device
--- hw/remote/vfio-user-obj.c | 80 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+)
hw/remote/vfio-user-obj.c +80 -0
@@ -41,6 +41,9 @@#include "hw/remote/machine.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qemu/notify.h" +#include "sysemu/sysemu.h" +#include "libvfio-user.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -82,13 +85,24 @@ struct VfuObject {char *device;Error *err; + +Notifier machine_done; + +vfu_ctx_t *vfu_ctx; }; +static void vfu_object_init_ctx (VfuObject *o,Error **errp); +static void vfu_object_set_socket (Object *obj,Visitor *v,const char *name,void *opaque,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +error_setg (errp,"vfu: Unable to set socket property - server busy" ); +return ; + } +qapi_free_SocketAddress (o->socket); o->socket =NULL ;
@@ -104,17 +118,69 @@ static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name, }trace_vfu_prop ("socket" , o->socket->u.q_unix.path); + +vfu_object_init_ctx (o, errp); }static void vfu_object_set_device (Object *obj,const char *str,Error **errp) {VfuObject *o =VFU_OBJECT (obj); +if (o->vfu_ctx) { +error_setg (errp,"vfu: Unable to set device property - server busy" ); +return ; + } +g_free (o->device); o->device =g_strdup (str);trace_vfu_prop ("device" , str); + +vfu_object_init_ctx (o, errp); +} + +/* +* TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' +* properties. It also depends on devices instantiated in QEMU. These +* dependencies are not available during the instance_init phase of this +* object's life-cycle. As such, the server is initialized after the +* machine is setup. machine_init_done_notifier notifies TYPE_VFU_OBJECT +* when the machine is setup, and the dependencies are available. +*/ +static void vfu_object_machine_done (Notifier *notifier,void *data) +{ +VfuObject *o =container_of (notifier, VfuObject, machine_done); +Error *err =NULL ; + +vfu_object_init_ctx (o, &err); + +if (err) { +error_propagate (&error_abort, err); + } +} + +static void vfu_object_init_ctx (VfuObject *o,Error **errp) +{ +ERRP_GUARD (); + +if (o->vfu_ctx || !o->socket || !o->device || + !phase_check (PHASE_MACHINE_READY )) { +return ; + } + +if (o->err) { +error_propagate (errp, o->err); + o->err =NULL ; +return ; + } + + o->vfu_ctx =vfu_create_ctx (VFU_TRANS_SOCK , o->socket->u.q_unix.path,0 , + o,VFU_DEV_TYPE_PCI ); +if (o->vfu_ctx ==NULL ) { +error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno)); +return ; + } }static void vfu_object_init (Object *obj)
@@ -124,6 +190,11 @@ static void vfu_object_init(Object *obj) k->nr_devs++; +if (!phase_check (PHASE_MACHINE_READY )) { + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done); + } +if (!object_dynamic_cast (OBJECT (current_machine),TYPE_REMOTE_MACHINE )) {error_setg (&o->err,"vfu: %s only compatible with %s machine" ,TYPE_VFU_OBJECT ,TYPE_REMOTE_MACHINE );
@@ -142,6 +213,10 @@ static void vfu_object_finalize(Object *obj) o->socket =NULL ; +if (o->vfu_ctx) { +vfu_destroy_ctx (o->vfu_ctx); + } +g_free (o->device); o->device =NULL ;
@@ -149,6 +224,11 @@ static void vfu_object_finalize(Object *obj)if (!k->nr_devs && k->auto_shutdown) {qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); } + +if (o->machine_done.notify) { +qemu_remove_machine_init_done_notifier (&o->machine_done); + o->machine_done.notify =NULL ; + } }static void vfu_object_class_init (ObjectClass *klass,void *data)
Signature · 1 line
2.20.1
Re: [PATCH v6 08/19] vfio-user: instantiate vfio-user context
hw/remote/vfio-user-obj.c static void vfu_object_init(Object *obj) · line 195
Quoted 7 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +3
-0@@ -124,6 +190,11 @@ static void vfu_object_init(Object *obj) k->nr_devs++; +if (!phase_check (PHASE_MACHINE_READY )) { + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done);
This probably has to happen after the next if statement since qemu_add_machine_init_done_notifier() can immediately call ->notify() and we'd try to initialize on a non-remote machine type.
Re: [PATCH v6 08/19] vfio-user: instantiate vfio-user context
Quoted 9 lines · Stefan Hajnoczi
Quoted 4 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +0
-0@@ -124,6 +190,11 @@ static void vfu_object_init(Object *obj) k->nr_devs++;
Quoted 7 lines
hw/remote/vfio-user-obj.c static void vfu_object_init(Object *obj) · line 195
Quoted 3 lines
hw/remote/vfio-user-obj.c +3
-0+if (!phase_check (PHASE_MACHINE_READY )) { + o->machine_done.notify = vfu_object_machine_done; +qemu_add_machine_init_done_notifier (&o->machine_done);This probably has to happen after the next if statement since
qemu_add_machine_init_done_notifier()can immediately call ->notify() and we'd try to initialize on a non-remote machine type.
OK, got it.
Thank you!
--
Jag
PATCH v6 9/19 vfio-user: find and init PCI device
In reply to Jagannathan RamanFind the PCI device with specified id. Initialize the device context with the QEMU PCI device
--- hw/remote/vfio-user-obj.c | 59 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+)
hw/remote/vfio-user-obj.c +59 -0
@@ -44,6 +44,8 @@#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" +#include "hw/qdev-core.h" +#include "hw/pci/pci.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
@@ -89,6 +91,10 @@ struct VfuObject {Notifier machine_done;vfu_ctx_t *vfu_ctx; + +PCIDevice *pci_dev; + +Error *unplug_blocker; };static void vfu_object_init_ctx (VfuObject *o,Error **errp);
@@ -163,6 +169,9 @@ static void vfu_object_machine_done(Notifier *notifier, void *data)static void vfu_object_init_ctx (VfuObject *o,Error **errp) {ERRP_GUARD (); +DeviceState *dev =NULL ; +vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +int ret;if (o->vfu_ctx || !o->socket || !o->device || !phase_check (PHASE_MACHINE_READY )) {
@@ -181,6 +190,48 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)error_setg (errp,"vfu: Failed to create context - %s" ,strerror (errno));return ; } + + dev =qdev_find_recursive (sysbus_get_default (), o->device); +if (dev ==NULL ) { +error_setg (errp,"vfu: Device %s not found" , o->device); + goto fail; + } + +if (!object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_setg (errp,"vfu: %s not a PCI device" , o->device); + goto fail; + } + + o->pci_dev =PCI_DEVICE (dev); + +if (pci_is_express (o->pci_dev)) { + pci_type =VFU_PCI_TYPE_EXPRESS ; + } + + ret =vfu_pci_init (o->vfu_ctx, pci_type,PCI_HEADER_TYPE_NORMAL ,0 ); +if (ret <0 ) { +error_setg (errp, +"vfu: Failed to attach PCI device %s to context - %s" , + o->device,strerror (errno)); + goto fail; + } + +error_setg (&o->unplug_blocker, +"vfu: %s for %s must be deleted before unplugging" , +TYPE_VFU_OBJECT , o->device); +qdev_add_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); + +return ; + +fail: +vfu_destroy_ctx (o->vfu_ctx); +if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + o->vfu_ctx =NULL ; + o->pci_dev =NULL ; }static void vfu_object_init (Object *obj)
@@ -221,6 +272,14 @@ static void vfu_object_finalize(Object *obj) o->device =NULL ; +if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + + o->pci_dev =NULL ; +if (!k->nr_devs && k->auto_shutdown) {qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); }
Signature · 1 line
2.20.1
Re: [PATCH v6 09/19] vfio-user: find and init PCI device
hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 281
Quoted 11 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +7
-0@@ -221,6 +272,14 @@ static void vfu_object_finalize(Object *obj) o->device =NULL ; +if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + + o->pci_dev =NULL ;
Since we don't hold a reference to o->pci_dev there is an assumption about the order of --object vs --device ->finalize() here. I think it will work because softmmu/runstate.c:qemu_cleanup() doesn't unref main_system_bus and only --object ->finalize() is called, but this seems fragile. We should probably hold a reference to pci_dev and call object_unref() on it.
Re: [PATCH v6 09/19] vfio-user: find and init PCI device
Quoted 9 lines · Stefan Hajnoczi
Quoted 4 lines · Jagannathan Raman
hw/remote/vfio-user-obj.c +0
-0@@ -221,6 +272,14 @@ static void vfu_object_finalize(Object *obj) o->device =NULL ;
Quoted 14 lines
hw/remote/vfio-user-obj.c static void vfu_object_finalize(Object *obj) · line 281
Quoted 7 lines
hw/remote/vfio-user-obj.c +7
-0+if (o->unplug_blocker && o->pci_dev) { +qdev_del_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); +error_free (o->unplug_blocker); + o->unplug_blocker =NULL ; + } + + o->pci_dev =NULL ;Since we don't hold a reference to
o->pci_devthere is an assumption about the order of --object vs --device ->finalize() here. I think it will work because softmmu/runstate.c:qemu_cleanup() doesn't unrefmain_system_busand only --object ->finalize() is called, but this seems fragile. We should probably hold a reference topci_devand callobject_unref()on it.
OK, will do.
Thank you!
--
Jag
PATCH v6 10/19 vfio-user: run vfio-user context
In reply to Jagannathan RamanSetup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
--- qapi/misc.json | 23 ++++++++++ hw/remote/vfio-user-obj.c | 96 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 1 deletion(-)
qapi/misc.json +23 -0
@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true } + +## +# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel +# +# @id: ID of the TYPE_VFIO_USER_SERVER object +# +# @device: ID of attached PCI device +# +# Since: 6.3 +# +# Example: +# +# <- { "event": "VFU_CLIENT_HANGUP", +# "data": { "id": "vfu1", +# "device": "lsi1" }, +# "timestamp": { "seconds": 1265044230, "microseconds": 450486 } } +# +## +{ 'event': 'VFU_CLIENT_HANGUP', + 'data': { 'id': 'str', 'device': 'str' } }
hw/remote/vfio-user-obj.c +95 -1
149 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v6 10/19] vfio-user: run vfio-user context
Quoted 11 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- qapi/misc.json | 23 ++++++++++ hw/remote/vfio-user-obj.c | 96 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 1 deletion(-)
Re: [PATCH v6 10/19] vfio-user: run vfio-user context
In reply to Jagannathan RamanQuoted 22 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- qapi/misc.json | 23 ++++++++++ hw/remote/vfio-user-obj.c | 96 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 1 deletion(-)qapi/misc.json +2
-0@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true } + +##
qapi/misc.json line 541
Quoted 10 lines
qapi/misc.json +10
-0+# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel +# +# @id: ID of the TYPE_VFIO_USER_SERVER object +# +# @device: ID of attached PCI device +# +# Since: 6.3
7.0
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH v6 10/19] vfio-user: run vfio-user context
Quoted 31 lines · Eric Blake
Quoted 26 lines · Jagannathan Raman
Setup a handler to run vfio-user context. The context is driven by messages to the file descriptor associated with it - get the fd for the context and hook up the handler with it
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
qapi/misc.json| 23 ++++++++++hw/remote/vfio-user-obj.c| 96 ++++++++++++++++++++++++++++++++++++++-2 files changed, 118 insertions(+), 1 deletion(-)qapi/misc.json +6
-0@@ -527,3 +527,26 @@ 'data': { '*option': 'str' }, 'returns': ['CommandLineOptionInfo'], 'allow-preconfig': true } + +## +# @VFU_CLIENT_HANGUP: +# +# Emitted when the client of a TYPE_VFIO_USER_SERVER closes the +# communication channel
Quoted 8 lines
qapi/misc.json line 541
Quoted 6 lines
qapi/misc.json +6
-0+# +# @id: ID of the TYPE_VFIO_USER_SERVER object +# +# @device: ID of attached PCI device +# +# Since: 6.37.0
OK, got it. Looks like the next release version is 7.0:
https://gitlab.com/qemu-project/qemu/-/milestones
Thank you!
--
Jag
Quoted 6 lines
Signature · 4 lines
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
PATCH v6 11/19 vfio-user: handle PCI config space accesses
In reply to Jagannathan RamanDefine and register handlers for PCI config space accesses
--- hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 47 insertions(+)
hw/remote/vfio-user-obj.c +45 -0
@@ -47,6 +47,7 @@#include "qapi/qapi-events-misc.h" #include "qemu/notify.h" #include "qemu/thread.h" +#include "qemu/main-loop.h" #include "sysemu/sysemu.h" #include "libvfio-user.h" #include "hw/qdev-core.h"
@@ -217,6 +218,39 @@ retry_attach:qemu_set_fd_handler (o->vfu_poll_fd, vfu_object_ctx_run,NULL , o); } +static ssize_t vfu_object_cfg_access (vfu_ctx_t *vfu_ctx,char *const buf, +size_t count,loff_t offset, +const bool is_write) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint32_t pci_access_width =sizeof (uint32_t ); +size_t bytes = count; +uint32_t val =0 ; +char *ptr = buf; +int len; + +while (bytes >0 ) { + len = (bytes > pci_access_width) ? pci_access_width : bytes; +if (is_write) { +memcpy (&val, ptr, len); +pci_host_config_write_common (o->pci_dev, offset, +pci_config_size (o->pci_dev), + val, len); +trace_vfu_cfg_write (offset, val); + }else { + val =pci_host_config_read_common (o->pci_dev, offset, +pci_config_size (o->pci_dev), len); +memcpy (ptr, &val, len); +trace_vfu_cfg_read (offset, val); + } + offset += len; + ptr += len; + bytes -= len; + } + +return count; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These
@@ -293,6 +327,17 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) TYPE_VFU_OBJECT, o->device);qdev_add_unplug_blocker (DEVICE (o->pci_dev), o->unplug_blocker); + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_CFG_REGION_IDX , +pci_config_size (o->pci_dev), &vfu_object_cfg_access, +VFU_REGION_FLAG_RW |VFU_REGION_FLAG_ALWAYS_CB , +NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (errp, +"vfu: Failed to setup config space handlers for %s- %s" , + o->device,strerror (errno)); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/trace-events +2 -0
@@ -5,3 +5,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, # vfio-user-obj.c vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" +vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" +vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x"
Signature · 1 line
2.20.1
Re: [PATCH v6 11/19] vfio-user: handle PCI config space accesses
Quoted 10 lines · Jagannathan Raman
Define and register handlers for PCI config space accesses
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>--- hw/remote/vfio-user-obj.c | 45 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 2 files changed, 47 insertions(+)
hw/pci/pci.c:pci_update_mappings() will unmap/map BARs when the vfio-user client touches BARs. Please add a comment that the remote machine type never dispatches memory accesses in the global memory address space and therefore we don't care that multiple remote devices may set up conflicting Memory and I/O Space BARs.
Re: [PATCH v6 11/19] vfio-user: handle PCI config space accesses
Quoted 16 lines · Stefan Hajnoczi
Quoted 10 lines · Jagannathan Raman
Define and register handlers for PCI config space accesses
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Reviewed-by Stefan Hajnoczi <[email protected]>---
hw/remote/vfio-user-obj.c| 45 +++++++++++++++++++++++++++++++++++++++
hw/remote/trace-events | 2 ++2 files changed, 47 insertions(+)
Quoted 5 lines
hw/pci/pci.c:pci_update_mappings() will unmap/map BARs when the vfio-user client touches BARs. Please add a comment that the remote machine type never dispatches memory accesses in the global memory address space and therefore we don't care that multiple remote devices may set up conflicting Memory and I/O Space BARs.
OK, will do.
Thank you!
--
Jag
PATCH v6 12/19 vfio-user: IOMMU support for remote device
In reply to Jagannathan RamanAssign separate address space for each device in the remote processes.
--- include/hw/remote/iommu.h | 18 +++++++++ hw/remote/iommu.c | 78 +++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 2 + hw/remote/meson.build | 1 + 4 files changed, 99 insertions(+) create mode 100644 include/hw/remote/iommu.h create mode 100644 hw/remote/iommu.c
include/hw/remote/iommu.h new file +18 -0
@@ -0,0 +1,18 @@ +/** +* Copyright © 2022 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef REMOTE_IOMMU_H +#define REMOTE_IOMMU_H + +#include "hw/pci/pci_bus.h" + +void remote_configure_iommu (PCIBus *pci_bus); + +void remote_iommu_del_device (PCIDevice *pci_dev); + +#endif
hw/remote/iommu.c new file +78 -0
78 lines not on this page · raw message
MAINTAINERS +2 -0
@@ -3569,6 +3569,8 @@ F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c +F: hw/remote/iommu.c +F: include/hw/remote/iommu.h EBPF: M: Jason Wang <[email protected]>
hw/remote/meson.build +1 -0
@@ -6,6 +6,7 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('message.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iohub.c')) +remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('iommu.c')) remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: files('vfio-user-obj.c')) remote_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_true: vfiouser)
Signature · 1 line
2.20.1
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;
A mutable global hash table requires synchronization when device emulation runs in multiple threads.
I suggest using pci_setup_iommu()'s iommu_opaque argument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.
hw/remote/iommu.c line 38
Quoted 12 lines
hw/remote/iommu.c new file +12
-0+ +#define INT2VOIDP (i) (void *)(uintptr_t)(i) + +static AddressSpace *remote_iommu_find_add_as (PCIBus *pci_bus, +void *opaque,int devfn) +{ +struct RemoteIommuElem *elem =NULL ; +int pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_bus), devfn); + +if (!remote_iommu_elem_by_bdf) { +return &address_space_memory; + }
When can this happen? remote_configure_iommu() allocates remote_iommu_elem_by_bdf so it should always be non-NULL.
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 15 lines · Stefan Hajnoczi
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
Quoted 3 lines
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.
OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
Quoted 16 lines
hw/remote/iommu.c line 38
Quoted 13 lines
hw/remote/iommu.c new file +12
-0+ +#define INT2VOIDP (i) (void *)(uintptr_t)(i) + +static AddressSpace *remote_iommu_find_add_as (PCIBus *pci_bus, +void *opaque,int devfn) +{ +struct RemoteIommuElem *elem =NULL ; +int pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_bus), devfn); + +if (!remote_iommu_elem_by_bdf) { +return &address_space_memory; + }When can this happen?
remote_configure_iommu()allocatesremote_iommu_elem_by_bdfso it should always be non-NULL.
I think we won’t hit this case. g_hash_table_new_full() would always succeed.
Thank you!
--
Jag
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 15 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
Quoted 7 lines
Quoted 3 lines
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
I thought the PCIe topology can be set up with a separate buf for each x-vfio-user-server? I remember something like that in the previous revision where a root port was instantiated for each x-vfio-user-server.
Stefan
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 24 lines · Stefan Hajnoczi
Quoted 19 lines · Jag Raman
Quoted 18 lines · Stefan Hajnoczi
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.
Quoted 7 lines
Quoted 3 lines
OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
I thought the PCIe topology can be set up with a separate buf for each x-vfio-user-server? I remember something like that in the previous revision where a root port was instantiated for each x-vfio-user-server.
Yes, we could setup the PCIe topology to be that way. But the user could add more than one device to the same bus, unless the bus type explicitly limits the number of devices to one (BusClass->max_dev).
Thank you!
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 28 lines · Jag Raman
Quoted 28 lines · Stefan Hajnoczi
Quoted 22 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
Quoted 3 lines
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
Quoted 7 lines
Quoted 3 lines
I thought the PCIe topology can be set up with a separate buf for each x-vfio-user-server? I remember something like that in the previous revision where a root port was instantiated for each x-vfio-user-server.
Yes, we could setup the PCIe topology to be that way. But the user could add more than one device to the same bus, unless the bus type explicitly limits the number of devices to one (
BusClass->max_dev).
Due to how the IOMMU is used to restrict the bus to the vfio-user client's DMA mappings, it seems like it's necesssary to limit the number of devices to 1 per bus anyway?
Stefan
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 37 lines · Stefan Hajnoczi
Quoted 32 lines · Jag Raman
Quoted 31 lines · Stefan Hajnoczi
Quoted 15 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
hw/remote/iommu.c line 26
Quoted 6 lines · Jagannathan Raman
hw/remote/iommu.c new file +6
-0+struct RemoteIommuElem { +AddressSpace as; +MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
Quoted 7 lines
Quoted 3 lines
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
I thought the PCIe topology can be set up with a separate buf for each x-vfio-user-server? I remember something like that in the previous revision where a root port was instantiated for each x-vfio-user-server.
Quoted 7 lines
Quoted 3 lines
Yes, we could setup the PCIe topology to be that way. But the user could add more than one device to the same bus, unless the bus type explicitly limits the number of devices to one (
BusClass->max_dev).Due to how the IOMMU is used to restrict the bus to the vfio-user client's DMA mappings, it seems like it's necesssary to limit the number of devices to 1 per bus anyway?
Hi Stefan,
“remote_iommu_elem_by_bdf” has a separate entry for each of the BDF combinations - it provides a separate DMA address space per device. As such, we don’t have to limit the number of devices to 1 per bus.
Thank you!
--
Jag
Quoted 2 lines
Stefan
Re: [PATCH v6 12/19] vfio-user: IOMMU support for remote device
Quoted 45 lines · Jag Raman
Quoted 37 lines · Stefan Hajnoczi
Quoted 32 lines · Jag Raman
Quoted 31 lines · Stefan Hajnoczi
Quoted 15 lines · Jag Raman
Quoted 15 lines · Stefan Hajnoczi
Review
Quoted 6 lines · Jagannathan Raman
(quoted diff) +6
-0+struct RemoteIommuElem { + AddressSpace as; + MemoryRegion mr; +}; + +GHashTable *remote_iommu_elem_by_bdf;A mutable global hash table requires synchronization when device emulation runs in multiple threads.
Quoted 7 lines
Quoted 3 lines
I suggest using pci_setup_iommu()'s
iommu_opaqueargument to avoid the global. If there is only 1 device per remote PCI bus, then there are no further synchronization concerns.OK, will avoid the global. We would need to access the hash table concurrently since there could be more than one device in the same bus - so a mutex would be needed here.
I thought the PCIe topology can be set up with a separate buf for each x-vfio-user-server? I remember something like that in the previous revision where a root port was instantiated for each x-vfio-user-server.
Quoted 7 lines
Quoted 3 lines
Yes, we could setup the PCIe topology to be that way. But the user could add more than one device to the same bus, unless the bus type explicitly limits the number of devices to one (
BusClass->max_dev).Due to how the IOMMU is used to restrict the bus to the vfio-user client's DMA mappings, it seems like it's necesssary to limit the number of devices to 1 per bus anyway?
Quoted 5 lines
Hi Stefan,
“remote_iommu_elem_by_bdf” has a separate entry for each of the BDF combinations - it provides a separate DMA address space per device. As such, we don’t have to limit the number of devices to 1 per bus.
I see, thanks!
Stefan
PATCH v6 13/19 vfio-user: handle DMA mappings
In reply to Jagannathan RamanDefine and register callbacks to manage the RAM regions used for device DMA
--- hw/remote/machine.c | 5 ++++ hw/remote/vfio-user-obj.c | 55 +++++++++++++++++++++++++++++++++++++++ hw/remote/trace-events | 2 ++ 3 files changed, 62 insertions(+)
hw/remote/machine.c +5 -0
@@ -22,6 +22,7 @@#include "hw/pci/pci_host.h" #include "hw/remote/iohub.h" #include "hw/qdev-core.h" +#include "hw/remote/iommu.h" static void remote_machine_init (MachineState *machine) {
@@ -51,6 +52,10 @@ static void remote_machine_init(MachineState *machine) pci_host =PCI_HOST_BRIDGE (rem_host); +if (s->vfio_user) { +remote_configure_iommu (pci_host->bus); + } +remote_iohub_init (&s->iohub); pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
hw/remote/vfio-user-obj.c +55 -0
67 lines not on this page · raw message
hw/remote/trace-events +2 -0
@@ -7,3 +7,5 @@ mpqemu_recv_io_error(int cmd, int size, int nfds) "failed to receive %d size %d, vfu_prop(const char *prop, const char *val) "vfu: setting %s as %s" vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" +vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" +vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
Signature · 1 line
2.20.1
PATCH v6 14/19 vfio-user: handle PCI BAR accesses
In reply to Jagannathan RamanDetermine the BARs used by the PCI device and register handlers to manage the access to the same.
--- include/exec/memory.h | 3 + hw/remote/vfio-user-obj.c | 166 ++++++++++++++++++++++++++++++++ softmmu/physmem.c | 4 +- tests/qtest/fuzz/generic_fuzz.c | 9 +- hw/remote/trace-events | 3 + 5 files changed, 179 insertions(+), 6 deletions(-)
include/exec/memory.h +3 -0
@@ -2810,6 +2810,9 @@ MemTxResult address_space_write_cached_slow(MemoryRegionCache *cache, hwaddr addr, const void *buf, hwaddr len); +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr); +bool prepare_mmio_access (MemoryRegion *mr); +static inline bool memory_access_is_direct (MemoryRegion *mr,bool is_write) {if (is_write) {
hw/remote/vfio-user-obj.c +166 -0
184 lines not on this page · raw message
softmmu/physmem.c +2 -2
@@ -2717,7 +2717,7 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } -static int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;
@@ -2744,7 +2744,7 @@ static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)return l; } -static bool prepare_mmio_access (MemoryRegion *mr) +bool prepare_mmio_access (MemoryRegion *mr) {bool release_lock = false;
tests/qtest/fuzz/generic_fuzz.c +5 -4
@@ -144,7 +144,7 @@ static void *pattern_alloc(pattern p, size_t len)return buf; } -static int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) +static int fuzz_memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;
@@ -242,11 +242,12 @@ void fuzz_dma_read_cb(size_t addr, size_t len, MemoryRegion *mr)/* * If mr1 isn't RAM, address_space_translate doesn't update l. Use -* memory_access_size to identify the number of bytes that it is safe -* to write without accidentally writing to another MemoryRegion. +* fuzz_memory_access_size to identify the number of bytes that it +* is safe to write without accidentally writing to another +* MemoryRegion. */ if (!memory_region_is_ram (mr1)) { - l =memory_access_size (mr1, l, addr1); + l =fuzz_memory_access_size (mr1, l, addr1); } if (memory_region_is_ram(mr1) || memory_region_is_romd(mr1) ||
hw/remote/trace-events +3 -0
@@ -9,3 +9,6 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" +vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" +vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" +vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""
Signature · 1 line
2.20.1
Re: [PATCH v6 14/19] vfio-user: handle PCI BAR accesses
Quoted 2 lines · Jagannathan Raman
Determine the BARs used by the PCI device and register handlers to manage the access to the same.
Hi Paolo,
Please review this from the memory API perspective. vfu_object_bar_rw() reimplements MemoryRegion read/write because we're dispatching to a MemoryRegion without going through an AddressSpace/FlatView.
Quoted 282 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/exec/memory.h | 3 + hw/remote/vfio-user-obj.c | 166 ++++++++++++++++++++++++++++++++ softmmu/physmem.c | 4 +- tests/qtest/fuzz/generic_fuzz.c | 9 +- hw/remote/trace-events | 3 + 5 files changed, 179 insertions(+), 6 deletions(-)include/exec/memory.h +3
-0@@ -2810,6 +2810,9 @@ MemTxResult address_space_write_cached_slow(MemoryRegionCache *cache, hwaddr addr, const void *buf, hwaddr len); +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr); +bool prepare_mmio_access (MemoryRegion *mr); +static inline bool memory_access_is_direct (MemoryRegion *mr,bool is_write) {if (is_write) {hw/remote/vfio-user-obj.c +166
-0@@ -53,6 +53,7 @@#include "hw/qdev-core.h" #include "hw/pci/pci.h" #include "qemu/timer.h" +#include "exec/memory.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -299,6 +300,169 @@ static void dma_unregister(vfu_ctx_t *vfu_ctx, vfu_dma_info_t *info)trace_vfu_dma_unregister ((uint64_t )info->iova.iov_base); } +static size_t vfu_object_bar_rw (PCIDevice *pci_dev,int pci_bar, +hwaddr offset,char *const buf, +hwaddr len,const bool is_write) +{ +uint8_t *ptr = (uint8_t *)buf; +uint8_t *ram_ptr =NULL ; +bool release_lock = false; +MemoryRegionSection section = {0 }; +MemoryRegion *mr =NULL ; +int access_size; +hwaddr size =0 ; +MemTxResult result; +uint64_t val; + + section =memory_region_find (pci_dev->io_regions[pci_bar].memory, + offset, len); + +if (!section.mr) { +return 0 ; + } + + mr = section.mr; + +if (is_write && mr->readonly) { +warn_report ("vfu: attempting to write to readonly region in " +"bar %d - [0x%" PRIx64" - 0x%" PRIx64"]" , + pci_bar, offset, (offset + len)); +return 0 ; + } + +if (memory_access_is_direct (mr, is_write)) { +/** +* Some devices expose a PCI expansion ROM, which could be buffer +* based as compared to other regions which are primarily based on +* MemoryRegionOps. memory_region_find() would already check +* for buffer overflow, we don't need to repeat it here. +*/ + ram_ptr =memory_region_get_ram_ptr (mr); + + size = len; + +if (is_write) { +memcpy (ram_ptr, buf, size); + }else { +memcpy (buf, ram_ptr, size); + } + + goto exit; + } + +while (len >0 ) { +/** +* The read/write logic used below is similar to the ones in +* flatview_read/write_continue() +*/ + release_lock =prepare_mmio_access (mr); + + access_size =memory_access_size (mr, len, offset); + +if (is_write) { + val =ldn_he_p (ptr, access_size); + + result =memory_region_dispatch_write (mr, offset, val, +size_memop (access_size), +MEMTXATTRS_UNSPECIFIED ); + }else { + result =memory_region_dispatch_read (mr, offset, &val, +size_memop (access_size), +MEMTXATTRS_UNSPECIFIED ); + +stn_he_p (ptr, access_size, val); + } + +if (release_lock) { +qemu_mutex_unlock_iothread (); + release_lock = false; + } + +if (result !=MEMTX_OK ) { +warn_report ("vfu: failed to %s 0x%" PRIx64"" , + is_write ?"write to" :"read from" , + (offset - size)); + + goto exit; + } + + len -= access_size; + size += access_size; + ptr += access_size; + offset += access_size; + } + +exit: +memory_region_unref (mr); + +return size; +} + +/** +* VFU_OBJECT_BAR_HANDLER - macro for defining handlers for PCI BARs. +* +* To create handler for BAR number 2, VFU_OBJECT_BAR_HANDLER(2) would +* define vfu_object_bar2_handler +*/ +#define VFU_OBJECT_BAR_HANDLER (BAR_NO ) \ + static ssize_t vfu_object_bar##BAR_NO##_handler(vfu_ctx_t *vfu_ctx, \ + char * const buf, size_t count, \ + loff_t offset, const bool is_write) \ + { \ + VfuObject *o = vfu_get_private(vfu_ctx); \ + PCIDevice *pci_dev = o->pci_dev; \ + \ + return vfu_object_bar_rw(pci_dev, BAR_NO, offset, \ + buf, count, is_write); \ + } \ + +VFU_OBJECT_BAR_HANDLER(0) +VFU_OBJECT_BAR_HANDLER(1) +VFU_OBJECT_BAR_HANDLER(2) +VFU_OBJECT_BAR_HANDLER(3) +VFU_OBJECT_BAR_HANDLER(4) +VFU_OBJECT_BAR_HANDLER(5) +VFU_OBJECT_BAR_HANDLER(6) + +static vfu_region_access_cb_t *vfu_object_bar_handlers[PCI_NUM_REGIONS ] = { + &vfu_object_bar0_handler, + &vfu_object_bar1_handler, + &vfu_object_bar2_handler, + &vfu_object_bar3_handler, + &vfu_object_bar4_handler, + &vfu_object_bar5_handler, + &vfu_object_bar6_handler, +}; + +/** +* vfu_object_register_bars - Identify active BAR regions of pdev and setup +* callbacks to handle read/write accesses +*/ +static void vfu_object_register_bars (vfu_ctx_t *vfu_ctx,PCIDevice *pdev) +{ +int flags =VFU_REGION_FLAG_RW ; +int i; + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { +if (!pdev->io_regions[i].size) { +continue ; + } + +if ((i ==VFU_PCI_DEV_ROM_REGION_IDX ) || + pdev->io_regions[i].memory->readonly) { + flags &= ~VFU_REGION_FLAG_WRITE ; + } + +vfu_setup_region (vfu_ctx,VFU_PCI_DEV_BAR0_REGION_IDX + i, + (size_t )pdev->io_regions[i].size, + vfu_object_bar_handlers[i], + flags,NULL ,0 ,-1 ,0 ); + +trace_vfu_bar_register (i, pdev->io_regions[i].addr, + pdev->io_regions[i].size); + } +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -393,6 +557,8 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } +vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",softmmu/physmem.c +2
-2@@ -2717,7 +2717,7 @@ void memory_region_flush_rom_device(MemoryRegion *mr, hwaddr addr, hwaddr size)invalidate_and_set_dirty (mr, addr, size); } -static int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) +int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -2744,7 +2744,7 @@ static int memory_access_size(MemoryRegion *mr, unsigned l, hwaddr addr)return l; } -static bool prepare_mmio_access (MemoryRegion *mr) +bool prepare_mmio_access (MemoryRegion *mr) {bool release_lock = false;tests/qtest/fuzz/generic_fuzz.c +5
-4@@ -144,7 +144,7 @@ static void *pattern_alloc(pattern p, size_t len)return buf; } -static int memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) +static int fuzz_memory_access_size (MemoryRegion *mr,unsigned l,hwaddr addr) {unsigned access_size_max = mr->ops->valid.max_access_size;@@ -242,11 +242,12 @@ void fuzz_dma_read_cb(size_t addr, size_t len, MemoryRegion *mr)/* * If mr1 isn't RAM, address_space_translate doesn't update l. Use -* memory_access_size to identify the number of bytes that it is safe -* to write without accidentally writing to another MemoryRegion. +* fuzz_memory_access_size to identify the number of bytes that it +* is safe to write without accidentally writing to another +* MemoryRegion. */ if (!memory_region_is_ram (mr1)) { - l =memory_access_size (mr1, l, addr1); + l =fuzz_memory_access_size (mr1, l, addr1); } if (memory_region_is_ram(mr1) || memory_region_is_romd(mr1) ||hw/remote/trace-events +3
-0@@ -9,3 +9,6 @@ vfu_cfg_read(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u -> 0x%x" vfu_cfg_write(uint32_t offset, uint32_t val) "vfu: cfg: 0x%u <- 0x%x" vfu_dma_register(uint64_t gpa, size_t len) "vfu: registering GPA 0x%"PRIx64", %zu bytes" vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" +vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" +vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" +vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64""Signature · 1 line
2.20.1
PATCH v6 15/19 vfio-user: handle device interrupts
In reply to Jagannathan RamanForward remote device's interrupts to the guest
--- include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-) create mode 100644 include/hw/remote/vfio-user-obj.h create mode 100644 stubs/vfio-user-obj.c
include/hw/pci/pci.h +6 -0
@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */
@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */
include/hw/remote/vfio-user-obj.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif
hw/pci/msi.c +12 -1
@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {
@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ; + dev->msi_notify = pci_msi_notify; +pci_set_word (dev->config +msi_flags_off (dev), flags);pci_set_word (dev->wmask +msi_flags_off (dev),PCI_MSI_FLAGS_QSIZE |PCI_MSI_FLAGS_ENABLE );
@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)return mask & (1U << vector); } -void msi_notify (PCIDevice *dev,unsigned int vector) +static void pci_msi_notify (PCIDevice *dev,unsigned int vector) {uint16_t flags =pci_get_word (dev->config +msi_flags_off (dev));bool msi64bit = flags &PCI_MSI_FLAGS_64BIT ;
@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); } +void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) { + dev->msi_notify (dev, vector); + } +} +void msi_send_message (PCIDevice *dev,MSIMessage msg) {MemTxAttrs attrs = {};
hw/pci/msix.c +11 -1
@@ -31,6 +31,8 @@#define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)#define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8) +static void pci_msix_notify (PCIDevice *dev,unsigned vector); +MSIMessage msix_get_message (PCIDevice *dev,unsigned vector) {uint8_t *table_entry = dev->msix_table + vector *PCI_MSIX_ENTRY_SIZE ;
@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries, dev->msix_table =g_malloc0 (table_size); dev->msix_pba =g_malloc0 (pba_size); dev->msix_entry_used =g_malloc0 (nentries *sizeof *dev->msix_entry_used); + dev->msix_notify = pci_msix_notify;msix_mask_all (dev, nentries);
@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev) }/* Send an MSI-X message */ -void msix_notify (PCIDevice *dev,unsigned vector) +static void pci_msix_notify (PCIDevice *dev,unsigned vector) {MSIMessage msg;
@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)msi_send_message (dev, msg); } +void msix_notify (PCIDevice *dev,unsigned vector) +{ +if (dev->msix_notify) { + dev->msix_notify (dev, vector); + } +} +void msix_reset (PCIDevice *dev) {if (!msix_present (dev)) {
hw/remote/machine.c +7 -4
@@ -23,6 +23,7 @@#include "hw/remote/iohub.h" #include "hw/qdev-core.h" #include "hw/remote/iommu.h" +#include "hw/remote/vfio-user-obj.h" static void remote_machine_init (MachineState *machine) {
@@ -54,12 +55,14 @@ static void remote_machine_init(MachineState *machine)if (s->vfio_user) {remote_configure_iommu (pci_host->bus); - } -remote_iohub_init (&s->iohub); +vfu_object_set_bus_irq (pci_host->bus); + }else { +remote_iohub_init (&s->iohub); -pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, - &s->iohub,REMOTE_IOHUB_NB_PIRQS ); +pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + }qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); }
hw/remote/vfio-user-obj.c +107 -0
152 lines not on this page · raw message
stubs/vfio-user-obj.c new file +6 -0
@@ -0,0 +1,6 @@ +#include "qemu/osdep.h" +#include "hw/remote/vfio-user-obj.h" + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +}
MAINTAINERS +1 -0
@@ -3569,6 +3569,7 @@ F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c +F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c F: include/hw/remote/iommu.h
hw/remote/trace-events +1 -0
@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64"" +vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"
stubs/meson.build +1 -0
@@ -57,3 +57,4 @@ if have_system else stub_ss.add(files('qdev.c')) endif +stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_false: files('vfio-user-obj.c'))
Signature · 1 line
2.20.1
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 70 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-) create mode 100644 include/hw/remote/vfio-user-obj.h create mode 100644 stubs/vfio-user-obj.cinclude/hw/pci/pci.h +6
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {
hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 4 lines
hw/pci/msi.c +1
-0@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ; + dev->msi_notify = pci_msi_notify;
Are you sure it's correct to skip the msi_is_masked() logic? I think the callback function should only override the behavior of msi_send_message(), not the entire msi_notify() function.
The same applies to MSI-X.
Quoted 14 lines
hw/pci/msi.c +2
-1+pci_set_word (dev->config +msi_flags_off (dev), flags);pci_set_word (dev->wmask +msi_flags_off (dev),PCI_MSI_FLAGS_QSIZE |PCI_MSI_FLAGS_ENABLE );@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)return mask & (1U << vector); } -void msi_notify (PCIDevice *dev,unsigned int vector) +static void pci_msi_notify (PCIDevice *dev,unsigned int vector) {uint16_t flags =pci_get_word (dev->config +msi_flags_off (dev));bool msi64bit = flags &PCI_MSI_FLAGS_64BIT ;
hw/pci/msi.c void msi_notify(PCIDevice *dev, unsigned int vector) · line 341
Quoted 6 lines
hw/pci/msi.c +3
-0@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); } +void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) {
Can this ever be NULL?
Quoted 38 lines
hw/pci/msi.c +4
-0+ dev->msi_notify (dev, vector); + } +} +void msi_send_message (PCIDevice *dev,MSIMessage msg) {MemTxAttrs attrs = {};hw/pci/msix.c +4
-1@@ -31,6 +31,8 @@#define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)#define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8) +static void pci_msix_notify (PCIDevice *dev,unsigned vector); +MSIMessage msix_get_message (PCIDevice *dev,unsigned vector) {uint8_t *table_entry = dev->msix_table + vector *PCI_MSIX_ENTRY_SIZE ;@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries, dev->msix_table =g_malloc0 (table_size); dev->msix_pba =g_malloc0 (pba_size); dev->msix_entry_used =g_malloc0 (nentries *sizeof *dev->msix_entry_used); + dev->msix_notify = pci_msix_notify;msix_mask_all (dev, nentries);@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev) }/* Send an MSI-X message */ -void msix_notify (PCIDevice *dev,unsigned vector) +static void pci_msix_notify (PCIDevice *dev,unsigned vector) {MSIMessage msg;
hw/pci/msix.c void msix_notify(PCIDevice *dev, unsigned vector) · line 511
Quoted 6 lines
hw/pci/msix.c +3
-0@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)msi_send_message (dev, msg); } +void msix_notify (PCIDevice *dev,unsigned vector) +{ +if (dev->msix_notify) {
Can this ever be NULL?
Quoted 53 lines
hw/pci/msix.c +4
-0+ dev->msix_notify (dev, vector); + } +} +void msix_reset (PCIDevice *dev) {if (!msix_present (dev)) {hw/remote/machine.c +7
-4@@ -23,6 +23,7 @@#include "hw/remote/iohub.h" #include "hw/qdev-core.h" #include "hw/remote/iommu.h" +#include "hw/remote/vfio-user-obj.h" static void remote_machine_init (MachineState *machine) {@@ -54,12 +55,14 @@ static void remote_machine_init(MachineState *machine)if (s->vfio_user) {remote_configure_iommu (pci_host->bus); - } -remote_iohub_init (&s->iohub); +vfu_object_set_bus_irq (pci_host->bus); + }else { +remote_iohub_init (&s->iohub); -pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, - &s->iohub,REMOTE_IOHUB_NB_PIRQS ); +pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + }qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); }hw/remote/vfio-user-obj.c +3
-0@@ -54,6 +54,9 @@#include "hw/pci/pci.h" #include "qemu/timer.h" #include "exec/memory.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/remote/vfio-user-obj.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )
hw/remote/vfio-user-obj.c struct VfuObject { · line 113
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -107,6 +110,10 @@ struct VfuObject {int vfu_poll_fd; }; +static GHashTable *vfu_object_bdf_to_ctx_table;
I suggest adding a void *msi_notify_opaque field to PCIDevice and passing the value as an argument to ->msi_notify(). vfio-user-obj.c can set the value to vfu_ctx and eliminate the vfu_object_bdf_to_ctx_table hash table.
This simplifies the code, makes it faster, and solves race conditions during hot plug/unplug if other instances are running in IOThreads.
Quoted 7 lines
hw/remote/vfio-user-obj.c +3
-0+ +#define INT2VOIDP (i) (void *)(uintptr_t)(i) +static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,
hw/remote/vfio-user-obj.c static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) · line 477
Quoted 8 lines
hw/remote/vfio-user-obj.c +5
-0@@ -463,6 +470,86 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) } } +static void vfu_object_irq_trigger (int pci_bdf,unsigned vector) +{ +vfu_ctx_t *vfu_ctx =NULL ; + +if (!vfu_object_bdf_to_ctx_table) {
Can this ever be NULL?
Quoted 119 lines
hw/remote/vfio-user-obj.c +91
-0+return ; + } + + vfu_ctx =g_hash_table_lookup (vfu_object_bdf_to_ctx_table, +INT2VOIDP (pci_bdf)); + +if (vfu_ctx) { +vfu_irq_trigger (vfu_ctx, vector); + } +} + +static int vfu_object_map_irq (PCIDevice *pci_dev,int intx) +{ +int pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), + pci_dev->devfn); + +return pci_bdf; +} + +static void vfu_object_set_irq (void *opaque,int pirq,int level) +{ +if (level) { +vfu_object_irq_trigger (pirq,0 ); + } +} + +static void vfu_object_msi_notify (PCIDevice *pci_dev,unsigned vector) +{ +int pci_bdf; + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +vfu_object_irq_trigger (pci_bdf, vector); +} + +static int vfu_object_setup_irqs (VfuObject *o,PCIDevice *pci_dev) +{ +vfu_ctx_t *vfu_ctx = o->vfu_ctx; +int ret, pci_bdf; + + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_INTX_IRQ ,1 ); +if (ret <0 ) { +return ret; + } + + ret =0 ; +if (msix_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSIX_IRQ , +msix_nr_vectors_allocated (pci_dev)); + + pci_dev->msix_notify = vfu_object_msi_notify; + }else if (msi_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSI_IRQ , +msi_nr_vectors_allocated (pci_dev)); + + pci_dev->msi_notify = vfu_object_msi_notify; + } + +if (ret <0 ) { +return ret; + } + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +g_hash_table_insert (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf), + o->vfu_ctx); + +return 0 ; +} + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -559,6 +646,13 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_object_setup_irqs (o, o->pci_dev); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup interrupts for %s" , + o->device); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",@@ -612,6 +706,7 @@ static void vfu_object_finalize(Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj);VfuObject *o =VFU_OBJECT (obj); +int pci_bdf; k->nr_devs--;@@ -638,9 +733,17 @@ static void vfu_object_finalize(Object *obj) o->unplug_blocker =NULL ; } +if (o->pci_dev) { + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (o->pci_dev)), + o->pci_dev->devfn); +g_hash_table_remove (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf)); + } + o->pci_dev =NULL ;if (!k->nr_devs && k->auto_shutdown) { +g_hash_table_destroy (vfu_object_bdf_to_ctx_table); + vfu_object_bdf_to_ctx_table =NULL ;qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); }
hw/remote/vfio-user-obj.c static void vfu_object_class_init(ObjectClass *klass, void *data) · line 764
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -658,6 +761,10 @@ static void vfu_object_class_init(ObjectClass *klass, void *data) k->auto_shutdown = true; + msi_nonbroken = true;
This should go in hw/remote/machine.c. It's a global variable related to the machine's interrupt controller capabilities. The value is not related to vfu_object_class_init(), which will be called by any QEMU binary that links hw/remote/vfio-user-obj.o regardless of which machine type is instantiated.
Quoted 51 lines
hw/remote/vfio-user-obj.c +3
-0+ + vfu_object_bdf_to_ctx_table =g_hash_table_new_full (NULL ,NULL ,NULL ,NULL ); +object_class_property_add (klass,"socket" ,"SocketAddress" ,NULL , vfu_object_set_socket,NULL ,NULL ); object_class_property_set_description(klass, "socket",stubs/vfio-user-obj.c new file +6
-0@@ -0,0 +1,6 @@ +#include "qemu/osdep.h" +#include "hw/remote/vfio-user-obj.h" + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +}MAINTAINERS +1
-0@@ -3569,6 +3569,7 @@ F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c +F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c F: include/hw/remote/iommu.hhw/remote/trace-events +1
-0@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64"" vfu_bar_register(int i, uint64_t addr, uint64_t size) "vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64"" vfu_bar_rw_enter(const char *op, uint64_t addr) "vfu: %s request for BAR address 0x%"PRIx64"" vfu_bar_rw_exit(const char *op, uint64_t addr) "vfu: Finished %s of BAR address 0x%"PRIx64"" +vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"stubs/meson.build +1
-0@@ -57,3 +57,4 @@ if have_system else stub_ss.add(files('qdev.c')) endif +stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_false: files('vfio-user-obj.c'))Signature · 1 line
2.20.1
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 78 lines · Stefan Hajnoczi
Quoted 73 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cinclude/hw/pci/pci.h +6
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ;
Quoted 3 lines
hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 1 line
hw/pci/msi.c +1
-0+ dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the
pci_msi_notify() callback includes the test for msi_is_masked() - that covers the non vfio-user case.
The vfio-user callback should add this test - will do it if we continue this approach. Thanks for the catch!
Quoted 2 lines
callback function should only override the behavior of
msi_send_message(), not the entiremsi_notify()function.
OK, this sounds like a better approach.
Quoted 20 lines
The same applies to MSI-X.
Quoted 17 lines
hw/pci/msi.c +2
-1+pci_set_word (dev->config +msi_flags_off (dev), flags);pci_set_word (dev->wmask +msi_flags_off (dev),PCI_MSI_FLAGS_QSIZE |PCI_MSI_FLAGS_ENABLE );@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)return mask & (1U << vector); } -void msi_notify (PCIDevice *dev,unsigned int vector) +static void pci_msi_notify (PCIDevice *dev,unsigned int vector) {uint16_t flags =pci_get_word (dev->config +msi_flags_off (dev));bool msi64bit = flags &PCI_MSI_FLAGS_64BIT ;@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); }
Quoted 5 lines
hw/pci/msi.c void msi_notify(PCIDevice *dev, unsigned int vector) · line 341
Quoted 3 lines
hw/pci/msi.c +3
-0+void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) {Can this ever be NULL?
Unlikely in the current code flow, but it could change in the future.
As a matter of principle, I thought that we should check if a function pointer is non-NULL before invoking it in QEMU. Is that not the case?
Quoted 111 lines
Quoted 39 lines
hw/pci/msi.c +4
-0+ dev->msi_notify (dev, vector); + } +} +void msi_send_message (PCIDevice *dev,MSIMessage msg) {MemTxAttrs attrs = {};hw/pci/msix.c +4
-1@@ -31,6 +31,8 @@#define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)#define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8) +static void pci_msix_notify (PCIDevice *dev,unsigned vector); +MSIMessage msix_get_message (PCIDevice *dev,unsigned vector) {uint8_t *table_entry = dev->msix_table + vector *PCI_MSIX_ENTRY_SIZE ;@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries, dev->msix_table =g_malloc0 (table_size); dev->msix_pba =g_malloc0 (pba_size); dev->msix_entry_used =g_malloc0 (nentries *sizeof *dev->msix_entry_used); + dev->msix_notify = pci_msix_notify;msix_mask_all (dev, nentries);@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev) }/* Send an MSI-X message */ -void msix_notify (PCIDevice *dev,unsigned vector) +static void pci_msix_notify (PCIDevice *dev,unsigned vector) {MSIMessage msg;hw/pci/msix.c void msix_notify(PCIDevice *dev, unsigned vector) · line 511
Quoted 6 lines
hw/pci/msix.c +3
-0@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)msi_send_message (dev, msg); } +void msix_notify (PCIDevice *dev,unsigned vector) +{ +if (dev->msix_notify) {Can this ever be NULL?
Quoted 53 lines
hw/pci/msix.c +4
-0+ dev->msix_notify (dev, vector); + } +} +void msix_reset (PCIDevice *dev) {if (!msix_present (dev)) {hw/remote/machine.c +7
-4@@ -23,6 +23,7 @@#include "hw/remote/iohub.h" #include "hw/qdev-core.h" #include "hw/remote/iommu.h" +#include "hw/remote/vfio-user-obj.h" static void remote_machine_init (MachineState *machine) {@@ -54,12 +55,14 @@ static void remote_machine_init(MachineState *machine)if (s->vfio_user) {remote_configure_iommu (pci_host->bus); - } -remote_iohub_init (&s->iohub); +vfu_object_set_bus_irq (pci_host->bus); + }else { +remote_iohub_init (&s->iohub); -pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, - &s->iohub,REMOTE_IOHUB_NB_PIRQS ); +pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + }qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); }hw/remote/vfio-user-obj.c +3
-0@@ -54,6 +54,9 @@#include "hw/pci/pci.h" #include "qemu/timer.h" #include "exec/memory.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/remote/vfio-user-obj.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )hw/remote/vfio-user-obj.c struct VfuObject { · line 113
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -107,6 +110,10 @@ struct VfuObject {int vfu_poll_fd; }; +static GHashTable *vfu_object_bdf_to_ctx_table;I suggest adding a void
*msi_notify_opaquefield to PCIDevice and passing the value as an argument to ->msi_notify().vfio-user-obj.ccan set the value tovfu_ctxand eliminate thevfu_object_bdf_to_ctx_tablehash table.
Quoted 2 lines
This simplifies the code, makes it faster, and solves race conditions during hot plug/unplug if other instances are running in IOThreads.
OK, will do.
Quoted 141 lines
Quoted 8 lines
hw/remote/vfio-user-obj.c +3
-0+ +#define INT2VOIDP (i) (void *)(uintptr_t)(i) +static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,hw/remote/vfio-user-obj.c static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) · line 477
Quoted 8 lines
hw/remote/vfio-user-obj.c +5
-0@@ -463,6 +470,86 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) } } +static void vfu_object_irq_trigger (int pci_bdf,unsigned vector) +{ +vfu_ctx_t *vfu_ctx =NULL ; + +if (!vfu_object_bdf_to_ctx_table) {Can this ever be NULL?
Quoted 122 lines
hw/remote/vfio-user-obj.c +91
-0+return ; + } + + vfu_ctx =g_hash_table_lookup (vfu_object_bdf_to_ctx_table, +INT2VOIDP (pci_bdf)); + +if (vfu_ctx) { +vfu_irq_trigger (vfu_ctx, vector); + } +} + +static int vfu_object_map_irq (PCIDevice *pci_dev,int intx) +{ +int pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), + pci_dev->devfn); + +return pci_bdf; +} + +static void vfu_object_set_irq (void *opaque,int pirq,int level) +{ +if (level) { +vfu_object_irq_trigger (pirq,0 ); + } +} + +static void vfu_object_msi_notify (PCIDevice *pci_dev,unsigned vector) +{ +int pci_bdf; + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +vfu_object_irq_trigger (pci_bdf, vector); +} + +static int vfu_object_setup_irqs (VfuObject *o,PCIDevice *pci_dev) +{ +vfu_ctx_t *vfu_ctx = o->vfu_ctx; +int ret, pci_bdf; + + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_INTX_IRQ ,1 ); +if (ret <0 ) { +return ret; + } + + ret =0 ; +if (msix_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSIX_IRQ , +msix_nr_vectors_allocated (pci_dev)); + + pci_dev->msix_notify = vfu_object_msi_notify; + }else if (msi_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSI_IRQ , +msi_nr_vectors_allocated (pci_dev)); + + pci_dev->msi_notify = vfu_object_msi_notify; + } + +if (ret <0 ) { +return ret; + } + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +g_hash_table_insert (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf), + o->vfu_ctx); + +return 0 ; +} + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -559,6 +646,13 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_object_setup_irqs (o, o->pci_dev); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup interrupts for %s" , + o->device); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",@@ -612,6 +706,7 @@ static void vfu_object_finalize(Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj);VfuObject *o =VFU_OBJECT (obj); +int pci_bdf; k->nr_devs--;@@ -638,9 +733,17 @@ static void vfu_object_finalize(Object *obj) o->unplug_blocker =NULL ; } +if (o->pci_dev) { + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (o->pci_dev)), + o->pci_dev->devfn); +g_hash_table_remove (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf)); + } + o->pci_dev =NULL ;if (!k->nr_devs && k->auto_shutdown) { +g_hash_table_destroy (vfu_object_bdf_to_ctx_table); + vfu_object_bdf_to_ctx_table =NULL ;qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); }@@ -658,6 +761,10 @@ static void vfu_object_class_init(ObjectClass *klass, void *data) k->auto_shutdown = true;
Quoted 7 lines
hw/remote/vfio-user-obj.c static void vfu_object_class_init(ObjectClass *klass, void *data) · line 764
Quoted 1 line
hw/remote/vfio-user-obj.c +1
-0+ msi_nonbroken = true;This should go in
hw/remote/machine.c. It's a global variable related to the machine's interrupt controller capabilities. The value is not related tovfu_object_class_init(), which will be called by any QEMU binary that linkshw/remote/vfio-user-obj.oregardless of which machine type is instantiated.
multiprocess QEMU, which also uses the remote machine, doesn’t support MSI and that’s why we placed it here originally. In subsequent series, we have added ‘vfio-user’ machine sub-option to discern vfio-user and multiprocess, so this could be moved to the machine initialization code as you just pointed out.
Thank you!
--
Jag
Quoted 52 lines
Quoted 52 lines
hw/remote/vfio-user-obj.c +3
-0+ + vfu_object_bdf_to_ctx_table =g_hash_table_new_full (NULL ,NULL ,NULL ,NULL ); +object_class_property_add (klass,"socket" ,"SocketAddress" ,NULL , vfu_object_set_socket,NULL ,NULL ); object_class_property_set_description(klass, "socket",stubs/vfio-user-obj.c new file +6
-0@@ -0,0 +1,6 @@ +#include "qemu/osdep.h" +#include "hw/remote/vfio-user-obj.h" + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +}MAINTAINERS +0
-0@@ -3569,6 +3569,7 @@ F: hw/remote/iohub.cF:
include/hw/remote/iohub.h
F: subprojects/libvfio-user
F:hw/remote/vfio-user-obj.c
+F:include/hw/remote/vfio-user-obj.h
F:hw/remote/iommu.c
F:include/hw/remote/iommu.hhw/remote/trace-events +0
-0@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
vfu_bar_register(int i, uint64_t addr, uint64_t size)"vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""vfu_bar_rw_enter(const char *op, uint64_t addr)"vfu: %s request for BAR address 0x%"PRIx64""vfu_bar_rw_exit(const char *op, uint64_t addr)"vfu: Finished %s of BAR address 0x%"PRIx64""hw/remote/trace-events +1
-0+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"stubs/meson.build +0
-0@@ -57,3 +57,4 @@ if have_systemelse
stub_ss.add(files('qdev.c'))
endif
+stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER',if_false: files('vfio-user-obj.c'))Signature · 1 line
2.20.1
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 12 lines · Jag Raman
Quoted 11 lines · Stefan Hajnoczi
hw/pci/msi.c void msi_notify(PCIDevice *dev, unsigned int vector) · line 341
Quoted 7 lines · Jagannathan Raman
hw/pci/msi.c +3
-0@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); } +void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) {Can this ever be NULL?
Quoted 4 lines
Unlikely in the current code flow, but it could change in the future.
As a matter of principle, I thought that we should check if a function pointer is non-NULL before invoking it in QEMU. Is that not the case?
No, it's better to dump core with a backtrace when a program invariant is violated than to silently suppress it. If msi_notify() is called but the function pointer is NULL then there is a bug in the program that needs to be fixed.
Stefan
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
In reply to Stefan HajnocziQuoted 78 lines · Stefan Hajnoczi
Quoted 73 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cinclude/hw/pci/pci.h +6
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ;
Quoted 7 lines
hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 1 line
hw/pci/msi.c +1
-0+ dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the callback function should only override the behavior ofmsi_send_message(), not the entiremsi_notify()function.The same applies to MSI-X.
Hi Stefan,
We noticed that the client is handling the masking and unmasking of MSIx interrupts.
Concerning MSIx, vfio_msix_vector_use() handles unmasking and vfio_msix_vector_release() handles masking operations. The server triggers an MSIx interrupt by signaling the eventfd associated with the vector. If the vector is unmasked, the interrupt bypasses the client/QEMU and takes this
path: “server -> KVM -> guest”. Whereas, if the vector is masked, it lands on the client via: “server -> vfio_msi_interrupt()”. vfio_msi_interrupt() suppresses the interrupt if the vector is masked. The use and release functions switch the server’s eventfd between VFIOPCIDevice->VFIOMSIVector[i]->kvm_interrupt and VFIOPCIDevice->VFIOMSIVector[i]->interrupt using the VFIO_DEVICE_SET_IRQS message.
Concerning MSI, the server should check if the vector is unmasked before triggering. The server is not doing it presently, will update it. For some reason, I had assumed that MSI handling is similar to MSIx in terms of masking - sorry about that. The masking and unmasking information for MSI is in the config space registers, so the server should have this information.
You had previously suggested using callbacks for msi_get_message & msi_send_message, considering the masking issue. Given MSIx masking (including the MSIx table BAR) is handled at the client, the masking information doesn’t reach the server - so msix_notify will never invoke the msi_send_message callback - all the vectors remain masked at the server end (msix_init() -> msix_mask_all()).
Thank you!
--
Jag
Quoted 336 lines
Quoted 15 lines
hw/pci/msi.c +2
-1+pci_set_word (dev->config +msi_flags_off (dev), flags);pci_set_word (dev->wmask +msi_flags_off (dev),PCI_MSI_FLAGS_QSIZE |PCI_MSI_FLAGS_ENABLE );@@ -307,7 +311,7 @@ bool msi_is_masked(const PCIDevice *dev, unsigned int vector)return mask & (1U << vector); } -void msi_notify (PCIDevice *dev,unsigned int vector) +static void pci_msi_notify (PCIDevice *dev,unsigned int vector) {uint16_t flags =pci_get_word (dev->config +msi_flags_off (dev));bool msi64bit = flags &PCI_MSI_FLAGS_64BIT ;hw/pci/msi.c void msi_notify(PCIDevice *dev, unsigned int vector) · line 341
Quoted 6 lines
hw/pci/msi.c +3
-0@@ -332,6 +336,13 @@ void msi_notify(PCIDevice *dev, unsigned int vector)msi_send_message (dev, msg); } +void msi_notify (PCIDevice *dev,unsigned int vector) +{ +if (dev->msi_notify) {Can this ever be NULL?
Quoted 38 lines
hw/pci/msi.c +4
-0+ dev->msi_notify (dev, vector); + } +} +void msi_send_message (PCIDevice *dev,MSIMessage msg) {MemTxAttrs attrs = {};hw/pci/msix.c +4
-1@@ -31,6 +31,8 @@#define MSIX_ENABLE_MASK (PCI_MSIX_FLAGS_ENABLE >> 8)#define MSIX_MASKALL_MASK (PCI_MSIX_FLAGS_MASKALL >> 8) +static void pci_msix_notify (PCIDevice *dev,unsigned vector); +MSIMessage msix_get_message (PCIDevice *dev,unsigned vector) {uint8_t *table_entry = dev->msix_table + vector *PCI_MSIX_ENTRY_SIZE ;@@ -334,6 +336,7 @@ int msix_init(struct PCIDevice *dev, unsigned short nentries, dev->msix_table =g_malloc0 (table_size); dev->msix_pba =g_malloc0 (pba_size); dev->msix_entry_used =g_malloc0 (nentries *sizeof *dev->msix_entry_used); + dev->msix_notify = pci_msix_notify;msix_mask_all (dev, nentries);@@ -485,7 +488,7 @@ int msix_enabled(PCIDevice *dev) }/* Send an MSI-X message */ -void msix_notify (PCIDevice *dev,unsigned vector) +static void pci_msix_notify (PCIDevice *dev,unsigned vector) {MSIMessage msg;hw/pci/msix.c void msix_notify(PCIDevice *dev, unsigned vector) · line 511
Quoted 6 lines
hw/pci/msix.c +3
-0@@ -503,6 +506,13 @@ void msix_notify(PCIDevice *dev, unsigned vector)msi_send_message (dev, msg); } +void msix_notify (PCIDevice *dev,unsigned vector) +{ +if (dev->msix_notify) {Can this ever be NULL?
Quoted 53 lines
hw/pci/msix.c +4
-0+ dev->msix_notify (dev, vector); + } +} +void msix_reset (PCIDevice *dev) {if (!msix_present (dev)) {hw/remote/machine.c +7
-4@@ -23,6 +23,7 @@#include "hw/remote/iohub.h" #include "hw/qdev-core.h" #include "hw/remote/iommu.h" +#include "hw/remote/vfio-user-obj.h" static void remote_machine_init (MachineState *machine) {@@ -54,12 +55,14 @@ static void remote_machine_init(MachineState *machine)if (s->vfio_user) {remote_configure_iommu (pci_host->bus); - } -remote_iohub_init (&s->iohub); +vfu_object_set_bus_irq (pci_host->bus); + }else { +remote_iohub_init (&s->iohub); -pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, - &s->iohub,REMOTE_IOHUB_NB_PIRQS ); +pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); + }qbus_set_hotplug_handler (BUS (pci_host->bus),OBJECT (s)); }hw/remote/vfio-user-obj.c +3
-0@@ -54,6 +54,9 @@#include "hw/pci/pci.h" #include "qemu/timer.h" #include "exec/memory.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/remote/vfio-user-obj.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )hw/remote/vfio-user-obj.c struct VfuObject { · line 113
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -107,6 +110,10 @@ struct VfuObject {int vfu_poll_fd; }; +static GHashTable *vfu_object_bdf_to_ctx_table;I suggest adding a void
*msi_notify_opaquefield to PCIDevice and passing the value as an argument to ->msi_notify().vfio-user-obj.ccan set the value tovfu_ctxand eliminate thevfu_object_bdf_to_ctx_tablehash table.This simplifies the code, makes it faster, and solves race conditions during hot plug/unplug if other instances are running in IOThreads.
Quoted 7 lines
hw/remote/vfio-user-obj.c +3
-0+ +#define INT2VOIDP (i) (void *)(uintptr_t)(i) +static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,hw/remote/vfio-user-obj.c static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) · line 477
Quoted 8 lines
hw/remote/vfio-user-obj.c +5
-0@@ -463,6 +470,86 @@ static void vfu_object_register_bars(vfu_ctx_t *vfu_ctx, PCIDevice *pdev) } } +static void vfu_object_irq_trigger (int pci_bdf,unsigned vector) +{ +vfu_ctx_t *vfu_ctx =NULL ; + +if (!vfu_object_bdf_to_ctx_table) {Can this ever be NULL?
Quoted 119 lines
hw/remote/vfio-user-obj.c +91
-0+return ; + } + + vfu_ctx =g_hash_table_lookup (vfu_object_bdf_to_ctx_table, +INT2VOIDP (pci_bdf)); + +if (vfu_ctx) { +vfu_irq_trigger (vfu_ctx, vector); + } +} + +static int vfu_object_map_irq (PCIDevice *pci_dev,int intx) +{ +int pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), + pci_dev->devfn); + +return pci_bdf; +} + +static void vfu_object_set_irq (void *opaque,int pirq,int level) +{ +if (level) { +vfu_object_irq_trigger (pirq,0 ); + } +} + +static void vfu_object_msi_notify (PCIDevice *pci_dev,unsigned vector) +{ +int pci_bdf; + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +vfu_object_irq_trigger (pci_bdf, vector); +} + +static int vfu_object_setup_irqs (VfuObject *o,PCIDevice *pci_dev) +{ +vfu_ctx_t *vfu_ctx = o->vfu_ctx; +int ret, pci_bdf; + + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_INTX_IRQ ,1 ); +if (ret <0 ) { +return ret; + } + + ret =0 ; +if (msix_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSIX_IRQ , +msix_nr_vectors_allocated (pci_dev)); + + pci_dev->msix_notify = vfu_object_msi_notify; + }else if (msi_nr_vectors_allocated (pci_dev)) { + ret =vfu_setup_device_nr_irqs (vfu_ctx,VFU_DEV_MSI_IRQ , +msi_nr_vectors_allocated (pci_dev)); + + pci_dev->msi_notify = vfu_object_msi_notify; + } + +if (ret <0 ) { +return ret; + } + + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (pci_dev)), pci_dev->devfn); + +g_hash_table_insert (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf), + o->vfu_ctx); + +return 0 ; +} + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -559,6 +646,13 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)vfu_object_register_bars (o->vfu_ctx, o->pci_dev); + ret =vfu_object_setup_irqs (o, o->pci_dev); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup interrupts for %s" , + o->device); + goto fail; + } + ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",@@ -612,6 +706,7 @@ static void vfu_object_finalize(Object *obj) {VfuObjectClass *k =VFU_OBJECT_GET_CLASS (obj);VfuObject *o =VFU_OBJECT (obj); +int pci_bdf; k->nr_devs--;@@ -638,9 +733,17 @@ static void vfu_object_finalize(Object *obj) o->unplug_blocker =NULL ; } +if (o->pci_dev) { + pci_bdf =PCI_BUILD_BDF (pci_bus_num (pci_get_bus (o->pci_dev)), + o->pci_dev->devfn); +g_hash_table_remove (vfu_object_bdf_to_ctx_table,INT2VOIDP (pci_bdf)); + } + o->pci_dev =NULL ;if (!k->nr_devs && k->auto_shutdown) { +g_hash_table_destroy (vfu_object_bdf_to_ctx_table); + vfu_object_bdf_to_ctx_table =NULL ;qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); }hw/remote/vfio-user-obj.c static void vfu_object_class_init(ObjectClass *klass, void *data) · line 764
Quoted 4 lines
hw/remote/vfio-user-obj.c +1
-0@@ -658,6 +761,10 @@ static void vfu_object_class_init(ObjectClass *klass, void *data) k->auto_shutdown = true; + msi_nonbroken = true;This should go in
hw/remote/machine.c. It's a global variable related to the machine's interrupt controller capabilities. The value is not related tovfu_object_class_init(), which will be called by any QEMU binary that linkshw/remote/vfio-user-obj.oregardless of which machine type is instantiated.Quoted 51 lines
hw/remote/vfio-user-obj.c +3
-0+ + vfu_object_bdf_to_ctx_table =g_hash_table_new_full (NULL ,NULL ,NULL ,NULL ); +object_class_property_add (klass,"socket" ,"SocketAddress" ,NULL , vfu_object_set_socket,NULL ,NULL ); object_class_property_set_description(klass, "socket",stubs/vfio-user-obj.c new file +6
-0@@ -0,0 +1,6 @@ +#include "qemu/osdep.h" +#include "hw/remote/vfio-user-obj.h" + +void vfu_object_set_bus_irq (PCIBus *pci_bus) +{ +}MAINTAINERS +0
-0@@ -3569,6 +3569,7 @@ F: hw/remote/iohub.cF:
include/hw/remote/iohub.h
F: subprojects/libvfio-user
F:hw/remote/vfio-user-obj.c
+F:include/hw/remote/vfio-user-obj.h
F:hw/remote/iommu.c
F:include/hw/remote/iommu.hhw/remote/trace-events +0
-0@@ -12,3 +12,4 @@ vfu_dma_unregister(uint64_t gpa) "vfu: unregistering GPA 0x%"PRIx64""
vfu_bar_register(int i, uint64_t addr, uint64_t size)"vfu: BAR %d: addr 0x%"PRIx64" size 0x%"PRIx64""vfu_bar_rw_enter(const char *op, uint64_t addr)"vfu: %s request for BAR address 0x%"PRIx64""vfu_bar_rw_exit(const char *op, uint64_t addr)"vfu: Finished %s of BAR address 0x%"PRIx64""hw/remote/trace-events +1
-0+vfu_interrupt(int pirq) "vfu: sending interrupt to device - PIRQ %d"stubs/meson.build +0
-0@@ -57,3 +57,4 @@ if have_systemelse
stub_ss.add(files('qdev.c'))
endif
+stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER',if_false: files('vfio-user-obj.c'))Signature · 1 line
2.20.1
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 108 lines · Jag Raman
Quoted 78 lines · Stefan Hajnoczi
Quoted 73 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cinclude/hw/pci/pci.h +6
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ;Quoted 7 lines
hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 1 line
hw/pci/msi.c +1
-0+ dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the callback function should only override the behavior ofmsi_send_message(), not the entiremsi_notify()function.The same applies to MSI-X.
Hi Stefan,
We noticed that the client is handling the masking and unmasking of MSIx interrupts.
Concerning MSIx,
vfio_msix_vector_use()handles unmasking andvfio_msix_vector_release()handles masking operations. The server triggers an MSIx interrupt by signaling the eventfd associated with the vector. If the vector is unmasked, the interrupt bypasses the client/QEMU and takes this
path: “server -> KVM -> guest”. Whereas, if the vector is masked, it lands on the client via: “server -> vfio_msi_interrupt()”.vfio_msi_interrupt()suppresses the interrupt if the vector is masked. The use and release functions switch the server’s eventfd between VFIOPCIDevice->VFIOMSIVector[i]->kvm_interrupt and VFIOPCIDevice->VFIOMSIVector[i]->interrupt using theVFIO_DEVICE_SET_IRQSmessage.Concerning MSI, the server should check if the vector is unmasked before triggering. The server is not doing it presently, will update it. For some reason, I had assumed that MSI handling is similar to MSIx in terms of masking - sorry about that. The masking and unmasking information for MSI is in the config space registers, so the server should have this information.
Quoted 6 lines
You had previously suggested using callbacks for
msi_get_message&msi_send_message, considering the masking issue. Given MSIx masking (including the MSIx table BAR) is handled at the client, the masking information doesn’t reach the server - somsix_notifywill never invoke themsi_send_messagecallback - all the vectors remain masked at the server end (msix_init()->msix_mask_all()).
I was expecting vfio-user devices to be involved in MSI-X masking so they can implement the Pending bit semantics described in the spec:
If a masked vector has its Pending bit Set, and the associated underlying interrupt events are somehow satisfied (usually by software though the exact manner is Function-specific), the Function must Clear the Pending bit, to avoid sending a spurious interrupt message later when software unmasks the vector.
Does QEMU VFIO support this?
What happens when a hw/net/e1000e_core.c vfio-user device uses msix_clr_pending() and related APIs?
Also, having the vfio-user daemon write to the eventfd while the vector is masked is a waste of CPU cycles. The PCIe spec describes using MSI-X masking for poll mode operation and going via eventfd is suboptimal:
Software is permitted to mask one or more vectors indefinitely, and service their associated interrupt events strictly based on polling their Pending bits. A Function must Set and Clear its Pending bits as necessary to support this “pure polling” mode of operation.
Maybe the answer is these issues don't matter in practice because MSI-X masking is not used much?
Stefan
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 123 lines · Stefan Hajnoczi
Quoted 108 lines · Jag Raman
Quoted 78 lines · Stefan Hajnoczi
Quoted 73 lines · Jagannathan Raman
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cinclude/hw/pci/pci.h +6
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice *pci_dev,typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector); typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba; + +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ;Quoted 7 lines
hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 1 line
hw/pci/msi.c +1
-0+ dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the callback function should only override the behavior ofmsi_send_message(), not the entiremsi_notify()function.The same applies to MSI-X.
Hi Stefan,
We noticed that the client is handling the masking and unmasking of MSIx interrupts.
Concerning MSIx,
vfio_msix_vector_use()handles unmasking andvfio_msix_vector_release()handles masking operations. The server triggers an MSIx interrupt by signaling the eventfd associated with the vector. If the vector is unmasked, the interrupt bypasses the client/QEMU and takes this
path: “server -> KVM -> guest”. Whereas, if the vector is masked, it lands on the client via: “server -> vfio_msi_interrupt()”.vfio_msi_interrupt()suppresses the interrupt if the vector is masked. The use and release functions switch the server’s eventfd between VFIOPCIDevice->VFIOMSIVector[i]->kvm_interrupt and VFIOPCIDevice->VFIOMSIVector[i]->interrupt using theVFIO_DEVICE_SET_IRQSmessage.Concerning MSI, the server should check if the vector is unmasked before triggering. The server is not doing it presently, will update it. For some reason, I had assumed that MSI handling is similar to MSIx in terms of masking - sorry about that. The masking and unmasking information for MSI is in the config space registers, so the server should have this information.
Quoted 6 lines
You had previously suggested using callbacks for
msi_get_message&msi_send_message, considering the masking issue. Given MSIx masking (including the MSIx table BAR) is handled at the client, the masking information doesn’t reach the server - somsix_notifywill never invoke themsi_send_messagecallback - all the vectors remain masked at the server end (msix_init()->msix_mask_all()).I was expecting vfio-user devices to be involved in MSI-X masking so they can implement the Pending bit semantics described in the spec:
Quoted 7 lines
If a masked vector has its Pending bit Set, and the associated underlying interrupt events are somehow satisfied (usually by software though the exact manner is Function-specific), the Function must Clear the Pending bit, to avoid sending a spurious interrupt message later when software unmasks the vector.
Does QEMU VFIO support this?
QEMU VFIO doesn’t seem to support it - I couldn’t find a place where an assigned/passthru PCI device clears the pending bits in QEMU.
Quoted 8 lines
What happens when a
hw/net/e1000e_core.cvfio-user device usesmsix_clr_pending()and related APIs?Also, having the vfio-user daemon write to the eventfd while the vector is masked is a waste of CPU cycles. The PCIe spec describes using MSI-X masking for poll mode operation and going via eventfd is suboptimal:
Quoted 7 lines
Software is permitted to mask one or more vectors indefinitely, and service their associated interrupt events strictly based on polling their Pending bits. A Function must Set and Clear its Pending bits as necessary to support this “pure polling” mode of operation.
Maybe the answer is these issues don't matter in practice because MSI-X masking is not used much?
From what I can tell, “pure polling” is used by ivshmem and virtio-pci devices in QEMU.
e1000e doesn’t use “pure polling”, but it does clear pending interrupts.
John Johnson, John Levon & Thanos,
Any thoughts?
Thank you!
--
Jag
Quoted 2 lines
Stefan
RE: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 136 lines
-----Original Message-----
From: Jag Raman <[email protected]>
Sent: 29 March 2022 20:07
To: Stefan Hajnoczi <[email protected]>
Cc: Alex Williamson <[email protected]>; qemu-devel <qemu- [email protected]>; Michael S. Tsirkin <[email protected]>; Philippe Mathieu- Daudé <[email protected]>; Paolo Bonzini <[email protected]>; Beraldo Leal <[email protected]>; Daniel P. Berrangé <[email protected]>; [email protected]; Marcel Apfelbaum <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Juan Quintela <[email protected]>; [email protected]; John Levon <[email protected]>; Thanos Makatos <[email protected]>; Elena Ufimtseva <[email protected]>; John Johnson <[email protected]>; Kanth Ghatraju <[email protected]>
Subject: Re: [PATCH v6 15/19] vfio-user: handle device interruptsQuoted 1 line
On Mar 29, 2022, at 10:24 AM, Stefan Hajnoczi <[email protected]>
Quoted 5 lines · wrote:
Quoted 3 lines · Jag Raman
Quoted 3 lines
On Mar 7, 2022, at 5:24 AM, Stefan Hajnoczi <[email protected]>
Quoted 22 lines · wrote:
Quoted 22 lines
Quoted 22 lines · Jagannathan Raman
Quoted 20 lines
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/hw/pci/pci.h typedef uint32_t PCIConfigReadFunc(PCIDevice
Quoted 5 lines
include/hw/pci/pci.h +0
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice
*pci_dev,Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/hw/pci/pci.h +2
-0typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector);Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/hw/pci/pci.h +0
-0typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba;Quoted 8 lines
Quoted 8 lines
include/hw/remote/vfio-
Quoted 8 lines
include/hw/pci/pci.h +4
-0+ +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio- renamed from include/hw/remote/vfio-user-obj.h +0
-0
user-obj.hQuoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 25 lines
new file mode 100644
index 0000000000..87ab78b875include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 4 lines
hw/pci/msi.c +1
-0@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ; + dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the callback function should only override the behavior ofmsi_send_message(), not the entiremsi_notify()function.The same applies to MSI-X.
Quoted 8 lines
Quoted 8 lines
Hi Stefan,
We noticed that the client is handling the masking and unmasking of MSIx interrupts.
Concerning MSIx,
vfio_msix_vector_use()handles unmasking andvfio_msix_vector_release()handles masking operations. The server triggers an MSIx interrupt by signaling the eventfd associated with the vector. If thevector
Quoted 2 lines
Quoted 2 lines
is unmasked, the interrupt bypasses the client/QEMU and takes this
path: “server -> KVM -> guest”. Whereas, if the vector is masked, it lands onthe
Quoted 1 line
Quoted 1 line
client via: “server -> vfio_msi_interrupt()”.
vfio_msi_interrupt()suppressesthe
Quoted 1 line
Quoted 1 line
interrupt if the vector is masked. The use and release functions switch the
server’s
Quoted 6 lines
Quoted 6 lines
eventfd between VFIOPCIDevice->VFIOMSIVector[i]->kvm_interrupt and VFIOPCIDevice->VFIOMSIVector[i]->interrupt using the
VFIO_DEVICE_SET_IRQSmessage.Concerning MSI, the server should check if the vector is unmasked before triggering. The server is not doing it presently, will update it. For some
reason,
Quoted 2 lines
Quoted 2 lines
I had assumed that MSI handling is similar to MSIx in terms of masking - sorry about that. The masking and unmasking information for MSI is in the config
Quoted 12 lines
space
Quoted 5 lines
Quoted 5 lines
registers, so the server should have this information.
You had previously suggested using callbacks for
msi_get_message&msi_send_message, considering the masking issue. Given MSIx masking (including the MSIx table BAR) is handled at the client, the maskinginformation
Quoted 5 lines
Quoted 3 lines
doesn’t reach the server - so
msix_notifywill never invoke themsi_send_messagecallback - all the vectors remain masked at the server end (msix_init()->msix_mask_all()).I was expecting vfio-user devices to be involved in MSI-X masking so
libvfio-user can't be involved in the first place since QEMU emulates MSI/X:
https://lore.kernel.org/all/[email protected]/T/
Quoted 29 lines
Quoted 9 lines
they can implement the Pending bit semantics described in the spec:
If a masked vector has its Pending bit Set, and the associated underlying interrupt events are somehow satisfied (usually by software though the exact manner is Function-specific), the Function must Clear the Pending bit, to avoid sending a spurious interrupt message later when software unmasks the vector.
Does QEMU VFIO support this?
QEMU VFIO doesn’t seem to support it - I couldn’t find a place where an assigned/passthru PCI device clears the pending bits in QEMU.
Quoted 8 lines
What happens when a
hw/net/e1000e_core.cvfio-user device usesmsix_clr_pending()and related APIs?Also, having the vfio-user daemon write to the eventfd while the vector is masked is a waste of CPU cycles. The PCIe spec describes using MSI-X masking for poll mode operation and going via eventfd is suboptimal:
Quoted 7 lines
Software is permitted to mask one or more vectors indefinitely, and service their associated interrupt events strictly based on polling their Pending bits. A Function must Set and Clear its Pending bits as necessary to support this “pure polling” mode of operation.
Maybe the answer is these issues don't matter in practice because MSI-X masking is not used much?
Quoted 8 lines
From what I can tell, “pure polling” is used by ivshmem and virtio-pci devices in QEMU.
e1000e doesn’t use “pure polling”, but it does clear pending interrupts.
John Johnson, John Levon & Thanos,
Any thoughts?
If QEMU stops emulating MSI/X then we libvfio-user would have to do it.
Re: [PATCH v6 15/19] vfio-user: handle device interrupts
Quoted 184 lines · Thanos Makatos
Quoted 136 lines
-----Original Message-----
From: Jag Raman <[email protected]>
Sent: 29 March 2022 20:07
To: Stefan Hajnoczi <[email protected]>
Cc: Alex Williamson <[email protected]>; qemu-devel <qemu- [email protected]>; Michael S. Tsirkin <[email protected]>; Philippe Mathieu- Daudé <[email protected]>; Paolo Bonzini <[email protected]>; Beraldo Leal <[email protected]>; Daniel P. Berrangé <[email protected]>; [email protected]; Marcel Apfelbaum <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Juan Quintela <[email protected]>; [email protected]; John Levon <[email protected]>; Thanos Makatos <[email protected]>; Elena Ufimtseva <[email protected]>; John Johnson <[email protected]>; Kanth Ghatraju <[email protected]>
Subject: Re: [PATCH v6 15/19] vfio-user: handle device interruptsQuoted 1 line
On Mar 29, 2022, at 10:24 AM, Stefan Hajnoczi <[email protected]>
Quoted 5 lines · wrote:
Quoted 3 lines · Jag Raman
Quoted 3 lines
On Mar 7, 2022, at 5:24 AM, Stefan Hajnoczi <[email protected]>
Quoted 22 lines · wrote:
Quoted 22 lines
Quoted 22 lines · Jagannathan Raman
Quoted 20 lines
Forward remote device's interrupts to the guest
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/hw/pci/pci.h | 6 ++ include/hw/remote/vfio-user-obj.h | 6 ++ hw/pci/msi.c | 13 +++- hw/pci/msix.c | 12 +++- hw/remote/machine.c | 11 +-- hw/remote/vfio-user-obj.c | 107 ++++++++++++++++++++++++++++++ stubs/vfio-user-obj.c | 6 ++ MAINTAINERS | 1 + hw/remote/trace-events | 1 + stubs/meson.build | 1 + 10 files changed, 158 insertions(+), 6 deletions(-)create mode 100644
include/hw/remote/vfio-user-obj.hcreate mode 100644stubs/vfio-user-obj.cQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/hw/pci/pci.h typedef uint32_t PCIConfigReadFunc(PCIDevice
Quoted 5 lines
include/hw/pci/pci.h +0
-0@@ -129,6 +129,8 @@ typedef uint32_t PCIConfigReadFunc(PCIDevice
*pci_dev,Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/hw/pci/pci.h +2
-0typedef void PCIMapIORegionFunc (PCIDevice *pci_dev,int region_num,pcibus_t addr,pcibus_t size,int type);typedef void PCIUnregisterFunc (PCIDevice *pci_dev); +typedef void PCIMSINotify (PCIDevice *pci_dev,unsigned vector); +typedef void PCIMSIxNotify (PCIDevice *pci_dev,unsigned vector);Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/hw/pci/pci.h +0
-0typedef struct PCIIORegion { pcibus_t addr; /* current PCI mapping address. -1 means not mapped */@@ -323,6 +325,10 @@ struct PCIDevice {/* Space to store MSIX table & pending bit array */ uint8_t *msix_table;uint8_t *msix_pba;Quoted 8 lines
Quoted 8 lines
include/hw/remote/vfio-
Quoted 8 lines
include/hw/pci/pci.h +4
-0+ +PCIMSINotify *msi_notify; +PCIMSIxNotify *msix_notify; +/* MemoryRegion container for msix exclusive BAR setup */ MemoryRegion msix_exclusive_bar;/* Memory Regions for MSIX table and pending bit entries. */ include/hw/remote/vfio- renamed from include/hw/remote/vfio-user-obj.h +0
-0
user-obj.hQuoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 25 lines
new file mode 100644
index 0000000000..87ab78b875include/hw/remote/vfio-user-obj.h new file +6
-0@@ -0,0 +1,6 @@ +#ifndef VFIO_USER_OBJ_H +#define VFIO_USER_OBJ_H + +void vfu_object_set_bus_irq (PCIBus *pci_bus); + +#endif hw/pci/msi.c +2
-0@@ -51,6 +51,8 @@*/ bool msi_nonbroken; +static void pci_msi_notify (PCIDevice *dev,unsigned int vector); +/* If we get rid of cap allocator, we won't need this. */ static inline uint8_t msi_cap_sizeof (uint16_t flags) {hw/pci/msi.c int msi_init(struct PCIDevice *dev, uint8_t offset, · line 230
Quoted 4 lines
hw/pci/msi.c +1
-0@@ -225,6 +227,8 @@ int msi_init(struct PCIDevice *dev, uint8_t offset, dev->msi_cap = config_offset; dev->cap_present |=QEMU_PCI_CAP_MSI ; + dev->msi_notify = pci_msi_notify;Are you sure it's correct to skip the
msi_is_masked()logic? I think the callback function should only override the behavior ofmsi_send_message(), not the entiremsi_notify()function.The same applies to MSI-X.
Quoted 8 lines
Quoted 8 lines
Hi Stefan,
We noticed that the client is handling the masking and unmasking of MSIx interrupts.
Concerning MSIx,
vfio_msix_vector_use()handles unmasking andvfio_msix_vector_release()handles masking operations. The server triggers an MSIx interrupt by signaling the eventfd associated with the vector. If thevector
Quoted 2 lines
Quoted 2 lines
is unmasked, the interrupt bypasses the client/QEMU and takes this
path: “server -> KVM -> guest”. Whereas, if the vector is masked, it lands onthe
Quoted 1 line
Quoted 1 line
client via: “server -> vfio_msi_interrupt()”.
vfio_msi_interrupt()suppressesthe
Quoted 1 line
Quoted 1 line
interrupt if the vector is masked. The use and release functions switch the
server’s
Quoted 6 lines
Quoted 6 lines
eventfd between VFIOPCIDevice->VFIOMSIVector[i]->kvm_interrupt and VFIOPCIDevice->VFIOMSIVector[i]->interrupt using the
VFIO_DEVICE_SET_IRQSmessage.Concerning MSI, the server should check if the vector is unmasked before triggering. The server is not doing it presently, will update it. For some
reason,
Quoted 2 lines
Quoted 2 lines
I had assumed that MSI handling is similar to MSIx in terms of masking - sorry about that. The masking and unmasking information for MSI is in the config
Quoted 12 lines
space
Quoted 5 lines
Quoted 5 lines
registers, so the server should have this information.
You had previously suggested using callbacks for
msi_get_message&msi_send_message, considering the masking issue. Given MSIx masking (including the MSIx table BAR) is handled at the client, the maskinginformation
Quoted 5 lines
Quoted 3 lines
doesn’t reach the server - so
msix_notifywill never invoke themsi_send_messagecallback - all the vectors remain masked at the server end (msix_init()->msix_mask_all()).I was expecting vfio-user devices to be involved in MSI-X masking so
libvfio-user can't be involved in the first place since QEMU emulates MSI/X:
https://lore.kernel.org/all/[email protected]/T/Quoted 32 lines
Quoted 9 lines
they can implement the Pending bit semantics described in the spec:
If a masked vector has its Pending bit Set, and the associated underlying interrupt events are somehow satisfied (usually by software though the exact manner is Function-specific), the Function must Clear the Pending bit, to avoid sending a spurious interrupt message later when software unmasks the vector.
Does QEMU VFIO support this?
QEMU VFIO doesn’t seem to support it - I couldn’t find a place where an assigned/passthru PCI device clears the pending bits in QEMU.
Quoted 8 lines
What happens when a
hw/net/e1000e_core.cvfio-user device usesmsix_clr_pending()and related APIs?Also, having the vfio-user daemon write to the eventfd while the vector is masked is a waste of CPU cycles. The PCIe spec describes using MSI-X masking for poll mode operation and going via eventfd is suboptimal:
Quoted 7 lines
Software is permitted to mask one or more vectors indefinitely, and service their associated interrupt events strictly based on polling their Pending bits. A Function must Set and Clear its Pending bits as necessary to support this “pure polling” mode of operation.
Maybe the answer is these issues don't matter in practice because MSI-X masking is not used much?
From what I can tell, “pure polling” is used by ivshmem and virtio-pci devices in QEMU.
Quoted 7 lines
Quoted 5 lines
e1000e doesn’t use “pure polling”, but it does clear pending interrupts.
John Johnson, John Levon & Thanos,
Any thoughts?
If QEMU stops emulating MSI/X then we libvfio-user would have to do it.
If Alex is happy with VFIO not supporting the exact Pending bits semantics in the PCIe spec, then vfio-user probably doesn't need to either.
I just wanted to raise this design issue and maybe we can document somewhere that this is intentional.
Stefan
PATCH v6 16/19 softmmu/vl: defer backend init
In reply to Jagannathan RamanAllow deferred initialization of backends. TYPE_REMOTE_MACHINE is agnostic to QEMU's RUN_STATE. It's state is driven by the QEMU client via the vfio-user protocol. Whereas, the backends presently defer initialization if QEMU is in RUN_STATE_INMIGRATE. Since the remote machine can't use RUN_STATE*, this commit allows it to ask for deferred initialization of backend device. It is primarily targeted towards block devices in this commit, but it needed not be limited to that.
--- include/sysemu/sysemu.h | 4 ++++ block/block-backend.c | 3 ++- blockdev.c | 2 +- softmmu/vl.c | 17 +++++++++++++++++ stubs/defer-backend-init.c | 7 +++++++ MAINTAINERS | 1 + stubs/meson.build | 1 + 7 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 stubs/defer-backend-init.c
include/sysemu/sysemu.h +4 -0
@@ -119,4 +119,8 @@ extern QemuOptsList qemu_net_opts;extern QemuOptsList qemu_global_opts;extern QemuOptsList qemu_semihosting_config_opts; +bool deferred_backend_init (void ); +void set_deferred_backend_init (void ); +void clear_deferred_backend_init (void ); +#endif
block/block-backend.c +2 -1
@@ -20,6 +20,7 @@#include "sysemu/blockdev.h" #include "sysemu/runstate.h" #include "sysemu/replay.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h"
@@ -935,7 +936,7 @@ int blk_attach_dev(BlockBackend *blk, DeviceState *dev)/* While migration is still incoming, we don't need to apply the * permissions of guest device BlockBackends. We might still have a block * job or NBD server writing to the image for storage migration. */ -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { blk->disable_perm = true; }
blockdev.c +1 -1
@@ -569,7 +569,7 @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts,qdict_set_default_str (bs_opts,BDRV_OPT_AUTO_READ_ONLY ,"on" );assert ((bdrv_flags &BDRV_O_CACHE_MASK ) ==0 ); -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { bdrv_flags |=BDRV_O_INACTIVE ; }
softmmu/vl.c +17 -0
@@ -496,6 +496,23 @@ static QemuOptsList qemu_action_opts = { }, }; +bool defer_backend_init; + +bool deferred_backend_init (void ) +{ +return defer_backend_init; +} + +void set_deferred_backend_init (void ) +{ + defer_backend_init = true; +} + +void clear_deferred_backend_init (void ) +{ + defer_backend_init = false; +} +const char *qemu_get_vm_name (void ) {return qemu_name;
stubs/defer-backend-init.c new file +7 -0
@@ -0,0 +1,7 @@ +#include "qemu/osdep.h" +#include "sysemu/sysemu.h" + +bool deferred_backend_init (void ) +{ +return false; +}
MAINTAINERS +1 -0
@@ -3572,6 +3572,7 @@ F: hw/remote/vfio-user-obj.c F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c F: include/hw/remote/iommu.h +F: stubs/defer-backend-init.c EBPF: M: Jason Wang <[email protected]>
stubs/meson.build +1 -0
@@ -58,3 +58,4 @@ else stub_ss.add(files('qdev.c')) endif stub_ss.add(when: 'CONFIG_VFIO_USER_SERVER', if_false: files('vfio-user-obj.c')) +stub_ss.add(files('defer-backend-init.c'))
Signature · 1 line
2.20.1
Re: [PATCH v6 16/19] softmmu/vl: defer backend init
Quoted 2 lines · Jagannathan Raman
Allow deferred initialization of backends.
TYPE_REMOTE_MACHINEis agnostic to QEMU'sRUN_STATE. It's state is driven by the QEMU client
s/It's/Its/
Quoted 5 lines
via the vfio-user protocol. Whereas, the backends presently defer initialization if QEMU is in
RUN_STATE_INMIGRATE. Since the remote machine can't use RUN_STATE*, this commit allows it to ask for deferred initialization of backend device. It is primarily targeted towards block devices in this commit, but it needed not be limited to that.
What is the purpose of this commit? I don't understand the description.
Quoted 41 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/sysemu/sysemu.h | 4 ++++ block/block-backend.c | 3 ++- blockdev.c | 2 +- softmmu/vl.c | 17 +++++++++++++++++ stubs/defer-backend-init.c | 7 +++++++ MAINTAINERS | 1 + stubs/meson.build | 1 + 7 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 stubs/defer-backend-init.cinclude/sysemu/sysemu.h +4
-0@@ -119,4 +119,8 @@ extern QemuOptsList qemu_net_opts;extern QemuOptsList qemu_global_opts;extern QemuOptsList qemu_semihosting_config_opts; +bool deferred_backend_init (void ); +void set_deferred_backend_init (void ); +void clear_deferred_backend_init (void ); +#endif block/block-backend.c +1
-0@@ -20,6 +20,7 @@#include "sysemu/blockdev.h" #include "sysemu/runstate.h" #include "sysemu/replay.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h"
block/block-backend.c int blk_attach_dev(BlockBackend *blk, DeviceState *dev) · line 941
Quoted 7 lines
block/block-backend.c +1
-1@@ -935,7 +936,7 @@ int blk_attach_dev(BlockBackend *blk, DeviceState *dev)/* While migration is still incoming, we don't need to apply the * permissions of guest device BlockBackends. We might still have a block * job or NBD server writing to the image for storage migration. */ -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { blk->disable_perm = true; }
Why is this necessary for vfio-user? Disk images shouldn't be in use by another process so we don't need to bypass permissions temporarily.
blockdev.c static BlockBackend *blockdev_init(const char *file, QDict *bs_opts, · line 573
Quoted 12 lines
blockdev.c +1
-1@@ -569,7 +569,7 @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts,qdict_set_default_str (bs_opts,BDRV_OPT_AUTO_READ_ONLY ,"on" );assert ((bdrv_flags &BDRV_O_CACHE_MASK ) ==0 ); -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { bdrv_flags |=BDRV_O_INACTIVE ;
Same here.
Re: [PATCH v6 16/19] softmmu/vl: defer backend init
Quoted 10 lines · Stefan Hajnoczi
Quoted 2 lines · Jagannathan Raman
Allow deferred initialization of backends.
TYPE_REMOTE_MACHINEis agnostic to QEMU'sRUN_STATE. It's state is driven by the QEMU clients/It's/Its/
Quoted 7 lines
Quoted 5 lines
via the vfio-user protocol. Whereas, the backends presently defer initialization if QEMU is in
RUN_STATE_INMIGRATE. Since the remote machine can't use RUN_STATE*, this commit allows it to ask for deferred initialization of backend device. It is primarily targeted towards block devices in this commit, but it needed not be limited to that.What is the purpose of this commit? I don't understand the description.
Sorry it’s not clear. This patch is needed to support vfio-user migration.
Just for background, this patch along with the next one helps to migrate individual devices from the source to the destination. For example, in a storage server daemon with 5 PCI controllers, we could migrate just 2 of the 5 controllers to the destination while the remaining 3 continue to run on the source. The destination could also be a server that is already running, it doesn’t have to be frozen for migration.
This patch specifically affects how block drives are initialized in the destination. In all the presently defined use cases, QEMU launches the destination in RUN_STATE_INMIGRATE. This is essentially a frozen state, which implicitly defers the initialization of the backends such as block drives until after the migration is complete. Whereas in vfio-user, the destination cannot be in RUN_STATE_INMIGRATE as it could already be running. Therefore, we need a way to tell backend devices to defer their initialization. This patch addresses the need to defer backend initialization for already running QEMU instances.
Quoted 44 lines
Quoted 44 lines
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
include/sysemu/sysemu.h | 4 ++++ block/block-backend.c | 3 ++- blockdev.c | 2 +- softmmu/vl.c | 17 +++++++++++++++++ stubs/defer-backend-init.c | 7 +++++++ MAINTAINERS | 1 + stubs/meson.build | 1 + 7 files changed, 33 insertions(+), 2 deletions(-)create mode 100644
stubs/defer-backend-init.cinclude/sysemu/sysemu.h +4
-0@@ -119,4 +119,8 @@ extern QemuOptsList qemu_net_opts;extern QemuOptsList qemu_global_opts;extern QemuOptsList qemu_semihosting_config_opts; +bool deferred_backend_init (void ); +void set_deferred_backend_init (void ); +void clear_deferred_backend_init (void ); +#endif block/block-backend.c +1
-0@@ -20,6 +20,7 @@#include "sysemu/blockdev.h" #include "sysemu/runstate.h" #include "sysemu/replay.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qapi/qapi-events-block.h" #include "qemu/id.h" @@ -935,7 +936,7 @@ int blk_attach_dev(BlockBackend *blk, DeviceState *dev)/* While migration is still incoming, we don't need to apply the * permissions of guest device BlockBackends. We might still have a block
Quoted 8 lines
block/block-backend.c int blk_attach_dev(BlockBackend *blk, DeviceState *dev) · line 941
Quoted 5 lines
block/block-backend.c +1
-1* job or NBD server writing to the image for storage migration. */ -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { blk->disable_perm = true; }Why is this necessary for vfio-user? Disk images shouldn't be in use by another process so we don't need to bypass permissions temporarily.
The destination in vfio-user migration needs this - the source would already be using the disk images.
Thank you!
--
Jag
Quoted 15 lines
blockdev.c static BlockBackend *blockdev_init(const char *file, QDict *bs_opts, · line 573
Quoted 13 lines
blockdev.c +1
-1@@ -569,7 +569,7 @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts,qdict_set_default_str (bs_opts,BDRV_OPT_AUTO_READ_ONLY ,"on" );assert ((bdrv_flags &BDRV_O_CACHE_MASK ) ==0 ); -if (runstate_check (RUN_STATE_INMIGRATE )) { +if (runstate_check (RUN_STATE_INMIGRATE ) ||deferred_backend_init ()) { bdrv_flags |=BDRV_O_INACTIVE ;Same here.
PATCH v6 17/19 vfio-user: register handlers to facilitate migration
In reply to Jagannathan RamanStore and load the device's state during migration. use libvfio-user's handlers for this purpose
--- include/block/block.h | 1 + include/migration/vmstate.h | 2 + migration/savevm.h | 2 + block.c | 5 + hw/remote/machine.c | 7 + hw/remote/vfio-user-obj.c | 467 ++++++++++++++++++++++++++++++++++++ migration/savevm.c | 89 +++++++ migration/vmstate.c | 19 ++ 8 files changed, 592 insertions(+)
include/block/block.h +1 -0
@@ -495,6 +495,7 @@ int generated_co_wrapper bdrv_invalidate_cache(BlockDriverState *bs, Error **errp);void bdrv_invalidate_cache_all (Error **errp);int bdrv_inactivate_all (void ); +int bdrv_inactivate (BlockDriverState *bs);/* Ensure contents are flushed to disk. */ int generated_co_wrapper bdrv_flush(BlockDriverState *bs);
include/migration/vmstate.h +2 -0
@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;#define VMSTATE_END_OF_LIST () \ {} +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev); +int vmstate_load_state (QEMUFile *f,const VMStateDescription *vmsd,void *opaque,int version_id); int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
migration/savevm.h +2 -0
@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif
block.c +5 -0
@@ -6565,6 +6565,11 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs)return 0 ; } +int bdrv_inactivate (BlockDriverState *bs) +{ +return bdrv_inactivate_recurse (bs); +} +int bdrv_inactivate_all (void ) {BlockDriverState *bs =NULL ;
hw/remote/machine.c +7 -0
@@ -24,6 +24,7 @@#include "hw/qdev-core.h" #include "hw/remote/iommu.h" #include "hw/remote/vfio-user-obj.h" +#include "sysemu/sysemu.h" static void remote_machine_init (MachineState *machine) {
@@ -86,6 +87,11 @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp) s->vfio_user = value; } +static void remote_machine_instance_init (Object *obj) +{ +set_deferred_backend_init (); +} +static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);
@@ -105,6 +111,7 @@ static const TypeInfo remote_machine = { .name = TYPE_REMOTE_MACHINE, .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), + .instance_init = remote_machine_instance_init, .class_init = remote_machine_class_init, .interfaces = (InterfaceInfo []) { {TYPE_HOTPLUG_HANDLER },
hw/remote/vfio-user-obj.c +467 -0
515 lines not on this page · raw message
migration/savevm.c +89 -0
107 lines not on this page · raw message
migration/vmstate.c +19 -0
@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)return size; } +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } + +return size; +} +static void vmstate_handle_alloc (void *ptr,const VMStateField *field,void *opaque) {
Signature · 1 line
2.20.1
Re: [PATCH v6 17/19] vfio-user: register handlers to facilitate migration
Quoted 6 lines · Jagannathan Raman
Store and load the device's state during migration. use libvfio-user's handlers for this purpose
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>
Why does no one call clear_deferred_backend_init?
Paolo
Quoted 766 lines
--- include/block/block.h | 1 + include/migration/vmstate.h | 2 + migration/savevm.h | 2 + block.c | 5 + hw/remote/machine.c | 7 + hw/remote/vfio-user-obj.c | 467 ++++++++++++++++++++++++++++++++++++ migration/savevm.c | 89 +++++++ migration/vmstate.c | 19 ++ 8 files changed, 592 insertions(+)include/block/block.h +1
-0@@ -495,6 +495,7 @@ int generated_co_wrapper bdrv_invalidate_cache(BlockDriverState *bs, Error **errp);void bdrv_invalidate_cache_all (Error **errp);int bdrv_inactivate_all (void ); +int bdrv_inactivate (BlockDriverState *bs);/* Ensure contents are flushed to disk. */ int generated_co_wrapper bdrv_flush(BlockDriverState *bs);include/migration/vmstate.h +2
-0@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;#define VMSTATE_END_OF_LIST () \ {} +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev); +int vmstate_load_state (QEMUFile *f,const VMStateDescription *vmsd,void *opaque,int version_id); int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,migration/savevm.h +2
-0@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif block.c +5
-0@@ -6565,6 +6565,11 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs)return 0 ; } +int bdrv_inactivate (BlockDriverState *bs) +{ +return bdrv_inactivate_recurse (bs); +} +int bdrv_inactivate_all (void ) {BlockDriverState *bs =NULL ;hw/remote/machine.c +7
-0@@ -24,6 +24,7 @@#include "hw/qdev-core.h" #include "hw/remote/iommu.h" #include "hw/remote/vfio-user-obj.h" +#include "sysemu/sysemu.h" static void remote_machine_init (MachineState *machine) {@@ -86,6 +87,11 @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp) s->vfio_user = value; } +static void remote_machine_instance_init (Object *obj) +{ +set_deferred_backend_init (); +} +static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -105,6 +111,7 @@ static const TypeInfo remote_machine = { .name = TYPE_REMOTE_MACHINE, .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), + .instance_init = remote_machine_instance_init, .class_init = remote_machine_class_init, .interfaces = (InterfaceInfo []) { {TYPE_HOTPLUG_HANDLER },hw/remote/vfio-user-obj.c +467
-0@@ -57,6 +57,13 @@#include "hw/pci/msi.h" #include "hw/pci/msix.h" #include "hw/remote/vfio-user-obj.h" +#include "migration/qemu-file.h" +#include "migration/savevm.h" +#include "migration/vmstate.h" +#include "migration/global_state.h" +#include "block/block.h" +#include "sysemu/block-backend.h" +#include "net/net.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -108,12 +115,49 @@ struct VfuObject {Error *unplug_blocker;int vfu_poll_fd; + +/* +* vfu_mig_buf holds the migration data. In the remote server, this +* buffer replaces the role of an IO channel which links the source +* and the destination. +* +* Whenever the client QEMU process initiates migration, the remote +* server gets notified via libvfio-user callbacks. The remote server +* sets up a QEMUFile object using this buffer as backend. The remote +* server passes this object to its migration subsystem, which slurps +* the VMSD of the device ('devid' above) referenced by this object +* and stores the VMSD in this buffer. +* +* The client subsequetly asks the remote server for any data that +* needs to be moved over to the destination via libvfio-user +* library's vfu_migration_callbacks_t callbacks. The remote hands +* over this buffer as data at this time. +* +* A reverse of this process happens at the destination. +*/ +uint8_t *vfu_mig_buf; + +uint64_t vfu_mig_buf_size; + +uint64_t vfu_mig_buf_pending; + +uint64_t vfu_mig_data_written; + +uint64_t vfu_mig_section_offset; + +QEMUFile *vfu_mig_file; + +vfu_migr_state_t vfu_state; };static GHashTable *vfu_object_bdf_to_ctx_table;#define INT2VOIDP (i) (void *)(uintptr_t)(i) +#define KB (x) ((size_t) (x) << 10) + +#define VFU_OBJECT_MIG_WINDOW KB(64) +static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,@@ -163,6 +207,394 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)vfu_object_init_ctx (o, errp); } +/** +* Migration helper functions +* +* vfu_mig_buf_read & vfu_mig_buf_write are used by QEMU's migration +* subsystem - qemu_remote_loadvm & qemu_remote_savevm. loadvm/savevm +* call these functions via QEMUFileOps to load/save the VMSD of a +* device into vfu_mig_buf +* +*/ +static ssize_t vfu_mig_buf_read (void *opaque,uint8_t *buf,int64_t pos, +size_t size,Error **errp) +{ +VfuObject *o = opaque; + +if (pos > o->vfu_mig_buf_size) { + size =0 ; + }else if ((pos + size) > o->vfu_mig_buf_size) { + size = o->vfu_mig_buf_size - pos; + } + +memcpy (buf, (o->vfu_mig_buf + pos), size); + +return size; +} + +static ssize_t vfu_mig_buf_write (void *opaque,struct iovec *iov,int iovcnt, +int64_t pos,Error **errp) +{ +ERRP_GUARD (); +VfuObject *o = opaque; +uint64_t end = pos +iov_size (iov, iovcnt); +int i; + +if (o->vfu_mig_buf_pending) { +error_setg (errp,"Migration is ongoing" ); +return 0 ; + } + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + } + +for (i =0 ; i < iovcnt; i++) { +memcpy ((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base, + iov[i].iov_len); + o->vfu_mig_buf_size += iov[i].iov_len; + } + +return iov_size (iov, iovcnt); +} + +static int vfu_mig_buf_shutdown (void *opaque,bool rd,bool wr,Error **errp) +{ +VfuObject *o = opaque; + + o->vfu_mig_buf_size =0 ; + +g_free (o->vfu_mig_buf); + + o->vfu_mig_buf =NULL ; + + o->vfu_mig_buf_pending =0 ; + + o->vfu_mig_data_written =0 ; + + o->vfu_mig_section_offset =0 ; + +return 0 ; +} + +static const QEMUFileOps vfu_mig_fops_save = { + .writev_buffer = vfu_mig_buf_write, + .shut_down = vfu_mig_buf_shutdown, +}; + +static const QEMUFileOps vfu_mig_fops_load = { + .get_buffer = vfu_mig_buf_read, + .shut_down = vfu_mig_buf_shutdown, +}; + +static BlockDriverState *vfu_object_find_bs_by_dev (DeviceState *dev) +{ +BlockBackend *blk =blk_by_dev (dev); + +if (!blk) { +return NULL ; + } + +return blk_bs (blk); +} + +static int vfu_object_bdrv_invalidate_cache_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; +Error *local_err =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +bdrv_invalidate_cache (bs, &local_err); +if (local_err) { +error_report_err (local_err); +return -1 ; + } + +return 0 ; +} + +static int vfu_object_bdrv_inactivate_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +return bdrv_inactivate (bs); +} + +static void vfu_object_start_stop_netdev (DeviceState *dev,bool start) +{ +NetClientState *nc =NULL ; +Error *local_err =NULL ; +char *netdev =NULL ; + + netdev =object_property_get_str (OBJECT (dev),"netdev" , &local_err); +if (local_err) { +/** +* object_property_get_str() sets Error if netdev property is +* not found, not necessarily an error in the context of +* this function +*/ +error_free (local_err); +return ; + } + +if (!netdev) { +return ; + } + + nc =qemu_find_netdev (netdev); + +if (!nc) { +return ; + } + +if (!start) { +qemu_flush_or_purge_queued_packets (nc, true); + +if (nc->info && nc->info->cleanup) { + nc->info->cleanup (nc); + } + }else if (nc->peer) { +qemu_flush_or_purge_queued_packets (nc->peer, false); + } +} + +static int vfu_object_start_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_invalidate_cache_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, true); + +return ret; +} + +static int vfu_object_stop_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_inactivate_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, false); + +return ret; +} + +/** +* handlers for vfu_migration_callbacks_t +* +* The libvfio-user library accesses these handlers to drive the migration +* at the remote end, and also to transport the data stored in vfu_mig_buf +* +*/ +static void vfu_mig_state_stop_and_copy (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_save, false); + } + + ret =qemu_remote_savevm (o->vfu_mig_file,DEVICE (o->pci_dev)); +if (ret) { +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; +return ; + } + +qemu_fflush (o->vfu_mig_file); +} + +static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (o->vfu_state !=VFU_MIGR_STATE_RESUME ) { + goto run_ctx; + } + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } + +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +run_ctx: + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_start_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to setup backends for %s" , + o->device); +return ; + } +} + +static void vfu_mig_state_stop (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_stop_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to inactivate backends for %s" , + o->device); + } +} + +static int vfu_mig_transition (vfu_ctx_t *vfu_ctx,vfu_migr_state_t state) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (o->vfu_state == state) { +return 0 ; + } + +switch (state) { +case VFU_MIGR_STATE_RESUME : +break ; +case VFU_MIGR_STATE_STOP_AND_COPY : +vfu_mig_state_stop_and_copy (vfu_ctx); +break ; +case VFU_MIGR_STATE_STOP : +vfu_mig_state_stop (vfu_ctx); +break ; +case VFU_MIGR_STATE_PRE_COPY : +break ; +case VFU_MIGR_STATE_RUNNING : +vfu_mig_state_running (vfu_ctx); +break ; +default : +warn_report ("vfu: Unknown migration state %d" , state); + } + + o->vfu_state = state; + +return 0 ; +} + +static uint64_t vfu_mig_get_pending_bytes (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +static bool mig_ongoing; + +if (!mig_ongoing && !o->vfu_mig_buf_pending) { + o->vfu_mig_buf_pending = o->vfu_mig_buf_size; + mig_ongoing = true; + } + +if (mig_ongoing && !o->vfu_mig_buf_pending) { + mig_ongoing = false; + } + +return o->vfu_mig_buf_pending; +} + +static int vfu_mig_prepare_data (vfu_ctx_t *vfu_ctx,uint64_t *offset, +uint64_t *size) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t data_size = o->vfu_mig_buf_pending; + +if (data_size >VFU_OBJECT_MIG_WINDOW ) { + data_size =VFU_OBJECT_MIG_WINDOW ; + } + + o->vfu_mig_section_offset = o->vfu_mig_buf_size - o->vfu_mig_buf_pending; + + o->vfu_mig_buf_pending -= data_size; + +if (offset) { + *offset =0 ; + } + +if (size) { + *size = data_size; + } + +return 0 ; +} + +static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t read_offset = o->vfu_mig_section_offset + offset; + +if (read_offset > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - offset outside range" ); +return -1 ; + } + +if ((read_offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - size outside range" ); + size = o->vfu_mig_buf_size - read_offset; + } + +memcpy (buf, (o->vfu_mig_buf + read_offset), size); + +return size; +} + +static ssize_t vfu_mig_write_data (vfu_ctx_t *vfu_ctx,void *data, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t end = o->vfu_mig_data_written + offset + size; + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + o->vfu_mig_buf_size = end; + } + +memcpy ((o->vfu_mig_buf + o->vfu_mig_data_written + offset), data, size); + +return size; +} + +static int vfu_mig_data_written (vfu_ctx_t *vfu_ctx,uint64_t count) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + + o->vfu_mig_data_written += count; + +return 0 ; +} + +static const vfu_migration_callbacks_t vfu_mig_cbs = { + .version =VFU_MIGR_CALLBACKS_VERS , + .transition = &vfu_mig_transition, + .get_pending_bytes = &vfu_mig_get_pending_bytes, + .prepare_data = &vfu_mig_prepare_data, + .read_data = &vfu_mig_read_data, + .data_written = &vfu_mig_data_written, + .write_data = &vfu_mig_write_data, +}; +static void vfu_object_ctx_run (void *opaque) {VfuObject *o = opaque;@@ -550,6 +982,13 @@ void vfu_object_set_bus_irq(PCIBus *pci_bus)pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); } +static bool vfu_object_migratable (VfuObject *o) +{ +DeviceClass *dc =DEVICE_GET_CLASS (o->pci_dev); + +return dc->vmsd && !dc->vmsd->unmigratable; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -575,6 +1014,7 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)ERRP_GUARD ();DeviceState *dev =NULL ;vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +uint64_t migr_regs_size, migr_size;int ret; if (o->vfu_ctx || !o->socket || !o->device ||@@ -653,6 +1093,31 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + migr_regs_size =vfu_get_migr_register_area_size (); + migr_size = migr_regs_size +VFU_OBJECT_MIG_WINDOW ; + + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_MIGR_REGION_IDX , + migr_size,NULL , +VFU_REGION_FLAG_RW ,NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to register migration BAR %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +if (!vfu_object_migratable (o)) { + goto realize_ctx; + } + + ret =vfu_setup_device_migration_callbacks (o->vfu_ctx, &vfu_mig_cbs, + migr_regs_size); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup migration %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +realize_ctx: ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",@@ -700,6 +1165,8 @@ static void vfu_object_init(Object *obj) } o->vfu_poll_fd =-1 ; + + o->vfu_state =VFU_MIGR_STATE_STOP ; }static void vfu_object_finalize (Object *obj)migration/savevm.c +89
-0@@ -66,6 +66,7 @@#include "net/announce.h" #include "qemu/yank.h" #include "yank_functions.h" +#include "hw/qdev-core.h" const unsigned int postcopy_ram_discard_version;@@ -1606,6 +1607,64 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)return ret; } +static SaveStateEntry *find_se_from_dev (DeviceState *dev) +{ +SaveStateEntry *se; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (se->opaque == dev) { +return se; + } + } + +return NULL ; +} + +static int qemu_remote_savevm_section_full (DeviceState *dev,void *opaque) +{ +QEMUFile *f = opaque; +SaveStateEntry *se; +int ret; + + se =find_se_from_dev (dev); +if (!se) { +return 0 ; + } + +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque) || + se->vmsd->unmigratable) { +return 0 ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL ); +if (ret) { +qemu_file_set_error (f, ret); +return ret; + } + +save_section_footer (f, se); + +return 0 ; +} + +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev) +{ +int ret =qdev_walk_children (dev,NULL ,NULL , + qemu_remote_savevm_section_full, +NULL , f); + +if (ret) { +return ret; + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f); + +return 0 ; +} +void qemu_savevm_live_state (QEMUFile *f) {/* save QEMU_VM_SECTION_END section */ @@ -2447,6 +2506,36 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)return 0 ; } +int qemu_remote_loadvm (QEMUFile *f) +{ +uint8_t section_type; +int ret =0 ; + +while (true) { + section_type =qemu_get_byte (f); + + ret =qemu_file_get_error (f); +if (ret) { +break ; + } + +switch (section_type) { +case QEMU_VM_SECTION_FULL : + ret =qemu_loadvm_section_start_full (f,NULL ); +if (ret <0 ) { +break ; + } +break ; +case QEMU_VM_EOF : +return ret; +default : +return -EINVAL ; + } + } + +return ret; +} +static int qemu_loadvm_section_part_end (QEMUFile *f,MigrationIncomingState *mis) {migration/vmstate.c +19
-0@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)return size; } +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } + +return size; +} +static void vmstate_handle_alloc (void *ptr,const VMStateField *field,void *opaque) {
Re: [PATCH v6 17/19] vfio-user: register handlers to facilitate migration
Quoted 12 lines · Paolo Bonzini
Quoted 5 lines · Jagannathan Raman
Store and load the device's state during migration. use libvfio-user's handlers for this purpose
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>Why does no one call
clear_deferred_backend_init?
We’ll clear it at the machine finalization. FWIW, the ‘x-remote’ machine operates in a deferred backend initialization mode for the entire lifecycle of the VM.
Thank you Paolo!
--
Jag
Quoted 746 lines
Paolo
Quoted 742 lines
--- include/block/block.h | 1 + include/migration/vmstate.h | 2 + migration/savevm.h | 2 + block.c | 5 + hw/remote/machine.c | 7 + hw/remote/vfio-user-obj.c | 467 ++++++++++++++++++++++++++++++++++++ migration/savevm.c | 89 +++++++ migration/vmstate.c | 19 ++ 8 files changed, 592 insertions(+)include/block/block.h +1
-0@@ -495,6 +495,7 @@ int generated_co_wrapper bdrv_invalidate_cache(BlockDriverState *bs, Error **errp);void bdrv_invalidate_cache_all (Error **errp);int bdrv_inactivate_all (void ); +int bdrv_inactivate (BlockDriverState *bs);/* Ensure contents are flushed to disk. */ int generated_co_wrapper bdrv_flush(BlockDriverState *bs);include/migration/vmstate.h +1
-0@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;#define VMSTATE_END_OF_LIST () \ {} +uint64_t vmstate_vmsd_size(PCIDevice *pci_dev); +int vmstate_load_state (QEMUFile *f,const VMStateDescription *vmsd,void *opaque,int version_id); int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,migration/savevm.h +2
-0@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif block.c +4
-0@@ -6565,6 +6565,11 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs)return 0 ; } +int bdrv_inactivate(BlockDriverState *bs) +{ +return bdrv_inactivate_recurse (bs); +} +int bdrv_inactivate_all (void ) {BlockDriverState *bs =NULL ;hw/remote/machine.c +6
-0@@ -24,6 +24,7 @@#include "hw/qdev-core.h" #include "hw/remote/iommu.h" #include "hw/remote/vfio-user-obj.h" +#include "sysemu/sysemu.h" static void remote_machine_init (MachineState *machine) {@@ -86,6 +87,11 @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp) s->vfio_user = value; } +static void remote_machine_instance_init(Object *obj) +{ +set_deferred_backend_init (); +} +static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -105,6 +111,7 @@ static const TypeInfo remote_machine = { .name = TYPE_REMOTE_MACHINE, .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), + .instance_init = remote_machine_instance_init, .class_init = remote_machine_class_init, .interfaces = (InterfaceInfo []) { {TYPE_HOTPLUG_HANDLER },hw/remote/vfio-user-obj.c +463
-0@@ -57,6 +57,13 @@#include "hw/pci/msi.h" #include "hw/pci/msix.h" #include "hw/remote/vfio-user-obj.h" +#include "migration/qemu-file.h" +#include "migration/savevm.h" +#include "migration/vmstate.h" +#include "migration/global_state.h" +#include "block/block.h" +#include "sysemu/block-backend.h" +#include "net/net.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -108,12 +115,49 @@ struct VfuObject {Error *unplug_blocker;int vfu_poll_fd; + +/* +* vfu_mig_buf holds the migration data. In the remote server, this +* buffer replaces the role of an IO channel which links the source +* and the destination. +* +* Whenever the client QEMU process initiates migration, the remote +* server gets notified via libvfio-user callbacks. The remote server +* sets up a QEMUFile object using this buffer as backend. The remote +* server passes this object to its migration subsystem, which slurps +* the VMSD of the device ('devid' above) referenced by this object +* and stores the VMSD in this buffer. +* +* The client subsequetly asks the remote server for any data that +* needs to be moved over to the destination via libvfio-user +* library's vfu_migration_callbacks_t callbacks. The remote hands +* over this buffer as data at this time. +* +* A reverse of this process happens at the destination. +*/ +uint8_t *vfu_mig_buf; + +uint64_t vfu_mig_buf_size; + +uint64_t vfu_mig_buf_pending; + +uint64_t vfu_mig_data_written; + +uint64_t vfu_mig_section_offset; + +QEMUFile *vfu_mig_file; + +vfu_migr_state_t vfu_state; };static GHashTable *vfu_object_bdf_to_ctx_table;#define INT2VOIDP (i) (void *)(uintptr_t)(i) +#define KB(x) ((size_t) (x) << 10) + +#define VFU_OBJECT_MIG_WINDOW KB(64) +static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,@@ -163,6 +207,394 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)vfu_object_init_ctx (o, errp); } +/** +* Migration helper functions +* +* vfu_mig_buf_read & vfu_mig_buf_write are used by QEMU's migration +* subsystem - qemu_remote_loadvm & qemu_remote_savevm. loadvm/savevm +* call these functions via QEMUFileOps to load/save the VMSD of a +* device into vfu_mig_buf +* +*/ +static ssize_t vfu_mig_buf_read(void *opaque, uint8_t *buf, int64_t pos, +size_t size,Error **errp) +{ +VfuObject *o = opaque; + +if (pos > o->vfu_mig_buf_size) { + size =0 ; + }else if ((pos + size) > o->vfu_mig_buf_size) { + size = o->vfu_mig_buf_size - pos; + } + +memcpy (buf, (o->vfu_mig_buf + pos), size); + +return size; +} + +static ssize_t vfu_mig_buf_write (void *opaque,struct iovec *iov,int iovcnt, +int64_t pos,Error **errp) +{ +ERRP_GUARD (); +VfuObject *o = opaque; +uint64_t end = pos +iov_size (iov, iovcnt); +int i; + +if (o->vfu_mig_buf_pending) { +error_setg (errp,"Migration is ongoing" ); +return 0 ; + } + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + } + +for (i =0 ; i < iovcnt; i++) { +memcpy ((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base, + iov[i].iov_len); + o->vfu_mig_buf_size += iov[i].iov_len; + } + +return iov_size (iov, iovcnt); +} + +static int vfu_mig_buf_shutdown (void *opaque,bool rd,bool wr,Error **errp) +{ +VfuObject *o = opaque; + + o->vfu_mig_buf_size =0 ; + +g_free (o->vfu_mig_buf); + + o->vfu_mig_buf =NULL ; + + o->vfu_mig_buf_pending =0 ; + + o->vfu_mig_data_written =0 ; + + o->vfu_mig_section_offset =0 ; + +return 0 ; +} + +static const QEMUFileOps vfu_mig_fops_save = { + .writev_buffer = vfu_mig_buf_write, + .shut_down = vfu_mig_buf_shutdown, +}; + +static const QEMUFileOps vfu_mig_fops_load = { + .get_buffer = vfu_mig_buf_read, + .shut_down = vfu_mig_buf_shutdown, +}; + +static BlockDriverState *vfu_object_find_bs_by_dev (DeviceState *dev) +{ +BlockBackend *blk =blk_by_dev (dev); + +if (!blk) { +return NULL ; + } + +return blk_bs (blk); +} + +static int vfu_object_bdrv_invalidate_cache_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; +Error *local_err =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +bdrv_invalidate_cache (bs, &local_err); +if (local_err) { +error_report_err (local_err); +return -1 ; + } + +return 0 ; +} + +static int vfu_object_bdrv_inactivate_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +return bdrv_inactivate (bs); +} + +static void vfu_object_start_stop_netdev (DeviceState *dev,bool start) +{ +NetClientState *nc =NULL ; +Error *local_err =NULL ; +char *netdev =NULL ; + + netdev =object_property_get_str (OBJECT (dev),"netdev" , &local_err); +if (local_err) { +/** +* object_property_get_str() sets Error if netdev property is +* not found, not necessarily an error in the context of +* this function +*/ +error_free (local_err); +return ; + } + +if (!netdev) { +return ; + } + + nc =qemu_find_netdev (netdev); + +if (!nc) { +return ; + } + +if (!start) { +qemu_flush_or_purge_queued_packets (nc, true); + +if (nc->info && nc->info->cleanup) { + nc->info->cleanup (nc); + } + }else if (nc->peer) { +qemu_flush_or_purge_queued_packets (nc->peer, false); + } +} + +static int vfu_object_start_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_invalidate_cache_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, true); + +return ret; +} + +static int vfu_object_stop_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_inactivate_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, false); + +return ret; +} + +/** +* handlers for vfu_migration_callbacks_t +* +* The libvfio-user library accesses these handlers to drive the migration +* at the remote end, and also to transport the data stored in vfu_mig_buf +* +*/ +static void vfu_mig_state_stop_and_copy (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_save, false); + } + + ret =qemu_remote_savevm (o->vfu_mig_file,DEVICE (o->pci_dev)); +if (ret) { +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; +return ; + } + +qemu_fflush (o->vfu_mig_file); +} + +static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (o->vfu_state !=VFU_MIGR_STATE_RESUME ) { + goto run_ctx; + } + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } + +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +run_ctx: + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_start_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to setup backends for %s" , + o->device); +return ; + } +} + +static void vfu_mig_state_stop (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_stop_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to inactivate backends for %s" , + o->device); + } +} + +static int vfu_mig_transition (vfu_ctx_t *vfu_ctx,vfu_migr_state_t state) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (o->vfu_state == state) { +return 0 ; + } + +switch (state) { +case VFU_MIGR_STATE_RESUME : +break ; +case VFU_MIGR_STATE_STOP_AND_COPY : +vfu_mig_state_stop_and_copy (vfu_ctx); +break ; +case VFU_MIGR_STATE_STOP : +vfu_mig_state_stop (vfu_ctx); +break ; +case VFU_MIGR_STATE_PRE_COPY : +break ; +case VFU_MIGR_STATE_RUNNING : +vfu_mig_state_running (vfu_ctx); +break ; +default : +warn_report ("vfu: Unknown migration state %d" , state); + } + + o->vfu_state = state; + +return 0 ; +} + +static uint64_t vfu_mig_get_pending_bytes (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +static bool mig_ongoing; + +if (!mig_ongoing && !o->vfu_mig_buf_pending) { + o->vfu_mig_buf_pending = o->vfu_mig_buf_size; + mig_ongoing = true; + } + +if (mig_ongoing && !o->vfu_mig_buf_pending) { + mig_ongoing = false; + } + +return o->vfu_mig_buf_pending; +} + +static int vfu_mig_prepare_data (vfu_ctx_t *vfu_ctx,uint64_t *offset, +uint64_t *size) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t data_size = o->vfu_mig_buf_pending; + +if (data_size >VFU_OBJECT_MIG_WINDOW ) { + data_size =VFU_OBJECT_MIG_WINDOW ; + } + + o->vfu_mig_section_offset = o->vfu_mig_buf_size - o->vfu_mig_buf_pending; + + o->vfu_mig_buf_pending -= data_size; + +if (offset) { + *offset =0 ; + } + +if (size) { + *size = data_size; + } + +return 0 ; +} + +static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t read_offset = o->vfu_mig_section_offset + offset; + +if (read_offset > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - offset outside range" ); +return -1 ; + } + +if ((read_offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - size outside range" ); + size = o->vfu_mig_buf_size - read_offset; + } + +memcpy (buf, (o->vfu_mig_buf + read_offset), size); + +return size; +} + +static ssize_t vfu_mig_write_data (vfu_ctx_t *vfu_ctx,void *data, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t end = o->vfu_mig_data_written + offset + size; + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + o->vfu_mig_buf_size = end; + } + +memcpy ((o->vfu_mig_buf + o->vfu_mig_data_written + offset), data, size); + +return size; +} + +static int vfu_mig_data_written (vfu_ctx_t *vfu_ctx,uint64_t count) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + + o->vfu_mig_data_written += count; + +return 0 ; +} + +static const vfu_migration_callbacks_t vfu_mig_cbs = { + .version =VFU_MIGR_CALLBACKS_VERS , + .transition = &vfu_mig_transition, + .get_pending_bytes = &vfu_mig_get_pending_bytes, + .prepare_data = &vfu_mig_prepare_data, + .read_data = &vfu_mig_read_data, + .data_written = &vfu_mig_data_written, + .write_data = &vfu_mig_write_data, +}; +static void vfu_object_ctx_run (void *opaque) {VfuObject *o = opaque;@@ -550,6 +982,13 @@ void vfu_object_set_bus_irq(PCIBus *pci_bus)pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); } +static bool vfu_object_migratable(VfuObject *o) +{ +DeviceClass *dc =DEVICE_GET_CLASS (o->pci_dev); + +return dc->vmsd && !dc->vmsd->unmigratable; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -575,6 +1014,7 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)ERRP_GUARD ();DeviceState *dev =NULL ;vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +uint64_t migr_regs_size, migr_size;int ret; if (o->vfu_ctx || !o->socket || !o->device ||@@ -653,6 +1093,31 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + migr_regs_size =vfu_get_migr_register_area_size (); + migr_size = migr_regs_size +VFU_OBJECT_MIG_WINDOW ; + + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_MIGR_REGION_IDX , + migr_size,NULL , +VFU_REGION_FLAG_RW ,NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to register migration BAR %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +if (!vfu_object_migratable (o)) { + goto realize_ctx; + } + + ret =vfu_setup_device_migration_callbacks (o->vfu_ctx, &vfu_mig_cbs, + migr_regs_size); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup migration %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +realize_ctx: ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",@@ -700,6 +1165,8 @@ static void vfu_object_init(Object *obj) } o->vfu_poll_fd =-1 ; + + o->vfu_state =VFU_MIGR_STATE_STOP ; }static void vfu_object_finalize (Object *obj)migration/savevm.c +1
-0@@ -66,6 +66,7 @@#include "net/announce.h" #include "qemu/yank.h" #include "yank_functions.h" +#include "hw/qdev-core.h" const unsigned int postcopy_ram_discard_version; @@ -1606,6 +1607,64 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp) return ret;}
+static SaveStateEntry *find_se_from_dev(DeviceState *dev)migration/savevm.c +86
-0+{ +SaveStateEntry *se; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (se->opaque == dev) { +return se; + } + } + +return NULL ; +} + +static int qemu_remote_savevm_section_full (DeviceState *dev,void *opaque) +{ +QEMUFile *f = opaque; +SaveStateEntry *se; +int ret; + + se =find_se_from_dev (dev); +if (!se) { +return 0 ; + } + +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque) || + se->vmsd->unmigratable) { +return 0 ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL ); +if (ret) { +qemu_file_set_error (f, ret); +return ret; + } + +save_section_footer (f, se); + +return 0 ; +} + +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev) +{ +int ret =qdev_walk_children (dev,NULL ,NULL , + qemu_remote_savevm_section_full, +NULL , f); + +if (ret) { +return ret; + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f); + +return 0 ; +} +void qemu_savevm_live_state (QEMUFile *f) {/* save QEMU_VM_SECTION_END section */ @@ -2447,6 +2506,36 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)return 0 ; } +int qemu_remote_loadvm(QEMUFile *f) +{ +uint8_t section_type; +int ret =0 ; + +while (true) { + section_type =qemu_get_byte (f); + + ret =qemu_file_get_error (f); +if (ret) { +break ; + } + +switch (section_type) { +case QEMU_VM_SECTION_FULL : + ret =qemu_loadvm_section_start_full (f,NULL ); +if (ret <0 ) { +break ; + } +break ; +case QEMU_VM_EOF : +return ret; +default : +return -EINVAL ; + } + } + +return ret; +} +static int qemu_loadvm_section_part_end (QEMUFile *f,MigrationIncomingState *mis) {migration/vmstate.c +18
-0@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)return size; } +uint64_t vmstate_vmsd_size(PCIDevice *pci_dev) +{ +DeviceClass *dc =DEVICE_GET_CLASS (DEVICE (pci_dev)); +const VMStateField *field =NULL ; +uint64_t size =0 ; + +if (!dc->vmsd) { +return 0 ; + } + + field = dc->vmsd->fields; +while (field && field->name) { + size +=vmstate_size (pci_dev, field); + field++; + } + +return size; +} +static void vmstate_handle_alloc (void *ptr,const VMStateField *field,void *opaque) {
Re: [PATCH v6 17/19] vfio-user: register handlers to facilitate migration
In reply to Jagannathan RamanQuoted 156 lines · Jagannathan Raman
Store and load the device's state during migration. use libvfio-user's handlers for this purpose
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- include/block/block.h | 1 + include/migration/vmstate.h | 2 + migration/savevm.h | 2 + block.c | 5 + hw/remote/machine.c | 7 + hw/remote/vfio-user-obj.c | 467 ++++++++++++++++++++++++++++++++++++ migration/savevm.c | 89 +++++++ migration/vmstate.c | 19 ++ 8 files changed, 592 insertions(+)include/block/block.h +1
-0@@ -495,6 +495,7 @@ int generated_co_wrapper bdrv_invalidate_cache(BlockDriverState *bs, Error **errp);void bdrv_invalidate_cache_all (Error **errp);int bdrv_inactivate_all (void ); +int bdrv_inactivate (BlockDriverState *bs);/* Ensure contents are flushed to disk. */ int generated_co_wrapper bdrv_flush(BlockDriverState *bs);include/migration/vmstate.h +2
-0@@ -1165,6 +1165,8 @@ extern const VMStateInfo vmstate_info_qlist;#define VMSTATE_END_OF_LIST () \ {} +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev); +int vmstate_load_state (QEMUFile *f,const VMStateDescription *vmsd,void *opaque,int version_id); int vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,migration/savevm.h +2
-0@@ -67,5 +67,7 @@ int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis);int qemu_load_device_state (QEMUFile *f);int qemu_savevm_state_complete_precopy_non_iterable (QEMUFile *f,bool in_postcopy,bool inactivate_disks); +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev); +int qemu_remote_loadvm (QEMUFile *f);#endif block.c +5
-0@@ -6565,6 +6565,11 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs)return 0 ; } +int bdrv_inactivate (BlockDriverState *bs) +{ +return bdrv_inactivate_recurse (bs); +} +int bdrv_inactivate_all (void ) {BlockDriverState *bs =NULL ;hw/remote/machine.c +7
-0@@ -24,6 +24,7 @@#include "hw/qdev-core.h" #include "hw/remote/iommu.h" #include "hw/remote/vfio-user-obj.h" +#include "sysemu/sysemu.h" static void remote_machine_init (MachineState *machine) {@@ -86,6 +87,11 @@ static void remote_machine_set_vfio_user(Object *obj, bool value, Error **errp) s->vfio_user = value; } +static void remote_machine_instance_init (Object *obj) +{ +set_deferred_backend_init (); +} +static void remote_machine_class_init (ObjectClass *oc,void *data) {MachineClass *mc =MACHINE_CLASS (oc);@@ -105,6 +111,7 @@ static const TypeInfo remote_machine = { .name = TYPE_REMOTE_MACHINE, .parent = TYPE_MACHINE, .instance_size = sizeof(RemoteMachineState), + .instance_init = remote_machine_instance_init, .class_init = remote_machine_class_init, .interfaces = (InterfaceInfo []) { {TYPE_HOTPLUG_HANDLER },hw/remote/vfio-user-obj.c +39
-0@@ -57,6 +57,13 @@#include "hw/pci/msi.h" #include "hw/pci/msix.h" #include "hw/remote/vfio-user-obj.h" +#include "migration/qemu-file.h" +#include "migration/savevm.h" +#include "migration/vmstate.h" +#include "migration/global_state.h" +#include "block/block.h" +#include "sysemu/block-backend.h" +#include "net/net.h" #define TYPE_VFU_OBJECT "x-vfio-user-server"OBJECT_DECLARE_TYPE (VfuObject, VfuObjectClass,VFU_OBJECT )@@ -108,12 +115,49 @@ struct VfuObject {Error *unplug_blocker;int vfu_poll_fd; + +/* +* vfu_mig_buf holds the migration data. In the remote server, this +* buffer replaces the role of an IO channel which links the source +* and the destination. +* +* Whenever the client QEMU process initiates migration, the remote +* server gets notified via libvfio-user callbacks. The remote server +* sets up a QEMUFile object using this buffer as backend. The remote +* server passes this object to its migration subsystem, which slurps +* the VMSD of the device ('devid' above) referenced by this object +* and stores the VMSD in this buffer. +* +* The client subsequetly asks the remote server for any data that +* needs to be moved over to the destination via libvfio-user +* library's vfu_migration_callbacks_t callbacks. The remote hands +* over this buffer as data at this time. +* +* A reverse of this process happens at the destination. +*/ +uint8_t *vfu_mig_buf; + +uint64_t vfu_mig_buf_size; + +uint64_t vfu_mig_buf_pending; + +uint64_t vfu_mig_data_written; + +uint64_t vfu_mig_section_offset; + +QEMUFile *vfu_mig_file; +
hw/remote/vfio-user-obj.c struct VfuObject { · line 159
Quoted 10 lines
hw/remote/vfio-user-obj.c +4
-0+vfu_migr_state_t vfu_state; };static GHashTable *vfu_object_bdf_to_ctx_table;#define INT2VOIDP (i) (void *)(uintptr_t)(i) +#define KB (x) ((size_t) (x) << 10) + +#define VFU_OBJECT_MIG_WINDOW KB(64)
Please use "qemu/units.h":
#include "qemu/units.h" ... #define VFU_OBJECT_MIG_WINDOW_SIZE (64 * KiB)
(Adding "_SIZE" to the name makes the purpose of the constant clearer.)
Quoted 153 lines
hw/remote/vfio-user-obj.c +146
-0+static void vfu_object_init_ctx (VfuObject *o,Error **errp); static void vfu_object_set_socket(Object *obj, Visitor *v, const char *name,@@ -163,6 +207,394 @@ static void vfu_object_set_device(Object *obj, const char *str, Error **errp)vfu_object_init_ctx (o, errp); } +/** +* Migration helper functions +* +* vfu_mig_buf_read & vfu_mig_buf_write are used by QEMU's migration +* subsystem - qemu_remote_loadvm & qemu_remote_savevm. loadvm/savevm +* call these functions via QEMUFileOps to load/save the VMSD of a +* device into vfu_mig_buf +* +*/ +static ssize_t vfu_mig_buf_read (void *opaque,uint8_t *buf,int64_t pos, +size_t size,Error **errp) +{ +VfuObject *o = opaque; + +if (pos > o->vfu_mig_buf_size) { + size =0 ; + }else if ((pos + size) > o->vfu_mig_buf_size) { + size = o->vfu_mig_buf_size - pos; + } + +memcpy (buf, (o->vfu_mig_buf + pos), size); + +return size; +} + +static ssize_t vfu_mig_buf_write (void *opaque,struct iovec *iov,int iovcnt, +int64_t pos,Error **errp) +{ +ERRP_GUARD (); +VfuObject *o = opaque; +uint64_t end = pos +iov_size (iov, iovcnt); +int i; + +if (o->vfu_mig_buf_pending) { +error_setg (errp,"Migration is ongoing" ); +return 0 ; + } + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + } + +for (i =0 ; i < iovcnt; i++) { +memcpy ((o->vfu_mig_buf + o->vfu_mig_buf_size), iov[i].iov_base, + iov[i].iov_len); + o->vfu_mig_buf_size += iov[i].iov_len; + } + +return iov_size (iov, iovcnt); +} + +static int vfu_mig_buf_shutdown (void *opaque,bool rd,bool wr,Error **errp) +{ +VfuObject *o = opaque; + + o->vfu_mig_buf_size =0 ; + +g_free (o->vfu_mig_buf); + + o->vfu_mig_buf =NULL ; + + o->vfu_mig_buf_pending =0 ; + + o->vfu_mig_data_written =0 ; + + o->vfu_mig_section_offset =0 ; + +return 0 ; +} + +static const QEMUFileOps vfu_mig_fops_save = { + .writev_buffer = vfu_mig_buf_write, + .shut_down = vfu_mig_buf_shutdown, +}; + +static const QEMUFileOps vfu_mig_fops_load = { + .get_buffer = vfu_mig_buf_read, + .shut_down = vfu_mig_buf_shutdown, +}; + +static BlockDriverState *vfu_object_find_bs_by_dev (DeviceState *dev) +{ +BlockBackend *blk =blk_by_dev (dev); + +if (!blk) { +return NULL ; + } + +return blk_bs (blk); +} + +static int vfu_object_bdrv_invalidate_cache_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; +Error *local_err =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +bdrv_invalidate_cache (bs, &local_err); +if (local_err) { +error_report_err (local_err); +return -1 ; + } + +return 0 ; +} + +static int vfu_object_bdrv_inactivate_by_dev (DeviceState *dev) +{ +BlockDriverState *bs =NULL ; + + bs =vfu_object_find_bs_by_dev (dev); +if (!bs) { +return 0 ; + } + +return bdrv_inactivate (bs); +} + +static void vfu_object_start_stop_netdev (DeviceState *dev,bool start) +{ +NetClientState *nc =NULL ; +Error *local_err =NULL ; +char *netdev =NULL ; + + netdev =object_property_get_str (OBJECT (dev),"netdev" , &local_err); +if (local_err) { +/** +* object_property_get_str() sets Error if netdev property is +* not found, not necessarily an error in the context of +* this function +*/ +error_free (local_err); +return ; + } + +if (!netdev) { +return ; + } + + nc =qemu_find_netdev (netdev); +
hw/remote/vfio-user-obj.c static void vfu_object_set_device(Object *obj, const char *str, Error **errp) · line 364
Quoted 10 lines
hw/remote/vfio-user-obj.c +10
-0+if (!nc) { +return ; + } + +if (!start) { +qemu_flush_or_purge_queued_packets (nc, true); + +if (nc->info && nc->info->cleanup) { + nc->info->cleanup (nc); + }
I'm not sure if this is correct. Do we actually want to clean up the NetClient (e.g. close the tap file descriptor)? If yes, why isn't the NetClient removed from the net_clients tailq?
Quoted 291 lines
hw/remote/vfio-user-obj.c +266
-0+ }else if (nc->peer) { +qemu_flush_or_purge_queued_packets (nc->peer, false); + } +} + +static int vfu_object_start_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_invalidate_cache_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, true); + +return ret; +} + +static int vfu_object_stop_devs (DeviceState *dev,void *opaque) +{ +int ret =vfu_object_bdrv_inactivate_by_dev (dev); + +if (ret) { +return ret; + } + +vfu_object_start_stop_netdev (dev, false); + +return ret; +} + +/** +* handlers for vfu_migration_callbacks_t +* +* The libvfio-user library accesses these handlers to drive the migration +* at the remote end, and also to transport the data stored in vfu_mig_buf +* +*/ +static void vfu_mig_state_stop_and_copy (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_save, false); + } + + ret =qemu_remote_savevm (o->vfu_mig_file,DEVICE (o->pci_dev)); +if (ret) { +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; +return ; + } + +qemu_fflush (o->vfu_mig_file); +} + +static void vfu_mig_state_running (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + +if (o->vfu_state !=VFU_MIGR_STATE_RESUME ) { + goto run_ctx; + } + +if (!o->vfu_mig_file) { + o->vfu_mig_file =qemu_fopen_ops (o, &vfu_mig_fops_load, false); + } + + ret =qemu_remote_loadvm (o->vfu_mig_file); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to restore device state" ); +return ; + } + +qemu_file_shutdown (o->vfu_mig_file); + o->vfu_mig_file =NULL ; + +run_ctx: + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_start_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to setup backends for %s" , + o->device); +return ; + } +} + +static void vfu_mig_state_stop (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +int ret; + + ret =qdev_walk_children (DEVICE (o->pci_dev),NULL ,NULL , + vfu_object_stop_devs, +NULL ,NULL ); +if (ret) { +VFU_OBJECT_ERROR (o,"vfu: failed to inactivate backends for %s" , + o->device); + } +} + +static int vfu_mig_transition (vfu_ctx_t *vfu_ctx,vfu_migr_state_t state) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (o->vfu_state == state) { +return 0 ; + } + +switch (state) { +case VFU_MIGR_STATE_RESUME : +break ; +case VFU_MIGR_STATE_STOP_AND_COPY : +vfu_mig_state_stop_and_copy (vfu_ctx); +break ; +case VFU_MIGR_STATE_STOP : +vfu_mig_state_stop (vfu_ctx); +break ; +case VFU_MIGR_STATE_PRE_COPY : +break ; +case VFU_MIGR_STATE_RUNNING : +vfu_mig_state_running (vfu_ctx); +break ; +default : +warn_report ("vfu: Unknown migration state %d" , state); + } + + o->vfu_state = state; + +return 0 ; +} + +static uint64_t vfu_mig_get_pending_bytes (vfu_ctx_t *vfu_ctx) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +static bool mig_ongoing; + +if (!mig_ongoing && !o->vfu_mig_buf_pending) { + o->vfu_mig_buf_pending = o->vfu_mig_buf_size; + mig_ongoing = true; + } + +if (mig_ongoing && !o->vfu_mig_buf_pending) { + mig_ongoing = false; + } + +return o->vfu_mig_buf_pending; +} + +static int vfu_mig_prepare_data (vfu_ctx_t *vfu_ctx,uint64_t *offset, +uint64_t *size) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t data_size = o->vfu_mig_buf_pending; + +if (data_size >VFU_OBJECT_MIG_WINDOW ) { + data_size =VFU_OBJECT_MIG_WINDOW ; + } + + o->vfu_mig_section_offset = o->vfu_mig_buf_size - o->vfu_mig_buf_pending; + + o->vfu_mig_buf_pending -= data_size; + +if (offset) { + *offset =0 ; + } + +if (size) { + *size = data_size; + } + +return 0 ; +} + +static ssize_t vfu_mig_read_data (vfu_ctx_t *vfu_ctx,void *buf, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t read_offset = o->vfu_mig_section_offset + offset; + +if (read_offset > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - offset outside range" ); +return -1 ; + } + +if ((read_offset + size) > o->vfu_mig_buf_size) { +warn_report ("vfu: buffer overflow - size outside range" ); + size = o->vfu_mig_buf_size - read_offset; + } + +memcpy (buf, (o->vfu_mig_buf + read_offset), size); + +return size; +} + +static ssize_t vfu_mig_write_data (vfu_ctx_t *vfu_ctx,void *data, +uint64_t size,uint64_t offset) +{ +VfuObject *o =vfu_get_private (vfu_ctx); +uint64_t end = o->vfu_mig_data_written + offset + size; + +if (end > o->vfu_mig_buf_size) { + o->vfu_mig_buf =g_realloc (o->vfu_mig_buf, end); + o->vfu_mig_buf_size = end; + } + +memcpy ((o->vfu_mig_buf + o->vfu_mig_data_written + offset), data, size); + +return size; +} + +static int vfu_mig_data_written (vfu_ctx_t *vfu_ctx,uint64_t count) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + + o->vfu_mig_data_written += count; + +return 0 ; +} + +static const vfu_migration_callbacks_t vfu_mig_cbs = { + .version =VFU_MIGR_CALLBACKS_VERS , + .transition = &vfu_mig_transition, + .get_pending_bytes = &vfu_mig_get_pending_bytes, + .prepare_data = &vfu_mig_prepare_data, + .read_data = &vfu_mig_read_data, + .data_written = &vfu_mig_data_written, + .write_data = &vfu_mig_write_data, +}; +static void vfu_object_ctx_run (void *opaque) {VfuObject *o = opaque;@@ -550,6 +982,13 @@ void vfu_object_set_bus_irq(PCIBus *pci_bus)pci_bus_irqs (pci_bus, vfu_object_set_irq, vfu_object_map_irq,NULL ,1 ); } +static bool vfu_object_migratable (VfuObject *o) +{ +DeviceClass *dc =DEVICE_GET_CLASS (o->pci_dev); + +return dc->vmsd && !dc->vmsd->unmigratable; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -575,6 +1014,7 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp)ERRP_GUARD ();DeviceState *dev =NULL ;vfu_pci_type_t pci_type =VFU_PCI_TYPE_CONVENTIONAL ; +uint64_t migr_regs_size, migr_size;int ret; if (o->vfu_ctx || !o->socket || !o->device ||@@ -653,6 +1093,31 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + migr_regs_size =vfu_get_migr_register_area_size (); + migr_size = migr_regs_size +VFU_OBJECT_MIG_WINDOW ; + + ret =vfu_setup_region (o->vfu_ctx,VFU_PCI_DEV_MIGR_REGION_IDX , + migr_size,NULL , +VFU_REGION_FLAG_RW ,NULL ,0 ,-1 ,0 ); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to register migration BAR %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +if (!vfu_object_migratable (o)) { + goto realize_ctx; + } + + ret =vfu_setup_device_migration_callbacks (o->vfu_ctx, &vfu_mig_cbs, + migr_regs_size); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup migration %s- %s" , + o->device,strerror (errno)); + goto fail; + } + +realize_ctx: ret =vfu_realize_ctx (o->vfu_ctx);if (ret <0 ) { error_setg(errp, "vfu: Failed to realize device %s- %s",
hw/remote/vfio-user-obj.c static void vfu_object_init(Object *obj) · line 1169
Quoted 5 lines
hw/remote/vfio-user-obj.c +2
-0@@ -700,6 +1165,8 @@ static void vfu_object_init(Object *obj) } o->vfu_poll_fd =-1 ; + + o->vfu_state =VFU_MIGR_STATE_STOP ;
I was expecting RUNNING instead of STOP. Can you explain the state machine? Perhaps --object x-vfio-user-server needs an incoming=on|off parameter that defaults to off?
Quoted 122 lines
hw/remote/vfio-user-obj.c +0
-0}static void vfu_object_finalize (Object *obj)migration/savevm.c +89
-0@@ -66,6 +66,7 @@#include "net/announce.h" #include "qemu/yank.h" #include "yank_functions.h" +#include "hw/qdev-core.h" const unsigned int postcopy_ram_discard_version;@@ -1606,6 +1607,64 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)return ret; } +static SaveStateEntry *find_se_from_dev (DeviceState *dev) +{ +SaveStateEntry *se; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { +if (se->opaque == dev) { +return se; + } + } + +return NULL ; +} + +static int qemu_remote_savevm_section_full (DeviceState *dev,void *opaque) +{ +QEMUFile *f = opaque; +SaveStateEntry *se; +int ret; + + se =find_se_from_dev (dev); +if (!se) { +return 0 ; + } + +if (!se->vmsd || !vmstate_save_needed (se->vmsd, se->opaque) || + se->vmsd->unmigratable) { +return 0 ; + } + +save_section_header (f, se,QEMU_VM_SECTION_FULL ); + + ret =vmstate_save (f, se,NULL ); +if (ret) { +qemu_file_set_error (f, ret); +return ret; + } + +save_section_footer (f, se); + +return 0 ; +} + +int qemu_remote_savevm (QEMUFile *f,DeviceState *dev) +{ +int ret =qdev_walk_children (dev,NULL ,NULL , + qemu_remote_savevm_section_full, +NULL , f); + +if (ret) { +return ret; + } + +qemu_put_byte (f,QEMU_VM_EOF ); +qemu_fflush (f); + +return 0 ; +} +void qemu_savevm_live_state (QEMUFile *f) {/* save QEMU_VM_SECTION_END section */ @@ -2447,6 +2506,36 @@ qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)return 0 ; } +int qemu_remote_loadvm (QEMUFile *f) +{ +uint8_t section_type; +int ret =0 ; + +while (true) { + section_type =qemu_get_byte (f); + + ret =qemu_file_get_error (f); +if (ret) { +break ; + } + +switch (section_type) { +case QEMU_VM_SECTION_FULL : + ret =qemu_loadvm_section_start_full (f,NULL ); +if (ret <0 ) { +break ; + } +break ; +case QEMU_VM_EOF : +return ret; +default : +return -EINVAL ; + } + } + +return ret; +} +static int qemu_loadvm_section_part_end (QEMUFile *f,MigrationIncomingState *mis) {migration/vmstate.c +0
-0
migration/vmstate.c static int vmstate_size(void *opaque, const VMStateField *field) · line 66
Quoted 4 lines
migration/vmstate.c +1
-0@@ -63,6 +63,25 @@ static int vmstate_size(void *opaque, const VMStateField *field)return size; } +uint64_t vmstate_vmsd_size (PCIDevice *pci_dev)
This function is no longer used and can be dropped.
PATCH v6 18/19 vfio-user: handle reset of remote device
In reply to Jagannathan RamanAdds handler to reset a remote device
--- hw/remote/vfio-user-obj.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
hw/remote/vfio-user-obj.c +19 -0
@@ -989,6 +989,19 @@ static bool vfu_object_migratable(VfuObject *o)return dc->vmsd && !dc->vmsd->unmigratable; } +static int vfu_object_device_reset (vfu_ctx_t *vfu_ctx,vfu_reset_type_t type) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (type ==VFU_RESET_LOST_CONN ) { +return 0 ; + } + +qdev_reset_all (DEVICE (o->pci_dev)); + +return 0 ; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These
@@ -1105,6 +1118,12 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + ret =vfu_setup_device_reset_cb (o->vfu_ctx, &vfu_object_device_reset); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup reset callback" ); + goto fail; + } +if (!vfu_object_migratable (o)) { goto realize_ctx; }
Signature · 1 line
2.20.1
Re: [PATCH v6 18/19] vfio-user: handle reset of remote device
Quoted 14 lines · Jagannathan Raman
Adds handler to reset a remote device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>--- hw/remote/vfio-user-obj.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)hw/remote/vfio-user-obj.c +0
-0
hw/remote/vfio-user-obj.c static bool vfu_object_migratable(VfuObject *o) · line 998
Quoted 10 lines
hw/remote/vfio-user-obj.c +7
-0@@ -989,6 +989,19 @@ static bool vfu_object_migratable(VfuObject *o)return dc->vmsd && !dc->vmsd->unmigratable; } +static int vfu_object_device_reset (vfu_ctx_t *vfu_ctx,vfu_reset_type_t type) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (type ==VFU_RESET_LOST_CONN ) { +return 0 ; + }
Why is a lost connection ignored? Should there be a QMP monitor event?
Quoted 25 lines
hw/remote/vfio-user-obj.c +12
-0+ +qdev_reset_all (DEVICE (o->pci_dev)); + +return 0 ; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -1105,6 +1118,12 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + ret =vfu_setup_device_reset_cb (o->vfu_ctx, &vfu_object_device_reset); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup reset callback" ); + goto fail; + } +if (!vfu_object_migratable (o)) { goto realize_ctx; }Signature · 1 line
2.20.1
Re: [PATCH v6 18/19] vfio-user: handle reset of remote device
Quoted 22 lines · Stefan Hajnoczi
Quoted 17 lines · Jagannathan Raman
Adds handler to reset a remote device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/vfio-user-obj.c| 19 +++++++++++++++++++1 file changed, 19 insertions(+)hw/remote/vfio-user-obj.c +0
-0@@ -989,6 +989,19 @@ static bool vfu_object_migratable(VfuObject *o)return dc->vmsd && !dc->vmsd->unmigratable; }
Quoted 9 lines
hw/remote/vfio-user-obj.c static bool vfu_object_migratable(VfuObject *o) · line 998
Quoted 7 lines
hw/remote/vfio-user-obj.c +7
-0+static int vfu_object_device_reset (vfu_ctx_t *vfu_ctx,vfu_reset_type_t type) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (type ==VFU_RESET_LOST_CONN ) { +return 0 ; + }Why is a lost connection ignored? Should there be a QMP monitor event?
We handle the lost connection case in vfu_object_ctx_run(), which is in PATCH 5 of this series. We are sending a QMP monitor event in this case.
Thank you!
--
Jag
Quoted 26 lines
Quoted 26 lines
hw/remote/vfio-user-obj.c +12
-0+ +qdev_reset_all (DEVICE (o->pci_dev)); + +return 0 ; +} +/* * TYPE_VFU_OBJECT depends on the availability of the 'socket' and 'device' * properties. It also depends on devices instantiated in QEMU. These @@ -1105,6 +1118,12 @@ static void vfu_object_init_ctx(VfuObject *o, Error **errp) goto fail; } + ret =vfu_setup_device_reset_cb (o->vfu_ctx, &vfu_object_device_reset); +if (ret <0 ) { +error_setg (errp,"vfu: Failed to setup reset callback" ); + goto fail; + } +if (!vfu_object_migratable (o)) { goto realize_ctx; }Signature · 1 line
2.20.1
Re: [PATCH v6 18/19] vfio-user: handle reset of remote device
Quoted 32 lines · Jag Raman
Quoted 22 lines · Stefan Hajnoczi
Quoted 17 lines · Jagannathan Raman
Adds handler to reset a remote device
Signed-off-by Elena Ufimtseva <[email protected]>Signed-off-by John G Johnson <[email protected]>Signed-off-by Jagannathan Raman <[email protected]>---
hw/remote/vfio-user-obj.c| 19 +++++++++++++++++++1 file changed, 19 insertions(+)hw/remote/vfio-user-obj.c +0
-0@@ -989,6 +989,19 @@ static bool vfu_object_migratable(VfuObject *o)return dc->vmsd && !dc->vmsd->unmigratable; }Quoted 9 lines
hw/remote/vfio-user-obj.c static bool vfu_object_migratable(VfuObject *o) · line 998
Quoted 7 lines
hw/remote/vfio-user-obj.c +7
-0+static int vfu_object_device_reset (vfu_ctx_t *vfu_ctx,vfu_reset_type_t type) +{ +VfuObject *o =vfu_get_private (vfu_ctx); + +if (type ==VFU_RESET_LOST_CONN ) { +return 0 ; + }Why is a lost connection ignored? Should there be a QMP monitor event?
Quoted 2 lines
We handle the lost connection case in
vfu_object_ctx_run(), which is in PATCH 5 of this series. We are sending a QMP monitor event in this case.
Great, please add a comment here.
Stefan
PATCH v6 19/19 vfio-user: avocado tests for vfio-user
In reply to Jagannathan RamanAvocado tests for libvfio-user in QEMU - tests startup, hotplug and migration of the server object
--- MAINTAINERS | 1 + tests/avocado/vfio-user.py | 234 +++++++++++++++++++++++++++++++++++++ 2 files changed, 235 insertions(+) create mode 100644 tests/avocado/vfio-user.py
MAINTAINERS +1 -0
@@ -3573,6 +3573,7 @@ F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c F: include/hw/remote/iommu.h F: stubs/defer-backend-init.c +F: tests/avocado/vfio-user.py EBPF: M: Jason Wang <[email protected]>
tests/avocado/vfio-user.py new file +234 -0
234 lines not on this page · raw message
Signature · 1 line
2.20.1