Initial support for multi-process qemu
v8 cover Initial support for multi-process qemu
Hello
This is the v8 of the patchset. Thank you very much for the detailed feedback for v7. We appreciate your time. We believe we have address all the comments for v7 in the current series.
The broader items we have addressed in this series are as follows:
- Removed the main channel / control channel.
- Enabled 1:1 mapping between irqfd:PCIDevice, moving away from shared interrupt lines to avoid collision between irqfds
- Defined an object type named “remote-object” that connects the device with its fd in the remote process. This object limits the number of remote devices to 1 per process to alleviate security concerns with multiple devices per process. We are addressing the authentication mechanism for multiple devices in the VFIO-over-socket proposal.
To touch upon the brief history of this project, we posted the Proof Of Concept patches before the BoF session in 2018. Subsequently, we posted RFC v1 [1], RFC v2 [2], RFC v3 [3], RFC v4 [4], v5 [5], v6 [6] and v7 [7] of the patch series. Following people contributed to the design and implementation of this project:
Stefan Hajnoczi <[email protected]>
Konrad Wilk <[email protected]>
Kanth Ghatraju <[email protected]>
John G Johnson <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>
We would like to thank QEMU community for your feedback in the design and implementation of this project.
For the full concept writeup about QEMU multi-process, please refer to docs/devel/qemu-multiprocess.rst. Also see docs/qemu-multiprocess.txt for usage information.
We welcome all your ideas, concerns, and questions for this patchset.
[POC]: https://www.mail-archive.com/[email protected]/msg566538.html
[1]: https://www.mail-archive.com/[email protected]/msg602285.html
[2]: https://www.mail-archive.com/[email protected]/msg624877.html
[3]: https://www.mail-archive.com/[email protected]/msg642000.html
[4]: https://www.mail-archive.com/[email protected]/msg655118.html
[5]: https://www.mail-archive.com/[email protected]/msg682429.html
[6]: https://www.mail-archive.com/[email protected]/msg697484.html
[7]: https://patchew.org/QEMU/[email protected]/
Elena Ufimtseva (8):
multi-process: add qio channel function to transmit
multi-process: define MPQemuMsg format and transmission functions
multi-process: add co-routines to communicate with remote
multi-process: introduce proxy object
multi-process: Forward PCI config space acceses to the remote process
multi-process: heartbeat messages to remote
multi-process: perform device reset in the remote process
multi-process: add configure and usage information
Jagannathan Raman (11):
memory: alloc RAM from file at offset
multi-process: Add config option for multi-process QEMU
multi-process: setup PCI host bridge for remote device
multi-process: setup a machine object for remote device process
multi-process: Initialize message handler in remote device
multi-process: Associate fd of a PCIDevice with its object
multi-process: setup memory manager for remote device
multi-process: PCI BAR read/write handling for proxy & remote
endpoints
multi-process: Synchronize remote memory
multi-process: create IOHUB object to handle irq
multi-process: Retrieve PCI info from remote process
John G Johnson (1):
multi-process: add the concept description to
docs/devel/qemu-multiprocess
MAINTAINERS | 26 ++ backends/hostmem-memfd.c | 2 +- configure | 11 + docs/devel/index.rst | 1 + docs/devel/multi-process.rst | 966 ++++++++++++++++++++++++++++++++++++++++ docs/multi-process.rst | 67 +++ exec.c | 11 +- hw/Makefile.objs | 1 + hw/i386/Makefile.objs | 4 + hw/i386/remote-memory.c | 58 +++ hw/i386/remote-msg.c | 266 +++++++++++ hw/i386/remote-obj.c | 127 ++++++ hw/i386/remote.c | 79 ++++ hw/misc/ivshmem.c | 3 +- hw/pci-host/Makefile.objs | 1 + hw/pci-host/remote.c | 76 ++++ hw/pci/Makefile.objs | 2 + hw/pci/memory-sync.c | 211 +++++++++ hw/pci/proxy.c | 417 +++++++++++++++++ hw/remote/Makefile.objs | 1 + hw/remote/iohub.c | 123 +++++ include/exec/memory.h | 2 + include/exec/ram_addr.h | 2 +- include/hw/i386/remote-memory.h | 19 + include/hw/i386/remote-obj.h | 42 ++ include/hw/i386/remote.h | 35 ++ include/hw/pci-host/remote.h | 31 ++ include/hw/pci/memory-sync.h | 27 ++ include/hw/pci/pci_ids.h | 3 + include/hw/pci/proxy.h | 48 ++ include/hw/remote/iohub.h | 42 ++ include/io/channel.h | 24 + include/io/mpqemu-link.h | 123 +++++ include/qemu/mmap-alloc.h | 3 +- io/Makefile.objs | 2 + io/channel.c | 45 ++ io/mpqemu-link.c | 303 +++++++++++++ scripts/mpqemu-launcher.py | 49 ++ softmmu/memory.c | 3 +- util/mmap-alloc.c | 7 +- util/oslib-posix.c | 2 +- 41 files changed, 3252 insertions(+), 13 deletions(-) create mode 100644 docs/devel/multi-process.rst create mode 100644 docs/multi-process.rst create mode 100644 hw/i386/remote-memory.c create mode 100644 hw/i386/remote-msg.c create mode 100644 hw/i386/remote-obj.c create mode 100644 hw/i386/remote.c create mode 100644 hw/pci-host/remote.c create mode 100644 hw/pci/memory-sync.c create mode 100644 hw/pci/proxy.c create mode 100644 hw/remote/Makefile.objs create mode 100644 hw/remote/iohub.c create mode 100644 include/hw/i386/remote-memory.h create mode 100644 include/hw/i386/remote-obj.h create mode 100644 include/hw/i386/remote.h create mode 100644 include/hw/pci-host/remote.h create mode 100644 include/hw/pci/memory-sync.h create mode 100644 include/hw/pci/proxy.h create mode 100644 include/hw/remote/iohub.h create mode 100644 include/io/mpqemu-link.h create mode 100644 io/mpqemu-link.c create mode 100755 scripts/mpqemu-launcher.py
Signature · 1 line
1.8.3.1
PATCH v8 1/20 memory: alloc RAM from file at offset
Allow RAM MemoryRegion to be created from an offset in a file, instead of allocating at offset of 0 by default. This is needed to synchronize RAM between QEMU & remote process.
--- backends/hostmem-memfd.c | 2 +- exec.c | 11 +++++++---- hw/misc/ivshmem.c | 3 ++- include/exec/memory.h | 2 ++ include/exec/ram_addr.h | 2 +- include/qemu/mmap-alloc.h | 3 ++- softmmu/memory.c | 3 ++- util/mmap-alloc.c | 7 ++++--- util/oslib-posix.c | 2 +- 9 files changed, 22 insertions(+), 13 deletions(-)
backends/hostmem-memfd.c +1 -1
@@ -56,7 +56,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) name =host_memory_backend_get_name (backend);memory_region_init_ram_from_fd (&backend->mr,OBJECT (backend), name, backend->size, - backend->share, fd, errp); + backend->share, fd,0 , errp);g_free (name); }
exec.c +7 -4
@@ -1832,6 +1832,7 @@ static void *file_ram_alloc(RAMBlock *block,ram_addr_t memory, int fd, bool truncate, + off_t offset, Error **errp) { void *area;
@@ -1882,7 +1883,8 @@ static void *file_ram_alloc(RAMBlock *block, } area =qemu_ram_mmap (fd, memory, block->mr->align, - block->flags &RAM_SHARED , block->flags &RAM_PMEM ); + block->flags &RAM_SHARED , block->flags &RAM_PMEM , + offset);if (area ==MAP_FAILED ) {error_setg_errno (errp, errno,"unable to map backing store for guest RAM" );
@@ -2314,7 +2316,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)#ifdef CONFIG_POSIX RAMBlock *qemu_ram_alloc_from_fd (ram_addr_t size,MemoryRegion *mr,uint32_t ram_flags,int fd, -Error **errp) +off_t offset,Error **errp) {RAMBlock *new_block;Error *local_err =NULL ;
@@ -2367,7 +2369,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr, new_block->used_length = size; new_block->max_length = size; new_block->flags = ram_flags; - new_block->host =file_ram_alloc (new_block, size, fd, !file_size, errp); + new_block->host =file_ram_alloc (new_block, size, fd, !file_size, offset, + errp);if (!new_block->host) {g_free (new_block);return NULL ;
@@ -2397,7 +2400,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,return NULL ; } - block =qemu_ram_alloc_from_fd (size, mr, ram_flags, fd, errp); + block =qemu_ram_alloc_from_fd (size, mr, ram_flags, fd,0 , errp);if (!block) {if (created) {unlink (mem_path);
hw/misc/ivshmem.c +2 -1
@@ -492,7 +492,8 @@ static void process_msg_shmem(IVShmemState *s, int fd, Error **errp)/* mmap the region and map into the BAR2 */ memory_region_init_ram_from_fd (&s->server_bar2,OBJECT (s), -"ivshmem.bar2" , size, true, fd, &local_err); +"ivshmem.bar2" , size, true, fd,0 , + &local_err);if (local_err) {error_propagate (errp, local_err);return ;
include/exec/memory.h +2 -0
@@ -908,6 +908,7 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,* @size: size of the region. * @share: %true if memory must be mmaped with the MAP_SHARED flag * @fd: the fd to mmap. +* @offset: offset within the file referenced by fd * @errp: pointer to Error*, to store an error if it happens. * * Note that this function does not do anything to cause the data in the
@@ -919,6 +920,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,uint64_t size, bool share, int fd, + ram_addr_t offset, Error **errp);#endif
include/exec/ram_addr.h +1 -1
@@ -121,7 +121,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr, Error **errp);RAMBlock *qemu_ram_alloc_from_fd (ram_addr_t size,MemoryRegion *mr,uint32_t ram_flags,int fd, -Error **errp); +off_t offset,Error **errp);RAMBlock *qemu_ram_alloc_from_ptr (ram_addr_t size,void *host,MemoryRegion *mr,Error **errp);
include/qemu/mmap-alloc.h +2 -1
@@ -25,7 +25,8 @@ void *qemu_ram_mmap(int fd,size_t size, size_t align, bool shared, - bool is_pmem); + bool is_pmem, + off_t start);void qemu_ram_munmap (int fd,void *ptr,size_t size);
softmmu/memory.c +2 -1
@@ -1576,6 +1576,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,uint64_t size, bool share, int fd, + ram_addr_t offset, Error **errp) { Error *err = NULL;
@@ -1585,7 +1586,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr, mr->destructor = memory_region_destructor_ram; mr->ram_block =qemu_ram_alloc_from_fd (size, mr, share ?RAM_SHARED :0 , - fd, &err); + fd, offset, &err); mr->dirty_log_mask =tcg_enabled () ? (1 <<DIRTY_MEMORY_CODE ) :0 ;if (err) { mr->size =int128_zero ();
util/mmap-alloc.c +4 -3
@@ -86,7 +86,8 @@ void *qemu_ram_mmap(int fd,size_t size, size_t align, bool shared, - bool is_pmem) + bool is_pmem, + off_t start) { int flags;int map_sync_flags =0 ;
@@ -147,7 +148,7 @@ void *qemu_ram_mmap(int fd, offset =QEMU_ALIGN_UP ((uintptr_t )guardptr, align) - (uintptr_t )guardptr; ptr =mmap (guardptr + offset, size,PROT_READ |PROT_WRITE , - flags | map_sync_flags, fd,0 ); + flags | map_sync_flags, fd, start);if (ptr ==MAP_FAILED && map_sync_flags) {if (errno ==ENOTSUP ) {
@@ -172,7 +173,7 @@ void *qemu_ram_mmap(int fd,* we will remove these flags to handle compatibility. */ ptr =mmap (guardptr + offset, size,PROT_READ |PROT_WRITE , - flags, fd,0 ); + flags, fd, start); }if (ptr ==MAP_FAILED ) {
util/oslib-posix.c +1 -1
@@ -227,7 +227,7 @@ void *qemu_memalign(size_t alignment, size_t size)void *qemu_anon_ram_alloc (size_t size,uint64_t *alignment,bool shared) {size_t align =QEMU_VMALLOC_ALIGN ; -void *ptr =qemu_ram_mmap (-1 , size, align, shared, false); +void *ptr =qemu_ram_mmap (-1 , size, align, shared, false,0 );if (ptr ==MAP_FAILED ) {return NULL ;
Signature · 1 line
1.8.3.1
PATCH v8 2/20 multi-process: Add config option for multi-process QEMU
In reply to Jagannathan RamanAdd a configuration option to separate multi-process code
--- configure | 11 +++++++++++ 1 file changed, 11 insertions(+)
configure +11 -0
@@ -519,6 +519,7 @@ fuzzing="no" rng_none="no" secret_keyring="" libdaxctl="" +mpqemu="no" supported_cpu="no" supported_os="no"
@@ -1640,6 +1641,10 @@ for opt do ;; --disable-libdaxctl) libdaxctl=no ;; + --enable-mpqemu) mpqemu=yes + ;; + --disable-mpqemu) mpqemu=no + ;; *) echo "ERROR: unknown option $opt" echo "Try '$0 --help' for more information"
@@ -1943,6 +1948,8 @@ disabled with --disable-FEATURE, default is enabled if available: xkbcommon xkbcommon support rng-none dummy RNG, avoid using /dev/(u)random and getrandom() libdaxctl libdaxctl support + mpqemu multi-process QEMU support + NOTE: The object files are built at the place where configure is launched EOF
@@ -7019,6 +7026,7 @@ echo "fuzzing support $fuzzing" echo "gdb $gdb_bin" echo "rng-none $rng_none" echo "Linux keyring $secret_keyring" +echo "multiprocess QEMU $mpqemu" if test "$supported_cpu" = "no"; then echo
@@ -7891,6 +7899,9 @@ fi if test "$have_mlockall" = "yes" ; then echo "HAVE_MLOCKALL=y" >> $config_host_mak fi +if test "$mpqemu" = "yes" ; then + echo "CONFIG_MPQEMU=y" >> $config_host_mak +fi if test "$fuzzing" = "yes" ; then if test "$have_fuzzer" = "yes"; then FUZZ_LDFLAGS=" -fsanitize=fuzzer"
Signature · 1 line
1.8.3.1
PATCH v8 3/20 multi-process: setup PCI host bridge for remote device
In reply to Jagannathan RamanPCI host bridge is setup for the remote device process. It is implemented using remote-pcihost object. It is an extension of the PCI host bridge setup by QEMU.
Remote-pcihost configures a PCI bus which could be used by the remote PCI device to latch on to.
--- MAINTAINERS | 8 +++++ hw/pci-host/Makefile.objs | 1 + hw/pci-host/remote.c | 76 ++++++++++++++++++++++++++++++++++++++++++++ include/hw/pci-host/remote.h | 30 +++++++++++++++++ 4 files changed, 115 insertions(+) create mode 100644 hw/pci-host/remote.c create mode 100644 include/hw/pci-host/remote.h
MAINTAINERS +8 -0
@@ -3035,6 +3035,14 @@ S: Maintained F: hw/semihosting/ F: include/hw/semihosting/ +Multi-process QEMU +M: Jagannathan Raman <[email protected]> +M: Elena Ufimtseva <[email protected]> +M: John G Johnson <[email protected]> +S: Maintained +F: hw/pci-host/remote.c +F: include/hw/pci-host/remote.h + Build and test automation ------------------------- Build and test automation
hw/pci-host/Makefile.objs +1 -0
@@ -18,6 +18,7 @@ common-obj-$(CONFIG_XEN_IGD_PASSTHROUGH) += xen_igd_pt.o common-obj-$(CONFIG_PCI_EXPRESS_Q35) += q35.o common-obj-$(CONFIG_PCI_EXPRESS_GENERIC_BRIDGE) += gpex.o common-obj-$(CONFIG_PCI_EXPRESS_XILINX) += xilinx-pcie.o +common-obj-$(CONFIG_MPQEMU) += remote.o common-obj-$(CONFIG_PCI_EXPRESS_DESIGNWARE) += designware.o obj-$(CONFIG_POWERNV) += pnv_phb4.o pnv_phb4_pec.o
hw/pci-host/remote.c new file +76 -0
76 lines not on this page · raw message
include/hw/pci-host/remote.h new file +30 -0
@@ -0,0 +1,30 @@ +/* +* PCI Host for remote device +* +* Copyright © 2018, 2020 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef REMOTE_PCIHOST_H +#define REMOTE_PCIHOST_H + +#include "exec/memory.h" +#include "hw/pci/pcie_host.h" + +#define TYPE_REMOTE_HOST_DEVICE "remote-pcihost" +#define REMOTE_HOST_DEVICE (obj) \ + OBJECT_CHECK(RemotePCIHost, (obj), TYPE_REMOTE_HOST_DEVICE) + +typedef struct RemotePCIHost { +/*< private >*/ +PCIExpressHost parent_obj; +/*< public >*/ + +MemoryRegion *mr_pci_mem; +MemoryRegion *mr_sys_io; +}RemotePCIHost ; + +#endif
Signature · 1 line
1.8.3.1
Re: [PATCH v8 03/20] multi-process: setup PCI host bridge for remote device
Quoted 3 lines · Jagannathan Raman
+ * This PCI host is purely a container for PCI devices. It's fake in the
+ * sense that the guest never sees this PCI host and has no way of
+ * accessing it. It's job is just to provide the environment that QEMU
s/It's/Its/ :)
hw/pci-host/remote.c line 42
Quoted 3 lines
hw/pci-host/remote.c new file +3
-0+static void remote_pcihost_realize (DeviceState *dev,Error **errp) +{ +char *busname =g_strdup_printf ("remote-pci-%ld" , (unsigned long )getpid ());
Why is the pid used here? I don't understand why this helps. If additional instances are created in this process then the name won't be unique. The name isn't exposed outside the process, so there is no need to uniquely identify the process either.
g_free(busname) is missing. A nice way to fix this is:
g_autofree char *busname = ...
This avoids the need to manually call g_free().
PATCH v8 4/20 multi-process: setup a machine object for remote device process
In reply to Jagannathan Ramanremote-machine object sets up various subsystems of the remote device process. Instantiate PCI host bridge object and initialize RAM, IO & PCI memory regions.
--- MAINTAINERS | 2 ++ hw/i386/Makefile.objs | 1 + hw/i386/remote.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ include/hw/i386/remote.h | 28 ++++++++++++++++++ include/hw/pci-host/remote.h | 1 + 5 files changed, 101 insertions(+) create mode 100644 hw/i386/remote.c create mode 100644 include/hw/i386/remote.h
MAINTAINERS +2 -0
@@ -3042,6 +3042,8 @@ M: John G Johnson <[email protected]> S: Maintained F: hw/pci-host/remote.c F: include/hw/pci-host/remote.h +F: hw/i386/remote.c +F: include/hw/i386/remote.h Build and test automation -------------------------
hw/i386/Makefile.objs +1 -0
@@ -14,6 +14,7 @@ obj-$(CONFIG_XEN) += ../xenpv/ xen/ obj-$(CONFIG_VMPORT) += vmport.o obj-$(CONFIG_VMMOUSE) += vmmouse.o obj-$(CONFIG_PC) += port92.o +obj-$(CONFIG_MPQEMU) += remote.o obj-y += kvmvapic.o obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote.c new file +69 -0
69 lines not on this page · raw message
include/hw/i386/remote.h new file +28 -0
@@ -0,0 +1,28 @@ +/* +* Remote machine configuration +* +* Copyright © 2018, 2020 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef REMOTE_MACHINE_H +#define REMOTE_MACHINE_H + +#include "qom/object.h" +#include "hw/boards.h" +#include "hw/pci-host/remote.h" + +typedef struct RemoteMachineState { +MachineState parent_obj; + +RemotePCIHost *host; +}RemoteMachineState ; + +#define TYPE_REMOTE_MACHINE "remote-machine" +#define REMOTE_MACHINE (obj) \ + OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE) + +#endif
include/hw/pci-host/remote.h +1 -0
@@ -25,6 +25,7 @@ typedef struct RemotePCIHost {MemoryRegion *mr_pci_mem;MemoryRegion *mr_sys_io; + MemoryRegion *mr_sys_mem; } RemotePCIHost;#endif
Signature · 1 line
1.8.3.1
PATCH v8 5/20 multi-process: add qio channel function to transmit
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
The entire array of the memory regions and file handlers. Will be used in the next patch.
--- include/io/channel.h | 24 ++++++++++++++++++++++++ io/channel.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+)
include/io/channel.h +24 -0
@@ -779,5 +779,29 @@ void qio_channel_set_aio_fd_handler(QIOChannel *ioc, IOHandler *io_read, IOHandler *io_write, void *opaque); +/** +* qio_channel_writev_full_all: +* @ioc: the channel object +* @iov: the array of memory regions to write data from +* @niov: the length of the @iov array +* @fds: an array of file handles to send +* @nfds: number of file handles in @fds +* @errp: pointer to a NULL-initialized error object +* +* +* Behaves like qio_channel_writev_full but will attempt +* to send all data passed (file handles and memory regions). +* The function will wait for all requested data +* to be written, yielding from the current coroutine +* if required. +* +* Returns: 0 if all bytes were written, or -1 on error +*/ + +int qio_channel_writev_full_all (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +int *fds,size_t nfds, +Error **errp);#endif /* QIO_CHANNEL_H */
io/channel.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v8 6/20 multi-process: define MPQemuMsg format and transmission functions
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Defines MPQemuMsg, which is the message that is sent to the remote process. This message is sent over QIOChannel and is used to command the remote process to perform various tasks.
Also defined the helper functions to send and receive messages over the QIOChannel
--- MAINTAINERS | 2 + include/io/mpqemu-link.h | 73 ++++++++++++++++++++ io/Makefile.objs | 2 + io/mpqemu-link.c | 175 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+) create mode 100644 include/io/mpqemu-link.h create mode 100644 io/mpqemu-link.c
MAINTAINERS +2 -0
@@ -3044,6 +3044,8 @@ F: hw/pci-host/remote.c F: include/hw/pci-host/remote.h F: hw/i386/remote.c F: include/hw/i386/remote.h +F: io/mpqemu-link.c +F: include/io/mpqemu-link.h Build and test automation -------------------------
include/io/mpqemu-link.h new file +73 -0
73 lines not on this page · raw message
io/Makefile.objs +2 -0
@@ -10,3 +10,5 @@ io-obj-y += channel-util.o io-obj-y += dns-resolver.o io-obj-y += net-listener.o io-obj-y += task.o + +io-obj-$(CONFIG_MPQEMU) += mpqemu-link.o
io/mpqemu-link.c new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v8 06/20] multi-process: define MPQemuMsg format and transmission functions
io/mpqemu-link.c line 48
Quoted 2 lines · Jagannathan Raman
io/mpqemu-link.c new file +2
-0+static int mpqemu_readv (QIOChannel *ioc,struct iovec *iov,int **fds, +size_t *nfds,Error **errp)
readv(2) and similar functions take an int iovcnt argument while mpqemu_readv() takes just a single struct iovec. The name mpqemu_read() seems more appopriate and iov argument could be replaced by void *buf, size_t len. That is cleaner because this function modifies the iov argument (callers need to be be careful!).
Quoted 14 lines
io/mpqemu-link.c new file +14
-0+{ +struct iovec *l_iov = iov; +size_t *l_nfds = nfds; +unsigned int niov =1 ; +int **l_fds = fds; +size_t size, len; +Error *local_err =NULL ; + + size = iov->iov_len; + +while (size >0 ) { + len =qio_channel_readv_full (ioc, l_iov, niov, l_fds, l_nfds, + &local_err); +
io/mpqemu-link.c line 72
Quoted 10 lines
io/mpqemu-link.c new file +10
-0+if (len ==QIO_CHANNEL_ERR_BLOCK ) { +if (qemu_in_coroutine ()) { +qio_channel_yield (ioc,G_IO_IN ); + }else { +qio_channel_wait (ioc,G_IO_IN ); + } +continue ; + } + +if (len <=0 ) {
size_t is unsigned so this does not detect negative values. Please use qio_channel_readv_full()'s return type (ssize_t) instead.
io/mpqemu-link.c line 85
Quoted 13 lines
io/mpqemu-link.c new file +13
-0+error_propagate (errp, local_err); +return -EIO ; + } + + l_fds =NULL ; + l_nfds =NULL ; + + size -= len; + + (void )iov_discard_front (&l_iov, &niov, len); + } + +return iov->iov_len;
read()-style functions return the number of bytes read. mpqemu_readv() returns the number of unread bytes remaining. Maintaining consistency with read() function conventions will reduce the chance of bugs, so I suggest returning the number of bytes read instead.
Quoted 6 lines
io/mpqemu-link.c new file +6
-0+} + +void mpqemu_msg_recv (MPQemuMsg *msg,QIOChannel *ioc,Error **errp) +{ +Error *local_err =NULL ; +int *fds =NULL ;
io/mpqemu-link.c line 101
Quoted 10 lines
io/mpqemu-link.c new file +10
-0+struct iovec hdr, data; +size_t nfds =0 ; + + hdr.iov_base = msg; + hdr.iov_len =MPQEMU_MSG_HDR_SIZE ; + +if (mpqemu_readv (ioc, &hdr, &fds, &nfds, &local_err) <0 ) { +error_propagate (errp, local_err); +return ; + }
Can we read less than MPQEMU_MSG_HDR_SIZE if the peer closes the socket? That case probably needs to be handled.
io/mpqemu-link.c line 173
Quoted 4 lines
io/mpqemu-link.c new file +4
-0+void mpqemu_msg_cleanup (MPQemuMsg *msg) +{ +if (msg->data2) { +free (msg->data2);
The buffer was allocated with g_malloc0() so g_free() should be used to free it.
PATCH v8 7/20 multi-process: add co-routines to communicate with remote
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
process to avoid blocking the main loop during the message exchanges. To be used by proxy device.
--- include/io/mpqemu-link.h | 15 +++++++++ io/mpqemu-link.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+)
include/io/mpqemu-link.h +15 -0
@@ -14,6 +14,7 @@#include "qom/object.h" #include "qemu/thread.h" #include "io/channel.h" +#include "io/channel-socket.h" #define REMOTE_MAX_FDS 8
@@ -27,6 +28,7 @@*/ typedef enum {INIT =0 , +RET_MSG ,MAX =INT_MAX , }MPQemuCmd ;
@@ -64,6 +66,19 @@ typedef struct {uint8_t *data2; } MPQemuMsg; +struct MPQemuRequest { +MPQemuMsg *msg; +QIOChannel *ioc; +Coroutine *co; +bool finished; +int error; +long ret; +}; + +typedef struct MPQemuRequest MPQemuRequest ; + +uint64_t mpqemu_msg_send_and_await_reply (MPQemuMsg *msg,QIOChannel *ioc, +Error **errp);void mpqemu_msg_send (MPQemuMsg *msg,QIOChannel *ioc,Error **errp);void mpqemu_msg_recv (MPQemuMsg *msg,QIOChannel *ioc,Error **errp);
io/mpqemu-link.c +82 -0
94 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v8 07/20] multi-process: add co-routines to communicate with remote
Quoted 18 lines · Jagannathan Raman
io/mpqemu-link.c +18
-0+/* +* Create if needed and enter co-routine to send the message to the +* remote channel ioc and wait for the reply. +* Returns the value from the reply message, sets the error on failure. +*/ + +uint64_t mpqemu_msg_send_and_await_reply (MPQemuMsg *msg,QIOChannel *ioc, +Error **errp) +{ +MPQemuRequest req = {0 }; +uint64_t ret =UINT64_MAX ; + + req.ioc = ioc; +if (!req.ioc) { +error_setg (errp,"Channel is set to NULL" ); +return ret; + } +
io/mpqemu-link.c void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp) · line 207
Quoted 10 lines
io/mpqemu-link.c +10
-0+ req.msg = msg; + req.ret =0 ; + req.finished = false; + + req.co =qemu_coroutine_create (mpqemu_msg_send_co, &req); +qemu_coroutine_enter (req.co); + +while (!req.finished) { +aio_poll (qemu_get_aio_context (), true); + }
This is called from vcpu threads. aio_poll() does not release the global mutex so all other vcpu threads are blocked until mpqemu communication completes.
The simplest solution is to unlock the global mutex and call blocking, non-coroutine versions of the send/recv code. That way other vcpu threads can execute.
This means that the QEMU process would not use the mpqemu IOChannel in the event loop.
The remote process would still use the IOChannel in the event loop, however.
Stefan
PATCH v8 8/20 multi-process: Initialize message handler in remote device
In reply to Jagannathan RamanInitializes the message handler function in the remote process. It is called whenever there's an event pending on QIOChannel that registers this function.
--- MAINTAINERS | 1 + hw/i386/Makefile.objs | 1 + hw/i386/remote-msg.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++ include/hw/i386/remote.h | 4 +++ 4 files changed, 70 insertions(+) create mode 100644 hw/i386/remote-msg.c
MAINTAINERS +1 -0
@@ -3046,6 +3046,7 @@ F: hw/i386/remote.c F: include/hw/i386/remote.h F: io/mpqemu-link.c F: include/io/mpqemu-link.h +F: hw/i386/remote-msg.c Build and test automation -------------------------
hw/i386/Makefile.objs +1 -0
@@ -15,6 +15,7 @@ obj-$(CONFIG_VMPORT) += vmport.o obj-$(CONFIG_VMMOUSE) += vmmouse.o obj-$(CONFIG_PC) += port92.o obj-$(CONFIG_MPQEMU) += remote.o +obj-$(CONFIG_MPQEMU) += remote-msg.o obj-y += kvmvapic.o obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-msg.c new file +64 -0
64 lines not on this page · raw message
include/hw/i386/remote.h +4 -0
@@ -14,6 +14,7 @@#include "qom/object.h" #include "hw/boards.h" #include "hw/pci-host/remote.h" +#include "io/channel.h" typedef struct RemoteMachineState { MachineState parent_obj;
@@ -25,4 +26,7 @@ typedef struct RemoteMachineState {#define REMOTE_MACHINE (obj) \ OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE) +gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond, +gpointer opaque); +#endif
Signature · 1 line
1.8.3.1
Re: [PATCH v8 08/20] multi-process: Initialize message handler in remote device
Quoted 5 lines · Jagannathan Raman
hw/i386/remote-msg.c new file +5
-0+gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond, +gpointer opaque) +{ +PCIDevice *pci_dev = (PCIDevice *)opaque; +Error *local_err =NULL ;
hw/i386/remote-msg.c line 33
Quoted 10 lines
hw/i386/remote-msg.c new file +10
-0+MPQemuMsg msg = {0 }; + +if (cond &G_IO_HUP ) { +qemu_system_shutdown_request (SHUTDOWN_CAUSE_GUEST_SHUTDOWN ); +return FALSE; + } + +if (cond & (G_IO_ERR |G_IO_NVAL )) { +error_report ("Error %d while processing message from proxy \ +in remote process pid=%d" , errno,getpid ());
String literals don't wrap with backslash in C. Please use:
"Error %d while processing message from proxy "
"in remote process pid=%d", ...
instead.
hw/i386/remote-msg.c line 44
Quoted 11 lines
hw/i386/remote-msg.c new file +11
-0+return FALSE; + } + +mpqemu_msg_recv (&msg, ioc, &local_err); +if (local_err) { + goto exit; + } + +if (!mpqemu_msg_valid (&msg)) { +error_report ("Received invalid message from proxy \ +in remote process pid=%d" ,getpid ());
Same here.
PATCH v8 9/20 multi-process: Associate fd of a PCIDevice with its object
In reply to Jagannathan RamanAssociate the file descriptor for a PCIDevice in remote process with DeviceState object.
--- MAINTAINERS | 2 + hw/i386/Makefile.objs | 1 + hw/i386/remote-obj.c | 127 +++++++++++++++++++++++++++++++++++++++++++ include/hw/i386/remote-obj.h | 42 ++++++++++++++ 4 files changed, 172 insertions(+) create mode 100644 hw/i386/remote-obj.c create mode 100644 include/hw/i386/remote-obj.h
MAINTAINERS +2 -0
@@ -3047,6 +3047,8 @@ F: include/hw/i386/remote.h F: io/mpqemu-link.c F: include/io/mpqemu-link.h F: hw/i386/remote-msg.c +F: include/hw/i386/remote-obj.h +F: hw/i386/remote-obj.c Build and test automation -------------------------
hw/i386/Makefile.objs +1 -0
@@ -16,6 +16,7 @@ obj-$(CONFIG_VMMOUSE) += vmmouse.o obj-$(CONFIG_PC) += port92.o obj-$(CONFIG_MPQEMU) += remote.o obj-$(CONFIG_MPQEMU) += remote-msg.o +obj-$(CONFIG_MPQEMU) += remote-obj.o obj-y += kvmvapic.o obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-obj.c new file +127 -0
127 lines not on this page · raw message
include/hw/i386/remote-obj.h new file +42 -0
42 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Re: [PATCH v8 09/20] multi-process: Associate fd of a PCIDevice with its object
hw/i386/remote-obj.c line 36
Quoted 5 lines · Jagannathan Raman
hw/i386/remote-obj.c new file +5
-0+static void remote_object_set_devid (Object *obj,const char *str,Error **errp) +{ +RemoteObject *o =REMOTE_OBJECT (obj); + + o->devid =g_strdup (str);
Setter functions can be called multiple times so g_free(o->devid) is needed to prevent memory leaks.
hw/i386/remote-obj.c line 40
Quoted 4 lines
hw/i386/remote-obj.c new file +4
-0+} + +static void remote_object_machine_done (Notifier *notifier,void *data) +{
Is UserCreatableClass->complete() called too early? If you can use it instead of the machine init done notifier then that would make error reporting cleaner. Both command-line and object_add monitor commands would fail as expected if an error occurs, whereas this patch prints an error and continues running.
hw/i386/remote-obj.c line 50
Quoted 10 lines
hw/i386/remote-obj.c new file +10
-0+RemoteObject *o =container_of (notifier, RemoteObject, machine_done); +DeviceState *dev =NULL ; +QIOChannel *ioc =NULL ; +Error *err =NULL ; + + dev =qdev_find_recursive (sysbus_get_default (), o->devid); +if (!dev || !object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +error_report ("%s is not a PCI device" , o->devid); +return ; + }
What happens when a "device_del" monitor command removes the device? I guess a use-after-free will occur since this object isn't aware that the device disappeared. QOM's Object has a reference count that you can use to keep the object alive, but that still might not play well with hot unplug where the device is being removed from its bus. One solution is to refuse hot unplugging the device while the remote object exists.
hw/i386/remote-obj.c line 55
Quoted 5 lines
hw/i386/remote-obj.c new file +5
-0+ + ioc =qio_channel_new_fd (o->fd, &err); +if (!ioc) { +error_free (err); +error_report ("Failed to allocate IO channel" );
Please print err, it contains a more detailed error message than "Failed to allocate IO channel".
PATCH v8 10/20 multi-process: setup memory manager for remote device
In reply to Jagannathan RamanSyncSysMemMsg message format is defined. It is used to send file descriptors of the RAM regions to remote device.
RAM on the remote device is configured with a set of file descriptors. Old RAM regions are deleted and new regions, each with an fd, is added to the RAM.
--- MAINTAINERS | 2 ++ hw/i386/Makefile.objs | 1 + hw/i386/remote-memory.c | 58 +++++++++++++++++++++++++++++++++++++++++ include/hw/i386/remote-memory.h | 19 ++++++++++++++ include/io/mpqemu-link.h | 13 +++++++++ io/mpqemu-link.c | 13 +++++++++ 6 files changed, 106 insertions(+) create mode 100644 hw/i386/remote-memory.c create mode 100644 include/hw/i386/remote-memory.h
MAINTAINERS +2 -0
@@ -3049,6 +3049,8 @@ F: include/io/mpqemu-link.h F: hw/i386/remote-msg.c F: include/hw/i386/remote-obj.h F: hw/i386/remote-obj.c +F: include/hw/i386/remote-memory.h +F: hw/i386/remote-memory.c Build and test automation -------------------------
hw/i386/Makefile.objs +1 -0
@@ -17,6 +17,7 @@ obj-$(CONFIG_PC) += port92.o obj-$(CONFIG_MPQEMU) += remote.o obj-$(CONFIG_MPQEMU) += remote-msg.o obj-$(CONFIG_MPQEMU) += remote-obj.o +obj-$(CONFIG_MPQEMU) += remote-memory.o obj-y += kvmvapic.o obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-memory.c new file +58 -0
58 lines not on this page · raw message
include/hw/i386/remote-memory.h new file +19 -0
@@ -0,0 +1,19 @@ +/* +* Memory manager for remote device +* +* Copyright © 2018, 2020 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef REMOTE_MEMORY_H +#define REMOTE_MEMORY_H + +#include "exec/hwaddr.h" +#include "io/mpqemu-link.h" + +void remote_sysmem_reconfig (MPQemuMsg *msg,Error **errp); + +#endif
include/io/mpqemu-link.h +13 -0
@@ -15,6 +15,8 @@#include "qemu/thread.h" #include "io/channel.h" #include "io/channel-socket.h" +#include "exec/cpu-common.h" +#include "exec/hwaddr.h" #define REMOTE_MAX_FDS 8
@@ -25,13 +27,22 @@* * MPQemuCmd enum type to specify the command to be executed on the remote * device. +* +* SYNC_SYSMEM Shares QEMU's RAM with remote device's RAM */ typedef enum {INIT =0 , +SYNC_SYSMEM ,RET_MSG ,MAX =INT_MAX , }MPQemuCmd ; +typedef struct { +hwaddr gpas[REMOTE_MAX_FDS ]; +uint64_t sizes[REMOTE_MAX_FDS ]; +off_t offsets[REMOTE_MAX_FDS ]; +}SyncSysmemMsg ; +/** * Maximum size of data2 field in the message to be transmitted. */
@@ -50,6 +61,7 @@ typedef enum {* MPQemuMsg Format of the message sent to the remote device from QEMU. * */ + typedef struct { int cmd; int bytestream;
@@ -57,6 +69,7 @@ typedef struct {union {uint64_t u64; +SyncSysmemMsg sync_sysmem; } data1;int fds[REMOTE_MAX_FDS ];
io/mpqemu-link.c +13 -0
@@ -245,6 +245,19 @@ bool mpqemu_msg_valid(MPQemuMsg *msg) } } } +/* Verify message specific fields. */ +switch (msg->cmd) { +case SYNC_SYSMEM : +if (msg->num_fds ==0 || msg->bytestream) { +return false; + } +if (msg->size !=sizeof (msg->data1)) { +return false; + } +break ; +default : +break ; + }return true; }
Signature · 1 line
1.8.3.1
Re: [PATCH v8 10/20] multi-process: setup memory manager for remote device
hw/i386/remote-memory.c line 37
Quoted 6 lines · Jagannathan Raman
hw/i386/remote-memory.c new file +6
-0+ QTAILQ_FOREACH_SAFE(subregion, &sysmem->subregions, subregions_link, next) { +if (subregion->ram) { +memory_region_del_subregion (sysmem, subregion); +object_unparent (OBJECT (subregion)); + } + }
Where is g_free(subregion) called?
hw/i386/remote-memory.c line 50
Quoted 13 lines
hw/i386/remote-memory.c new file +13
-0+ +for (region =0 ; region < msg->num_fds; region++) { + subregion =g_new (MemoryRegion,1 ); + name =g_strdup_printf ("remote-mem-%u" , suffix++); +memory_region_init_ram_from_fd (subregion,NULL , + name, sysmem_info->sizes[region], +RAM_SHARED , msg->fds[region], + sysmem_info->offsets[region], + &local_err); +g_free (name); +if (local_err) { +error_propagate (errp, local_err); +break ;
Where is g_free(subregion) called?
PATCH v8 11/20 multi-process: introduce proxy object
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Defines a PCI Device proxy object as a child of TYPE_PCI_DEVICE.
--- MAINTAINERS | 2 ++ hw/pci/Makefile.objs | 1 + hw/pci/proxy.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++ include/hw/pci/proxy.h | 26 ++++++++++++++++ 4 files changed, 109 insertions(+) create mode 100644 hw/pci/proxy.c create mode 100644 include/hw/pci/proxy.h
MAINTAINERS +2 -0
@@ -3051,6 +3051,8 @@ F: include/hw/i386/remote-obj.h F: hw/i386/remote-obj.c F: include/hw/i386/remote-memory.h F: hw/i386/remote-memory.c +F: hw/pci/proxy.c +F: include/hw/pci/proxy.h Build and test automation -------------------------
hw/pci/Makefile.objs +1 -0
@@ -12,3 +12,4 @@ common-obj-$(CONFIG_PCI_EXPRESS) += pcie_port.o pcie_host.o common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o common-obj-$(CONFIG_ALL) += pci-stub.o +obj-$(CONFIG_MPQEMU) += proxy.o
hw/pci/proxy.c new file +80 -0
80 lines not on this page · raw message
include/hw/pci/proxy.h new file +26 -0
@@ -0,0 +1,26 @@ +/* +* Copyright © 2018, 2020 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef PROXY_H +#define PROXY_H + +#include "hw/pci/pci.h" +#include "io/channel.h" + +#define TYPE_PCI_PROXY_DEV "pci-proxy-dev" + +#define PCI_PROXY_DEV (obj) \ + OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV) + +typedef struct PCIProxyDev { +PCIDevice parent_dev; +char *fd; +QIOChannel *ioc; +}PCIProxyDev ; + +#endif /* PROXY_H */
Signature · 1 line
1.8.3.1
PATCH v8 12/20 multi-process: Forward PCI config space acceses to the remote process
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
The Proxy Object sends the PCI config space accesses as messages to the remote process over the communication channel
TODO:
Investigate if the local PCI config writes can be dropped. Without the proxy local PCI config space writes for the device, the driver in the guest times out on the probing.
We have tried to only refer to the remote for the PCI config writes, but the driver timeout in the guest forced as to left this as it is (removing local PCI config only).
--- hw/i386/remote-msg.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++ hw/pci/proxy.c | 55 ++++++++++++++++++++++++++++++++++++++++ include/io/mpqemu-link.h | 8 ++++++ io/mpqemu-link.c | 14 +++++++++++ 4 files changed, 142 insertions(+)
hw/i386/remote-msg.c +65 -0
@@ -15,6 +15,12 @@#include "io/mpqemu-link.h" #include "qapi/error.h" #include "sysemu/runstate.h" +#include "hw/pci/pci.h" + +static void process_config_write (QIOChannel *ioc,PCIDevice *dev, +MPQemuMsg *msg); +static void process_config_read (QIOChannel *ioc,PCIDevice *dev, +MPQemuMsg *msg);gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond,gpointer opaque)
@@ -46,6 +52,12 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond, }switch (msg.cmd) { +case PCI_CONFIG_WRITE : +process_config_write (ioc, pci_dev, &msg); +break ; +case PCI_CONFIG_READ : +process_config_read (ioc, pci_dev, &msg); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
@@ -62,3 +74,56 @@ exit:return TRUE; } + +static void process_config_write (QIOChannel *ioc,PCIDevice *dev, +MPQemuMsg *msg) +{ +ConfDataMsg *conf = (ConfDataMsg *)msg->data2; +MPQemuMsg ret = {0 }; +Error *local_err =NULL ; + +if ((conf->addr +sizeof (conf->val)) >pci_config_size (dev)) { +error_report ("Bad address received when writing PCI config, pid %d" , +getpid ()); + ret.data1.u64 =UINT64_MAX ; + }else { +pci_default_write_config (dev, conf->addr, conf->val, conf->l); + } + + ret.cmd =RET_MSG ; + ret.bytestream =0 ; + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_report ("Could not send message to proxy from pid %d" , +getpid ()); + } +} + +static void process_config_read (QIOChannel *ioc,PCIDevice *dev, +MPQemuMsg *msg) +{ +ConfDataMsg *conf = (ConfDataMsg *)msg->data2; +MPQemuMsg ret = {0 }; +Error *local_err =NULL ; + +if ((conf->addr +sizeof (conf->val)) >pci_config_size (dev)) { +error_report ("Bad address received when reading PCI config, pid %d" , +getpid ()); + ret.data1.u64 =UINT64_MAX ; + }else { + ret.data1.u64 =pci_default_read_config (dev, conf->addr, conf->l); + } + + ret.cmd =RET_MSG ; + ret.bytestream =0 ; + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_report ("Could not send message to proxy from pid %d" , +getpid ()); + } + +}
hw/pci/proxy.c +55 -0
73 lines not on this page · raw message
include/io/mpqemu-link.h +8 -0
@@ -34,6 +34,8 @@ typedef enum {INIT =0 ,SYNC_SYSMEM ,RET_MSG , +PCI_CONFIG_WRITE , +PCI_CONFIG_READ ,MAX =INT_MAX , } MPQemuCmd;
@@ -43,6 +45,12 @@ typedef struct {off_t offsets[REMOTE_MAX_FDS ]; } SyncSysmemMsg; +typedef struct { +uint32_t addr; +uint32_t val; +int l; +}ConfDataMsg ; +/** * Maximum size of data2 field in the message to be transmitted. */
io/mpqemu-link.c +14 -0
@@ -255,6 +255,20 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)return false; }break ; +case PCI_CONFIG_WRITE : +case PCI_CONFIG_READ : +if (msg->size !=sizeof (ConfDataMsg)) { +return false; + } +if (!msg->bytestream) { +return false; + } +ConfDataMsg *conf = (ConfDataMsg *)msg->data2; + +if (conf->l !=1 && conf->l !=2 && conf->l !=4 ) { +return false; + } +break ;default :break ; }
Signature · 1 line
1.8.3.1
PATCH v8 13/20 multi-process: PCI BAR read/write handling for proxy & remote endpoints
In reply to Jagannathan RamanProxy device object implements handler for PCI BAR writes and reads. The handler uses BAR_WRITE/BAR_READ message to communicate to the remote process with the BAR address and value to be written/read. The remote process implements handler for BAR_WRITE/BAR_READ message.
--- hw/i386/remote-msg.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++ hw/pci/proxy.c | 61 +++++++++++++++++++++++++++++++++ include/hw/pci/proxy.h | 16 +++++++-- include/io/mpqemu-link.h | 10 ++++++ io/mpqemu-link.c | 6 ++++ 5 files changed, 179 insertions(+), 2 deletions(-)
hw/i386/remote-msg.c +88 -0
@@ -16,11 +16,14 @@#include "qapi/error.h" #include "sysemu/runstate.h" #include "hw/pci/pci.h" +#include "exec/memattrs.h" static void process_config_write (QIOChannel *ioc,PCIDevice *dev,MPQemuMsg *msg);static void process_config_read (QIOChannel *ioc,PCIDevice *dev,MPQemuMsg *msg); +static void process_bar_write (QIOChannel *ioc,MPQemuMsg *msg,Error **errp); +static void process_bar_read (QIOChannel *ioc,MPQemuMsg *msg,Error **errp);gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond,gpointer opaque)
@@ -58,6 +61,12 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,case PCI_CONFIG_READ :process_config_read (ioc, pci_dev, &msg);break ; +case BAR_WRITE : +process_bar_write (ioc, &msg, &local_err); +break ; +case BAR_READ : +process_bar_read (ioc, &msg, &local_err); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
@@ -127,3 +136,82 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev, } } + +static void process_bar_write (QIOChannel *ioc,MPQemuMsg *msg,Error **errp) +{ +BarAccessMsg *bar_access = &msg->data1.bar_access; +AddressSpace *as = + bar_access->memory ? &address_space_memory : &address_space_io; +MPQemuMsg ret = {0 }; +MemTxResult res; +uint64_t val; +Error *local_err =NULL ; + +if (!is_power_of_2 (bar_access->size) || + (bar_access->size >sizeof (uint64_t ))) { + ret.data1.u64 =UINT64_MAX ; + goto fail; + } + + val =cpu_to_le64 (bar_access->val); + + res =address_space_rw (as, bar_access->addr,MEMTXATTRS_UNSPECIFIED , + (void *)&val, bar_access->size, true); + +if (res !=MEMTX_OK ) { +error_setg (errp,"Could not perform address space write operation," +" inaccessible address: %lx in pid %d." , + bar_access->addr,getpid ()); + ret.data1.u64 =-1 ; + } + +fail: + ret.cmd =RET_MSG ; + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_setg (errp,"Error while sending message to proxy " +"in remote process pid=%d" ,getpid ()); + } +} + +static void process_bar_read (QIOChannel *ioc,MPQemuMsg *msg,Error **errp) +{ +BarAccessMsg *bar_access = &msg->data1.bar_access; +MPQemuMsg ret = {0 }; +AddressSpace *as; +MemTxResult res; +uint64_t val =0 ; +Error *local_err =NULL ; + + as = bar_access->memory ? &address_space_memory : &address_space_io; + +if (!is_power_of_2 (bar_access->size) || + (bar_access->size >sizeof (uint64_t ))) { + val =UINT64_MAX ; + goto fail; + } + + res =address_space_rw (as, bar_access->addr,MEMTXATTRS_UNSPECIFIED , + (void *)&val, bar_access->size, false); + +if (res !=MEMTX_OK ) { +error_setg (errp,"Could not perform address space read operation," +" inaccessible address: %lx in pid %d." , + bar_access->addr,getpid ()); + val =UINT64_MAX ; + goto fail; + } + +fail: + ret.cmd =RET_MSG ; + ret.data1.u64 =le64_to_cpu (val); + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_setg (errp,"Error while sending message to proxy " +"in remote process pid=%d" ,getpid ()); + } +}
hw/pci/proxy.c +61 -0
64 lines not on this page · raw message
include/hw/pci/proxy.h +14 -2
@@ -17,10 +17,22 @@#define PCI_PROXY_DEV (obj) \ OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV) -typedef struct PCIProxyDev { +typedef struct PCIProxyDev PCIProxyDev ; + +typedef struct ProxyMemoryRegion { +PCIProxyDev *dev; +MemoryRegion mr; +bool memory; +bool present; +uint8_t type; +}ProxyMemoryRegion ; + +struct PCIProxyDev {PCIDevice parent_dev;char *fd;QIOChannel *ioc; -}PCIProxyDev ; + +ProxyMemoryRegion region[PCI_NUM_REGIONS ]; +};#endif /* PROXY_H */
include/io/mpqemu-link.h +10 -0
@@ -36,6 +36,8 @@ typedef enum {RET_MSG ,PCI_CONFIG_WRITE ,PCI_CONFIG_READ , +BAR_WRITE , +BAR_READ ,MAX =INT_MAX , } MPQemuCmd;
@@ -51,6 +53,13 @@ typedef struct {int l; } ConfDataMsg; +typedef struct { +hwaddr addr; +uint64_t val; +unsigned size; +bool memory; +}BarAccessMsg ; +/** * Maximum size of data2 field in the message to be transmitted. */
@@ -78,6 +87,7 @@ typedef struct {union {uint64_t u64;SyncSysmemMsg sync_sysmem; +BarAccessMsg bar_access; } data1;int fds[REMOTE_MAX_FDS ];
io/mpqemu-link.c +6 -0
@@ -269,6 +269,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)return false; }break ; +case BAR_WRITE : +case BAR_READ : +if ((msg->size !=sizeof (msg->data1)) || (msg->num_fds !=0 )) { +return false; + } +break ;default :break ; }
Signature · 1 line
1.8.3.1
Re: [PATCH v8 13/20] multi-process: PCI BAR read/write handling for proxy & remote endpoints
Quoted 26 lines · Jagannathan Raman
hw/i386/remote-msg.c +26
-0+static void process_bar_write (QIOChannel *ioc,MPQemuMsg *msg,Error **errp) +{ +BarAccessMsg *bar_access = &msg->data1.bar_access; +AddressSpace *as = + bar_access->memory ? &address_space_memory : &address_space_io; +MPQemuMsg ret = {0 }; +MemTxResult res; +uint64_t val; +Error *local_err =NULL ; + +if (!is_power_of_2 (bar_access->size) || + (bar_access->size >sizeof (uint64_t ))) { + ret.data1.u64 =UINT64_MAX ; + goto fail; + } + + val =cpu_to_le64 (bar_access->val); + + res =address_space_rw (as, bar_access->addr,MEMTXATTRS_UNSPECIFIED , + (void *)&val, bar_access->size, true); + +if (res !=MEMTX_OK ) { +error_setg (errp,"Could not perform address space write operation," +" inaccessible address: %lx in pid %d." , + bar_access->addr,getpid ()); + ret.data1.u64 =-1 ;
hw/i386/remote-msg.c static void process_config_read(QIOChannel *ioc, PCIDevice *dev, · line 175
Quoted 10 lines
hw/i386/remote-msg.c +10
-0+ } + +fail: + ret.cmd =RET_MSG ; + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_setg (errp,"Error while sending message to proxy " +"in remote process pid=%d" ,getpid ());
There is an assertion failure if res != MEMTX_OK because errp was already set. error_setg() must not be called on an Error pointer that has already been set.
It is simplest to do:
mpqemu_msg_send(&ret, ioc, (errp && *errp) ? NULL : &local_err);
Quoted 29 lines
hw/i386/remote-msg.c +29
-0+ } +} + +static void process_bar_read (QIOChannel *ioc,MPQemuMsg *msg,Error **errp) +{ +BarAccessMsg *bar_access = &msg->data1.bar_access; +MPQemuMsg ret = {0 }; +AddressSpace *as; +MemTxResult res; +uint64_t val =0 ; +Error *local_err =NULL ; + + as = bar_access->memory ? &address_space_memory : &address_space_io; + +if (!is_power_of_2 (bar_access->size) || + (bar_access->size >sizeof (uint64_t ))) { + val =UINT64_MAX ; + goto fail; + } + + res =address_space_rw (as, bar_access->addr,MEMTXATTRS_UNSPECIFIED , + (void *)&val, bar_access->size, false); + +if (res !=MEMTX_OK ) { +error_setg (errp,"Could not perform address space read operation," +" inaccessible address: %lx in pid %d." , + bar_access->addr,getpid ()); + val =UINT64_MAX ; + goto fail;
hw/i386/remote-msg.c static void process_config_read(QIOChannel *ioc, PCIDevice *dev, · line 214
Quoted 10 lines
hw/i386/remote-msg.c +10
-0+ } + +fail: + ret.cmd =RET_MSG ; + ret.data1.u64 =le64_to_cpu (val); + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { + error_setg(errp, "Error while sending message to proxy "
Same here.
hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 142
Quoted 6 lines
hw/pci/proxy.c +6
-0+static void send_bar_access_msg (PCIProxyDev *pdev,MemoryRegion *mr, +bool write,hwaddr addr,uint64_t *val, +unsigned size,bool memory) +{ +MPQemuMsg msg = {0 }; +long ret = -EINVAL ;
long is not guaranteed to be 64-bit. This function supports 64-bit accesses to BARs so uint64_t is needed here.
Quoted 8 lines
hw/pci/proxy.c +8
-0+Error *local_err =NULL ; + + msg.bytestream =0 ; + msg.size =sizeof (msg.data1); + msg.data1.bar_access.addr = mr->addr + addr; + msg.data1.bar_access.size = size; + msg.data1.bar_access.memory = memory; +
hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 160
Quoted 10 lines
hw/pci/proxy.c +10
-0+if (write) { + msg.cmd =BAR_WRITE ; + msg.data1.bar_access.val = *val; + }else { + msg.cmd =BAR_READ ; + } + + ret =mpqemu_msg_send_and_await_reply (&msg, pdev->ioc, &local_err); +if (local_err) { +error_report ("Failed to send BAR command to the remote process." );
Leaks local_err. Please report the error message from local_err and then free it.
hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 194
Quoted 7 lines
hw/pci/proxy.c +7
-0+const MemoryRegionOps proxy_mr_ops = { + .read = proxy_bar_read, + .write = proxy_bar_write, + .endianness =DEVICE_NATIVE_ENDIAN , + .impl = { + .min_access_size =1 , + .max_access_size =1 ,
Should this be .max_access_size = 8?
io/mpqemu-link.c bool mpqemu_msg_valid(MPQemuMsg *msg) · line 274
Quoted 11 lines
io/mpqemu-link.c +3
-0@@ -269,6 +269,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)return false; }break ; +case BAR_WRITE : +case BAR_READ : +if ((msg->size !=sizeof (msg->data1)) || (msg->num_fds !=0 )) {
What about bytestream? It would be cleanest not to send bytestream over the wire. Since it is sent today the receiver can be confused if it has the wrong value and some mpqemu_msg_valid() cases validate bytestream. It is not validate here for some reason.
PATCH v8 14/20 multi-process: Synchronize remote memory
In reply to Jagannathan RamanAdd memory-listener object which is used to keep the view of the RAM in sync between QEMU and remote process.
A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message.
--- MAINTAINERS | 2 + hw/i386/remote-msg.c | 4 + hw/pci/Makefile.objs | 1 + hw/pci/memory-sync.c | 211 +++++++++++++++++++++++++++++++++++++++++++ hw/pci/proxy.c | 6 ++ include/hw/pci/memory-sync.h | 27 ++++++ include/hw/pci/proxy.h | 3 + 7 files changed, 254 insertions(+) create mode 100644 hw/pci/memory-sync.c create mode 100644 include/hw/pci/memory-sync.h
MAINTAINERS +2 -0
@@ -3053,6 +3053,8 @@ F: include/hw/i386/remote-memory.h F: hw/i386/remote-memory.c F: hw/pci/proxy.c F: include/hw/pci/proxy.h +F: hw/pci/memory-sync.c +F: include/hw/pci/memory-sync.h Build and test automation -------------------------
hw/i386/remote-msg.c +4 -0
@@ -17,6 +17,7 @@#include "sysemu/runstate.h" #include "hw/pci/pci.h" #include "exec/memattrs.h" +#include "hw/i386/remote-memory.h" static void process_config_write (QIOChannel *ioc,PCIDevice *dev,MPQemuMsg *msg);
@@ -67,6 +68,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,case BAR_READ :process_bar_read (ioc, &msg, &local_err);break ; +case SYNC_SYSMEM : +remote_sysmem_reconfig (&msg, &local_err); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
hw/pci/Makefile.objs +1 -0
@@ -13,3 +13,4 @@ common-obj-$(CONFIG_PCI_EXPRESS) += pcie_port.o pcie_host.o common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o common-obj-$(CONFIG_ALL) += pci-stub.o obj-$(CONFIG_MPQEMU) += proxy.o +obj-$(CONFIG_MPQEMU) += memory-sync.o
hw/pci/memory-sync.c new file +211 -0
211 lines not on this page · raw message
hw/pci/proxy.c +6 -0
@@ -17,6 +17,8 @@#include "monitor/monitor.h" #include "io/mpqemu-link.h" #include "qemu/error-report.h" +#include "hw/pci/memory-sync.h" +#include "qom/object.h" static void proxy_set_socket (PCIProxyDev *pdev,int fd,Error **errp) {
@@ -44,6 +46,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)error_setg (errp,"fd parameter not specified for %s" ,DEVICE (device)->id); } + +configure_memory_sync (&dev->sync, dev->ioc); }static void pci_proxy_dev_exit (PCIDevice *pdev)
@@ -51,6 +55,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev)PCIProxyDev *dev =PCI_PROXY_DEV (pdev);qio_channel_close (dev->ioc,NULL ); + +deconfigure_memory_sync (&dev->sync); } static int config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val,
include/hw/pci/memory-sync.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +* Copyright © 2018, 2020 Oracle and/or its affiliates. +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef MEMORY_SYNC_H +#define MEMORY_SYNC_H + +#include "exec/memory.h" +#include "io/channel.h" + +typedef struct RemoteMemSync { +MemoryListener listener; + +int n_mr_sections; +MemoryRegionSection *mr_sections; + +QIOChannel *ioc; +}RemoteMemSync ; + +void configure_memory_sync (RemoteMemSync *sync,QIOChannel *ioc); +void deconfigure_memory_sync (RemoteMemSync *sync); + +#endif
include/hw/pci/proxy.h +3 -0
@@ -11,6 +11,7 @@#include "hw/pci/pci.h" #include "io/channel.h" +#include "hw/pci/memory-sync.h" #define TYPE_PCI_PROXY_DEV "pci-proxy-dev"
@@ -32,6 +33,8 @@ struct PCIProxyDev {char *fd;QIOChannel *ioc; +RemoteMemSync sync; +ProxyMemoryRegion region[PCI_NUM_REGIONS ]; };
Signature · 1 line
1.8.3.1
PATCH v8 15/20 multi-process: create IOHUB object to handle irq
In reply to Jagannathan RamanIOHUB object is added to manage PCI IRQs. It uses KVM_IRQFD ioctl to create irqfd to injecting PCI interrupts to the guest. IOHUB object forwards the irqfd to the remote process. Remote process uses this fd to directly send interrupts to the guest, bypassing QEMU.
--- MAINTAINERS | 2 + hw/Makefile.objs | 1 + hw/i386/remote-msg.c | 4 ++ hw/i386/remote.c | 10 ++++ hw/pci/proxy.c | 58 ++++++++++++++++++++++ hw/remote/Makefile.objs | 1 + hw/remote/iohub.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++ include/hw/i386/remote.h | 3 ++ include/hw/pci/pci_ids.h | 3 ++ include/hw/pci/proxy.h | 5 ++ include/hw/remote/iohub.h | 42 ++++++++++++++++ include/io/mpqemu-link.h | 1 + io/mpqemu-link.c | 5 ++ 13 files changed, 258 insertions(+) create mode 100644 hw/remote/Makefile.objs create mode 100644 hw/remote/iohub.c create mode 100644 include/hw/remote/iohub.h
MAINTAINERS +2 -0
@@ -3055,6 +3055,8 @@ F: hw/pci/proxy.c F: include/hw/pci/proxy.h F: hw/pci/memory-sync.c F: include/hw/pci/memory-sync.h +F: hw/remote/iohub.c +F: include/hw/remote/iohub.h Build and test automation -------------------------
hw/Makefile.objs +1 -0
@@ -40,6 +40,7 @@ devices-dirs-$(CONFIG_MEM_DEVICE) += mem/ devices-dirs-$(CONFIG_NUBUS) += nubus/ devices-dirs-y += semihosting/ devices-dirs-y += smbios/ +devices-dirs-y += remote/ endif common-obj-y += $(devices-dirs-y)
hw/i386/remote-msg.c +4 -0
@@ -18,6 +18,7 @@#include "hw/pci/pci.h" #include "exec/memattrs.h" #include "hw/i386/remote-memory.h" +#include "hw/remote/iohub.h" static void process_config_write (QIOChannel *ioc,PCIDevice *dev,MPQemuMsg *msg);
@@ -71,6 +72,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,case SYNC_SYSMEM :remote_sysmem_reconfig (&msg, &local_err);break ; +case SET_IRQFD : +process_set_irqfd_msg (pci_dev, &msg); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
hw/i386/remote.c +10 -0
@@ -20,12 +20,15 @@#include "exec/address-spaces.h" #include "exec/memory.h" #include "qapi/error.h" +#include "hw/pci/pci_host.h" +#include "hw/remote/iohub.h" static void remote_machine_init (MachineState *machine) {MemoryRegion *system_memory, *system_io, *pci_memory;RemoteMachineState *s =REMOTE_MACHINE (machine);RemotePCIHost *rem_host; +PCIHostState *pci_host; system_memory =get_system_memory (); system_io =get_system_io ();
@@ -45,6 +48,13 @@ static void remote_machine_init(MachineState *machine)memory_region_add_subregion_overlap (system_memory,0x0 , pci_memory,-1 );qdev_realize (DEVICE (rem_host),sysbus_get_default (), &error_fatal); + + pci_host =PCI_HOST_BRIDGE (rem_host); + +remote_iohub_init (&s->iohub); + +pci_bus_irqs (pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq, + &s->iohub,REMOTE_IOHUB_NB_PIRQS ); }static void remote_machine_class_init (ObjectClass *oc,void *data)
hw/pci/proxy.c +58 -0
82 lines not on this page · raw message
hw/remote/Makefile.objs new file +1 -0
@@ -0,0 +1 @@ +common-obj-$(CONFIG_MPQEMU) += iohub.o
hw/remote/iohub.c new file +123 -0
123 lines not on this page · raw message
include/hw/i386/remote.h +3 -0
@@ -15,11 +15,14 @@#include "hw/boards.h" #include "hw/pci-host/remote.h" #include "io/channel.h" +#include "hw/remote/iohub.h" typedef struct RemoteMachineState {MachineState parent_obj;RemotePCIHost *host; + +RemoteIOHubState iohub; }RemoteMachineState ;#define TYPE_REMOTE_MACHINE "remote-machine"
include/hw/pci/pci_ids.h +3 -0
@@ -192,6 +192,9 @@#define PCI_DEVICE_ID_SUN_SIMBA 0x5000#define PCI_DEVICE_ID_SUN_SABRE 0xa000 +#define PCI_VENDOR_ID_ORACLE 0x108e +#define PCI_DEVICE_ID_REMOTE_IOHUB 0xb000 +#define PCI_VENDOR_ID_CMD 0x1095#define PCI_DEVICE_ID_CMD_646 0x0646
include/hw/pci/proxy.h +5 -0
@@ -12,6 +12,7 @@#include "hw/pci/pci.h" #include "io/channel.h" #include "hw/pci/memory-sync.h" +#include "qemu/event_notifier.h" #define TYPE_PCI_PROXY_DEV "pci-proxy-dev"
@@ -35,6 +36,10 @@ struct PCIProxyDev {RemoteMemSync sync; +int virq; +EventNotifier intr; +EventNotifier resample; +ProxyMemoryRegion region[PCI_NUM_REGIONS ]; };
include/hw/remote/iohub.h new file +42 -0
42 lines not on this page · raw message
include/io/mpqemu-link.h +1 -0
@@ -38,6 +38,7 @@ typedef enum {PCI_CONFIG_READ ,BAR_WRITE ,BAR_READ , +SET_IRQFD ,MAX =INT_MAX , } MPQemuCmd;
io/mpqemu-link.c +5 -0
@@ -275,6 +275,11 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)return false; }break ; +case SET_IRQFD : +if (msg->size || (msg->num_fds !=2 )) { +return false; + } +break ;default :break ; }
Signature · 1 line
1.8.3.1
PATCH v8 16/20 multi-process: Retrieve PCI info from remote process
In reply to Jagannathan RamanRetrieve PCI configuration info about the remote device and configure the Proxy PCI object based on the returned information
--- hw/pci/proxy.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++ include/io/mpqemu-link.h | 1 + 2 files changed, 86 insertions(+)
hw/pci/proxy.c +85 -0
@@ -23,6 +23,8 @@#include "sysemu/kvm.h" #include "util/event_notifier-posix.c" +static void probe_pci_info (PCIDevice *dev,Error **errp); +static void proxy_set_socket (PCIProxyDev *pdev,int fd,Error **errp) { pdev->ioc =qio_channel_new_fd (fd, errp);
@@ -86,6 +88,7 @@ static void setup_irqfd(PCIProxyDev *dev)static void pci_proxy_dev_realize (PCIDevice *device,Error **errp) {PCIProxyDev *dev =PCI_PROXY_DEV (device); +uint8_t *pci_conf = device->config;int fd;if (dev->fd) {
@@ -100,9 +103,14 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp) DEVICE(device)->id); } + pci_conf[PCI_LATENCY_TIMER ] =0xff ; + pci_conf[PCI_INTERRUPT_PIN ] =0x01 ; +configure_memory_sync (&dev->sync, dev->ioc);setup_irqfd (dev); + +probe_pci_info (PCI_DEVICE (dev), errp); }static void pci_proxy_dev_exit (PCIDevice *pdev)
@@ -258,3 +266,80 @@ const MemoryRegionOps proxy_mr_ops = { .max_access_size = 1, }, }; + +static void probe_pci_info (PCIDevice *dev,Error **errp) +{ +PCIDeviceClass *pc =PCI_DEVICE_GET_CLASS (dev); +uint32_t orig_val, new_val, base_class, val; +PCIProxyDev *pdev =PCI_PROXY_DEV (dev); +DeviceClass *dc =DEVICE_CLASS (pc); +uint8_t type; +int i, size; +char *name; + +config_op_send (pdev,PCI_VENDOR_ID , &val,2 ,PCI_CONFIG_READ ); + pc->vendor_id = (uint16_t )val; + +config_op_send (pdev,PCI_DEVICE_ID , &val,2 ,PCI_CONFIG_READ ); + pc->device_id = (uint16_t )val; + +config_op_send (pdev,PCI_CLASS_DEVICE , &val,2 ,PCI_CONFIG_READ ); + pc->class_id = (uint16_t )val; + +config_op_send (pdev,PCI_SUBSYSTEM_ID , &val,2 ,PCI_CONFIG_READ ); + pc->subsystem_id = (uint16_t )val; + + base_class = pc->class_id >>4 ; +switch (base_class) { +case PCI_BASE_CLASS_BRIDGE : +set_bit (DEVICE_CATEGORY_BRIDGE , dc->categories); +break ; +case PCI_BASE_CLASS_STORAGE : +set_bit (DEVICE_CATEGORY_STORAGE , dc->categories); +break ; +case PCI_BASE_CLASS_NETWORK : +set_bit (DEVICE_CATEGORY_NETWORK , dc->categories); +break ; +case PCI_BASE_CLASS_INPUT : +set_bit (DEVICE_CATEGORY_INPUT , dc->categories); +break ; +case PCI_BASE_CLASS_DISPLAY : +set_bit (DEVICE_CATEGORY_DISPLAY , dc->categories); +break ; +case PCI_BASE_CLASS_PROCESSOR : +set_bit (DEVICE_CATEGORY_CPU , dc->categories); +break ; +default : +set_bit (DEVICE_CATEGORY_MISC , dc->categories); +break ; + } + +for (i =0 ; i <PCI_NUM_REGIONS ; i++) { +config_op_send (pdev,PCI_BASE_ADDRESS_0 + (4 * i), &orig_val,4 , +PCI_CONFIG_READ ); + new_val =0xffffffff ; +config_op_send (pdev,PCI_BASE_ADDRESS_0 + (4 * i), &new_val,4 , +PCI_CONFIG_WRITE ); +config_op_send (pdev,PCI_BASE_ADDRESS_0 + (4 * i), &new_val,4 , +PCI_CONFIG_READ ); + size = (~(new_val &0xFFFFFFF0 )) +1 ; +config_op_send (pdev,PCI_BASE_ADDRESS_0 + (4 * i), &orig_val,4 , +PCI_CONFIG_WRITE ); + type = (new_val &0x1 ) ? +PCI_BASE_ADDRESS_SPACE_IO :PCI_BASE_ADDRESS_SPACE_MEMORY ; + +if (size) { + pdev->region[i].dev = pdev; + pdev->region[i].present = true; +if (type ==PCI_BASE_ADDRESS_SPACE_MEMORY ) { + pdev->region[i].memory = true; + } + name =g_strdup_printf ("bar-region-%d" , i); +memory_region_init_io (&pdev->region[i].mr,OBJECT (pdev), + &proxy_mr_ops, &pdev->region[i], + name, size); +pci_register_bar (dev, i, type, &pdev->region[i].mr); +g_free (name); + } + } +}
include/io/mpqemu-link.h +1 -0
@@ -39,6 +39,7 @@ typedef enum {BAR_WRITE ,BAR_READ ,SET_IRQFD , +GET_PCI_INFO ,MAX =INT_MAX , } MPQemuCmd;
Signature · 1 line
1.8.3.1
PATCH v8 17/20 multi-process: heartbeat messages to remote
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
In order to detect remote processes which are hung, the proxy periodically sends heartbeat messages to confirm if the remote process is alive. The remote process responds to this heartbeat message to confirm it is alive.
--- hw/i386/remote-msg.c | 19 ++++++++++++++++++ hw/pci/proxy.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ include/hw/pci/proxy.h | 2 ++ include/io/mpqemu-link.h | 1 + io/mpqemu-link.c | 8 ++++++++ 5 files changed, 82 insertions(+)
hw/i386/remote-msg.c +19 -0
@@ -26,6 +26,7 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev, MPQemuMsg *msg);static void process_bar_write (QIOChannel *ioc,MPQemuMsg *msg,Error **errp);static void process_bar_read (QIOChannel *ioc,MPQemuMsg *msg,Error **errp); +static void process_proxy_ping_msg (QIOChannel *ioc,Error **errp);gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond,gpointer opaque)
@@ -75,6 +76,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,case SET_IRQFD :process_set_irqfd_msg (pci_dev, &msg);break ; +case PROXY_PING : +process_proxy_ping_msg (ioc, &local_err); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
@@ -223,3 +227,18 @@ fail: "in remote process pid=%d", getpid()); } } + +static void process_proxy_ping_msg (QIOChannel *ioc,Error **errp) +{ +MPQemuMsg ret = {0 }; +Error *local_err =NULL ; + + ret.cmd =RET_MSG ; + ret.size =sizeof (ret.data1); + +mpqemu_msg_send (&ret, ioc, &local_err); +if (local_err) { +error_setg (errp,"Error while sending message to proxy " +"in remote process pid=%d" ,getpid ()); + } +}
hw/pci/proxy.c +52 -0
73 lines not on this page · raw message
include/hw/pci/proxy.h +2 -0
@@ -40,6 +40,8 @@ struct PCIProxyDev {EventNotifier intr;EventNotifier resample; +QEMUTimer *hb_timer; +ProxyMemoryRegion region[PCI_NUM_REGIONS ]; };
include/io/mpqemu-link.h +1 -0
@@ -40,6 +40,7 @@ typedef enum {BAR_READ ,SET_IRQFD ,GET_PCI_INFO , +PROXY_PING ,MAX =INT_MAX , } MPQemuCmd;
io/mpqemu-link.c +8 -0
@@ -280,6 +280,14 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)return false; }break ; +case PROXY_PING : +if (msg->bytestream || msg->num_fds) { +return false; + } +if (msg->size) { +return false; + } +break ;default :break ; }
Signature · 1 line
1.8.3.1
Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote
Quoted 12 lines · Jagannathan Raman
hw/pci/proxy.c +8
-0@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp) } } } + +static void hb_msg (PCIProxyDev *dev) +{ +DeviceState *ds =DEVICE (dev); +Error *local_err =NULL ; +MPQemuMsg msg = {0 }; + + msg.cmd =PROXY_PING ;
hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369
Quoted 10 lines
hw/pci/proxy.c +10
-0+ msg.bytestream =0 ; + msg.size =0 ; + + (void )mpqemu_msg_send_and_await_reply (&msg, dev->ioc, &local_err); +if (local_err) { +error_report_err (local_err); +qio_channel_close (dev->ioc, &local_err); +error_setg (&error_fatal,"Lost contact with device %s" , ds->id); + } +}
Here is my feedback from the last revision. Was this addressed?
This patch seems incomplete since no action is taken when the device fails to respond. vCPU threads that access the device will still get stuck.
The simplest way to make this useful is to close the connection when a timeout occurs. Then the G_IO_HUP handler for the UNIX domain socket should perform connection cleanup. At that point there are a few choices:
1. Stop guest execution and wait for the host admin to restore the mplink so execution can resume. This is similar to how -drive rerror=stop pauses the guest when a disk I/O error is encountered.
2. Stop guest execution but defer it until this stale device is actually accessed. This maximizes guest uptime. Guests that rarely access the device may not notice at all.
3. Return 0 from MemoryRegion read operations and ignore writes. The guest continues executing but the device is broken. This is risky because device drivers inside the guest may not be ready to deal with this. The result could be data loss or corruption.
4. Raise a bus-level event. Maybe PCI error reporting can be used to offline the device.
5. Terminate the guest with an error message.
6. ?
Until the heartbeat is fully implemented and tested I suggest dropping it from this patch series. Remember the G_IO_HUP will happen anyway if the remote device process terminates.
Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote
Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jagannathan Raman
hw/pci/proxy.c +12
-0@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp) } } } + +static void hb_msg (PCIProxyDev *dev) +{ +DeviceState *ds =DEVICE (dev); +Error *local_err =NULL ; +MPQemuMsg msg = {0 }; + + msg.cmd =PROXY_PING ; + msg.bytestream =0 ; + msg.size =0 ; + + (void )mpqemu_msg_send_and_await_reply (&msg, dev->ioc, &local_err);
Quoted 9 lines
hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369
Quoted 6 lines
hw/pci/proxy.c +6
-0+if (local_err) { +error_report_err (local_err); +qio_channel_close (dev->ioc, &local_err); +error_setg (&error_fatal,"Lost contact with device %s" , ds->id); + } +}Here is my feedback from the last revision. Was this addressed?
Hi Stefan,
Thank you for reviewing the patchset. In this version we decided to shutdown the guest when the heartbeat did not get a reply from the remote by setting the error_fatal.
Should we approach it differently or you prefer us to get rid of the heartbeat in this form?
Thank you,
Elena
Quoted 32 lines
This patch seems incomplete since no action is taken when the device fails to respond. vCPU threads that access the device will still get stuck.
The simplest way to make this useful is to close the connection when a timeout occurs. Then the
G_IO_HUPhandler for the UNIX domain socket should perform connection cleanup. At that point there are a few choices:1. Stop guest execution and wait for the host admin to restore the mplink so execution can resume. This is similar to how -drive rerror=stop pauses the guest when a disk I/O error is encountered.
2. Stop guest execution but defer it until this stale device is actually accessed. This maximizes guest uptime. Guests that rarely access the device may not notice at all.
3. Return 0 from MemoryRegion read operations and ignore writes. The guest continues executing but the device is broken. This is risky because device drivers inside the guest may not be ready to deal with this. The result could be data loss or corruption.
4. Raise a bus-level event. Maybe PCI error reporting can be used to offline the device.
5. Terminate the guest with an error message.
6. ?
Until the heartbeat is fully implemented and tested I suggest dropping it from this patch series. Remember the
G_IO_HUPwill happen anyway if the remote device process terminates.
Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote
Quoted 28 lines · Elena Ufimtseva
Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jagannathan Raman
hw/pci/proxy.c +12
-0@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp) } } } + +static void hb_msg (PCIProxyDev *dev) +{ +DeviceState *ds =DEVICE (dev); +Error *local_err =NULL ; +MPQemuMsg msg = {0 }; + + msg.cmd =PROXY_PING ; + msg.bytestream =0 ; + msg.size =0 ; + + (void )mpqemu_msg_send_and_await_reply (&msg, dev->ioc, &local_err);Quoted 9 lines
hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369
Quoted 6 lines
hw/pci/proxy.c +6
-0+if (local_err) { +error_report_err (local_err); +qio_channel_close (dev->ioc, &local_err); +error_setg (&error_fatal,"Lost contact with device %s" , ds->id); + } +}Here is my feedback from the last revision. Was this addressed?
Quoted 7 lines
Hi Stefan,
Thank you for reviewing the patchset. In this version we decided to shutdown the guest when the heartbeat did not get a reply from the remote by setting the
error_fatal.
Should we approach it differently or you prefer us to get rid of the heartbeat in this form?
I think the only case that this patch handles is when the mpqemu channel is closed.
The VM hangs when the channel is still open but the remote is unresponsive. (mpqemu_msg_send_and_await_reply() calls aio_poll() with the global mutex held so vcpus cannot make progress.)
The heartbeat mechanism needs to handle the case where the other side isn't responding. It can't hang QEMU.
I suggest dropping this patch. It can be done later.
Stefan
PATCH v8 18/20 multi-process: perform device reset in the remote process
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
Perform device reset in the remote process when QEMU performs device reset. This is required to reset the internal state (like registers, etc...) of emulated devices
--- hw/i386/remote-msg.c | 22 ++++++++++++++++++++++ hw/pci/proxy.c | 20 ++++++++++++++++++++ include/io/mpqemu-link.h | 1 + 3 files changed, 43 insertions(+)
hw/i386/remote-msg.c +22 -0
@@ -19,6 +19,7 @@#include "exec/memattrs.h" #include "hw/i386/remote-memory.h" #include "hw/remote/iohub.h" +#include "sysemu/reset.h" static void process_config_write (QIOChannel *ioc,PCIDevice *dev,MPQemuMsg *msg);
@@ -27,6 +28,8 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,static void process_bar_write (QIOChannel *ioc,MPQemuMsg *msg,Error **errp);static void process_bar_read (QIOChannel *ioc,MPQemuMsg *msg,Error **errp);static void process_proxy_ping_msg (QIOChannel *ioc,Error **errp); +static void process_device_reset_msg (QIOChannel *ioc,PCIDevice *dev, +Error **errp);gboolean mpqemu_process_msg (QIOChannel *ioc,GIOCondition cond,gpointer opaque)
@@ -79,6 +82,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,case PROXY_PING :process_proxy_ping_msg (ioc, &local_err);break ; +case DEVICE_RESET : +process_device_reset_msg (ioc, pci_dev, &local_err); +break ;default : error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)",
@@ -242,3 +248,19 @@ static void process_proxy_ping_msg(QIOChannel *ioc, Error **errp) "in remote process pid=%d", getpid()); } } + +static void process_device_reset_msg (QIOChannel *ioc,PCIDevice *dev, +Error **errp) +{ +DeviceClass *dc =DEVICE_GET_CLASS (dev); +DeviceState *s =DEVICE (dev); +MPQemuMsg ret = {0 }; + +if (dc->reset) { + dc->reset (s); + } + + ret.cmd =RET_MSG ; + +mpqemu_msg_send (&ret, ioc, errp); +}
hw/pci/proxy.c +20 -0
@@ -26,6 +26,7 @@static void probe_pci_info (PCIDevice *dev,Error **errp);static void start_hb_timer (PCIProxyDev *dev);static void stop_hb_timer (PCIProxyDev *dev); +static void proxy_device_reset (DeviceState *dev);static void proxy_set_socket (PCIProxyDev *pdev,int fd,Error **errp) {
@@ -191,6 +192,8 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data) k->config_read = pci_proxy_read_config; k->config_write = pci_proxy_write_config; + dc->reset = proxy_device_reset; +device_class_set_props (dc, proxy_properties); }
@@ -395,3 +398,20 @@ static void stop_hb_timer(PCIProxyDev *dev)timer_del (dev->hb_timer);timer_free (dev->hb_timer); } + +static void proxy_device_reset (DeviceState *dev) +{ +PCIProxyDev *pdev =PCI_PROXY_DEV (dev); +MPQemuMsg msg = {0 }; +Error *local_err =NULL ; + + msg.bytestream =0 ; + msg.size =sizeof (msg.data1); + msg.cmd =DEVICE_RESET ; + + (void )mpqemu_msg_send_and_await_reply (&msg, pdev->ioc, &local_err); +if (local_err) { +error_report ("Failed to send DEVICE_RESET to the remote process" ); + } + +}
include/io/mpqemu-link.h +1 -0
@@ -41,6 +41,7 @@ typedef enum {SET_IRQFD ,GET_PCI_INFO ,PROXY_PING , +DEVICE_RESET ,MAX =INT_MAX , } MPQemuCmd;
Signature · 1 line
1.8.3.1
PATCH v8 19/20 multi-process: add the concept description to docs/devel/qemu-multiprocess
In reply to Jagannathan RamanFrom: John G Johnson <[email protected]>
--- MAINTAINERS | 1 + docs/devel/index.rst | 1 + docs/devel/multi-process.rst | 966 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 968 insertions(+) create mode 100644 docs/devel/multi-process.rst
MAINTAINERS +1 -0
@@ -3057,6 +3057,7 @@ F: hw/pci/memory-sync.c F: include/hw/pci/memory-sync.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h +F: docs/devel/multi-process.rst Build and test automation -------------------------
docs/devel/index.rst +1 -0
@@ -30,3 +30,4 @@ Contents: reset s390-dasd-ipl clocks + multi-process
docs/devel/multi-process.rst new file +966 -0
966 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v8 20/20 multi-process: add configure and usage information
In reply to Jagannathan RamanFrom: Elena Ufimtseva <[email protected]>
--- MAINTAINERS | 2 ++ docs/multi-process.rst | 67 ++++++++++++++++++++++++++++++++++++++++++++++ scripts/mpqemu-launcher.py | 49 +++++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 docs/multi-process.rst create mode 100755 scripts/mpqemu-launcher.py
MAINTAINERS +2 -0
@@ -3058,6 +3058,8 @@ F: include/hw/pci/memory-sync.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h F: docs/devel/multi-process.rst +F: scripts/mpqemu-launcher.py +F: scripts/mpqemu-launcher-perf-mode.py Build and test automation -------------------------
docs/multi-process.rst new file +67 -0
67 lines not on this page · raw message
scripts/mpqemu-launcher.py new file +49 -0
49 lines not on this page · raw message
Signature · 1 line
1.8.3.1
I have finished reviewing this revision.
Please update the wiki page with the latest status, especially limitations like no dirty memory logging (used by migration, TCG, VGA), no MSI interrupts, no PCIe, no IOMMU support, etc.
https://wiki.qemu.org/Features/MultiProcessQEMU
Please name the machine type x-remote and the object -object x-remote-object. "x-" is the convention for experimental features in QEMU. It means that the interface is not stable and might change between releases.
Stefan