Skip to contentSkip to conversation

Initial support for multi-process qemu

PATCH Superseded Jagannathan Raman 3 participants 32 messages Jul 31, 2020 – Aug 19, 2020 Arm STM32

v8 of 21 Superseded Latest is v21 →

lore
Share & tools
lore
#0 lore raw

v8 cover Initial support for multi-process qemu

Hello

This is the v8 of the patchset. Thank you very much for the detailed feedback for v7. We appreciate your time. We believe we have address all the comments for v7 in the current series.

The broader items we have addressed in this series are as follows:
- Removed the main channel / control channel.
- Enabled 1:1 mapping between irqfd:PCIDevice, moving away from shared interrupt lines to avoid collision between irqfds
- Defined an object type named “remote-object” that connects the device with its fd in the remote process. This object limits the number of remote devices to 1 per process to alleviate security concerns with multiple devices per process. We are addressing the authentication mechanism for multiple devices in the VFIO-over-socket proposal.

To touch upon the brief history of this project, we posted the Proof Of Concept patches before the BoF session in 2018. Subsequently, we posted RFC v1 [1], RFC v2 [2], RFC v3 [3], RFC v4 [4], v5 [5], v6 [6] and v7 [7] of the patch series. Following people contributed to the design and implementation of this project:

Stefan Hajnoczi <[email protected]>
Konrad Wilk <[email protected]>
Kanth Ghatraju <[email protected]>
John G Johnson <[email protected]>
Elena Ufimtseva <[email protected]>
Jagannathan Raman <[email protected]>

We would like to thank QEMU community for your feedback in the design and implementation of this project.

For the full concept writeup about QEMU multi-process, please refer to docs/devel/qemu-multiprocess.rst. Also see docs/qemu-multiprocess.txt for usage information.

We welcome all your ideas, concerns, and questions for this patchset.

[POC]: https://www.mail-archive.com/[email protected]/msg566538.html
[1]: https://www.mail-archive.com/[email protected]/msg602285.html
[2]: https://www.mail-archive.com/[email protected]/msg624877.html
[3]: https://www.mail-archive.com/[email protected]/msg642000.html
[4]: https://www.mail-archive.com/[email protected]/msg655118.html
[5]: https://www.mail-archive.com/[email protected]/msg682429.html
[6]: https://www.mail-archive.com/[email protected]/msg697484.html
[7]: https://patchew.org/QEMU/[email protected]/

Elena Ufimtseva (8):
multi-process: add qio channel function to transmit
multi-process: define MPQemuMsg format and transmission functions
multi-process: add co-routines to communicate with remote
multi-process: introduce proxy object
multi-process: Forward PCI config space acceses to the remote process
multi-process: heartbeat messages to remote
multi-process: perform device reset in the remote process
multi-process: add configure and usage information

Jagannathan Raman (11):
memory: alloc RAM from file at offset
multi-process: Add config option for multi-process QEMU
multi-process: setup PCI host bridge for remote device
multi-process: setup a machine object for remote device process
multi-process: Initialize message handler in remote device
multi-process: Associate fd of a PCIDevice with its object
multi-process: setup memory manager for remote device
multi-process: PCI BAR read/write handling for proxy & remote
endpoints
multi-process: Synchronize remote memory
multi-process: create IOHUB object to handle irq
multi-process: Retrieve PCI info from remote process

John G Johnson (1):
multi-process: add the concept description to
docs/devel/qemu-multiprocess

 MAINTAINERS                     |  26 ++
 backends/hostmem-memfd.c        |   2 +-
 configure                       |  11 +
 docs/devel/index.rst            |   1 +
 docs/devel/multi-process.rst    | 966 ++++++++++++++++++++++++++++++++++++++++
 docs/multi-process.rst          |  67 +++
 exec.c                          |  11 +-
 hw/Makefile.objs                |   1 +
 hw/i386/Makefile.objs           |   4 +
 hw/i386/remote-memory.c         |  58 +++
 hw/i386/remote-msg.c            | 266 +++++++++++
 hw/i386/remote-obj.c            | 127 ++++++
 hw/i386/remote.c                |  79 ++++
 hw/misc/ivshmem.c               |   3 +-
 hw/pci-host/Makefile.objs       |   1 +
 hw/pci-host/remote.c            |  76 ++++
 hw/pci/Makefile.objs            |   2 +
 hw/pci/memory-sync.c            | 211 +++++++++
 hw/pci/proxy.c                  | 417 +++++++++++++++++
 hw/remote/Makefile.objs         |   1 +
 hw/remote/iohub.c               | 123 +++++
 include/exec/memory.h           |   2 +
 include/exec/ram_addr.h         |   2 +-
 include/hw/i386/remote-memory.h |  19 +
 include/hw/i386/remote-obj.h    |  42 ++
 include/hw/i386/remote.h        |  35 ++
 include/hw/pci-host/remote.h    |  31 ++
 include/hw/pci/memory-sync.h    |  27 ++
 include/hw/pci/pci_ids.h        |   3 +
 include/hw/pci/proxy.h          |  48 ++
 include/hw/remote/iohub.h       |  42 ++
 include/io/channel.h            |  24 +
 include/io/mpqemu-link.h        | 123 +++++
 include/qemu/mmap-alloc.h       |   3 +-
 io/Makefile.objs                |   2 +
 io/channel.c                    |  45 ++
 io/mpqemu-link.c                | 303 +++++++++++++
 scripts/mpqemu-launcher.py      |  49 ++
 softmmu/memory.c                |   3 +-
 util/mmap-alloc.c               |   7 +-
 util/oslib-posix.c              |   2 +-
 41 files changed, 3252 insertions(+), 13 deletions(-)
 create mode 100644 docs/devel/multi-process.rst
 create mode 100644 docs/multi-process.rst
 create mode 100644 hw/i386/remote-memory.c
 create mode 100644 hw/i386/remote-msg.c
 create mode 100644 hw/i386/remote-obj.c
 create mode 100644 hw/i386/remote.c
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 hw/pci/memory-sync.c
 create mode 100644 hw/pci/proxy.c
 create mode 100644 hw/remote/Makefile.objs
 create mode 100644 hw/remote/iohub.c
 create mode 100644 include/hw/i386/remote-memory.h
 create mode 100644 include/hw/i386/remote-obj.h
 create mode 100644 include/hw/i386/remote.h
 create mode 100644 include/hw/pci-host/remote.h
 create mode 100644 include/hw/pci/memory-sync.h
 create mode 100644 include/hw/pci/proxy.h
 create mode 100644 include/hw/remote/iohub.h
 create mode 100644 include/io/mpqemu-link.h
 create mode 100644 io/mpqemu-link.c
 create mode 100755 scripts/mpqemu-launcher.py
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v8 1/20 memory: alloc RAM from file at offset

Allow RAM MemoryRegion to be created from an offset in a file, instead of allocating at offset of 0 by default. This is needed to synchronize RAM between QEMU & remote process.

---
 backends/hostmem-memfd.c  |  2 +-
 exec.c                    | 11 +++++++----
 hw/misc/ivshmem.c         |  3 ++-
 include/exec/memory.h     |  2 ++
 include/exec/ram_addr.h   |  2 +-
 include/qemu/mmap-alloc.h |  3 ++-
 softmmu/memory.c          |  3 ++-
 util/mmap-alloc.c         |  7 ++++---
 util/oslib-posix.c        |  2 +-
 9 files changed, 22 insertions(+), 13 deletions(-)
backends/hostmem-memfd.c +1 -1
index 4c040a7..19c4f63 100644
@@ -56,7 +56,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     name = host_memory_backend_get_name(backend);
     memory_region_init_ram_from_fd(&backend->mr, OBJECT(backend),
                                    name, backend->size,
-                                   backend->share, fd, errp);
+                                   backend->share, fd, 0, errp);
     g_free(name);
 }
 
exec.c +7 -4
index 6f381f9..564a6b7 100644
@@ -1832,6 +1832,7 @@ static void *file_ram_alloc(RAMBlock *block,
                             ram_addr_t memory,
                             int fd,
                             bool truncate,
+                            off_t offset,
                             Error **errp)
 {
     void *area;
@@ -1882,7 +1883,8 @@ static void *file_ram_alloc(RAMBlock *block,
     }
 
     area = qemu_ram_mmap(fd, memory, block->mr->align,
-                         block->flags & RAM_SHARED, block->flags & RAM_PMEM);
+                         block->flags & RAM_SHARED, block->flags & RAM_PMEM,
+                         offset);
     if (area == MAP_FAILED) {
         error_setg_errno(errp, errno,
                          "unable to map backing store for guest RAM");
@@ -2314,7 +2316,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)
 #ifdef CONFIG_POSIX
 RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  uint32_t ram_flags, int fd,
-                                 Error **errp)
+                                 off_t offset, Error **errp)
 {
     RAMBlock *new_block;
     Error *local_err = NULL;
@@ -2367,7 +2369,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
     new_block->used_length = size;
     new_block->max_length = size;
     new_block->flags = ram_flags;
-    new_block->host = file_ram_alloc(new_block, size, fd, !file_size, errp);
+    new_block->host = file_ram_alloc(new_block, size, fd, !file_size, offset,
+                                     errp);
     if (!new_block->host) {
         g_free(new_block);
         return NULL;
@@ -2397,7 +2400,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,
         return NULL;
     }
 
-    block = qemu_ram_alloc_from_fd(size, mr, ram_flags, fd, errp);
+    block = qemu_ram_alloc_from_fd(size, mr, ram_flags, fd, 0, errp);
     if (!block) {
         if (created) {
             unlink(mem_path);
hw/misc/ivshmem.c +2 -1
index 2b6882f..5ffaa72 100644
@@ -492,7 +492,8 @@ static void process_msg_shmem(IVShmemState *s, int fd, Error **errp)
 
     /* mmap the region and map into the BAR2 */
     memory_region_init_ram_from_fd(&s->server_bar2, OBJECT(s),
-                                   "ivshmem.bar2", size, true, fd, &local_err);
+                                   "ivshmem.bar2", size, true, fd, 0,
+                                   &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
         return;
include/exec/memory.h +2 -0
index 307e527..608a73f 100644
@@ -908,6 +908,7 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,
  * @size: size of the region.
  * @share: %true if memory must be mmaped with the MAP_SHARED flag
  * @fd: the fd to mmap.
+ * @offset: offset within the file referenced by fd
  * @errp: pointer to Error*, to store an error if it happens.
  *
  * Note that this function does not do anything to cause the data in the
@@ -919,6 +920,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
                                     uint64_t size,
                                     bool share,
                                     int fd,
+                                    ram_addr_t offset,
                                     Error **errp);
 #endif
 
include/exec/ram_addr.h +1 -1
index 3ef729a..34b2bce 100644
@@ -121,7 +121,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,
                                    Error **errp);
 RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  uint32_t ram_flags, int fd,
-                                 Error **errp);
+                                 off_t offset, Error **errp);
 
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                   MemoryRegion *mr, Error **errp);
include/qemu/mmap-alloc.h +2 -1
index e786266..4f57985 100644
@@ -25,7 +25,8 @@ void *qemu_ram_mmap(int fd,
                     size_t size,
                     size_t align,
                     bool shared,
-                    bool is_pmem);
+                    bool is_pmem,
+                    off_t start);
 
 void qemu_ram_munmap(int fd, void *ptr, size_t size);
 
softmmu/memory.c +2 -1
index af25987..993c5b3 100644
@@ -1576,6 +1576,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
                                     uint64_t size,
                                     bool share,
                                     int fd,
+                                    ram_addr_t offset,
                                     Error **errp)
 {
     Error *err = NULL;
@@ -1585,7 +1586,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
     mr->destructor = memory_region_destructor_ram;
     mr->ram_block = qemu_ram_alloc_from_fd(size, mr,
                                            share ? RAM_SHARED : 0,
-                                           fd, &err);
+                                           fd, offset, &err);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
     if (err) {
         mr->size = int128_zero();
util/mmap-alloc.c +4 -3
index 27dcccd..a28f702 100644
@@ -86,7 +86,8 @@ void *qemu_ram_mmap(int fd,
                     size_t size,
                     size_t align,
                     bool shared,
-                    bool is_pmem)
+                    bool is_pmem,
+                    off_t start)
 {
     int flags;
     int map_sync_flags = 0;
@@ -147,7 +148,7 @@ void *qemu_ram_mmap(int fd,
     offset = QEMU_ALIGN_UP((uintptr_t)guardptr, align) - (uintptr_t)guardptr;
 
     ptr = mmap(guardptr + offset, size, PROT_READ | PROT_WRITE,
-               flags | map_sync_flags, fd, 0);
+               flags | map_sync_flags, fd, start);
 
     if (ptr == MAP_FAILED && map_sync_flags) {
         if (errno == ENOTSUP) {
@@ -172,7 +173,7 @@ void *qemu_ram_mmap(int fd,
          * we will remove these flags to handle compatibility.
          */
         ptr = mmap(guardptr + offset, size, PROT_READ | PROT_WRITE,
-                   flags, fd, 0);
+                   flags, fd, start);
     }
 
     if (ptr == MAP_FAILED) {
util/oslib-posix.c +1 -1
index ad8001a..72ad0b9 100644
@@ -227,7 +227,7 @@ void *qemu_memalign(size_t alignment, size_t size)
 void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment, bool shared)
 {
     size_t align = QEMU_VMALLOC_ALIGN;
-    void *ptr = qemu_ram_mmap(-1, size, align, shared, false);
+    void *ptr = qemu_ram_mmap(-1, size, align, shared, false, 0);
 
     if (ptr == MAP_FAILED) {
         return NULL;
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v8 2/20 multi-process: Add config option for multi-process QEMU

In reply to Jagannathan Raman

Add a configuration option to separate multi-process code

---
 configure | 11 +++++++++++
 1 file changed, 11 insertions(+)
configure +11 -0
index 2acc4d1..7b615c5 100755
@@ -519,6 +519,7 @@ fuzzing="no"
 rng_none="no"
 secret_keyring=""
 libdaxctl=""
+mpqemu="no"
 
 supported_cpu="no"
 supported_os="no"
@@ -1640,6 +1641,10 @@ for opt do
   ;;
   --disable-libdaxctl) libdaxctl=no
   ;;
+  --enable-mpqemu) mpqemu=yes
+  ;;
+  --disable-mpqemu) mpqemu=no
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1943,6 +1948,8 @@ disabled with --disable-FEATURE, default is enabled if available:
   xkbcommon       xkbcommon support
   rng-none        dummy RNG, avoid using /dev/(u)random and getrandom()
   libdaxctl       libdaxctl support
+  mpqemu          multi-process QEMU support
+
 
 NOTE: The object files are built at the place where configure is launched
 EOF
@@ -7019,6 +7026,7 @@ echo "fuzzing support   $fuzzing"
 echo "gdb               $gdb_bin"
 echo "rng-none          $rng_none"
 echo "Linux keyring     $secret_keyring"
+echo "multiprocess QEMU $mpqemu"
 
 if test "$supported_cpu" = "no"; then
     echo
@@ -7891,6 +7899,9 @@ fi
 if test "$have_mlockall" = "yes" ; then
   echo "HAVE_MLOCKALL=y" >> $config_host_mak
 fi
+if test "$mpqemu" = "yes" ; then
+  echo "CONFIG_MPQEMU=y" >> $config_host_mak
+fi
 if test "$fuzzing" = "yes" ; then
   if test "$have_fuzzer" = "yes"; then
     FUZZ_LDFLAGS=" -fsanitize=fuzzer"
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH v8 3/20 multi-process: setup PCI host bridge for remote device

In reply to Jagannathan Raman

PCI host bridge is setup for the remote device process. It is implemented using remote-pcihost object. It is an extension of the PCI host bridge setup by QEMU.
Remote-pcihost configures a PCI bus which could be used by the remote PCI device to latch on to.

---
 MAINTAINERS                  |  8 +++++
 hw/pci-host/Makefile.objs    |  1 +
 hw/pci-host/remote.c         | 76 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci-host/remote.h | 30 +++++++++++++++++
 4 files changed, 115 insertions(+)
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 include/hw/pci-host/remote.h
MAINTAINERS +8 -0
index 0886eb3..d781a28 100644
@@ -3035,6 +3035,14 @@ S: Maintained
 F: hw/semihosting/
 F: include/hw/semihosting/
 
+Multi-process QEMU
+M: Jagannathan Raman <[email protected]>
+M: Elena Ufimtseva <[email protected]>
+M: John G Johnson <[email protected]>
+S: Maintained
+F: hw/pci-host/remote.c
+F: include/hw/pci-host/remote.h
+
 Build and test automation
 -------------------------
 Build and test automation
hw/pci-host/Makefile.objs +1 -0
index e422e0a..daf9007 100644
@@ -18,6 +18,7 @@ common-obj-$(CONFIG_XEN_IGD_PASSTHROUGH) += xen_igd_pt.o
 common-obj-$(CONFIG_PCI_EXPRESS_Q35) += q35.o
 common-obj-$(CONFIG_PCI_EXPRESS_GENERIC_BRIDGE) += gpex.o
 common-obj-$(CONFIG_PCI_EXPRESS_XILINX) += xilinx-pcie.o
+common-obj-$(CONFIG_MPQEMU) += remote.o
 
 common-obj-$(CONFIG_PCI_EXPRESS_DESIGNWARE) += designware.o
 obj-$(CONFIG_POWERNV) += pnv_phb4.o pnv_phb4_pec.o
hw/pci-host/remote.c new file +76 -0

76 lines not on this page · raw message

include/hw/pci-host/remote.h new file +30 -0
new file mode 100644
index 0000000..bab6d3c
@@ -0,0 +1,30 @@
+/*
+ * PCI Host for remote device
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_PCIHOST_H
+#define REMOTE_PCIHOST_H
+
+#include "exec/memory.h"
+#include "hw/pci/pcie_host.h"
+
+#define TYPE_REMOTE_HOST_DEVICE "remote-pcihost"
+#define REMOTE_HOST_DEVICE(obj) \
+    OBJECT_CHECK(RemotePCIHost, (obj), TYPE_REMOTE_HOST_DEVICE)
+
+typedef struct RemotePCIHost {
+    /*< private >*/
+    PCIExpressHost parent_obj;
+    /*< public >*/
+
+    MemoryRegion *mr_pci_mem;
+    MemoryRegion *mr_sys_io;
+} RemotePCIHost;
+
+#endif
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [PATCH v8 03/20] multi-process: setup PCI host bridge for remote device

Quoted 3 lines · Jagannathan Raman

+ * This PCI host is purely a container for PCI devices. It's fake in the
+ * sense that the guest never sees this PCI host and has no way of
+ * accessing it. It's job is just to provide the environment that QEMU

s/It's/Its/ :)

hw/pci-host/remote.c line 42

Quoted 3 lines
hw/pci-host/remote.c new file +3 -0
+static void remote_pcihost_realize(DeviceState *dev, Error **errp)
+{
+    char *busname = g_strdup_printf("remote-pci-%ld", (unsigned long)getpid());

Why is the pid used here? I don't understand why this helps. If additional instances are created in this process then the name won't be unique. The name isn't exposed outside the process, so there is no need to uniquely identify the process either.

g_free(busname) is missing. A nice way to fix this is:

g_autofree char *busname = ...

This avoids the need to manually call g_free().

#5 lore raw

PATCH v8 4/20 multi-process: setup a machine object for remote device process

In reply to Jagannathan Raman

remote-machine object sets up various subsystems of the remote device process. Instantiate PCI host bridge object and initialize RAM, IO & PCI memory regions.

---
 MAINTAINERS                  |  2 ++
 hw/i386/Makefile.objs        |  1 +
 hw/i386/remote.c             | 69 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote.h     | 28 ++++++++++++++++++
 include/hw/pci-host/remote.h |  1 +
 5 files changed, 101 insertions(+)
 create mode 100644 hw/i386/remote.c
 create mode 100644 include/hw/i386/remote.h
MAINTAINERS +2 -0
index d781a28..32e264e 100644
@@ -3042,6 +3042,8 @@ M: John G Johnson <[email protected]>
 S: Maintained
 F: hw/pci-host/remote.c
 F: include/hw/pci-host/remote.h
+F: hw/i386/remote.c
+F: include/hw/i386/remote.h
 
 Build and test automation
 -------------------------
hw/i386/Makefile.objs +1 -0
index 6abc745..5ead266 100644
@@ -14,6 +14,7 @@ obj-$(CONFIG_XEN) += ../xenpv/ xen/
 obj-$(CONFIG_VMPORT) += vmport.o
 obj-$(CONFIG_VMMOUSE) += vmmouse.o
 obj-$(CONFIG_PC) += port92.o
+obj-$(CONFIG_MPQEMU) += remote.o
 
 obj-y += kvmvapic.o
 obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote.c new file +69 -0

69 lines not on this page · raw message

include/hw/i386/remote.h new file +28 -0
new file mode 100644
index 0000000..5b36b25
@@ -0,0 +1,28 @@
+/*
+ * Remote machine configuration
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_MACHINE_H
+#define REMOTE_MACHINE_H
+
+#include "qom/object.h"
+#include "hw/boards.h"
+#include "hw/pci-host/remote.h"
+
+typedef struct RemoteMachineState {
+    MachineState parent_obj;
+
+    RemotePCIHost *host;
+} RemoteMachineState;
+
+#define TYPE_REMOTE_MACHINE "remote-machine"
+#define REMOTE_MACHINE(obj) \
+    OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE)
+
+#endif
include/hw/pci-host/remote.h +1 -0
index bab6d3c..cc0fff4 100644
@@ -25,6 +25,7 @@ typedef struct RemotePCIHost {
 
     MemoryRegion *mr_pci_mem;
     MemoryRegion *mr_sys_io;
+    MemoryRegion *mr_sys_mem;
 } RemotePCIHost;
 
 #endif
Signature · 1 line
1.8.3.1
#6 lore raw

PATCH v8 5/20 multi-process: add qio channel function to transmit

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The entire array of the memory regions and file handlers. Will be used in the next patch.

---
 include/io/channel.h | 24 ++++++++++++++++++++++++
 io/channel.c         | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
include/io/channel.h +24 -0
index d4557f0..c2e3eae 100644
@@ -779,5 +779,29 @@ void qio_channel_set_aio_fd_handler(QIOChannel *ioc,
                                     IOHandler *io_read,
                                     IOHandler *io_write,
                                     void *opaque);
+/**
+ * qio_channel_writev_full_all:
+ * @ioc: the channel object
+ * @iov: the array of memory regions to write data from
+ * @niov: the length of the @iov array
+ * @fds: an array of file handles to send
+ * @nfds: number of file handles in @fds
+ * @errp: pointer to a NULL-initialized error object
+ *
+ *
+ * Behaves like qio_channel_writev_full but will attempt
+ * to send all data passed (file handles and memory regions).
+ * The function will wait for all requested data
+ * to be written, yielding from the current coroutine
+ * if required.
+ *
+ * Returns: 0 if all bytes were written, or -1 on error
+ */
+
+int qio_channel_writev_full_all(QIOChannel *ioc,
+                           const struct iovec *iov,
+                           size_t niov,
+                           int *fds, size_t nfds,
+                           Error **errp);
 
 #endif /* QIO_CHANNEL_H */
io/channel.c +45 -0

51 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#7 lore raw

PATCH v8 6/20 multi-process: define MPQemuMsg format and transmission functions

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Defines MPQemuMsg, which is the message that is sent to the remote process. This message is sent over QIOChannel and is used to command the remote process to perform various tasks.

Also defined the helper functions to send and receive messages over the QIOChannel

---
 MAINTAINERS              |   2 +
 include/io/mpqemu-link.h |  73 ++++++++++++++++++++
 io/Makefile.objs         |   2 +
 io/mpqemu-link.c         | 175 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 252 insertions(+)
 create mode 100644 include/io/mpqemu-link.h
 create mode 100644 io/mpqemu-link.c
MAINTAINERS +2 -0
index 32e264e..41fe809 100644
@@ -3044,6 +3044,8 @@ F: hw/pci-host/remote.c
 F: include/hw/pci-host/remote.h
 F: hw/i386/remote.c
 F: include/hw/i386/remote.h
+F: io/mpqemu-link.c
+F: include/io/mpqemu-link.h
 
 Build and test automation
 -------------------------
include/io/mpqemu-link.h new file +73 -0

73 lines not on this page · raw message

io/Makefile.objs +2 -0
index 9a20fce..5875ab0 100644
@@ -10,3 +10,5 @@ io-obj-y += channel-util.o
 io-obj-y += dns-resolver.o
 io-obj-y += net-listener.o
 io-obj-y += task.o
+
+io-obj-$(CONFIG_MPQEMU) += mpqemu-link.o
io/mpqemu-link.c new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#8 lore raw

Re: [PATCH v8 06/20] multi-process: define MPQemuMsg format and transmission functions

io/mpqemu-link.c line 48

Quoted 2 lines · Jagannathan Raman
io/mpqemu-link.c new file +2 -0
+static int mpqemu_readv(QIOChannel *ioc, struct iovec *iov, int **fds,
+                        size_t *nfds, Error **errp)

readv(2) and similar functions take an int iovcnt argument while mpqemu_readv() takes just a single struct iovec. The name mpqemu_read() seems more appopriate and iov argument could be replaced by void *buf, size_t len. That is cleaner because this function modifies the iov argument (callers need to be be careful!).

Quoted 14 lines
io/mpqemu-link.c new file +14 -0
+{
+    struct iovec *l_iov = iov;
+    size_t *l_nfds = nfds;
+    unsigned int niov = 1;
+    int **l_fds = fds;
+    size_t size, len;
+    Error *local_err = NULL;
+
+    size = iov->iov_len;
+
+    while (size > 0) {
+        len = qio_channel_readv_full(ioc, l_iov, niov, l_fds, l_nfds,
+                                     &local_err);
+

io/mpqemu-link.c line 72

Quoted 10 lines
io/mpqemu-link.c new file +10 -0
+        if (len == QIO_CHANNEL_ERR_BLOCK) {
+            if (qemu_in_coroutine()) {
+                qio_channel_yield(ioc, G_IO_IN);
+            } else {
+                qio_channel_wait(ioc, G_IO_IN);
+            }
+            continue;
+        }
+
+        if (len <= 0) {

size_t is unsigned so this does not detect negative values. Please use qio_channel_readv_full()'s return type (ssize_t) instead.

io/mpqemu-link.c line 85

Quoted 13 lines
io/mpqemu-link.c new file +13 -0
+            error_propagate(errp, local_err);
+            return -EIO;
+        }
+
+        l_fds = NULL;
+        l_nfds = NULL;
+
+        size -= len;
+
+        (void)iov_discard_front(&l_iov, &niov, len);
+    }
+
+    return iov->iov_len;

read()-style functions return the number of bytes read. mpqemu_readv() returns the number of unread bytes remaining. Maintaining consistency with read() function conventions will reduce the chance of bugs, so I suggest returning the number of bytes read instead.

Quoted 6 lines
io/mpqemu-link.c new file +6 -0
+}
+
+void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp)
+{
+    Error *local_err = NULL;
+    int *fds = NULL;

io/mpqemu-link.c line 101

Quoted 10 lines
io/mpqemu-link.c new file +10 -0
+    struct iovec hdr, data;
+    size_t nfds = 0;
+
+    hdr.iov_base = msg;
+    hdr.iov_len = MPQEMU_MSG_HDR_SIZE;
+
+    if (mpqemu_readv(ioc, &hdr, &fds, &nfds, &local_err) < 0) {
+        error_propagate(errp, local_err);
+        return;
+    }

Can we read less than MPQEMU_MSG_HDR_SIZE if the peer closes the socket? That case probably needs to be handled.

io/mpqemu-link.c line 173

Quoted 4 lines
io/mpqemu-link.c new file +4 -0
+void mpqemu_msg_cleanup(MPQemuMsg *msg)
+{
+    if (msg->data2) {
+        free(msg->data2);

The buffer was allocated with g_malloc0() so g_free() should be used to free it.

#9 lore raw

PATCH v8 7/20 multi-process: add co-routines to communicate with remote

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

process to avoid blocking the main loop during the message exchanges. To be used by proxy device.

---
 include/io/mpqemu-link.h | 15 +++++++++
 io/mpqemu-link.c         | 82 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+)
include/io/mpqemu-link.h +15 -0
index ae7008e..8591ad2 100644
@@ -14,6 +14,7 @@
 #include "qom/object.h"
 #include "qemu/thread.h"
 #include "io/channel.h"
+#include "io/channel-socket.h"
 
 #define REMOTE_MAX_FDS 8
 
@@ -27,6 +28,7 @@
  */
 typedef enum {
     INIT = 0,
+    RET_MSG,
     MAX = INT_MAX,
 } MPQemuCmd;
 
@@ -64,6 +66,19 @@ typedef struct {
     uint8_t *data2;
 } MPQemuMsg;
 
+struct MPQemuRequest {
+    MPQemuMsg *msg;
+    QIOChannel *ioc;
+    Coroutine *co;
+    bool finished;
+    int error;
+    long ret;
+};
+
+typedef struct MPQemuRequest MPQemuRequest;
+
+uint64_t mpqemu_msg_send_and_await_reply(MPQemuMsg *msg, QIOChannel *ioc,
+                                  Error **errp);
 void mpqemu_msg_send(MPQemuMsg *msg, QIOChannel *ioc, Error **errp);
 void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp);
 
io/mpqemu-link.c +82 -0

94 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#10 lore raw

Re: [PATCH v8 07/20] multi-process: add co-routines to communicate with remote

Quoted 18 lines · Jagannathan Raman
io/mpqemu-link.c +18 -0
+/*
+ * Create if needed and enter co-routine to send the message to the
+ * remote channel ioc and wait for the reply.
+ * Returns the value from the reply message, sets the error on failure.
+ */
+
+uint64_t mpqemu_msg_send_and_await_reply(MPQemuMsg *msg, QIOChannel *ioc,
+                                  Error **errp)
+{
+    MPQemuRequest req = {0};
+    uint64_t ret = UINT64_MAX;
+
+    req.ioc = ioc;
+    if (!req.ioc) {
+        error_setg(errp, "Channel is set to NULL");
+        return ret;
+    }
+

io/mpqemu-link.c void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp) · line 207

Quoted 10 lines
io/mpqemu-link.c +10 -0
+    req.msg = msg;
+    req.ret = 0;
+    req.finished = false;
+
+    req.co = qemu_coroutine_create(mpqemu_msg_send_co, &req);
+    qemu_coroutine_enter(req.co);
+
+    while (!req.finished) {
+        aio_poll(qemu_get_aio_context(), true);
+    }

This is called from vcpu threads. aio_poll() does not release the global mutex so all other vcpu threads are blocked until mpqemu communication completes.

The simplest solution is to unlock the global mutex and call blocking, non-coroutine versions of the send/recv code. That way other vcpu threads can execute.

This means that the QEMU process would not use the mpqemu IOChannel in the event loop.

The remote process would still use the IOChannel in the event loop, however.

Stefan

#11 lore raw

PATCH v8 8/20 multi-process: Initialize message handler in remote device

In reply to Jagannathan Raman

Initializes the message handler function in the remote process. It is called whenever there's an event pending on QIOChannel that registers this function.

---
 MAINTAINERS              |  1 +
 hw/i386/Makefile.objs    |  1 +
 hw/i386/remote-msg.c     | 64 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote.h |  4 +++
 4 files changed, 70 insertions(+)
 create mode 100644 hw/i386/remote-msg.c
MAINTAINERS +1 -0
index 41fe809..40e0654 100644
@@ -3046,6 +3046,7 @@ F: hw/i386/remote.c
 F: include/hw/i386/remote.h
 F: io/mpqemu-link.c
 F: include/io/mpqemu-link.h
+F: hw/i386/remote-msg.c
 
 Build and test automation
 -------------------------
hw/i386/Makefile.objs +1 -0
index 5ead266..8396958 100644
@@ -15,6 +15,7 @@ obj-$(CONFIG_VMPORT) += vmport.o
 obj-$(CONFIG_VMMOUSE) += vmmouse.o
 obj-$(CONFIG_PC) += port92.o
 obj-$(CONFIG_MPQEMU) += remote.o
+obj-$(CONFIG_MPQEMU) += remote-msg.o
 
 obj-y += kvmvapic.o
 obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-msg.c new file +64 -0

64 lines not on this page · raw message

include/hw/i386/remote.h +4 -0
index 5b36b25..7f9028f 100644
@@ -14,6 +14,7 @@
 #include "qom/object.h"
 #include "hw/boards.h"
 #include "hw/pci-host/remote.h"
+#include "io/channel.h"
 
 typedef struct RemoteMachineState {
     MachineState parent_obj;
@@ -25,4 +26,7 @@ typedef struct RemoteMachineState {
 #define REMOTE_MACHINE(obj) \
     OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE)
 
+gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
+                            gpointer opaque);
+
 #endif
Signature · 1 line
1.8.3.1
#12 lore raw

Re: [PATCH v8 08/20] multi-process: Initialize message handler in remote device

Quoted 5 lines · Jagannathan Raman
hw/i386/remote-msg.c new file +5 -0
+gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
+                            gpointer opaque)
+{
+    PCIDevice *pci_dev = (PCIDevice *)opaque;
+    Error *local_err = NULL;

hw/i386/remote-msg.c line 33

Quoted 10 lines
hw/i386/remote-msg.c new file +10 -0
+    MPQemuMsg msg = { 0 };
+
+    if (cond & G_IO_HUP) {
+        qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
+        return FALSE;
+    }
+
+    if (cond & (G_IO_ERR | G_IO_NVAL)) {
+        error_report("Error %d while processing message from proxy \
+                   in remote process pid=%d", errno, getpid());

String literals don't wrap with backslash in C. Please use:

"Error %d while processing message from proxy "
"in remote process pid=%d", ...

instead.

hw/i386/remote-msg.c line 44

Quoted 11 lines
hw/i386/remote-msg.c new file +11 -0
+        return FALSE;
+    }
+
+    mpqemu_msg_recv(&msg, ioc, &local_err);
+    if (local_err) {
+        goto exit;
+    }
+
+    if (!mpqemu_msg_valid(&msg)) {
+        error_report("Received invalid message from proxy \
+                     in remote process pid=%d", getpid());

Same here.

#13 lore raw

PATCH v8 9/20 multi-process: Associate fd of a PCIDevice with its object

In reply to Jagannathan Raman

Associate the file descriptor for a PCIDevice in remote process with DeviceState object.

---
 MAINTAINERS                  |   2 +
 hw/i386/Makefile.objs        |   1 +
 hw/i386/remote-obj.c         | 127 +++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote-obj.h |  42 ++++++++++++++
 4 files changed, 172 insertions(+)
 create mode 100644 hw/i386/remote-obj.c
 create mode 100644 include/hw/i386/remote-obj.h
MAINTAINERS +2 -0
index 40e0654..9fd278f 100644
@@ -3047,6 +3047,8 @@ F: include/hw/i386/remote.h
 F: io/mpqemu-link.c
 F: include/io/mpqemu-link.h
 F: hw/i386/remote-msg.c
+F: include/hw/i386/remote-obj.h
+F: hw/i386/remote-obj.c
 
 Build and test automation
 -------------------------
hw/i386/Makefile.objs +1 -0
index 8396958..30d7102 100644
@@ -16,6 +16,7 @@ obj-$(CONFIG_VMMOUSE) += vmmouse.o
 obj-$(CONFIG_PC) += port92.o
 obj-$(CONFIG_MPQEMU) += remote.o
 obj-$(CONFIG_MPQEMU) += remote-msg.o
+obj-$(CONFIG_MPQEMU) += remote-obj.o
 
 obj-y += kvmvapic.o
 obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-obj.c new file +127 -0

127 lines not on this page · raw message

include/hw/i386/remote-obj.h new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#14 lore raw

Re: [PATCH v8 09/20] multi-process: Associate fd of a PCIDevice with its object

hw/i386/remote-obj.c line 36

Quoted 5 lines · Jagannathan Raman
hw/i386/remote-obj.c new file +5 -0
+static void remote_object_set_devid(Object *obj, const char *str, Error **errp)
+{
+    RemoteObject *o = REMOTE_OBJECT(obj);
+
+    o->devid = g_strdup(str);

Setter functions can be called multiple times so g_free(o->devid) is needed to prevent memory leaks.

hw/i386/remote-obj.c line 40

Quoted 4 lines
hw/i386/remote-obj.c new file +4 -0
+}
+
+static void remote_object_machine_done(Notifier *notifier, void *data)
+{

Is UserCreatableClass->complete() called too early? If you can use it instead of the machine init done notifier then that would make error reporting cleaner. Both command-line and object_add monitor commands would fail as expected if an error occurs, whereas this patch prints an error and continues running.

hw/i386/remote-obj.c line 50

Quoted 10 lines
hw/i386/remote-obj.c new file +10 -0
+    RemoteObject *o = container_of(notifier, RemoteObject, machine_done);
+    DeviceState *dev = NULL;
+    QIOChannel *ioc = NULL;
+    Error *err = NULL;
+
+    dev = qdev_find_recursive(sysbus_get_default(), o->devid);
+    if (!dev || !object_dynamic_cast(OBJECT(dev), TYPE_PCI_DEVICE)) {
+        error_report("%s is not a PCI device", o->devid);
+        return;
+    }

What happens when a "device_del" monitor command removes the device? I guess a use-after-free will occur since this object isn't aware that the device disappeared. QOM's Object has a reference count that you can use to keep the object alive, but that still might not play well with hot unplug where the device is being removed from its bus. One solution is to refuse hot unplugging the device while the remote object exists.

hw/i386/remote-obj.c line 55

Quoted 5 lines
hw/i386/remote-obj.c new file +5 -0
+
+    ioc = qio_channel_new_fd(o->fd, &err);
+    if (!ioc) {
+        error_free(err);
+        error_report("Failed to allocate IO channel");

Please print err, it contains a more detailed error message than "Failed to allocate IO channel".

#15 lore raw

PATCH v8 10/20 multi-process: setup memory manager for remote device

In reply to Jagannathan Raman

SyncSysMemMsg message format is defined. It is used to send file descriptors of the RAM regions to remote device.
RAM on the remote device is configured with a set of file descriptors. Old RAM regions are deleted and new regions, each with an fd, is added to the RAM.

---
 MAINTAINERS                     |  2 ++
 hw/i386/Makefile.objs           |  1 +
 hw/i386/remote-memory.c         | 58 +++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote-memory.h | 19 ++++++++++++++
 include/io/mpqemu-link.h        | 13 +++++++++
 io/mpqemu-link.c                | 13 +++++++++
 6 files changed, 106 insertions(+)
 create mode 100644 hw/i386/remote-memory.c
 create mode 100644 include/hw/i386/remote-memory.h
MAINTAINERS +2 -0
index 9fd278f..2da7680 100644
@@ -3049,6 +3049,8 @@ F: include/io/mpqemu-link.h
 F: hw/i386/remote-msg.c
 F: include/hw/i386/remote-obj.h
 F: hw/i386/remote-obj.c
+F: include/hw/i386/remote-memory.h
+F: hw/i386/remote-memory.c
 
 Build and test automation
 -------------------------
hw/i386/Makefile.objs +1 -0
index 30d7102..4ba47e0 100644
@@ -17,6 +17,7 @@ obj-$(CONFIG_PC) += port92.o
 obj-$(CONFIG_MPQEMU) += remote.o
 obj-$(CONFIG_MPQEMU) += remote-msg.o
 obj-$(CONFIG_MPQEMU) += remote-obj.o
+obj-$(CONFIG_MPQEMU) += remote-memory.o
 
 obj-y += kvmvapic.o
 obj-$(CONFIG_ACPI) += acpi-common.o
hw/i386/remote-memory.c new file +58 -0

58 lines not on this page · raw message

include/hw/i386/remote-memory.h new file +19 -0
new file mode 100644
index 0000000..aab3e4d
@@ -0,0 +1,19 @@
+/*
+ * Memory manager for remote device
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_MEMORY_H
+#define REMOTE_MEMORY_H
+
+#include "exec/hwaddr.h"
+#include "io/mpqemu-link.h"
+
+void remote_sysmem_reconfig(MPQemuMsg *msg, Error **errp);
+
+#endif
include/io/mpqemu-link.h +13 -0
index 8591ad2..98383f9 100644
@@ -15,6 +15,8 @@
 #include "qemu/thread.h"
 #include "io/channel.h"
 #include "io/channel-socket.h"
+#include "exec/cpu-common.h"
+#include "exec/hwaddr.h"
 
 #define REMOTE_MAX_FDS 8
 
@@ -25,13 +27,22 @@
  *
  * MPQemuCmd enum type to specify the command to be executed on the remote
  * device.
+ *
+ * SYNC_SYSMEM      Shares QEMU's RAM with remote device's RAM
  */
 typedef enum {
     INIT = 0,
+    SYNC_SYSMEM,
     RET_MSG,
     MAX = INT_MAX,
 } MPQemuCmd;
 
+typedef struct {
+    hwaddr gpas[REMOTE_MAX_FDS];
+    uint64_t sizes[REMOTE_MAX_FDS];
+    off_t offsets[REMOTE_MAX_FDS];
+} SyncSysmemMsg;
+
 /**
  * Maximum size of data2 field in the message to be transmitted.
  */
@@ -50,6 +61,7 @@ typedef enum {
  * MPQemuMsg Format of the message sent to the remote device from QEMU.
  *
  */
+
 typedef struct {
     int cmd;
     int bytestream;
@@ -57,6 +69,7 @@ typedef struct {
 
     union {
         uint64_t u64;
+        SyncSysmemMsg sync_sysmem;
     } data1;
 
     int fds[REMOTE_MAX_FDS];
io/mpqemu-link.c +13 -0
index d4dd0fe..aad1d77 100644
@@ -245,6 +245,19 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             }
         }
     }
+     /* Verify message specific fields. */
+    switch (msg->cmd) {
+    case SYNC_SYSMEM:
+        if (msg->num_fds == 0 || msg->bytestream) {
+            return false;
+        }
+        if (msg->size != sizeof(msg->data1)) {
+            return false;
+        }
+        break;
+    default:
+        break;
+    }
 
     return true;
 }
Signature · 1 line
1.8.3.1
#16 lore raw

Re: [PATCH v8 10/20] multi-process: setup memory manager for remote device

hw/i386/remote-memory.c line 37

Quoted 6 lines · Jagannathan Raman
hw/i386/remote-memory.c new file +6 -0
+    QTAILQ_FOREACH_SAFE(subregion, &sysmem->subregions, subregions_link, next) {
+        if (subregion->ram) {
+            memory_region_del_subregion(sysmem, subregion);
+            object_unparent(OBJECT(subregion));
+        }
+    }

Where is g_free(subregion) called?

hw/i386/remote-memory.c line 50

Quoted 13 lines
hw/i386/remote-memory.c new file +13 -0
+
+    for (region = 0; region < msg->num_fds; region++) {
+        subregion = g_new(MemoryRegion, 1);
+        name = g_strdup_printf("remote-mem-%u", suffix++);
+        memory_region_init_ram_from_fd(subregion, NULL,
+                                       name, sysmem_info->sizes[region],
+                                       RAM_SHARED, msg->fds[region],
+                                       sysmem_info->offsets[region],
+                                       &local_err);
+        g_free(name);
+        if (local_err) {
+            error_propagate(errp, local_err);
+            break;

Where is g_free(subregion) called?

#17 lore raw

PATCH v8 11/20 multi-process: introduce proxy object

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Defines a PCI Device proxy object as a child of TYPE_PCI_DEVICE.

---
 MAINTAINERS            |  2 ++
 hw/pci/Makefile.objs   |  1 +
 hw/pci/proxy.c         | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h | 26 ++++++++++++++++
 4 files changed, 109 insertions(+)
 create mode 100644 hw/pci/proxy.c
 create mode 100644 include/hw/pci/proxy.h
MAINTAINERS +2 -0
index 2da7680..a8f097a 100644
@@ -3051,6 +3051,8 @@ F: include/hw/i386/remote-obj.h
 F: hw/i386/remote-obj.c
 F: include/hw/i386/remote-memory.h
 F: hw/i386/remote-memory.c
+F: hw/pci/proxy.c
+F: include/hw/pci/proxy.h
 
 Build and test automation
 -------------------------
hw/pci/Makefile.objs +1 -0
index c78f2fb..515dda5 100644
@@ -12,3 +12,4 @@ common-obj-$(CONFIG_PCI_EXPRESS) += pcie_port.o pcie_host.o
 
 common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
 common-obj-$(CONFIG_ALL) += pci-stub.o
+obj-$(CONFIG_MPQEMU) += proxy.o
hw/pci/proxy.c new file +80 -0

80 lines not on this page · raw message

include/hw/pci/proxy.h new file +26 -0
new file mode 100644
index 0000000..0a66ddc
@@ -0,0 +1,26 @@
+/*
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PROXY_H
+#define PROXY_H
+
+#include "hw/pci/pci.h"
+#include "io/channel.h"
+
+#define TYPE_PCI_PROXY_DEV "pci-proxy-dev"
+
+#define PCI_PROXY_DEV(obj) \
+            OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV)
+
+typedef struct PCIProxyDev {
+    PCIDevice parent_dev;
+    char *fd;
+    QIOChannel *ioc;
+} PCIProxyDev;
+
+#endif /* PROXY_H */
Signature · 1 line
1.8.3.1
#18 lore raw

PATCH v8 12/20 multi-process: Forward PCI config space acceses to the remote process

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The Proxy Object sends the PCI config space accesses as messages to the remote process over the communication channel

TODO:
Investigate if the local PCI config writes can be dropped. Without the proxy local PCI config space writes for the device, the driver in the guest times out on the probing.
We have tried to only refer to the remote for the PCI config writes, but the driver timeout in the guest forced as to left this as it is (removing local PCI config only).

---
 hw/i386/remote-msg.c     | 65 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/pci/proxy.c           | 55 ++++++++++++++++++++++++++++++++++++++++
 include/io/mpqemu-link.h |  8 ++++++
 io/mpqemu-link.c         | 14 +++++++++++
 4 files changed, 142 insertions(+)
hw/i386/remote-msg.c +65 -0
index 4315d2c..c5b8651 100644
@@ -15,6 +15,12 @@
 #include "io/mpqemu-link.h"
 #include "qapi/error.h"
 #include "sysemu/runstate.h"
+#include "hw/pci/pci.h"
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg);
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg);
 
 gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
                             gpointer opaque)
@@ -46,6 +52,12 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     }
 
     switch (msg.cmd) {
+    case PCI_CONFIG_WRITE:
+        process_config_write(ioc, pci_dev, &msg);
+        break;
+    case PCI_CONFIG_READ:
+        process_config_read(ioc, pci_dev, &msg);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
@@ -62,3 +74,56 @@ exit:
 
     return TRUE;
 }
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)msg->data2;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when writing PCI config, pid %d",
+                     getpid());
+        ret.data1.u64 = UINT64_MAX;
+    } else {
+        pci_default_write_config(dev, conf->addr, conf->val, conf->l);
+    }
+
+    ret.cmd = RET_MSG;
+    ret.bytestream = 0;
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+}
+
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)msg->data2;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when reading PCI config, pid %d",
+                     getpid());
+        ret.data1.u64 = UINT64_MAX;
+    } else {
+        ret.data1.u64 = pci_default_read_config(dev, conf->addr, conf->l);
+    }
+
+    ret.cmd = RET_MSG;
+    ret.bytestream = 0;
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+
+}
hw/pci/proxy.c +55 -0

73 lines not on this page · raw message

include/io/mpqemu-link.h +8 -0
index 98383f9..9bd754e 100644
@@ -34,6 +34,8 @@ typedef enum {
     INIT = 0,
     SYNC_SYSMEM,
     RET_MSG,
+    PCI_CONFIG_WRITE,
+    PCI_CONFIG_READ,
     MAX = INT_MAX,
 } MPQemuCmd;
 
@@ -43,6 +45,12 @@ typedef struct {
     off_t offsets[REMOTE_MAX_FDS];
 } SyncSysmemMsg;
 
+typedef struct {
+    uint32_t addr;
+    uint32_t val;
+    int l;
+} ConfDataMsg;
+
 /**
  * Maximum size of data2 field in the message to be transmitted.
  */
io/mpqemu-link.c +14 -0
index aad1d77..5d04b81 100644
@@ -255,6 +255,20 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case PCI_CONFIG_WRITE:
+    case PCI_CONFIG_READ:
+        if (msg->size != sizeof(ConfDataMsg)) {
+            return false;
+        }
+        if (!msg->bytestream) {
+            return false;
+        }
+        ConfDataMsg *conf = (ConfDataMsg *)msg->data2;
+
+        if (conf->l != 1 && conf->l != 2 && conf->l != 4) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#19 lore raw

PATCH v8 13/20 multi-process: PCI BAR read/write handling for proxy & remote endpoints

In reply to Jagannathan Raman

Proxy device object implements handler for PCI BAR writes and reads. The handler uses BAR_WRITE/BAR_READ message to communicate to the remote process with the BAR address and value to be written/read. The remote process implements handler for BAR_WRITE/BAR_READ message.

---
 hw/i386/remote-msg.c     | 88 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/pci/proxy.c           | 61 +++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h   | 16 +++++++--
 include/io/mpqemu-link.h | 10 ++++++
 io/mpqemu-link.c         |  6 ++++
 5 files changed, 179 insertions(+), 2 deletions(-)
hw/i386/remote-msg.c +88 -0
index c5b8651..7ccdd63 100644
@@ -16,11 +16,14 @@
 #include "qapi/error.h"
 #include "sysemu/runstate.h"
 #include "hw/pci/pci.h"
+#include "exec/memattrs.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
 static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
                                 MPQemuMsg *msg);
+static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
+static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 
 gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
                             gpointer opaque)
@@ -58,6 +61,12 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     case PCI_CONFIG_READ:
         process_config_read(ioc, pci_dev, &msg);
         break;
+    case BAR_WRITE:
+        process_bar_write(ioc, &msg, &local_err);
+        break;
+    case BAR_READ:
+        process_bar_read(ioc, &msg, &local_err);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
@@ -127,3 +136,82 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
     }
 
 }
+
+static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data1.bar_access;
+    AddressSpace *as =
+        bar_access->memory ? &address_space_memory : &address_space_io;
+    MPQemuMsg ret = { 0 };
+    MemTxResult res;
+    uint64_t val;
+    Error *local_err = NULL;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        ret.data1.u64 = UINT64_MAX;
+        goto fail;
+    }
+
+    val = cpu_to_le64(bar_access->val);
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, true);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space write operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        ret.data1.u64 = -1;
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());
+    }
+}
+
+static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data1.bar_access;
+    MPQemuMsg ret = { 0 };
+    AddressSpace *as;
+    MemTxResult res;
+    uint64_t val = 0;
+    Error *local_err = NULL;
+
+    as = bar_access->memory ? &address_space_memory : &address_space_io;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        val = UINT64_MAX;
+        goto fail;
+    }
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, false);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space read operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        val = UINT64_MAX;
+        goto fail;
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.data1.u64 = le64_to_cpu(val);
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());
+    }
+}
hw/pci/proxy.c +61 -0

64 lines not on this page · raw message

include/hw/pci/proxy.h +14 -2
index 0a66ddc..9d57483 100644
@@ -17,10 +17,22 @@
 #define PCI_PROXY_DEV(obj) \
             OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV)
 
-typedef struct PCIProxyDev {
+typedef struct PCIProxyDev PCIProxyDev;
+
+typedef struct ProxyMemoryRegion {
+    PCIProxyDev *dev;
+    MemoryRegion mr;
+    bool memory;
+    bool present;
+    uint8_t type;
+} ProxyMemoryRegion;
+
+struct PCIProxyDev {
     PCIDevice parent_dev;
     char *fd;
     QIOChannel *ioc;
-} PCIProxyDev;
+
+    ProxyMemoryRegion region[PCI_NUM_REGIONS];
+};
 
 #endif /* PROXY_H */
include/io/mpqemu-link.h +10 -0
index 9bd754e..ee3b44f 100644
@@ -36,6 +36,8 @@ typedef enum {
     RET_MSG,
     PCI_CONFIG_WRITE,
     PCI_CONFIG_READ,
+    BAR_WRITE,
+    BAR_READ,
     MAX = INT_MAX,
 } MPQemuCmd;
 
@@ -51,6 +53,13 @@ typedef struct {
     int l;
 } ConfDataMsg;
 
+typedef struct {
+    hwaddr addr;
+    uint64_t val;
+    unsigned size;
+    bool memory;
+} BarAccessMsg;
+
 /**
  * Maximum size of data2 field in the message to be transmitted.
  */
@@ -78,6 +87,7 @@ typedef struct {
     union {
         uint64_t u64;
         SyncSysmemMsg sync_sysmem;
+        BarAccessMsg bar_access;
     } data1;
 
     int fds[REMOTE_MAX_FDS];
io/mpqemu-link.c +6 -0
index 5d04b81..82b8465 100644
@@ -269,6 +269,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case BAR_WRITE:
+    case BAR_READ:
+        if ((msg->size != sizeof(msg->data1)) || (msg->num_fds != 0)) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#20 lore raw

Re: [PATCH v8 13/20] multi-process: PCI BAR read/write handling for proxy & remote endpoints

Quoted 26 lines · Jagannathan Raman
hw/i386/remote-msg.c +26 -0
+static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data1.bar_access;
+    AddressSpace *as =
+        bar_access->memory ? &address_space_memory : &address_space_io;
+    MPQemuMsg ret = { 0 };
+    MemTxResult res;
+    uint64_t val;
+    Error *local_err = NULL;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        ret.data1.u64 = UINT64_MAX;
+        goto fail;
+    }
+
+    val = cpu_to_le64(bar_access->val);
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, true);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space write operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        ret.data1.u64 = -1;

hw/i386/remote-msg.c static void process_config_read(QIOChannel *ioc, PCIDevice *dev, · line 175

Quoted 10 lines
hw/i386/remote-msg.c +10 -0
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());

There is an assertion failure if res != MEMTX_OK because errp was already set. error_setg() must not be called on an Error pointer that has already been set.

It is simplest to do:

  mpqemu_msg_send(&ret, ioc, (errp && *errp) ? NULL : &local_err);
Quoted 29 lines
hw/i386/remote-msg.c +29 -0
+    }
+}
+
+static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data1.bar_access;
+    MPQemuMsg ret = { 0 };
+    AddressSpace *as;
+    MemTxResult res;
+    uint64_t val = 0;
+    Error *local_err = NULL;
+
+    as = bar_access->memory ? &address_space_memory : &address_space_io;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        val = UINT64_MAX;
+        goto fail;
+    }
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, false);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space read operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        val = UINT64_MAX;
+        goto fail;

hw/i386/remote-msg.c static void process_config_read(QIOChannel *ioc, PCIDevice *dev, · line 214

Quoted 10 lines
hw/i386/remote-msg.c +10 -0
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.data1.u64 = le64_to_cpu(val);
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "

Same here.

hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 142

Quoted 6 lines
hw/pci/proxy.c +6 -0
+static void send_bar_access_msg(PCIProxyDev *pdev, MemoryRegion *mr,
+                                bool write, hwaddr addr, uint64_t *val,
+                                unsigned size, bool memory)
+{
+    MPQemuMsg msg = { 0 };
+    long ret = -EINVAL;

long is not guaranteed to be 64-bit. This function supports 64-bit accesses to BARs so uint64_t is needed here.

Quoted 8 lines
hw/pci/proxy.c +8 -0
+    Error *local_err = NULL;
+
+    msg.bytestream = 0;
+    msg.size = sizeof(msg.data1);
+    msg.data1.bar_access.addr = mr->addr + addr;
+    msg.data1.bar_access.size = size;
+    msg.data1.bar_access.memory = memory;
+

hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 160

Quoted 10 lines
hw/pci/proxy.c +10 -0
+    if (write) {
+        msg.cmd = BAR_WRITE;
+        msg.data1.bar_access.val = *val;
+    } else {
+        msg.cmd = BAR_READ;
+    }
+
+    ret = mpqemu_msg_send_and_await_reply(&msg, pdev->ioc, &local_err);
+    if (local_err) {
+        error_report("Failed to send BAR command to the remote process.");

Leaks local_err. Please report the error message from local_err and then free it.

hw/pci/proxy.c static void pci_proxy_dev_register_types(void) · line 194

Quoted 7 lines
hw/pci/proxy.c +7 -0
+const MemoryRegionOps proxy_mr_ops = {
+    .read = proxy_bar_read,
+    .write = proxy_bar_write,
+    .endianness = DEVICE_NATIVE_ENDIAN,
+    .impl = {
+        .min_access_size = 1,
+        .max_access_size = 1,

Should this be .max_access_size = 8?

io/mpqemu-link.c bool mpqemu_msg_valid(MPQemuMsg *msg) · line 274

Quoted 11 lines
io/mpqemu-link.c +3 -0
index 5d04b81..82b8465 100644
@@ -269,6 +269,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case BAR_WRITE:
+    case BAR_READ:
+        if ((msg->size != sizeof(msg->data1)) || (msg->num_fds != 0)) {

What about bytestream? It would be cleanest not to send bytestream over the wire. Since it is sent today the receiver can be confused if it has the wrong value and some mpqemu_msg_valid() cases validate bytestream. It is not validate here for some reason.

#21 lore raw

PATCH v8 14/20 multi-process: Synchronize remote memory

In reply to Jagannathan Raman

Add memory-listener object which is used to keep the view of the RAM in sync between QEMU and remote process.
A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message.

---
 MAINTAINERS                  |   2 +
 hw/i386/remote-msg.c         |   4 +
 hw/pci/Makefile.objs         |   1 +
 hw/pci/memory-sync.c         | 211 +++++++++++++++++++++++++++++++++++++++++++
 hw/pci/proxy.c               |   6 ++
 include/hw/pci/memory-sync.h |  27 ++++++
 include/hw/pci/proxy.h       |   3 +
 7 files changed, 254 insertions(+)
 create mode 100644 hw/pci/memory-sync.c
 create mode 100644 include/hw/pci/memory-sync.h
MAINTAINERS +2 -0
index a8f097a..bd4f4ec 100644
@@ -3053,6 +3053,8 @@ F: include/hw/i386/remote-memory.h
 F: hw/i386/remote-memory.c
 F: hw/pci/proxy.c
 F: include/hw/pci/proxy.h
+F: hw/pci/memory-sync.c
+F: include/hw/pci/memory-sync.h
 
 Build and test automation
 -------------------------
hw/i386/remote-msg.c +4 -0
index 7ccdd63..584498d 100644
@@ -17,6 +17,7 @@
 #include "sysemu/runstate.h"
 #include "hw/pci/pci.h"
 #include "exec/memattrs.h"
+#include "hw/i386/remote-memory.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -67,6 +68,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     case BAR_READ:
         process_bar_read(ioc, &msg, &local_err);
         break;
+    case SYNC_SYSMEM:
+        remote_sysmem_reconfig(&msg, &local_err);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
hw/pci/Makefile.objs +1 -0
index 515dda5..c90acd5 100644
@@ -13,3 +13,4 @@ common-obj-$(CONFIG_PCI_EXPRESS) += pcie_port.o pcie_host.o
 common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o
 common-obj-$(CONFIG_ALL) += pci-stub.o
 obj-$(CONFIG_MPQEMU) += proxy.o
+obj-$(CONFIG_MPQEMU) += memory-sync.o
hw/pci/memory-sync.c new file +211 -0

211 lines not on this page · raw message

hw/pci/proxy.c +6 -0
index 179f0c7..28b8c76 100644
@@ -17,6 +17,8 @@
 #include "monitor/monitor.h"
 #include "io/mpqemu-link.h"
 #include "qemu/error-report.h"
+#include "hw/pci/memory-sync.h"
+#include "qom/object.h"
 
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
@@ -44,6 +46,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
         error_setg(errp, "fd parameter not specified for %s",
                    DEVICE(device)->id);
     }
+
+    configure_memory_sync(&dev->sync, dev->ioc);
 }
 
 static void pci_proxy_dev_exit(PCIDevice *pdev)
@@ -51,6 +55,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev)
     PCIProxyDev *dev = PCI_PROXY_DEV(pdev);
 
     qio_channel_close(dev->ioc, NULL);
+
+    deconfigure_memory_sync(&dev->sync);
 }
 
 static int config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val,
include/hw/pci/memory-sync.h new file +27 -0
new file mode 100644
index 0000000..785f76a
@@ -0,0 +1,27 @@
+/*
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef MEMORY_SYNC_H
+#define MEMORY_SYNC_H
+
+#include "exec/memory.h"
+#include "io/channel.h"
+
+typedef struct RemoteMemSync {
+    MemoryListener listener;
+
+    int n_mr_sections;
+    MemoryRegionSection *mr_sections;
+
+    QIOChannel *ioc;
+} RemoteMemSync;
+
+void configure_memory_sync(RemoteMemSync *sync, QIOChannel *ioc);
+void deconfigure_memory_sync(RemoteMemSync *sync);
+
+#endif
include/hw/pci/proxy.h +3 -0
index 9d57483..3ff79f4 100644
@@ -11,6 +11,7 @@
 
 #include "hw/pci/pci.h"
 #include "io/channel.h"
+#include "hw/pci/memory-sync.h"
 
 #define TYPE_PCI_PROXY_DEV "pci-proxy-dev"
 
@@ -32,6 +33,8 @@ struct PCIProxyDev {
     char *fd;
     QIOChannel *ioc;
 
+    RemoteMemSync sync;
+
     ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
Signature · 1 line
1.8.3.1
#22 lore raw

PATCH v8 15/20 multi-process: create IOHUB object to handle irq

In reply to Jagannathan Raman

IOHUB object is added to manage PCI IRQs. It uses KVM_IRQFD ioctl to create irqfd to injecting PCI interrupts to the guest. IOHUB object forwards the irqfd to the remote process. Remote process uses this fd to directly send interrupts to the guest, bypassing QEMU.

---
 MAINTAINERS               |   2 +
 hw/Makefile.objs          |   1 +
 hw/i386/remote-msg.c      |   4 ++
 hw/i386/remote.c          |  10 ++++
 hw/pci/proxy.c            |  58 ++++++++++++++++++++++
 hw/remote/Makefile.objs   |   1 +
 hw/remote/iohub.c         | 123 ++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote.h  |   3 ++
 include/hw/pci/pci_ids.h  |   3 ++
 include/hw/pci/proxy.h    |   5 ++
 include/hw/remote/iohub.h |  42 ++++++++++++++++
 include/io/mpqemu-link.h  |   1 +
 io/mpqemu-link.c          |   5 ++
 13 files changed, 258 insertions(+)
 create mode 100644 hw/remote/Makefile.objs
 create mode 100644 hw/remote/iohub.c
 create mode 100644 include/hw/remote/iohub.h
MAINTAINERS +2 -0
index bd4f4ec..e75d010 100644
@@ -3055,6 +3055,8 @@ F: hw/pci/proxy.c
 F: include/hw/pci/proxy.h
 F: hw/pci/memory-sync.c
 F: include/hw/pci/memory-sync.h
+F: hw/remote/iohub.c
+F: include/hw/remote/iohub.h
 
 Build and test automation
 -------------------------
hw/Makefile.objs +1 -0
index 14b7ea4..4b2fa39 100644
@@ -40,6 +40,7 @@ devices-dirs-$(CONFIG_MEM_DEVICE) += mem/
 devices-dirs-$(CONFIG_NUBUS) += nubus/
 devices-dirs-y += semihosting/
 devices-dirs-y += smbios/
+devices-dirs-y += remote/
 endif
 
 common-obj-y += $(devices-dirs-y)
hw/i386/remote-msg.c +4 -0
index 584498d..756b710 100644
@@ -18,6 +18,7 @@
 #include "hw/pci/pci.h"
 #include "exec/memattrs.h"
 #include "hw/i386/remote-memory.h"
+#include "hw/remote/iohub.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -71,6 +72,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     case SYNC_SYSMEM:
         remote_sysmem_reconfig(&msg, &local_err);
         break;
+    case SET_IRQFD:
+        process_set_irqfd_msg(pci_dev, &msg);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
hw/i386/remote.c +10 -0
index a67be33..d4b5e33 100644
@@ -20,12 +20,15 @@
 #include "exec/address-spaces.h"
 #include "exec/memory.h"
 #include "qapi/error.h"
+#include "hw/pci/pci_host.h"
+#include "hw/remote/iohub.h"
 
 static void remote_machine_init(MachineState *machine)
 {
     MemoryRegion *system_memory, *system_io, *pci_memory;
     RemoteMachineState *s = REMOTE_MACHINE(machine);
     RemotePCIHost *rem_host;
+    PCIHostState *pci_host;
 
     system_memory = get_system_memory();
     system_io = get_system_io();
@@ -45,6 +48,13 @@ static void remote_machine_init(MachineState *machine)
     memory_region_add_subregion_overlap(system_memory, 0x0, pci_memory, -1);
 
     qdev_realize(DEVICE(rem_host), sysbus_get_default(), &error_fatal);
+
+    pci_host = PCI_HOST_BRIDGE(rem_host);
+
+    remote_iohub_init(&s->iohub);
+
+    pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 }
 
 static void remote_machine_class_init(ObjectClass *oc, void *data)
hw/pci/proxy.c +58 -0

82 lines not on this page · raw message

hw/remote/Makefile.objs new file +1 -0
new file mode 100644
index 0000000..635ce5e
@@ -0,0 +1 @@
+common-obj-$(CONFIG_MPQEMU) += iohub.o
hw/remote/iohub.c new file +123 -0

123 lines not on this page · raw message

include/hw/i386/remote.h +3 -0
index 7f9028f..01deccd 100644
@@ -15,11 +15,14 @@
 #include "hw/boards.h"
 #include "hw/pci-host/remote.h"
 #include "io/channel.h"
+#include "hw/remote/iohub.h"
 
 typedef struct RemoteMachineState {
     MachineState parent_obj;
 
     RemotePCIHost *host;
+
+    RemoteIOHubState iohub;
 } RemoteMachineState;
 
 #define TYPE_REMOTE_MACHINE "remote-machine"
include/hw/pci/pci_ids.h +3 -0
index 11f8ab7..bd0c17d 100644
@@ -192,6 +192,9 @@
 #define PCI_DEVICE_ID_SUN_SIMBA          0x5000
 #define PCI_DEVICE_ID_SUN_SABRE          0xa000
 
+#define PCI_VENDOR_ID_ORACLE             0x108e
+#define PCI_DEVICE_ID_REMOTE_IOHUB       0xb000
+
 #define PCI_VENDOR_ID_CMD                0x1095
 #define PCI_DEVICE_ID_CMD_646            0x0646
 
include/hw/pci/proxy.h +5 -0
index 3ff79f4..15cc381 100644
@@ -12,6 +12,7 @@
 #include "hw/pci/pci.h"
 #include "io/channel.h"
 #include "hw/pci/memory-sync.h"
+#include "qemu/event_notifier.h"
 
 #define TYPE_PCI_PROXY_DEV "pci-proxy-dev"
 
@@ -35,6 +36,10 @@ struct PCIProxyDev {
 
     RemoteMemSync sync;
 
+    int virq;
+    EventNotifier intr;
+    EventNotifier resample;
+
     ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
include/hw/remote/iohub.h new file +42 -0

42 lines not on this page · raw message

include/io/mpqemu-link.h +1 -0
index ee3b44f..6625de6 100644
@@ -38,6 +38,7 @@ typedef enum {
     PCI_CONFIG_READ,
     BAR_WRITE,
     BAR_READ,
+    SET_IRQFD,
     MAX = INT_MAX,
 } MPQemuCmd;
 
io/mpqemu-link.c +5 -0
index 82b8465..6fa4665 100644
@@ -275,6 +275,11 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case SET_IRQFD:
+        if (msg->size || (msg->num_fds != 2)) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#23 lore raw

PATCH v8 16/20 multi-process: Retrieve PCI info from remote process

In reply to Jagannathan Raman

Retrieve PCI configuration info about the remote device and configure the Proxy PCI object based on the returned information

---
 hw/pci/proxy.c           | 85 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/io/mpqemu-link.h |  1 +
 2 files changed, 86 insertions(+)
hw/pci/proxy.c +85 -0
index 1c1b682..50a806c 100644
@@ -23,6 +23,8 @@
 #include "sysemu/kvm.h"
 #include "util/event_notifier-posix.c"
 
+static void probe_pci_info(PCIDevice *dev, Error **errp);
+
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
     pdev->ioc = qio_channel_new_fd(fd, errp);
@@ -86,6 +88,7 @@ static void setup_irqfd(PCIProxyDev *dev)
 static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
 {
     PCIProxyDev *dev = PCI_PROXY_DEV(device);
+    uint8_t *pci_conf = device->config;
     int fd;
 
     if (dev->fd) {
@@ -100,9 +103,14 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
                    DEVICE(device)->id);
     }
 
+    pci_conf[PCI_LATENCY_TIMER] = 0xff;
+    pci_conf[PCI_INTERRUPT_PIN] = 0x01;
+
     configure_memory_sync(&dev->sync, dev->ioc);
 
     setup_irqfd(dev);
+
+    probe_pci_info(PCI_DEVICE(dev), errp);
 }
 
 static void pci_proxy_dev_exit(PCIDevice *pdev)
@@ -258,3 +266,80 @@ const MemoryRegionOps proxy_mr_ops = {
         .max_access_size = 1,
     },
 };
+
+static void probe_pci_info(PCIDevice *dev, Error **errp)
+{
+    PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(dev);
+    uint32_t orig_val, new_val, base_class, val;
+    PCIProxyDev *pdev = PCI_PROXY_DEV(dev);
+    DeviceClass *dc = DEVICE_CLASS(pc);
+    uint8_t type;
+    int i, size;
+    char *name;
+
+    config_op_send(pdev, PCI_VENDOR_ID, &val, 2, PCI_CONFIG_READ);
+    pc->vendor_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_DEVICE_ID, &val, 2, PCI_CONFIG_READ);
+    pc->device_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_CLASS_DEVICE, &val, 2, PCI_CONFIG_READ);
+    pc->class_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_SUBSYSTEM_ID, &val, 2, PCI_CONFIG_READ);
+    pc->subsystem_id = (uint16_t)val;
+
+    base_class = pc->class_id >> 4;
+    switch (base_class) {
+    case PCI_BASE_CLASS_BRIDGE:
+        set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+        break;
+    case PCI_BASE_CLASS_STORAGE:
+        set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+        break;
+    case PCI_BASE_CLASS_NETWORK:
+        set_bit(DEVICE_CATEGORY_NETWORK, dc->categories);
+        break;
+    case PCI_BASE_CLASS_INPUT:
+        set_bit(DEVICE_CATEGORY_INPUT, dc->categories);
+        break;
+    case PCI_BASE_CLASS_DISPLAY:
+        set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
+        break;
+    case PCI_BASE_CLASS_PROCESSOR:
+        set_bit(DEVICE_CATEGORY_CPU, dc->categories);
+        break;
+    default:
+        set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+        break;
+    }
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &orig_val, 4,
+                       PCI_CONFIG_READ);
+        new_val = 0xffffffff;
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &new_val, 4,
+                       PCI_CONFIG_WRITE);
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &new_val, 4,
+                       PCI_CONFIG_READ);
+        size = (~(new_val & 0xFFFFFFF0)) + 1;
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &orig_val, 4,
+                       PCI_CONFIG_WRITE);
+        type = (new_val & 0x1) ?
+                   PCI_BASE_ADDRESS_SPACE_IO : PCI_BASE_ADDRESS_SPACE_MEMORY;
+
+        if (size) {
+            pdev->region[i].dev = pdev;
+            pdev->region[i].present = true;
+            if (type == PCI_BASE_ADDRESS_SPACE_MEMORY) {
+                pdev->region[i].memory = true;
+            }
+            name = g_strdup_printf("bar-region-%d", i);
+            memory_region_init_io(&pdev->region[i].mr, OBJECT(pdev),
+                                  &proxy_mr_ops, &pdev->region[i],
+                                  name, size);
+            pci_register_bar(dev, i, type, &pdev->region[i].mr);
+            g_free(name);
+        }
+    }
+}
include/io/mpqemu-link.h +1 -0
index 6625de6..a3844e1 100644
@@ -39,6 +39,7 @@ typedef enum {
     BAR_WRITE,
     BAR_READ,
     SET_IRQFD,
+    GET_PCI_INFO,
     MAX = INT_MAX,
 } MPQemuCmd;
 
Signature · 1 line
1.8.3.1
#24 lore raw

PATCH v8 17/20 multi-process: heartbeat messages to remote

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

In order to detect remote processes which are hung, the proxy periodically sends heartbeat messages to confirm if the remote process is alive. The remote process responds to this heartbeat message to confirm it is alive.

---
 hw/i386/remote-msg.c     | 19 ++++++++++++++++++
 hw/pci/proxy.c           | 52 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h   |  2 ++
 include/io/mpqemu-link.h |  1 +
 io/mpqemu-link.c         |  8 ++++++++
 5 files changed, 82 insertions(+)
hw/i386/remote-msg.c +19 -0
index 756b710..2a4d7f1 100644
@@ -26,6 +26,7 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
                                 MPQemuMsg *msg);
 static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
+static void process_proxy_ping_msg(QIOChannel *ioc, Error **errp);
 
 gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
                             gpointer opaque)
@@ -75,6 +76,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     case SET_IRQFD:
         process_set_irqfd_msg(pci_dev, &msg);
         break;
+    case PROXY_PING:
+        process_proxy_ping_msg(ioc, &local_err);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
@@ -223,3 +227,18 @@ fail:
                    "in remote process pid=%d", getpid());
     }
 }
+
+static void process_proxy_ping_msg(QIOChannel *ioc, Error **errp)
+{
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data1);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());
+    }
+}
hw/pci/proxy.c +52 -0

73 lines not on this page · raw message

include/hw/pci/proxy.h +2 -0
index 15cc381..d784328 100644
@@ -40,6 +40,8 @@ struct PCIProxyDev {
     EventNotifier intr;
     EventNotifier resample;
 
+    QEMUTimer *hb_timer;
+
     ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
include/io/mpqemu-link.h +1 -0
index a3844e1..5f2913f 100644
@@ -40,6 +40,7 @@ typedef enum {
     BAR_READ,
     SET_IRQFD,
     GET_PCI_INFO,
+    PROXY_PING,
     MAX = INT_MAX,
 } MPQemuCmd;
 
io/mpqemu-link.c +8 -0
index 6fa4665..0222e81 100644
@@ -280,6 +280,14 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case PROXY_PING:
+        if (msg->bytestream || msg->num_fds) {
+            return false;
+        }
+        if (msg->size) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#25 lore raw

Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote

Quoted 12 lines · Jagannathan Raman
hw/pci/proxy.c +8 -0
@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp)
         }
     }
 }
+
+static void hb_msg(PCIProxyDev *dev)
+{
+    DeviceState *ds = DEVICE(dev);
+    Error *local_err = NULL;
+    MPQemuMsg msg = { 0 };
+
+    msg.cmd = PROXY_PING;

hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369

Quoted 10 lines
hw/pci/proxy.c +10 -0
+    msg.bytestream = 0;
+    msg.size = 0;
+
+    (void)mpqemu_msg_send_and_await_reply(&msg, dev->ioc, &local_err);
+    if (local_err) {
+        error_report_err(local_err);
+        qio_channel_close(dev->ioc, &local_err);
+        error_setg(&error_fatal, "Lost contact with device %s", ds->id);
+    }
+}

Here is my feedback from the last revision. Was this addressed?

This patch seems incomplete since no action is taken when the device fails to respond. vCPU threads that access the device will still get stuck.

The simplest way to make this useful is to close the connection when a timeout occurs. Then the G_IO_HUP handler for the UNIX domain socket should perform connection cleanup. At that point there are a few choices:

1. Stop guest execution and wait for the host admin to restore the mplink so execution can resume. This is similar to how -drive rerror=stop pauses the guest when a disk I/O error is encountered.

2. Stop guest execution but defer it until this stale device is actually accessed. This maximizes guest uptime. Guests that rarely access the device may not notice at all.

3. Return 0 from MemoryRegion read operations and ignore writes. The guest continues executing but the device is broken. This is risky because device drivers inside the guest may not be ready to deal with this. The result could be data loss or corruption.

4. Raise a bus-level event. Maybe PCI error reporting can be used to offline the device.

5. Terminate the guest with an error message.

6. ?

Until the heartbeat is fully implemented and tested I suggest dropping it from this patch series. Remember the G_IO_HUP will happen anyway if the remote device process terminates.

#26 lore raw

Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote

Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jagannathan Raman
hw/pci/proxy.c +12 -0
@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp)
         }
     }
 }
+
+static void hb_msg(PCIProxyDev *dev)
+{
+    DeviceState *ds = DEVICE(dev);
+    Error *local_err = NULL;
+    MPQemuMsg msg = { 0 };
+
+    msg.cmd = PROXY_PING;
+    msg.bytestream = 0;
+    msg.size = 0;
+
+    (void)mpqemu_msg_send_and_await_reply(&msg, dev->ioc, &local_err);
Quoted 9 lines

hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369

Quoted 6 lines
hw/pci/proxy.c +6 -0
+    if (local_err) {
+        error_report_err(local_err);
+        qio_channel_close(dev->ioc, &local_err);
+        error_setg(&error_fatal, "Lost contact with device %s", ds->id);
+    }
+}

Here is my feedback from the last revision. Was this addressed?

Hi Stefan,

Thank you for reviewing the patchset. In this version we decided to shutdown the guest when the heartbeat did not get a reply from the remote by setting the error_fatal.
Should we approach it differently or you prefer us to get rid of the heartbeat in this form?

Thank you,
Elena

Quoted 32 lines

This patch seems incomplete since no action is taken when the device fails to respond. vCPU threads that access the device will still get stuck.

The simplest way to make this useful is to close the connection when a timeout occurs. Then the G_IO_HUP handler for the UNIX domain socket should perform connection cleanup. At that point there are a few choices:

1. Stop guest execution and wait for the host admin to restore the mplink so execution can resume. This is similar to how -drive rerror=stop pauses the guest when a disk I/O error is encountered.

2. Stop guest execution but defer it until this stale device is actually accessed. This maximizes guest uptime. Guests that rarely access the device may not notice at all.

3. Return 0 from MemoryRegion read operations and ignore writes. The guest continues executing but the device is broken. This is risky because device drivers inside the guest may not be ready to deal with this. The result could be data loss or corruption.

4. Raise a bus-level event. Maybe PCI error reporting can be used to offline the device.

5. Terminate the guest with an error message.

6. ?

Until the heartbeat is fully implemented and tested I suggest dropping it from this patch series. Remember the G_IO_HUP will happen anyway if the remote device process terminates.

#27 lore raw

Re: [PATCH v8 17/20] multi-process: heartbeat messages to remote

Quoted 28 lines · Elena Ufimtseva
Quoted 17 lines · Stefan Hajnoczi
Quoted 16 lines · Jagannathan Raman
hw/pci/proxy.c +12 -0
@@ -343,3 +349,49 @@ static void probe_pci_info(PCIDevice *dev, Error **errp)
         }
     }
 }
+
+static void hb_msg(PCIProxyDev *dev)
+{
+    DeviceState *ds = DEVICE(dev);
+    Error *local_err = NULL;
+    MPQemuMsg msg = { 0 };
+
+    msg.cmd = PROXY_PING;
+    msg.bytestream = 0;
+    msg.size = 0;
+
+    (void)mpqemu_msg_send_and_await_reply(&msg, dev->ioc, &local_err);
Quoted 9 lines

hw/pci/proxy.c static void probe_pci_info(PCIDevice *dev, Error **errp) · line 369

Quoted 6 lines
hw/pci/proxy.c +6 -0
+    if (local_err) {
+        error_report_err(local_err);
+        qio_channel_close(dev->ioc, &local_err);
+        error_setg(&error_fatal, "Lost contact with device %s", ds->id);
+    }
+}

Here is my feedback from the last revision. Was this addressed?

Quoted 7 lines

Hi Stefan,

Thank you for reviewing the patchset. In this version we decided to shutdown the guest when the heartbeat did not get a reply from the remote by setting the error_fatal.
Should we approach it differently or you prefer us to get rid of the heartbeat in this form?

I think the only case that this patch handles is when the mpqemu channel is closed.

The VM hangs when the channel is still open but the remote is unresponsive. (mpqemu_msg_send_and_await_reply() calls aio_poll() with the global mutex held so vcpus cannot make progress.)

The heartbeat mechanism needs to handle the case where the other side isn't responding. It can't hang QEMU.

I suggest dropping this patch. It can be done later.

Stefan

#28 lore raw

PATCH v8 18/20 multi-process: perform device reset in the remote process

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Perform device reset in the remote process when QEMU performs device reset. This is required to reset the internal state (like registers, etc...) of emulated devices

---
 hw/i386/remote-msg.c     | 22 ++++++++++++++++++++++
 hw/pci/proxy.c           | 20 ++++++++++++++++++++
 include/io/mpqemu-link.h |  1 +
 3 files changed, 43 insertions(+)
hw/i386/remote-msg.c +22 -0
index 2a4d7f1..55db631 100644
@@ -19,6 +19,7 @@
 #include "exec/memattrs.h"
 #include "hw/i386/remote-memory.h"
 #include "hw/remote/iohub.h"
+#include "sysemu/reset.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -27,6 +28,8 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
 static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 static void process_proxy_ping_msg(QIOChannel *ioc, Error **errp);
+static void process_device_reset_msg(QIOChannel *ioc, PCIDevice *dev,
+                                     Error **errp);
 
 gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
                             gpointer opaque)
@@ -79,6 +82,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond,
     case PROXY_PING:
         process_proxy_ping_msg(ioc, &local_err);
         break;
+    case DEVICE_RESET:
+        process_device_reset_msg(ioc, pci_dev, &local_err);
+        break;
     default:
         error_setg(&local_err,
                    "Unknown command (%d) received for device %s (pid=%d)",
@@ -242,3 +248,19 @@ static void process_proxy_ping_msg(QIOChannel *ioc, Error **errp)
                    "in remote process pid=%d", getpid());
     }
 }
+
+static void process_device_reset_msg(QIOChannel *ioc, PCIDevice *dev,
+                                     Error **errp)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(dev);
+    DeviceState *s = DEVICE(dev);
+    MPQemuMsg ret = { 0 };
+
+    if (dc->reset) {
+        dc->reset(s);
+    }
+
+    ret.cmd = RET_MSG;
+
+    mpqemu_msg_send(&ret, ioc, errp);
+}
hw/pci/proxy.c +20 -0
index 490093c..b6cc3fc 100644
@@ -26,6 +26,7 @@
 static void probe_pci_info(PCIDevice *dev, Error **errp);
 static void start_hb_timer(PCIProxyDev *dev);
 static void stop_hb_timer(PCIProxyDev *dev);
+static void proxy_device_reset(DeviceState *dev);
 
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
@@ -191,6 +192,8 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data)
     k->config_read = pci_proxy_read_config;
     k->config_write = pci_proxy_write_config;
 
+    dc->reset = proxy_device_reset;
+
     device_class_set_props(dc, proxy_properties);
 }
 
@@ -395,3 +398,20 @@ static void stop_hb_timer(PCIProxyDev *dev)
     timer_del(dev->hb_timer);
     timer_free(dev->hb_timer);
 }
+
+static void proxy_device_reset(DeviceState *dev)
+{
+    PCIProxyDev *pdev = PCI_PROXY_DEV(dev);
+    MPQemuMsg msg = { 0 };
+    Error *local_err = NULL;
+
+    msg.bytestream = 0;
+    msg.size = sizeof(msg.data1);
+    msg.cmd = DEVICE_RESET;
+
+    (void)mpqemu_msg_send_and_await_reply(&msg, pdev->ioc, &local_err);
+    if (local_err) {
+        error_report("Failed to send DEVICE_RESET to the remote process");
+    }
+
+}
include/io/mpqemu-link.h +1 -0
index 5f2913f..28c68cb 100644
@@ -41,6 +41,7 @@ typedef enum {
     SET_IRQFD,
     GET_PCI_INFO,
     PROXY_PING,
+    DEVICE_RESET,
     MAX = INT_MAX,
 } MPQemuCmd;
 
Signature · 1 line
1.8.3.1
#29 lore raw

PATCH v8 19/20 multi-process: add the concept description to docs/devel/qemu-multiprocess

In reply to Jagannathan Raman

From: John G Johnson <[email protected]>

---
 MAINTAINERS                  |   1 +
 docs/devel/index.rst         |   1 +
 docs/devel/multi-process.rst | 966 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 968 insertions(+)
 create mode 100644 docs/devel/multi-process.rst
MAINTAINERS +1 -0
index e75d010..f41b54c 100644
@@ -3057,6 +3057,7 @@ F: hw/pci/memory-sync.c
 F: include/hw/pci/memory-sync.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
+F: docs/devel/multi-process.rst
 
 Build and test automation
 -------------------------
docs/devel/index.rst +1 -0
index ae6eac7..8356c6a 100644
@@ -30,3 +30,4 @@ Contents:
    reset
    s390-dasd-ipl
    clocks
+   multi-process
docs/devel/multi-process.rst new file +966 -0

966 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#30 lore raw

PATCH v8 20/20 multi-process: add configure and usage information

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

---
 MAINTAINERS                |  2 ++
 docs/multi-process.rst     | 67 ++++++++++++++++++++++++++++++++++++++++++++++
 scripts/mpqemu-launcher.py | 49 +++++++++++++++++++++++++++++++++
 3 files changed, 118 insertions(+)
 create mode 100644 docs/multi-process.rst
 create mode 100755 scripts/mpqemu-launcher.py
MAINTAINERS +2 -0
index f41b54c..3df9441 100644
@@ -3058,6 +3058,8 @@ F: include/hw/pci/memory-sync.h
 F: hw/remote/iohub.c
 F: include/hw/remote/iohub.h
 F: docs/devel/multi-process.rst
+F: scripts/mpqemu-launcher.py
+F: scripts/mpqemu-launcher-perf-mode.py
 
 Build and test automation
 -------------------------
docs/multi-process.rst new file +67 -0

67 lines not on this page · raw message

scripts/mpqemu-launcher.py new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#31 lore raw
In reply to Jagannathan Raman

I have finished reviewing this revision.

Please update the wiki page with the latest status, especially limitations like no dirty memory logging (used by migration, TCG, VGA), no MSI interrupts, no PCIe, no IOMMU support, etc.

https://wiki.qemu.org/Features/MultiProcessQEMU

Please name the machine type x-remote and the object -object x-remote-object. "x-" is the convention for experimental features in QEMU. It means that the interface is not stable and might change between releases.

Stefan