Skip to contentSkip to conversation

Initial support for multi-process Qemu

PATCH Superseded Jagannathan Raman 4 participants 31 messages Oct 15, 2020 – Dec 4, 2020

Subsystems (2)

v11 of 21 Superseded Latest is v21 →

lore
Share & tools
lore
#0 lore raw

v11 cover Initial support for multi-process Qemu

Hello,

This is the v11 of the patchset. Thank you very much for the review of the v10 of the series. We are glad to hear the patchset is getting in a better shape.

We made changes to the following patches in this series.

[PATCH v10 06/19] multi-process: define MPQemuMsg format and transmission functions [PATCH v10 08/19] multi-process: Associate fd of a PCIDevice with its object [PATCH v10 10/19] multi-process: introduce proxy object

To touch upon the history of this project, we posted the Proof Of Concept patches before the BoF session in 2018. Subsequently, we posted 9 versions on the qemu-devel mailist. You can find them by following the links below ([1] - [9]).

Following people contributed to the design and
implementation of this project:
Jagannathan Raman <[email protected]>
Elena Ufimtseva <[email protected]>
John G Johnson <[email protected]>
Stefan Hajnoczi <[email protected]>
Konrad Wilk <[email protected]>
Kanth Ghatraju <[email protected]>

We would like to thank QEMU community for your feedback in the design and implementation of this project.

Qemu wiki page:
https://wiki.qemu.org/Features/MultiProcessQEMU

For the full concept writeup about QEMU multi-process, please refer to docs/devel/qemu-multiprocess.rst. Also see docs/qemu-multiprocess.txt for usage information.

We welcome all your ideas, concerns, and questions for this patchset. [POC]: https://www.mail-archive.com/[email protected]/msg566538.html
[1]: https://www.mail-archive.com/[email protected]/msg602285.html
[2]: https://www.mail-archive.com/[email protected]/msg624877.html
[3]: https://www.mail-archive.com/[email protected]/msg642000.html
[4]: https://www.mail-archive.com/[email protected]/msg655118.html
[5]: https://www.mail-archive.com/[email protected]/msg682429.html
[6]: https://www.mail-archive.com/[email protected]/msg697484.html
[7]: https://patchew.org/QEMU/[email protected]/
[8]: https://www.mail-archive.com/[email protected]/msg727007.html
[9]: https://www.mail-archive.com/[email protected]/msg734275.html
[10]: https://www.mail-archive.com/[email protected]/msg747638.html

Elena Ufimtseva (7):
multi-process: add qio channel function to transmit
multi-process: define MPQemuMsg format and transmission functions
multi-process: introduce proxy object
multi-process: add proxy communication functions
multi-process: Forward PCI config space acceses to the remote process
multi-process: perform device reset in the remote process
multi-process: add configure and usage information

Jagannathan Raman (11):
memory: alloc RAM from file at offset
multi-process: Add config option for multi-process QEMU
multi-process: setup PCI host bridge for remote device
multi-process: setup a machine object for remote device process
multi-process: Initialize message handler in remote device
multi-process: Associate fd of a PCIDevice with its object
multi-process: setup memory manager for remote device
multi-process: PCI BAR read/write handling for proxy & remote
endpoints
multi-process: Synchronize remote memory
multi-process: create IOHUB object to handle irq
multi-process: Retrieve PCI info from remote process

John G Johnson (1):
multi-process: add the concept description to
docs/devel/qemu-multiprocess

 MAINTAINERS                     |  26 ++
 backends/hostmem-memfd.c        |   2 +-
 configure                       |  10 +
 docs/devel/index.rst            |   1 +
 docs/devel/multi-process.rst    | 966 ++++++++++++++++++++++++++++++++++++++++
 docs/multi-process.rst          |  67 +++
 hw/i386/meson.build             |   5 +
 hw/i386/remote-iohub.c          | 123 +++++
 hw/i386/remote-memory.c         |  58 +++
 hw/i386/remote-msg.c            | 241 ++++++++++
 hw/i386/remote-obj.c            | 154 +++++++
 hw/i386/remote.c                |  79 ++++
 hw/misc/ivshmem.c               |   3 +-
 hw/pci-host/meson.build         |   1 +
 hw/pci-host/remote.c            |  75 ++++
 hw/pci/memory-sync.c            | 210 +++++++++
 hw/pci/meson.build              |   3 +
 hw/pci/proxy.c                  | 377 ++++++++++++++++
 include/exec/memory.h           |   2 +
 include/exec/ram_addr.h         |   2 +-
 include/hw/i386/remote-iohub.h  |  42 ++
 include/hw/i386/remote-memory.h |  19 +
 include/hw/i386/remote-obj.h    |  42 ++
 include/hw/i386/remote.h        |  40 ++
 include/hw/pci-host/remote.h    |  31 ++
 include/hw/pci/memory-sync.h    |  27 ++
 include/hw/pci/pci_ids.h        |   3 +
 include/hw/pci/proxy.h          |  53 +++
 include/io/channel.h            |  24 +
 include/io/mpqemu-link.h        |  98 ++++
 include/qemu/mmap-alloc.h       |   3 +-
 io/channel.c                    |  45 ++
 io/meson.build                  |   2 +
 io/mpqemu-link.c                | 303 +++++++++++++
 meson.build                     |   1 +
 scripts/mpqemu-launcher.py      |  49 ++
 softmmu/memory.c                |   3 +-
 softmmu/physmem.c               |  11 +-
 util/mmap-alloc.c               |   7 +-
 util/oslib-posix.c              |   2 +-
 40 files changed, 3197 insertions(+), 13 deletions(-)
 create mode 100644 docs/devel/multi-process.rst
 create mode 100644 docs/multi-process.rst
 create mode 100644 hw/i386/remote-iohub.c
 create mode 100644 hw/i386/remote-memory.c
 create mode 100644 hw/i386/remote-msg.c
 create mode 100644 hw/i386/remote-obj.c
 create mode 100644 hw/i386/remote.c
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 hw/pci/memory-sync.c
 create mode 100644 hw/pci/proxy.c
 create mode 100644 include/hw/i386/remote-iohub.h
 create mode 100644 include/hw/i386/remote-memory.h
 create mode 100644 include/hw/i386/remote-obj.h
 create mode 100644 include/hw/i386/remote.h
 create mode 100644 include/hw/pci-host/remote.h
 create mode 100644 include/hw/pci/memory-sync.h
 create mode 100644 include/hw/pci/proxy.h
 create mode 100644 include/io/mpqemu-link.h
 create mode 100644 io/mpqemu-link.c
 create mode 100755 scripts/mpqemu-launcher.py
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v11 1/19 memory: alloc RAM from file at offset

Allow RAM MemoryRegion to be created from an offset in a file, instead of allocating at offset of 0 by default. This is needed to synchronize RAM between QEMU & remote process.

---
 backends/hostmem-memfd.c  |  2 +-
 hw/misc/ivshmem.c         |  3 ++-
 include/exec/memory.h     |  2 ++
 include/exec/ram_addr.h   |  2 +-
 include/qemu/mmap-alloc.h |  3 ++-
 softmmu/memory.c          |  3 ++-
 softmmu/physmem.c         | 11 +++++++----
 util/mmap-alloc.c         |  7 ++++---
 util/oslib-posix.c        |  2 +-
 9 files changed, 22 insertions(+), 13 deletions(-)
backends/hostmem-memfd.c +1 -1
index e5626d4..69b0ae3 100644
@@ -55,7 +55,7 @@ memfd_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     name = host_memory_backend_get_name(backend);
     memory_region_init_ram_from_fd(&backend->mr, OBJECT(backend),
                                    name, backend->size,
-                                   backend->share, fd, errp);
+                                   backend->share, fd, 0, errp);
     g_free(name);
 }
 
hw/misc/ivshmem.c +2 -1
index e321e5c..8d3e1ee 100644
@@ -494,7 +494,8 @@ static void process_msg_shmem(IVShmemState *s, int fd, Error **errp)
 
     /* mmap the region and map into the BAR2 */
     memory_region_init_ram_from_fd(&s->server_bar2, OBJECT(s),
-                                   "ivshmem.bar2", size, true, fd, &local_err);
+                                   "ivshmem.bar2", size, true, fd, 0,
+                                   &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
         return;
include/exec/memory.h +2 -0
index 622207b..5eb2ece 100644
@@ -934,6 +934,7 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,
  * @size: size of the region.
  * @share: %true if memory must be mmaped with the MAP_SHARED flag
  * @fd: the fd to mmap.
+ * @offset: offset within the file referenced by fd
  * @errp: pointer to Error*, to store an error if it happens.
  *
  * Note that this function does not do anything to cause the data in the
@@ -945,6 +946,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
                                     uint64_t size,
                                     bool share,
                                     int fd,
+                                    ram_addr_t offset,
                                     Error **errp);
 #endif
 
include/exec/ram_addr.h +1 -1
index c6d2ef1..d465a48 100644
@@ -121,7 +121,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,
                                    Error **errp);
 RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  uint32_t ram_flags, int fd,
-                                 Error **errp);
+                                 off_t offset, Error **errp);
 
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                   MemoryRegion *mr, Error **errp);
include/qemu/mmap-alloc.h +2 -1
index e786266..4f57985 100644
@@ -25,7 +25,8 @@ void *qemu_ram_mmap(int fd,
                     size_t size,
                     size_t align,
                     bool shared,
-                    bool is_pmem);
+                    bool is_pmem,
+                    off_t start);
 
 void qemu_ram_munmap(int fd, void *ptr, size_t size);
 
softmmu/memory.c +2 -1
index 403ff3a..f7d81ee 100644
@@ -1591,6 +1591,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
                                     uint64_t size,
                                     bool share,
                                     int fd,
+                                    ram_addr_t offset,
                                     Error **errp)
 {
     Error *err = NULL;
@@ -1600,7 +1601,7 @@ void memory_region_init_ram_from_fd(MemoryRegion *mr,
     mr->destructor = memory_region_destructor_ram;
     mr->ram_block = qemu_ram_alloc_from_fd(size, mr,
                                            share ? RAM_SHARED : 0,
-                                           fd, &err);
+                                           fd, offset, &err);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
     if (err) {
         mr->size = int128_zero();
softmmu/physmem.c +7 -4
index e319fb2..6203ea3 100644
@@ -1461,6 +1461,7 @@ static void *file_ram_alloc(RAMBlock *block,
                             ram_addr_t memory,
                             int fd,
                             bool truncate,
+                            off_t offset,
                             Error **errp)
 {
     void *area;
@@ -1511,7 +1512,8 @@ static void *file_ram_alloc(RAMBlock *block,
     }
 
     area = qemu_ram_mmap(fd, memory, block->mr->align,
-                         block->flags & RAM_SHARED, block->flags & RAM_PMEM);
+                         block->flags & RAM_SHARED, block->flags & RAM_PMEM,
+                         offset);
     if (area == MAP_FAILED) {
         error_setg_errno(errp, errno,
                          "unable to map backing store for guest RAM");
@@ -1943,7 +1945,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)
 #ifdef CONFIG_POSIX
 RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  uint32_t ram_flags, int fd,
-                                 Error **errp)
+                                 off_t offset, Error **errp)
 {
     RAMBlock *new_block;
     Error *local_err = NULL;
@@ -1996,7 +1998,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
     new_block->used_length = size;
     new_block->max_length = size;
     new_block->flags = ram_flags;
-    new_block->host = file_ram_alloc(new_block, size, fd, !file_size, errp);
+    new_block->host = file_ram_alloc(new_block, size, fd, !file_size, offset,
+                                     errp);
     if (!new_block->host) {
         g_free(new_block);
         return NULL;
@@ -2026,7 +2029,7 @@ RAMBlock *qemu_ram_alloc_from_file(ram_addr_t size, MemoryRegion *mr,
         return NULL;
     }
 
-    block = qemu_ram_alloc_from_fd(size, mr, ram_flags, fd, errp);
+    block = qemu_ram_alloc_from_fd(size, mr, ram_flags, fd, 0, errp);
     if (!block) {
         if (created) {
             unlink(mem_path);
util/mmap-alloc.c +4 -3
index 27dcccd..a28f702 100644
@@ -86,7 +86,8 @@ void *qemu_ram_mmap(int fd,
                     size_t size,
                     size_t align,
                     bool shared,
-                    bool is_pmem)
+                    bool is_pmem,
+                    off_t start)
 {
     int flags;
     int map_sync_flags = 0;
@@ -147,7 +148,7 @@ void *qemu_ram_mmap(int fd,
     offset = QEMU_ALIGN_UP((uintptr_t)guardptr, align) - (uintptr_t)guardptr;
 
     ptr = mmap(guardptr + offset, size, PROT_READ | PROT_WRITE,
-               flags | map_sync_flags, fd, 0);
+               flags | map_sync_flags, fd, start);
 
     if (ptr == MAP_FAILED && map_sync_flags) {
         if (errno == ENOTSUP) {
@@ -172,7 +173,7 @@ void *qemu_ram_mmap(int fd,
          * we will remove these flags to handle compatibility.
          */
         ptr = mmap(guardptr + offset, size, PROT_READ | PROT_WRITE,
-                   flags, fd, 0);
+                   flags, fd, start);
     }
 
     if (ptr == MAP_FAILED) {
util/oslib-posix.c +1 -1
index f15234b..93874df 100644
@@ -227,7 +227,7 @@ void *qemu_memalign(size_t alignment, size_t size)
 void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment, bool shared)
 {
     size_t align = QEMU_VMALLOC_ALIGN;
-    void *ptr = qemu_ram_mmap(-1, size, align, shared, false);
+    void *ptr = qemu_ram_mmap(-1, size, align, shared, false, 0);
 
     if (ptr == MAP_FAILED) {
         return NULL;
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v11 2/19 multi-process: Add config option for multi-process QEMU

In reply to Jagannathan Raman

Add a configuration option to separate multi-process code

---
 configure   | 10 ++++++++++
 meson.build |  1 +
 2 files changed, 11 insertions(+)
configure +10 -0
index f839c2a..2025e89 100755
@@ -447,6 +447,7 @@ meson=""
 ninja=""
 skip_meson=no
 gettext=""
+mpqemu="no"
 
 bogus_os="no"
 malloc_trim="auto"
@@ -1520,6 +1521,10 @@ for opt do
   ;;
   --disable-libdaxctl) libdaxctl=no
   ;;
+  --enable-mpqemu) mpqemu=yes
+  ;;
+  --disable-mpqemu) mpqemu=no
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1838,6 +1843,8 @@ disabled with --disable-FEATURE, default is enabled if available:
   xkbcommon       xkbcommon support
   rng-none        dummy RNG, avoid using /dev/(u)random and getrandom()
   libdaxctl       libdaxctl support
+  mpqemu          multi-process QEMU support
+
 
 NOTE: The object files are built at the place where configure is launched
 EOF
@@ -6737,6 +6744,9 @@ fi
 if test "$have_mlockall" = "yes" ; then
   echo "HAVE_MLOCKALL=y" >> $config_host_mak
 fi
+if test "$mpqemu" = "yes" ; then
+  echo "CONFIG_MPQEMU=y" >> $config_host_mak
+fi
 if test "$fuzzing" = "yes" ; then
   # If LIB_FUZZING_ENGINE is set, assume we are running on OSS-Fuzz, and the
   # needed CFLAGS have already been provided
meson.build +1 -0
index 1a4a482..bbcf524 100644
@@ -2091,6 +2091,7 @@ endif
 summary_info += {'thread sanitizer':  config_host.has_key('CONFIG_TSAN')}
 summary_info += {'rng-none':          config_host.has_key('CONFIG_RNG_NONE')}
 summary_info += {'Linux keyring':     config_host.has_key('CONFIG_SECRET_KEYRING')}
+summary_info += {'Multi-Process QEMU': config_host.has_key('CONFIG_MPQEMU')}
 summary(summary_info, bool_yn: true)
 
 if not supported_cpus.contains(cpu)
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [PATCH v11 02/19] multi-process: Add config option for multi-process QEMU

Quoted 35 lines · Jagannathan Raman

Add a configuration option to separate multi-process code

---
  configure   | 10 ++++++++++
  meson.build |  1 +
  2 files changed, 11 insertions(+)
configure +5 -0
index f839c2a..2025e89 100755
@@ -447,6 +447,7 @@ meson=""
  ninja=""
  skip_meson=no
  gettext=""
+mpqemu="no"
 
  bogus_os="no"
  malloc_trim="auto"
@@ -1520,6 +1521,10 @@ for opt do
    ;;
    --disable-libdaxctl) libdaxctl=no
    ;;
+  --enable-mpqemu) mpqemu=yes
+  ;;
+  --disable-mpqemu) mpqemu=no
+  ;;
    *)
        echo "ERROR: unknown option $opt"
        echo "Try '$0 --help' for more information"

configure disabled with --disable-FEATURE, default is enabled if available: · line 1846

Quoted 4 lines
configure +1 -0
@@ -1838,6 +1843,8 @@ disabled with --disable-FEATURE, default is enabled if available:
    xkbcommon       xkbcommon support
    rng-none        dummy RNG, avoid using /dev/(u)random and getrandom()
    libdaxctl       libdaxctl support
+  mpqemu          multi-process QEMU support

When I first heard about this, I was expecting something else. It could be better named as "remote-device emulation" or maybe "manifold (device) emulation" :) Anyway too late I presume.

#4 lore raw

PATCH v11 3/19 multi-process: setup PCI host bridge for remote device

In reply to Jagannathan Raman

PCI host bridge is setup for the remote device process. It is implemented using remote-pcihost object. It is an extension of the PCI host bridge setup by QEMU.
Remote-pcihost configures a PCI bus which could be used by the remote PCI device to latch on to.

---
 MAINTAINERS                  |  8 +++++
 hw/pci-host/meson.build      |  1 +
 hw/pci-host/remote.c         | 75 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci-host/remote.h | 30 ++++++++++++++++++
 4 files changed, 114 insertions(+)
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 include/hw/pci-host/remote.h
MAINTAINERS +8 -0
index 47dd38a..bb1d701 100644
@@ -3095,6 +3095,14 @@ S: Maintained
 F: hw/semihosting/
 F: include/hw/semihosting/
 
+Multi-process QEMU
+M: Jagannathan Raman <[email protected]>
+M: Elena Ufimtseva <[email protected]>
+M: John G Johnson <[email protected]>
+S: Maintained
+F: hw/pci-host/remote.c
+F: include/hw/pci-host/remote.h
+
 Build and test automation
 -------------------------
 Build and test automation
hw/pci-host/meson.build +1 -0
index e6d1b89..1bcecf6 100644
@@ -9,6 +9,7 @@ pci_ss.add(when: 'CONFIG_PCI_EXPRESS_XILINX', if_true: files('xilinx-pcie.c'))
 pci_ss.add(when: 'CONFIG_PCI_I440FX', if_true: files('i440fx.c'))
 pci_ss.add(when: 'CONFIG_PCI_SABRE', if_true: files('sabre.c'))
 pci_ss.add(when: 'CONFIG_XEN_IGD_PASSTHROUGH', if_true: files('xen_igd_pt.c'))
+pci_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 
 # PPC devices
 pci_ss.add(when: 'CONFIG_PREP_PCI', if_true: files('prep.c'))
hw/pci-host/remote.c new file +75 -0

75 lines not on this page · raw message

include/hw/pci-host/remote.h new file +30 -0
new file mode 100644
index 0000000..bab6d3c
@@ -0,0 +1,30 @@
+/*
+ * PCI Host for remote device
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_PCIHOST_H
+#define REMOTE_PCIHOST_H
+
+#include "exec/memory.h"
+#include "hw/pci/pcie_host.h"
+
+#define TYPE_REMOTE_HOST_DEVICE "remote-pcihost"
+#define REMOTE_HOST_DEVICE(obj) \
+    OBJECT_CHECK(RemotePCIHost, (obj), TYPE_REMOTE_HOST_DEVICE)
+
+typedef struct RemotePCIHost {
+    /*< private >*/
+    PCIExpressHost parent_obj;
+    /*< public >*/
+
+    MemoryRegion *mr_pci_mem;
+    MemoryRegion *mr_sys_io;
+} RemotePCIHost;
+
+#endif
Signature · 1 line
1.8.3.1
#5 lore raw

PATCH v11 4/19 multi-process: setup a machine object for remote device process

In reply to Jagannathan Raman

remote-machine object sets up various subsystems of the remote device process. Instantiate PCI host bridge object and initialize RAM, IO & PCI memory regions.

---
 MAINTAINERS                  |  2 ++
 hw/i386/meson.build          |  1 +
 hw/i386/remote.c             | 69 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote.h     | 28 ++++++++++++++++++
 include/hw/pci-host/remote.h |  1 +
 5 files changed, 101 insertions(+)
 create mode 100644 hw/i386/remote.c
 create mode 100644 include/hw/i386/remote.h
MAINTAINERS +2 -0
index bb1d701..12945ee 100644
@@ -3102,6 +3102,8 @@ M: John G Johnson <[email protected]>
 S: Maintained
 F: hw/pci-host/remote.c
 F: include/hw/pci-host/remote.h
+F: hw/i386/remote.c
+F: include/hw/i386/remote.h
 
 Build and test automation
 -------------------------
hw/i386/meson.build +1 -0
index e5d109f..d9d4f40 100644
@@ -24,6 +24,7 @@ i386_ss.add(when: 'CONFIG_PC', if_true: files(
   'pc_sysfw.c',
   'acpi-build.c',
   'port92.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 
 subdir('kvm')
 subdir('xen')
hw/i386/remote.c new file +69 -0

69 lines not on this page · raw message

include/hw/i386/remote.h new file +28 -0
new file mode 100644
index 0000000..d312972
@@ -0,0 +1,28 @@
+/*
+ * Remote machine configuration
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_MACHINE_H
+#define REMOTE_MACHINE_H
+
+#include "qom/object.h"
+#include "hw/boards.h"
+#include "hw/pci-host/remote.h"
+
+typedef struct RemoteMachineState {
+    MachineState parent_obj;
+
+    RemotePCIHost *host;
+} RemoteMachineState;
+
+#define TYPE_REMOTE_MACHINE "x-remote-machine"
+#define REMOTE_MACHINE(obj) \
+    OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE)
+
+#endif
include/hw/pci-host/remote.h +1 -0
index bab6d3c..cc0fff4 100644
@@ -25,6 +25,7 @@ typedef struct RemotePCIHost {
 
     MemoryRegion *mr_pci_mem;
     MemoryRegion *mr_sys_io;
+    MemoryRegion *mr_sys_mem;
 } RemotePCIHost;
 
 #endif
Signature · 1 line
1.8.3.1
#6 lore raw

Re: [PATCH v11 04/19] multi-process: setup a machine object for remote device process

Quoted 87 lines · Jagannathan Raman

remote-machine object sets up various subsystems of the remote device process. Instantiate PCI host bridge object and initialize RAM, IO & PCI memory regions.

---
  MAINTAINERS                  |  2 ++
  hw/i386/meson.build          |  1 +
  hw/i386/remote.c             | 69 ++++++++++++++++++++++++++++++++++++++++++++
  include/hw/i386/remote.h     | 28 ++++++++++++++++++
  include/hw/pci-host/remote.h |  1 +
  5 files changed, 101 insertions(+)

create mode 100644 hw/i386/remote.c
create mode 100644 include/hw/i386/remote.h

MAINTAINERS +2 -0
index bb1d701..12945ee 100644
@@ -3102,6 +3102,8 @@ M: John G Johnson <[email protected]>
  S: Maintained
  F: hw/pci-host/remote.c
  F: include/hw/pci-host/remote.h
+F: hw/i386/remote.c
+F: include/hw/i386/remote.h
 
  Build and test automation
  -------------------------
hw/i386/meson.build +1 -0
index e5d109f..d9d4f40 100644
@@ -24,6 +24,7 @@ i386_ss.add(when: 'CONFIG_PC', if_true: files(
    'pc_sysfw.c',
    'acpi-build.c',
    'port92.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 
  subdir('kvm')
  subdir('xen')
hw/i386/remote.c new file +38 -0
new file mode 100644
index 0000000..a67be33
@@ -0,0 +1,69 @@
+/*
+ * Machine for remote device
+ *
+ *  This machine type is used by the remote device process in multi-process
+ *  QEMU. QEMU device models depend on parent busses, interrupt controllers,
+ *  memory regions, etc. The remote machine type offers this environment so
+ *  that QEMU device models can be used as remote devices.
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+
+#include "hw/i386/remote.h"
+#include "exec/address-spaces.h"
+#include "exec/memory.h"
+#include "qapi/error.h"
+
+static void remote_machine_init(MachineState *machine)
+{
+    MemoryRegion *system_memory, *system_io, *pci_memory;
+    RemoteMachineState *s = REMOTE_MACHINE(machine);
+    RemotePCIHost *rem_host;
+
+    system_memory = get_system_memory();
+    system_io = get_system_io();
+
+    pci_memory = g_new(MemoryRegion, 1);
+    memory_region_init(pci_memory, NULL, "pci", UINT64_MAX);
+
+    rem_host = REMOTE_HOST_DEVICE(qdev_new(TYPE_REMOTE_HOST_DEVICE));
+
+    rem_host->mr_pci_mem = pci_memory;

hw/i386/remote.c line 48

Quoted 10 lines
hw/i386/remote.c new file +10 -0
+    rem_host->mr_sys_mem = system_memory;
+    rem_host->mr_sys_io = system_io;
+
+    s->host = rem_host;
+
+    object_property_add_child(OBJECT(s), "remote-device", OBJECT(rem_host));
+    memory_region_add_subregion_overlap(system_memory, 0x0, pci_memory, -1);
+
+    qdev_realize(DEVICE(rem_host), sysbus_get_default(), &error_fatal);
+}

Nothing is x86-specific in this machine. Why not make it a generic machine, like the 'none' one?

#7 lore raw

PATCH v11 5/19 multi-process: add qio channel function to transmit

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The entire array of the memory regions and file handlers. Will be used in the next patch.

---
 include/io/channel.h | 24 ++++++++++++++++++++++++
 io/channel.c         | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
include/io/channel.h +24 -0
index 3c04f0e..6807506 100644
@@ -773,5 +773,29 @@ void qio_channel_set_aio_fd_handler(QIOChannel *ioc,
                                     IOHandler *io_read,
                                     IOHandler *io_write,
                                     void *opaque);
+/**
+ * qio_channel_writev_full_all:
+ * @ioc: the channel object
+ * @iov: the array of memory regions to write data from
+ * @niov: the length of the @iov array
+ * @fds: an array of file handles to send
+ * @nfds: number of file handles in @fds
+ * @errp: pointer to a NULL-initialized error object
+ *
+ *
+ * Behaves like qio_channel_writev_full but will attempt
+ * to send all data passed (file handles and memory regions).
+ * The function will wait for all requested data
+ * to be written, yielding from the current coroutine
+ * if required.
+ *
+ * Returns: 0 if all bytes were written, or -1 on error
+ */
+
+int qio_channel_writev_full_all(QIOChannel *ioc,
+                           const struct iovec *iov,
+                           size_t niov,
+                           int *fds, size_t nfds,
+                           Error **errp);
 
 #endif /* QIO_CHANNEL_H */
io/channel.c +45 -0

51 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#8 lore raw

Re: [PATCH v11 05/19] multi-process: add qio channel function to transmit

Quoted 3 lines · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The entire array of the memory regions and file handlers.

I don't understand the description... Did you forgot the verb?

Quoted 10 lines

Will be used in the next patch.

---
  include/io/channel.h | 24 ++++++++++++++++++++++++
  io/channel.c         | 45 +++++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 69 insertions(+)
#9 lore raw

Re: [PATCH v11 05/19] multi-process: add qio channel function to transmit

In reply to Jagannathan Raman
Quoted 53 lines · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The entire array of the memory regions and file handlers. Will be used in the next patch.

---
 include/io/channel.h | 24 ++++++++++++++++++++++++
 io/channel.c         | 45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)
include/io/channel.h +24 -0
index 3c04f0e..6807506 100644
@@ -773,5 +773,29 @@ void qio_channel_set_aio_fd_handler(QIOChannel *ioc,
                                     IOHandler *io_read,
                                     IOHandler *io_write,
                                     void *opaque);
+/**
+ * qio_channel_writev_full_all:
+ * @ioc: the channel object
+ * @iov: the array of memory regions to write data from
+ * @niov: the length of the @iov array
+ * @fds: an array of file handles to send
+ * @nfds: number of file handles in @fds
+ * @errp: pointer to a NULL-initialized error object
+ *
+ *
+ * Behaves like qio_channel_writev_full but will attempt
+ * to send all data passed (file handles and memory regions).
+ * The function will wait for all requested data
+ * to be written, yielding from the current coroutine
+ * if required.
+ *
+ * Returns: 0 if all bytes were written, or -1 on error
+ */
+
+int qio_channel_writev_full_all(QIOChannel *ioc,
+                           const struct iovec *iov,
+                           size_t niov,
+                           int *fds, size_t nfds,
+                           Error **errp);
 
 #endif /* QIO_CHANNEL_H */
io/channel.c +0 -0
index e4376eb..22c10c5 100644

io/channel.c int qio_channel_writev_all(QIOChannel *ioc, · line 199

Quoted 10 lines
io/channel.c +6 -0
@@ -190,6 +190,51 @@ int qio_channel_writev_all(QIOChannel *ioc,
     return ret;
 }
 
+int qio_channel_writev_full_all(QIOChannel *ioc,
+                                const struct iovec *iov,
+                                size_t niov,
+                                int *fds, size_t nfds,
+                                Error **errp)
+{
 

Please make qio_channel_writev_all() call qio_channel_writev_full_all() to avoid logic duplication.

thanks

+ int ret = -1;

Quoted 45 lines
io/channel.c +38 -1
+    struct iovec *local_iov = g_new(struct iovec, niov);
+    struct iovec *local_iov_head = local_iov;
+    unsigned int nlocal_iov = niov;
+
+    nlocal_iov = iov_copy(local_iov, nlocal_iov,
+                          iov, niov,
+                          0, iov_size(iov, niov));
+
+    while (nlocal_iov > 0) {
+        ssize_t len;
+        len = qio_channel_writev_full(ioc, local_iov, nlocal_iov, fds,
+                                      nfds, errp);
+        if (len == QIO_CHANNEL_ERR_BLOCK) {
+            if (qemu_in_coroutine()) {
+                qio_channel_yield(ioc, G_IO_OUT);
+            } else {
+                qio_channel_wait(ioc, G_IO_OUT);
+            }
+            continue;
+        }
+        if (len < 0) {
+            goto cleanup;
+        }
+
+        iov_discard_front(&local_iov, &nlocal_iov, len);
+
+        if (len > 0) {
+            fds = NULL;
+            nfds = 0;
+        }
+    }
+
+    ret = 0;
+ cleanup:
+    g_free(local_iov_head);
+    return ret;
+}
+
 ssize_t qio_channel_readv(QIOChannel *ioc,
                           const struct iovec *iov,
                           size_t niov,
--

1.8.3.1

Signature · 1 line
Marc-André Lureau
#10 lore raw

PATCH v11 6/19 multi-process: define MPQemuMsg format and transmission functions

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Defines MPQemuMsg, which is the message that is sent to the remote process. This message is sent over QIOChannel and is used to command the remote process to perform various tasks.
Define transmission functions used by proxy and by remote.
There are certain restrictions on where its safe to use these functions:
- From main loop in co-routine context. Will block the main loop if not in

    co-routine context;
  - From vCPU thread with no co-routine context and if the channel is not part
    of the main loop handling;
  - From IOThread within co-routine context, outside of co-routine context will
    block IOThread;
---
 MAINTAINERS              |   2 +
 include/io/mpqemu-link.h |  60 ++++++++++++
 io/meson.build           |   2 +
 io/mpqemu-link.c         | 237 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 301 insertions(+)
 create mode 100644 include/io/mpqemu-link.h
 create mode 100644 io/mpqemu-link.c
MAINTAINERS +2 -0
index 12945ee..42b5da5 100644
@@ -3104,6 +3104,8 @@ F: hw/pci-host/remote.c
 F: include/hw/pci-host/remote.h
 F: hw/i386/remote.c
 F: include/hw/i386/remote.h
+F: io/mpqemu-link.c
+F: include/io/mpqemu-link.h
 
 Build and test automation
 -------------------------
include/io/mpqemu-link.h new file +60 -0

60 lines not on this page · raw message

io/meson.build +2 -0
index bcd8b1e..d25529c 100644
@@ -13,3 +13,5 @@ io_ss.add(files(
   'net-listener.c',
   'task.c',
 ))
+
+io_ss.add(when: 'CONFIG_MPQEMU', if_true: files('mpqemu-link.c'))
io/mpqemu-link.c new file +237 -0

237 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#11 lore raw

Re: [PATCH v11 06/19] multi-process: define MPQemuMsg format and transmission functions

Quoted 29 lines · Jagannathan Raman
io/mpqemu-link.c new file +29 -0
+void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp)
+{
+    Error *local_err = NULL;
+    int *fds = NULL;
+    size_t nfds = 0;
+    ssize_t len;
+
+    len = mpqemu_read(ioc, (void *)msg, MPQEMU_MSG_HDR_SIZE, &fds, &nfds,
+                      &local_err);
+    if (!local_err) {
+        if (len == -EIO) {
+            error_setg(&local_err, "Connection closed.");
+            goto fail;
+        }
+        if (len < 0) {
+            error_setg(&local_err, "Message length is less than 0");
+            goto fail;
+        }
+        if (len != MPQEMU_MSG_HDR_SIZE) {
+            error_setg(&local_err, "Message header corrupted");
+            goto fail;
+        }
+    } else {
+        goto fail;
+    }
+
+    if (msg->size > sizeof(msg->data)) {
+        error_setg(&local_err, "Invalid size for message");
+        goto fail;

io/mpqemu-link.c line 199

Quoted 10 lines
io/mpqemu-link.c new file +10 -0
+    }
+
+    if (mpqemu_read(ioc, (void *)&msg->data, msg->size, NULL, NULL,
+                    &local_err) < 0) {
+        goto fail;
+    }
+
+    msg->num_fds = nfds;
+    if (nfds) {
+        memcpy(msg->fds, fds, nfds * sizeof(int));

I can't find anything that limits nfds to REMOTE_MAX_FDS. This looks like a buffer overflow.

#12 lore raw

PATCH v11 7/19 multi-process: Initialize message handler in remote device

In reply to Jagannathan Raman

Initializes the message handler function in the remote process. It is called whenever there's an event pending on QIOChannel that registers this function.

---
 MAINTAINERS              |  1 +
 hw/i386/meson.build      |  1 +
 hw/i386/remote-msg.c     | 61 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote.h |  9 +++++++
 4 files changed, 72 insertions(+)
 create mode 100644 hw/i386/remote-msg.c
MAINTAINERS +1 -0
index 42b5da5..7314536 100644
@@ -3106,6 +3106,7 @@ F: hw/i386/remote.c
 F: include/hw/i386/remote.h
 F: io/mpqemu-link.c
 F: include/io/mpqemu-link.h
+F: hw/i386/remote-msg.c
 
 Build and test automation
 -------------------------
hw/i386/meson.build +1 -0
index d9d4f40..3a2aa6e 100644
@@ -25,6 +25,7 @@ i386_ss.add(when: 'CONFIG_PC', if_true: files(
   'acpi-build.c',
   'port92.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-msg.c'))
 
 subdir('kvm')
 subdir('xen')
hw/i386/remote-msg.c new file +61 -0

61 lines not on this page · raw message

include/hw/i386/remote.h +9 -0
index d312972..3073db6 100644
@@ -14,6 +14,7 @@
 #include "qom/object.h"
 #include "hw/boards.h"
 #include "hw/pci-host/remote.h"
+#include "io/channel.h"
 
 typedef struct RemoteMachineState {
     MachineState parent_obj;
@@ -21,8 +22,16 @@ typedef struct RemoteMachineState {
     RemotePCIHost *host;
 } RemoteMachineState;
 
+/* Used to pass to co-routine device and ioc. */
+typedef struct RemoteCommDev {
+    PCIDevice *dev;
+    QIOChannel *ioc;
+} RemoteCommDev;
+
 #define TYPE_REMOTE_MACHINE "x-remote-machine"
 #define REMOTE_MACHINE(obj) \
     OBJECT_CHECK(RemoteMachineState, (obj), TYPE_REMOTE_MACHINE)
 
+void coroutine_fn mpqemu_remote_msg_loop_co(void *data);
+
 #endif
Signature · 1 line
1.8.3.1
#13 lore raw

PATCH v11 8/19 multi-process: Associate fd of a PCIDevice with its object

In reply to Jagannathan Raman

Associate the file descriptor for a PCIDevice in remote process with DeviceState object.

---
 MAINTAINERS                  |   2 +
 hw/i386/meson.build          |   1 +
 hw/i386/remote-msg.c         |   1 +
 hw/i386/remote-obj.c         | 154 +++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote-obj.h |  42 ++++++++++++
 5 files changed, 200 insertions(+)
 create mode 100644 hw/i386/remote-obj.c
 create mode 100644 include/hw/i386/remote-obj.h
MAINTAINERS +2 -0
index 7314536..487ab13 100644
@@ -3107,6 +3107,8 @@ F: include/hw/i386/remote.h
 F: io/mpqemu-link.c
 F: include/io/mpqemu-link.h
 F: hw/i386/remote-msg.c
+F: include/hw/i386/remote-obj.h
+F: hw/i386/remote-obj.c
 
 Build and test automation
 -------------------------
hw/i386/meson.build +1 -0
index 3a2aa6e..9735f6b 100644
@@ -26,6 +26,7 @@ i386_ss.add(when: 'CONFIG_PC', if_true: files(
   'port92.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-msg.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-obj.c'))
 
 subdir('kvm')
 subdir('xen')
hw/i386/remote-msg.c +1 -0
index 83fd7bd..6451b77 100644
@@ -56,6 +56,7 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         }
     }
     qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);
+    g_free(com);
 
     return;
 }
hw/i386/remote-obj.c new file +154 -0

154 lines not on this page · raw message

include/hw/i386/remote-obj.h new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
1.8.3.1
Reviewed-by on v11 8/19
#14 lore raw

Re: [PATCH v11 08/19] multi-process: Associate fd of a PCIDevice with its object

Quoted 6 lines · Jagannathan Raman

Associate the file descriptor for a PCIDevice in remote process with DeviceState object.

Quoted 9 lines
---
 MAINTAINERS                  |   2 +
 hw/i386/meson.build          |   1 +
 hw/i386/remote-msg.c         |   1 +
 hw/i386/remote-obj.c         | 154 +++++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote-obj.h |  42 ++++++++++++
 5 files changed, 200 insertions(+)
 create mode 100644 hw/i386/remote-obj.c
 create mode 100644 include/hw/i386/remote-obj.h
#15 lore raw

PATCH v11 9/19 multi-process: setup memory manager for remote device

In reply to Jagannathan Raman

SyncSysMemMsg message format is defined. It is used to send file descriptors of the RAM regions to remote device.
RAM on the remote device is configured with a set of file descriptors. Old RAM regions are deleted and new regions, each with an fd, is added to the RAM.

---
 MAINTAINERS                     |  2 ++
 hw/i386/meson.build             |  1 +
 hw/i386/remote-memory.c         | 58 +++++++++++++++++++++++++++++++++++++++++
 include/hw/i386/remote-memory.h | 19 ++++++++++++++
 include/io/mpqemu-link.h        | 13 +++++++++
 io/mpqemu-link.c                | 11 ++++++++
 6 files changed, 104 insertions(+)
 create mode 100644 hw/i386/remote-memory.c
 create mode 100644 include/hw/i386/remote-memory.h
MAINTAINERS +2 -0
index 487ab13..4aa00ec 100644
@@ -3109,6 +3109,8 @@ F: include/io/mpqemu-link.h
 F: hw/i386/remote-msg.c
 F: include/hw/i386/remote-obj.h
 F: hw/i386/remote-obj.c
+F: include/hw/i386/remote-memory.h
+F: hw/i386/remote-memory.c
 
 Build and test automation
 -------------------------
hw/i386/meson.build +1 -0
index 9735f6b..b698b36 100644
@@ -27,6 +27,7 @@ i386_ss.add(when: 'CONFIG_PC', if_true: files(
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-msg.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-obj.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-memory.c'))
 
 subdir('kvm')
 subdir('xen')
hw/i386/remote-memory.c new file +58 -0

58 lines not on this page · raw message

include/hw/i386/remote-memory.h new file +19 -0
new file mode 100644
index 0000000..aab3e4d
@@ -0,0 +1,19 @@
+/*
+ * Memory manager for remote device
+ *
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef REMOTE_MEMORY_H
+#define REMOTE_MEMORY_H
+
+#include "exec/hwaddr.h"
+#include "io/mpqemu-link.h"
+
+void remote_sysmem_reconfig(MPQemuMsg *msg, Error **errp);
+
+#endif
include/io/mpqemu-link.h +13 -0
index 3505e06..e33181a 100644
@@ -14,6 +14,7 @@
 #include "qom/object.h"
 #include "qemu/thread.h"
 #include "io/channel.h"
+#include "exec/hwaddr.h"
 
 #define REMOTE_MAX_FDS 8
 
@@ -24,12 +25,22 @@
  *
  * MPQemuCmd enum type to specify the command to be executed on the remote
  * device.
+ *
+ * SYNC_SYSMEM      Shares QEMU's RAM with remote device's RAM
  */
 typedef enum {
     MPQEMU_CMD_INIT,
+    SYNC_SYSMEM,
+    RET_MSG,
     MPQEMU_CMD_MAX,
 } MPQemuCmd;
 
+typedef struct {
+    hwaddr gpas[REMOTE_MAX_FDS];
+    uint64_t sizes[REMOTE_MAX_FDS];
+    off_t offsets[REMOTE_MAX_FDS];
+} SyncSysmemMsg;
+
 /**
  * MPQemuMsg:
  * @cmd: The remote command
@@ -40,12 +51,14 @@ typedef enum {
  * MPQemuMsg Format of the message sent to the remote device from QEMU.
  *
  */
+
 typedef struct {
     int cmd;
     size_t size;
 
     union {
         uint64_t u64;
+        SyncSysmemMsg sync_sysmem;
     } data;
 
     int fds[REMOTE_MAX_FDS];
io/mpqemu-link.c +11 -0
index d646aad..b30e27c 100644
@@ -233,5 +233,16 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
         }
     }
 
+     /* Verify message specific fields. */
+    switch (msg->cmd) {
+    case SYNC_SYSMEM:
+        if (msg->num_fds == 0 || msg->size != sizeof(SyncSysmemMsg)) {
+            return false;
+        }
+        break;
+    default:
+        break;
+    }
+
     return true;
 }
Signature · 1 line
1.8.3.1
#16 lore raw

PATCH v11 10/19 multi-process: introduce proxy object

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Defines a PCI Device proxy object as a child of TYPE_PCI_DEVICE.

---
 MAINTAINERS            |  2 ++
 hw/pci/meson.build     |  1 +
 hw/pci/proxy.c         | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h | 36 +++++++++++++++++++
 4 files changed, 137 insertions(+)
 create mode 100644 hw/pci/proxy.c
 create mode 100644 include/hw/pci/proxy.h
MAINTAINERS +2 -0
index 4aa00ec..202db86 100644
@@ -3111,6 +3111,8 @@ F: include/hw/i386/remote-obj.h
 F: hw/i386/remote-obj.c
 F: include/hw/i386/remote-memory.h
 F: hw/i386/remote-memory.c
+F: hw/pci/proxy.c
+F: include/hw/pci/proxy.h
 
 Build and test automation
 -------------------------
hw/pci/meson.build +1 -0
index 5c4bbac..0df3017 100644
@@ -12,6 +12,7 @@ pci_ss.add(files(
 # allow plugging PCIe devices into PCI buses, include them even if
 # CONFIG_PCI_EXPRESS=n.
 pci_ss.add(files('pcie.c', 'pcie_aer.c'))
+pci_ss.add(when: 'CONFIG_MPQEMU', if_true: files('proxy.c'))
 softmmu_ss.add(when: 'CONFIG_PCI_EXPRESS', if_true: files('pcie_port.c', 'pcie_host.c'))
 softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
 
hw/pci/proxy.c new file +98 -0

98 lines not on this page · raw message

include/hw/pci/proxy.h new file +36 -0
new file mode 100644
index 0000000..923432a
@@ -0,0 +1,36 @@
+/*
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PROXY_H
+#define PROXY_H
+
+#include "hw/pci/pci.h"
+#include "io/channel.h"
+
+#define TYPE_PCI_PROXY_DEV "x-pci-proxy-dev"
+
+#define PCI_PROXY_DEV(obj) \
+            OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV)
+typedef struct PCIProxyDev PCIProxyDev;
+
+struct PCIProxyDev {
+    PCIDevice parent_dev;
+    char *fd;
+
+    /*
+     * Mutex used to protect the QIOChannel fd from
+     * the concurrent access by the VCPUs since proxy
+     * blocks while awaiting for the replies from the
+     * process remote.
+     */
+    QemuMutex io_mutex;
+    QIOChannel *ioc;
+    Error *migration_blocker;
+};
+
+#endif /* PROXY_H */
Signature · 1 line
1.8.3.1
Reviewed-by on v11 10/19
#17 lore raw

Re: [PATCH v11 10/19] multi-process: introduce proxy object

Quoted 7 lines · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Defines a PCI Device proxy object as a child of TYPE_PCI_DEVICE.

Quoted 8 lines
---
 MAINTAINERS            |  2 ++
 hw/pci/meson.build     |  1 +
 hw/pci/proxy.c         | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h | 36 +++++++++++++++++++
 4 files changed, 137 insertions(+)
 create mode 100644 hw/pci/proxy.c
 create mode 100644 include/hw/pci/proxy.h
#18 lore raw

PATCH v11 11/19 multi-process: add proxy communication functions

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

---
 include/io/mpqemu-link.h |  4 ++++
 io/mpqemu-link.c         | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
include/io/mpqemu-link.h +4 -0
index e33181a..4bea5da 100644
@@ -15,6 +15,8 @@
 #include "qemu/thread.h"
 #include "io/channel.h"
 #include "exec/hwaddr.h"
+#include "io/channel-socket.h"
+#include "hw/pci/proxy.h"
 
 #define REMOTE_MAX_FDS 8
 
@@ -65,6 +67,8 @@ typedef struct {
     int num_fds;
 } QEMU_PACKED MPQemuMsg;
 
+uint64_t mpqemu_msg_send_and_await_reply(MPQemuMsg *msg, PCIProxyDev *pdev,
+                                         Error **errp);
 void mpqemu_msg_send(MPQemuMsg *msg, QIOChannel *ioc, Error **errp);
 void mpqemu_msg_recv(MPQemuMsg *msg, QIOChannel *ioc, Error **errp);
 
io/mpqemu-link.c +38 -0
index b30e27c..1339cc7 100644
@@ -17,6 +17,7 @@
 #include "qemu/iov.h"
 #include "qemu/error-report.h"
 #include "qemu/main-loop.h"
+#include "io/channel.h"
 
 /*
  * Send message over the ioc QIOChannel.
@@ -214,6 +215,43 @@ fail:
     }
 }
 
+/*
+ * Called from VCPU thread in non-coroutine context.
+ */
+uint64_t mpqemu_msg_send_and_await_reply(MPQemuMsg *msg, PCIProxyDev *pdev,
+                                         Error **errp)
+{
+    MPQemuMsg msg_reply = {0};
+    uint64_t ret = UINT64_MAX;
+    Error *local_err = NULL;
+
+    qemu_mutex_lock(&pdev->io_mutex);
+    mpqemu_msg_send(msg, pdev->ioc, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        goto exit_send;
+    }
+
+    mpqemu_msg_recv(&msg_reply, pdev->ioc, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        goto exit_send;
+    }
+
+    if (!mpqemu_msg_valid(&msg_reply) || msg_reply.cmd != RET_MSG) {
+        error_setg(errp, "ERROR: Invalid reply received for command %d",
+                         msg->cmd);
+        goto exit_send;
+    } else {
+        ret = msg_reply.data.u64;
+    }
+
+ exit_send:
+    qemu_mutex_unlock(&pdev->io_mutex);
+
+    return ret;
+}
+
 bool mpqemu_msg_valid(MPQemuMsg *msg)
 {
     if (msg->cmd >= MPQEMU_CMD_MAX && msg->cmd < 0) {
Signature · 1 line
1.8.3.1
#19 lore raw

PATCH v11 12/19 multi-process: Forward PCI config space acceses to the remote process

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The Proxy Object sends the PCI config space accesses as messages to the remote process over the communication channel

TODO:
Investigate if the local PCI config writes can be dropped. Without the proxy local PCI config space writes for the device, the driver in the guest times out on the probing.
We have tried to only refer to the remote for the PCI config writes, but the driver timeout in the guest forced as to left this as it is (removing local PCI config only).

---
 hw/i386/remote-msg.c     | 62 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/pci/proxy.c           | 50 ++++++++++++++++++++++++++++++++++++++
 include/io/mpqemu-link.h |  9 +++++++
 io/mpqemu-link.c         |  6 +++++
 4 files changed, 127 insertions(+)
hw/i386/remote-msg.c +62 -0
index 6451b77..94937db 100644
@@ -15,6 +15,12 @@
 #include "io/mpqemu-link.h"
 #include "qapi/error.h"
 #include "sysemu/runstate.h"
+#include "hw/pci/pci.h"
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg);
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg);
 
 void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
 {
@@ -43,6 +49,12 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         }
 
         switch (msg.cmd) {
+        case PCI_CONFIG_WRITE:
+            process_config_write(com->ioc, pci_dev, &msg);
+            break;
+        case PCI_CONFIG_READ:
+            process_config_read(com->ioc, pci_dev, &msg);
+            break;
         default:
             error_setg(&local_err,
                        "Unknown command (%d) received for device %s (pid=%d)",
@@ -60,3 +72,53 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
 
     return;
 }
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)&msg->data.conf_data;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when writing PCI config, pid %d",
+                     getpid());
+        ret.data.u64 = UINT64_MAX;
+    } else {
+        pci_default_write_config(dev, conf->addr, conf->val, conf->l);
+    }
+
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+}
+
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)&msg->data.conf_data;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when reading PCI config, pid %d",
+                     getpid());
+        ret.data.u64 = UINT64_MAX;
+    } else {
+        ret.data.u64 = pci_default_read_config(dev, conf->addr, conf->l);
+    }
+
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+}
hw/pci/proxy.c +50 -0

68 lines not on this page · raw message

include/io/mpqemu-link.h +9 -0
index 4bea5da..459d345 100644
@@ -34,6 +34,8 @@ typedef enum {
     MPQEMU_CMD_INIT,
     SYNC_SYSMEM,
     RET_MSG,
+    PCI_CONFIG_WRITE,
+    PCI_CONFIG_READ,
     MPQEMU_CMD_MAX,
 } MPQemuCmd;
 
@@ -43,6 +45,12 @@ typedef struct {
     off_t offsets[REMOTE_MAX_FDS];
 } SyncSysmemMsg;
 
+typedef struct {
+    uint32_t addr;
+    uint32_t val;
+    int l;
+} ConfDataMsg;
+
 /**
  * MPQemuMsg:
  * @cmd: The remote command
@@ -60,6 +68,7 @@ typedef struct {
 
     union {
         uint64_t u64;
+        ConfDataMsg conf_data;
         SyncSysmemMsg sync_sysmem;
     } data;
 
io/mpqemu-link.c +6 -0
index 1339cc7..5ef82da 100644
@@ -278,6 +278,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case PCI_CONFIG_WRITE:
+    case PCI_CONFIG_READ:
+        if (msg->size != sizeof(ConfDataMsg)) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#20 lore raw

Re: [PATCH v11 12/19] multi-process: Forward PCI config space acceses to the remote process

Quoted 5 lines · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The Proxy Object sends the PCI config space accesses as messages to the remote process over the communication channel

Quoted 7 lines

TODO:
Investigate if the local PCI config writes can be dropped. Without the proxy local PCI config space writes for the device, the driver in the guest times out on the probing.
We have tried to only refer to the remote for the PCI config writes, but the driver timeout in the guest forced as to left this as it is (removing local PCI config only).

I expect this TODO to be lost in git history. Better to commit it in the code and remove it once resolved IMHO.

Quoted 11 lines
---
  hw/i386/remote-msg.c     | 62 ++++++++++++++++++++++++++++++++++++++++++++++++
  hw/pci/proxy.c           | 50 ++++++++++++++++++++++++++++++++++++++
  include/io/mpqemu-link.h |  9 +++++++
  io/mpqemu-link.c         |  6 +++++
  4 files changed, 127 insertions(+)

Please have a look at scripts/git.orderfile, it helps reviewers to avoid scrolling down/up the patches.

Quoted 64 lines
hw/i386/remote-msg.c +41 -0
index 6451b77..94937db 100644
@@ -15,6 +15,12 @@
  #include "io/mpqemu-link.h"
  #include "qapi/error.h"
  #include "sysemu/runstate.h"
+#include "hw/pci/pci.h"
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg);
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg);
 
  void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
  {
@@ -43,6 +49,12 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
          }
 
          switch (msg.cmd) {
+        case PCI_CONFIG_WRITE:
+            process_config_write(com->ioc, pci_dev, &msg);
+            break;
+        case PCI_CONFIG_READ:
+            process_config_read(com->ioc, pci_dev, &msg);
+            break;
          default:
              error_setg(&local_err,
                         "Unknown command (%d) received for device %s (pid=%d)",
@@ -60,3 +72,53 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
 
      return;
  }
+
+static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
+                                 MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)&msg->data.conf_data;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when writing PCI config, pid %d",
+                     getpid());
+        ret.data.u64 = UINT64_MAX;
+    } else {
+        pci_default_write_config(dev, conf->addr, conf->val, conf->l);
+    }
+
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+}
+
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg)
+{

hw/i386/remote-msg.c void coroutine_fn mpqemu_remote_msg_loop_co(void *data) · line 113

Quoted 10 lines
hw/i386/remote-msg.c +10 -0
+    ConfDataMsg *conf = (ConfDataMsg *)&msg->data.conf_data;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when reading PCI config, pid %d",
+                     getpid());
+        ret.data.u64 = UINT64_MAX;
+    } else {
+        ret.data.u64 = pci_default_read_config(dev, conf->addr, conf->l);

Isn't it endianess issue here? It might makes sense to declare ConfDataMsg and PCI_CONFIG_READ/PCI_CONFIG_WRITE packets in little endian.

Quoted 100 lines
hw/i386/remote-msg.c +11 -0
+    }
+
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_report("Could not send message to proxy from pid %d",
+                     getpid());
+    }
+}
hw/pci/proxy.c +50 -0
index 7a12f23..27714fe 100644
@@ -16,6 +16,8 @@
  #include "hw/qdev-properties.h"
  #include "monitor/monitor.h"
  #include "migration/blocker.h"
+#include "io/mpqemu-link.h"
+#include "qemu/error-report.h"
 
  static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
  {
@@ -69,6 +71,51 @@ static void pci_proxy_dev_exit(PCIDevice *pdev)
      error_free(dev->migration_blocker);
  }
 
+static int config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val,
+                          int l, unsigned int op)
+{
+    MPQemuMsg msg = { 0 };
+    uint64_t ret = -EINVAL;
+    Error *local_err = NULL;
+
+    msg.cmd = op;
+    msg.data.conf_data.addr = addr;
+    msg.data.conf_data.val = (op == PCI_CONFIG_WRITE) ? *val : 0;
+    msg.data.conf_data.l = l;
+    msg.size = sizeof(ConfDataMsg);
+
+    ret = mpqemu_msg_send_and_await_reply(&msg, pdev, &local_err);
+    if (local_err) {
+        error_report_err(local_err);
+    }
+    if (op == PCI_CONFIG_READ) {
+        *val = (uint32_t)ret;
+    }
+
+    return ret;
+}
+
+static uint32_t pci_proxy_read_config(PCIDevice *d, uint32_t addr, int len)
+{
+    uint32_t val;
+
+    (void)config_op_send(PCI_PROXY_DEV(d), addr, &val, len, PCI_CONFIG_READ);
+
+    return val;
+}
+
+static void pci_proxy_write_config(PCIDevice *d, uint32_t addr, uint32_t val,
+                                   int l)
+{
+    /*
+     * Some of the functions access the copy of the remote device
+     * PCI config space, therefore maintain it updated.
+     */
+    pci_default_write_config(d, addr, val, l);
+
+    (void)config_op_send(PCI_PROXY_DEV(d), addr, &val, l, PCI_CONFIG_WRITE);
+}
+
  static void pci_proxy_dev_class_init(ObjectClass *klass, void *data)
  {
      DeviceClass *dc = DEVICE_CLASS(klass);
@@ -76,6 +123,9 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data)
 
      k->realize = pci_proxy_dev_realize;
      k->exit = pci_proxy_dev_exit;
+    k->config_read = pci_proxy_read_config;
+    k->config_write = pci_proxy_write_config;
+
      device_class_set_props(dc, proxy_properties);
  }
 
include/io/mpqemu-link.h +2 -0
index 4bea5da..459d345 100644
@@ -34,6 +34,8 @@ typedef enum {
      MPQEMU_CMD_INIT,
      SYNC_SYSMEM,
      RET_MSG,
+    PCI_CONFIG_WRITE,
+    PCI_CONFIG_READ,
      MPQEMU_CMD_MAX,
  } MPQemuCmd;
 

include/io/mpqemu-link.h typedef struct { · line 52

Quoted 8 lines
include/io/mpqemu-link.h +5 -0
@@ -43,6 +45,12 @@ typedef struct {
      off_t offsets[REMOTE_MAX_FDS];
  } SyncSysmemMsg;
 
+typedef struct {
+    uint32_t addr;
+    uint32_t val;
+    int l;
+} ConfDataMsg;

Maybe name this PciConfDataMsg?

Quoted 30 lines
include/io/mpqemu-link.h +2 -0
+
  /**
   * MPQemuMsg:
   * @cmd: The remote command
@@ -60,6 +68,7 @@ typedef struct {
 
      union {
          uint64_t u64;
+        ConfDataMsg conf_data;
          SyncSysmemMsg sync_sysmem;
      } data;
 
io/mpqemu-link.c +6 -0
index 1339cc7..5ef82da 100644
@@ -278,6 +278,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
              return false;
          }
          break;
+    case PCI_CONFIG_WRITE:
+    case PCI_CONFIG_READ:
+        if (msg->size != sizeof(ConfDataMsg)) {
+            return false;
+        }
+        break;
      default:
          break;
      }
#21 lore raw

Re: [PATCH v11 12/19] multi-process: Forward PCI config space acceses to the remote process

Quoted 5 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

The Proxy Object sends the PCI config space accesses as messages to the remote process over the communication channel

...

Quoted 11 lines
Quoted 11 lines
hw/i386/remote-msg.c +9 -0
+static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
+                                MPQemuMsg *msg)
+{
+    ConfDataMsg *conf = (ConfDataMsg *)&msg->data.conf_data;
+    MPQemuMsg ret = { 0 };
+    Error *local_err = NULL;
+
+    if ((conf->addr + sizeof(conf->val)) > pci_config_size(dev)) {
+        error_report("Bad address received when reading PCI config, 

pid %d",
+                     getpid());

Quoted 8 lines
Quoted 4 lines

+        ret.data.u64 = UINT64_MAX;
+    } else {
+        ret.data.u64 = pci_default_read_config(dev, conf->addr, conf->l);

Isn't it endianess issue here? It might makes sense to declare ConfDataMsg and PCI_CONFIG_READ/PCI_CONFIG_WRITE packets in little endian.

Bah Stefan told me to look at the specs which are included in the latest patch (docs/multi-process.rst). As I process patches of this series in order I haven't looked at it yet. Still, why not put it first in your series?

Regards,

Phil.

#22 lore raw

PATCH v11 13/19 multi-process: PCI BAR read/write handling for proxy & remote endpoints

In reply to Jagannathan Raman

Proxy device object implements handler for PCI BAR writes and reads. The handler uses BAR_WRITE/BAR_READ message to communicate to the remote process with the BAR address and value to be written/read. The remote process implements handler for BAR_WRITE/BAR_READ message.

---
 hw/i386/remote-msg.c     | 87 ++++++++++++++++++++++++++++++++++++++++++++++++
 hw/pci/proxy.c           | 60 +++++++++++++++++++++++++++++++++
 include/hw/pci/proxy.h   | 10 ++++++
 include/io/mpqemu-link.h | 10 ++++++
 io/mpqemu-link.c         |  6 ++++
 5 files changed, 173 insertions(+)
hw/i386/remote-msg.c +87 -0
index 94937db..9b9ca7e 100644
@@ -16,11 +16,14 @@
 #include "qapi/error.h"
 #include "sysemu/runstate.h"
 #include "hw/pci/pci.h"
+#include "exec/memattrs.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
 static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
                                 MPQemuMsg *msg);
+static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
+static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 
 void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
 {
@@ -55,6 +58,12 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         case PCI_CONFIG_READ:
             process_config_read(com->ioc, pci_dev, &msg);
             break;
+        case BAR_WRITE:
+            process_bar_write(com->ioc, &msg, &local_err);
+            break;
+        case BAR_READ:
+            process_bar_read(com->ioc, &msg, &local_err);
+            break;
         default:
             error_setg(&local_err,
                        "Unknown command (%d) received for device %s (pid=%d)",
@@ -122,3 +131,81 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
                      getpid());
     }
 }
+
+static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data.bar_access;
+    AddressSpace *as =
+        bar_access->memory ? &address_space_memory : &address_space_io;
+    MPQemuMsg ret = { 0 };
+    MemTxResult res;
+    uint64_t val;
+    Error *local_err = NULL;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        ret.data.u64 = UINT64_MAX;
+        goto fail;
+    }
+
+    val = cpu_to_le64(bar_access->val);
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, true);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space write operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        ret.data.u64 = -1;
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());
+    }
+}
+
+static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp)
+{
+    BarAccessMsg *bar_access = &msg->data.bar_access;
+    MPQemuMsg ret = { 0 };
+    AddressSpace *as;
+    MemTxResult res;
+    uint64_t val = 0;
+    Error *local_err = NULL;
+
+    as = bar_access->memory ? &address_space_memory : &address_space_io;
+
+    if (!is_power_of_2(bar_access->size) ||
+       (bar_access->size > sizeof(uint64_t))) {
+        val = UINT64_MAX;
+        goto fail;
+    }
+
+    res = address_space_rw(as, bar_access->addr, MEMTXATTRS_UNSPECIFIED,
+                           (void *)&val, bar_access->size, false);
+
+    if (res != MEMTX_OK) {
+        error_setg(errp, "Could not perform address space read operation,"
+                   " inaccessible address: %lx in pid %d.",
+                   bar_access->addr, getpid());
+        val = UINT64_MAX;
+    }
+
+fail:
+    ret.cmd = RET_MSG;
+    ret.data.u64 = le64_to_cpu(val);
+    ret.size = sizeof(ret.data.u64);
+
+    mpqemu_msg_send(&ret, ioc, &local_err);
+    if (local_err) {
+        error_setg(errp, "Error while sending message to proxy "
+                   "in remote process pid=%d", getpid());
+    }
+}
hw/pci/proxy.c +60 -0

63 lines not on this page · raw message

include/hw/pci/proxy.h +10 -0
index 923432a..e29c61b 100644
@@ -16,8 +16,17 @@
 
 #define PCI_PROXY_DEV(obj) \
             OBJECT_CHECK(PCIProxyDev, (obj), TYPE_PCI_PROXY_DEV)
+
 typedef struct PCIProxyDev PCIProxyDev;
 
+typedef struct ProxyMemoryRegion {
+    PCIProxyDev *dev;
+    MemoryRegion mr;
+    bool memory;
+    bool present;
+    uint8_t type;
+} ProxyMemoryRegion;
+
 struct PCIProxyDev {
     PCIDevice parent_dev;
     char *fd;
@@ -31,6 +40,7 @@ struct PCIProxyDev {
     QemuMutex io_mutex;
     QIOChannel *ioc;
     Error *migration_blocker;
+    ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
 #endif /* PROXY_H */
include/io/mpqemu-link.h +10 -0
index 459d345..234ac82 100644
@@ -36,6 +36,8 @@ typedef enum {
     RET_MSG,
     PCI_CONFIG_WRITE,
     PCI_CONFIG_READ,
+    BAR_WRITE,
+    BAR_READ,
     MPQEMU_CMD_MAX,
 } MPQemuCmd;
 
@@ -51,6 +53,13 @@ typedef struct {
     int l;
 } ConfDataMsg;
 
+typedef struct {
+    hwaddr addr;
+    uint64_t val;
+    unsigned size;
+    bool memory;
+} BarAccessMsg;
+
 /**
  * MPQemuMsg:
  * @cmd: The remote command
@@ -70,6 +79,7 @@ typedef struct {
         uint64_t u64;
         ConfDataMsg conf_data;
         SyncSysmemMsg sync_sysmem;
+        BarAccessMsg bar_access;
     } data;
 
     int fds[REMOTE_MAX_FDS];
io/mpqemu-link.c +6 -0
index 5ef82da..7e297ef 100644
@@ -284,6 +284,12 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case BAR_WRITE:
+    case BAR_READ:
+        if ((msg->size != sizeof(BarAccessMsg)) || (msg->num_fds != 0)) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#23 lore raw

PATCH v11 14/19 multi-process: Synchronize remote memory

In reply to Jagannathan Raman

Add memory-listener object which is used to keep the view of the RAM in sync between QEMU and remote process.
A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message.

---
 MAINTAINERS                  |   2 +
 hw/i386/remote-msg.c         |   5 ++
 hw/pci/memory-sync.c         | 210 +++++++++++++++++++++++++++++++++++++++++++
 hw/pci/meson.build           |   2 +
 hw/pci/proxy.c               |   6 ++
 include/hw/pci/memory-sync.h |  27 ++++++
 include/hw/pci/proxy.h       |   2 +
 7 files changed, 254 insertions(+)
 create mode 100644 hw/pci/memory-sync.c
 create mode 100644 include/hw/pci/memory-sync.h
MAINTAINERS +2 -0
index 202db86..8ee717c 100644
@@ -3113,6 +3113,8 @@ F: include/hw/i386/remote-memory.h
 F: hw/i386/remote-memory.c
 F: hw/pci/proxy.c
 F: include/hw/pci/proxy.h
+F: hw/pci/memory-sync.c
+F: include/hw/pci/memory-sync.h
 
 Build and test automation
 -------------------------
hw/i386/remote-msg.c +5 -0
index 9b9ca7e..3465a9d 100644
@@ -17,6 +17,7 @@
 #include "sysemu/runstate.h"
 #include "hw/pci/pci.h"
 #include "exec/memattrs.h"
+#include "hw/i386/remote-memory.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -64,6 +65,10 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         case BAR_READ:
             process_bar_read(com->ioc, &msg, &local_err);
             break;
+        case SYNC_SYSMEM:
+            remote_sysmem_reconfig(&msg, &local_err);
+            break;
+
         default:
             error_setg(&local_err,
                        "Unknown command (%d) received for device %s (pid=%d)",
hw/pci/memory-sync.c new file +210 -0

210 lines not on this page · raw message

hw/pci/meson.build +2 -0
index 0df3017..7947a62 100644
@@ -18,3 +18,5 @@ softmmu_ss.add_all(when: 'CONFIG_PCI', if_true: pci_ss)
 
 softmmu_ss.add(when: 'CONFIG_PCI', if_false: files('pci-stub.c'))
 softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('pci-stub.c'))
+
+specific_ss.add(when: 'CONFIG_MPQEMU', if_true: files('memory-sync.c'))
hw/pci/proxy.c +6 -0
index 4ae4919..908c82e 100644
@@ -18,6 +18,8 @@
 #include "migration/blocker.h"
 #include "io/mpqemu-link.h"
 #include "qemu/error-report.h"
+#include "hw/pci/memory-sync.h"
+#include "qom/object.h"
 
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
@@ -58,6 +60,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
 
     qemu_mutex_init(&dev->io_mutex);
     qio_channel_set_blocking(dev->ioc, true, NULL);
+
+    configure_memory_sync(&dev->sync, dev->ioc);
 }
 
 static void pci_proxy_dev_exit(PCIDevice *pdev)
@@ -69,6 +73,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev)
     migrate_del_blocker(dev->migration_blocker);
 
     error_free(dev->migration_blocker);
+
+    deconfigure_memory_sync(&dev->sync);
 }
 
 static int config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val,
include/hw/pci/memory-sync.h new file +27 -0
new file mode 100644
index 0000000..785f76a
@@ -0,0 +1,27 @@
+/*
+ * Copyright © 2018, 2020 Oracle and/or its affiliates.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef MEMORY_SYNC_H
+#define MEMORY_SYNC_H
+
+#include "exec/memory.h"
+#include "io/channel.h"
+
+typedef struct RemoteMemSync {
+    MemoryListener listener;
+
+    int n_mr_sections;
+    MemoryRegionSection *mr_sections;
+
+    QIOChannel *ioc;
+} RemoteMemSync;
+
+void configure_memory_sync(RemoteMemSync *sync, QIOChannel *ioc);
+void deconfigure_memory_sync(RemoteMemSync *sync);
+
+#endif
include/hw/pci/proxy.h +2 -0
index e29c61b..52f3c43 100644
@@ -11,6 +11,7 @@
 
 #include "hw/pci/pci.h"
 #include "io/channel.h"
+#include "hw/pci/memory-sync.h"
 
 #define TYPE_PCI_PROXY_DEV "x-pci-proxy-dev"
 
@@ -40,6 +41,7 @@ struct PCIProxyDev {
     QemuMutex io_mutex;
     QIOChannel *ioc;
     Error *migration_blocker;
+    RemoteMemSync sync;
     ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
Signature · 1 line
1.8.3.1
#24 lore raw

PATCH v11 15/19 multi-process: create IOHUB object to handle irq

In reply to Jagannathan Raman

IOHUB object is added to manage PCI IRQs. It uses KVM_IRQFD ioctl to create irqfd to injecting PCI interrupts to the guest. IOHUB object forwards the irqfd to the remote process. Remote process uses this fd to directly send interrupts to the guest, bypassing QEMU.

---
 MAINTAINERS                    |   2 +
 hw/i386/meson.build            |   1 +
 hw/i386/remote-iohub.c         | 123 +++++++++++++++++++++++++++++++++++++++++
 hw/i386/remote-msg.c           |   4 ++
 hw/i386/remote.c               |  10 ++++
 hw/pci/proxy.c                 |  58 +++++++++++++++++++
 include/hw/i386/remote-iohub.h |  42 ++++++++++++++
 include/hw/i386/remote.h       |   3 +
 include/hw/pci/pci_ids.h       |   3 +
 include/hw/pci/proxy.h         |   5 ++
 include/io/mpqemu-link.h       |   1 +
 io/mpqemu-link.c               |   5 ++
 12 files changed, 257 insertions(+)
 create mode 100644 hw/i386/remote-iohub.c
 create mode 100644 include/hw/i386/remote-iohub.h
MAINTAINERS +2 -0
index 8ee717c..f33de17 100644
@@ -3115,6 +3115,8 @@ F: hw/pci/proxy.c
 F: include/hw/pci/proxy.h
 F: hw/pci/memory-sync.c
 F: include/hw/pci/memory-sync.h
+F: hw/i386/remote-iohub.c
+F: include/hw/i386/remote-iohub.h
 
 Build and test automation
 -------------------------
hw/i386/meson.build +1 -0
index b698b36..1442775 100644
@@ -28,6 +28,7 @@ i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-msg.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-obj.c'))
 i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-memory.c'))
+i386_ss.add(when: 'CONFIG_MPQEMU', if_true: files('remote-iohub.c'))
 
 subdir('kvm')
 subdir('xen')
hw/i386/remote-iohub.c new file +123 -0

123 lines not on this page · raw message

hw/i386/remote-msg.c +4 -0
index 3465a9d..a318a9f 100644
@@ -18,6 +18,7 @@
 #include "hw/pci/pci.h"
 #include "exec/memattrs.h"
 #include "hw/i386/remote-memory.h"
+#include "hw/i386/remote-iohub.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -68,6 +69,9 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         case SYNC_SYSMEM:
             remote_sysmem_reconfig(&msg, &local_err);
             break;
+        case SET_IRQFD:
+            process_set_irqfd_msg(pci_dev, &msg);
+            break;
 
         default:
             error_setg(&local_err,
hw/i386/remote.c +10 -0
index a67be33..95de652 100644
@@ -20,12 +20,15 @@
 #include "exec/address-spaces.h"
 #include "exec/memory.h"
 #include "qapi/error.h"
+#include "hw/pci/pci_host.h"
+#include "hw/i386/remote-iohub.h"
 
 static void remote_machine_init(MachineState *machine)
 {
     MemoryRegion *system_memory, *system_io, *pci_memory;
     RemoteMachineState *s = REMOTE_MACHINE(machine);
     RemotePCIHost *rem_host;
+    PCIHostState *pci_host;
 
     system_memory = get_system_memory();
     system_io = get_system_io();
@@ -45,6 +48,13 @@ static void remote_machine_init(MachineState *machine)
     memory_region_add_subregion_overlap(system_memory, 0x0, pci_memory, -1);
 
     qdev_realize(DEVICE(rem_host), sysbus_get_default(), &error_fatal);
+
+    pci_host = PCI_HOST_BRIDGE(rem_host);
+
+    remote_iohub_init(&s->iohub);
+
+    pci_bus_irqs(pci_host->bus, remote_iohub_set_irq, remote_iohub_map_irq,
+                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
 }
 
 static void remote_machine_class_init(ObjectClass *oc, void *data)
hw/pci/proxy.c +58 -0

82 lines not on this page · raw message

include/hw/i386/remote-iohub.h new file +42 -0

42 lines not on this page · raw message

include/hw/i386/remote.h +3 -0
index 3073db6..dec072f 100644
@@ -15,11 +15,14 @@
 #include "hw/boards.h"
 #include "hw/pci-host/remote.h"
 #include "io/channel.h"
+#include "hw/i386/remote-iohub.h"
 
 typedef struct RemoteMachineState {
     MachineState parent_obj;
 
     RemotePCIHost *host;
+
+    RemoteIOHubState iohub;
 } RemoteMachineState;
 
 /* Used to pass to co-routine device and ioc. */
include/hw/pci/pci_ids.h +3 -0
index 11f8ab7..bd0c17d 100644
@@ -192,6 +192,9 @@
 #define PCI_DEVICE_ID_SUN_SIMBA          0x5000
 #define PCI_DEVICE_ID_SUN_SABRE          0xa000
 
+#define PCI_VENDOR_ID_ORACLE             0x108e
+#define PCI_DEVICE_ID_REMOTE_IOHUB       0xb000
+
 #define PCI_VENDOR_ID_CMD                0x1095
 #define PCI_DEVICE_ID_CMD_646            0x0646
 
include/hw/pci/proxy.h +5 -0
index 52f3c43..1482217 100644
@@ -12,6 +12,7 @@
 #include "hw/pci/pci.h"
 #include "io/channel.h"
 #include "hw/pci/memory-sync.h"
+#include "qemu/event_notifier.h"
 
 #define TYPE_PCI_PROXY_DEV "x-pci-proxy-dev"
 
@@ -42,6 +43,10 @@ struct PCIProxyDev {
     QIOChannel *ioc;
     Error *migration_blocker;
     RemoteMemSync sync;
+    int virq;
+    EventNotifier intr;
+    EventNotifier resample;
+
     ProxyMemoryRegion region[PCI_NUM_REGIONS];
 };
 
include/io/mpqemu-link.h +1 -0
index 234ac82..4cf81f5 100644
@@ -38,6 +38,7 @@ typedef enum {
     PCI_CONFIG_READ,
     BAR_WRITE,
     BAR_READ,
+    SET_IRQFD,
     MPQEMU_CMD_MAX,
 } MPQemuCmd;
 
io/mpqemu-link.c +5 -0
index 7e297ef..2d1e576 100644
@@ -290,6 +290,11 @@ bool mpqemu_msg_valid(MPQemuMsg *msg)
             return false;
         }
         break;
+    case SET_IRQFD:
+        if (msg->size || (msg->num_fds != 2)) {
+            return false;
+        }
+        break;
     default:
         break;
     }
Signature · 1 line
1.8.3.1
#25 lore raw

PATCH v11 16/19 multi-process: Retrieve PCI info from remote process

In reply to Jagannathan Raman

Retrieve PCI configuration info about the remote device and configure the Proxy PCI object based on the returned information

---
 hw/pci/proxy.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
hw/pci/proxy.c +85 -0
index 93cffcc..ccffddb 100644
@@ -24,6 +24,8 @@
 #include "sysemu/kvm.h"
 #include "util/event_notifier-posix.c"
 
+static void probe_pci_info(PCIDevice *dev, Error **errp);
+
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
     pdev->ioc = qio_channel_new_fd(fd, errp);
@@ -87,6 +89,7 @@ static void setup_irqfd(PCIProxyDev *dev)
 static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
 {
     PCIProxyDev *dev = PCI_PROXY_DEV(device);
+    uint8_t *pci_conf = device->config;
     int fd;
 
     if (dev->fd) {
@@ -114,9 +117,14 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp)
     qemu_mutex_init(&dev->io_mutex);
     qio_channel_set_blocking(dev->ioc, true, NULL);
 
+    pci_conf[PCI_LATENCY_TIMER] = 0xff;
+    pci_conf[PCI_INTERRUPT_PIN] = 0x01;
+
     configure_memory_sync(&dev->sync, dev->ioc);
 
     setup_irqfd(dev);
+
+    probe_pci_info(PCI_DEVICE(dev), errp);
 }
 
 static void pci_proxy_dev_exit(PCIDevice *pdev)
@@ -270,3 +278,80 @@ const MemoryRegionOps proxy_mr_ops = {
         .max_access_size = 8,
     },
 };
+
+static void probe_pci_info(PCIDevice *dev, Error **errp)
+{
+    PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(dev);
+    uint32_t orig_val, new_val, base_class, val;
+    PCIProxyDev *pdev = PCI_PROXY_DEV(dev);
+    DeviceClass *dc = DEVICE_CLASS(pc);
+    uint8_t type;
+    int i, size;
+    char *name;
+
+    config_op_send(pdev, PCI_VENDOR_ID, &val, 2, PCI_CONFIG_READ);
+    pc->vendor_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_DEVICE_ID, &val, 2, PCI_CONFIG_READ);
+    pc->device_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_CLASS_DEVICE, &val, 2, PCI_CONFIG_READ);
+    pc->class_id = (uint16_t)val;
+
+    config_op_send(pdev, PCI_SUBSYSTEM_ID, &val, 2, PCI_CONFIG_READ);
+    pc->subsystem_id = (uint16_t)val;
+
+    base_class = pc->class_id >> 4;
+    switch (base_class) {
+    case PCI_BASE_CLASS_BRIDGE:
+        set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+        break;
+    case PCI_BASE_CLASS_STORAGE:
+        set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+        break;
+    case PCI_BASE_CLASS_NETWORK:
+        set_bit(DEVICE_CATEGORY_NETWORK, dc->categories);
+        break;
+    case PCI_BASE_CLASS_INPUT:
+        set_bit(DEVICE_CATEGORY_INPUT, dc->categories);
+        break;
+    case PCI_BASE_CLASS_DISPLAY:
+        set_bit(DEVICE_CATEGORY_DISPLAY, dc->categories);
+        break;
+    case PCI_BASE_CLASS_PROCESSOR:
+        set_bit(DEVICE_CATEGORY_CPU, dc->categories);
+        break;
+    default:
+        set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+        break;
+    }
+
+    for (i = 0; i < PCI_NUM_REGIONS; i++) {
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &orig_val, 4,
+                       PCI_CONFIG_READ);
+        new_val = 0xffffffff;
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &new_val, 4,
+                       PCI_CONFIG_WRITE);
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &new_val, 4,
+                       PCI_CONFIG_READ);
+        size = (~(new_val & 0xFFFFFFF0)) + 1;
+        config_op_send(pdev, PCI_BASE_ADDRESS_0 + (4 * i), &orig_val, 4,
+                       PCI_CONFIG_WRITE);
+        type = (new_val & 0x1) ?
+                   PCI_BASE_ADDRESS_SPACE_IO : PCI_BASE_ADDRESS_SPACE_MEMORY;
+
+        if (size) {
+            pdev->region[i].dev = pdev;
+            pdev->region[i].present = true;
+            if (type == PCI_BASE_ADDRESS_SPACE_MEMORY) {
+                pdev->region[i].memory = true;
+            }
+            name = g_strdup_printf("bar-region-%d", i);
+            memory_region_init_io(&pdev->region[i].mr, OBJECT(pdev),
+                                  &proxy_mr_ops, &pdev->region[i],
+                                  name, size);
+            pci_register_bar(dev, i, type, &pdev->region[i].mr);
+            g_free(name);
+        }
+    }
+}
Signature · 1 line
1.8.3.1
#26 lore raw

PATCH v11 17/19 multi-process: perform device reset in the remote process

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Perform device reset in the remote process when QEMU performs device reset. This is required to reset the internal state (like registers, etc...) of emulated devices

---
 hw/i386/remote-msg.c     | 23 ++++++++++++++++++++++-
 hw/pci/proxy.c           | 20 ++++++++++++++++++++
 include/io/mpqemu-link.h |  1 +
 3 files changed, 43 insertions(+), 1 deletion(-)
hw/i386/remote-msg.c +22 -1
index a318a9f..82f1b4f 100644
@@ -19,6 +19,7 @@
 #include "exec/memattrs.h"
 #include "hw/i386/remote-memory.h"
 #include "hw/i386/remote-iohub.h"
+#include "sysemu/reset.h"
 
 static void process_config_write(QIOChannel *ioc, PCIDevice *dev,
                                  MPQemuMsg *msg);
@@ -26,6 +27,8 @@ static void process_config_read(QIOChannel *ioc, PCIDevice *dev,
                                 MPQemuMsg *msg);
 static void process_bar_write(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
 static void process_bar_read(QIOChannel *ioc, MPQemuMsg *msg, Error **errp);
+static void process_device_reset_msg(QIOChannel *ioc, PCIDevice *dev,
+                                     Error **errp);
 
 void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
 {
@@ -72,7 +75,9 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data)
         case SET_IRQFD:
             process_set_irqfd_msg(pci_dev, &msg);
             break;
-
+        case DEVICE_RESET:
+            process_device_reset_msg(com->ioc, pci_dev, &local_err);
+            break;
         default:
             error_setg(&local_err,
                        "Unknown command (%d) received for device %s (pid=%d)",
@@ -218,3 +223,19 @@ fail:
                    "in remote process pid=%d", getpid());
     }
 }
+
+static void process_device_reset_msg(QIOChannel *ioc, PCIDevice *dev,
+                                     Error **errp)
+{
+    DeviceClass *dc = DEVICE_GET_CLASS(dev);
+    DeviceState *s = DEVICE(dev);
+    MPQemuMsg ret = { 0 };
+
+    if (dc->reset) {
+        dc->reset(s);
+    }
+
+    ret.cmd = RET_MSG;
+
+    mpqemu_msg_send(&ret, ioc, errp);
+}
hw/pci/proxy.c +20 -0
index ccffddb..e5b74bb 100644
@@ -25,6 +25,7 @@
 #include "util/event_notifier-posix.c"
 
 static void probe_pci_info(PCIDevice *dev, Error **errp);
+static void proxy_device_reset(DeviceState *dev);
 
 static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp)
 {
@@ -198,6 +199,8 @@ static void pci_proxy_dev_class_init(ObjectClass *klass, void *data)
     k->config_read = pci_proxy_read_config;
     k->config_write = pci_proxy_write_config;
 
+    dc->reset = proxy_device_reset;
+
     device_class_set_props(dc, proxy_properties);
 }
 
@@ -355,3 +358,20 @@ static void probe_pci_info(PCIDevice *dev, Error **errp)
         }
     }
 }
+
+static void proxy_device_reset(DeviceState *dev)
+{
+    PCIProxyDev *pdev = PCI_PROXY_DEV(dev);
+    MPQemuMsg msg = { 0 };
+    Error *local_err = NULL;
+
+    msg.cmd = DEVICE_RESET;
+    msg.size = 0;
+
+    (void)mpqemu_msg_send_and_await_reply(&msg, pdev, &local_err);
+    if (local_err) {
+        error_report("Failed to send DEVICE_RESET to the remote process");
+        error_free(local_err);
+    }
+
+}
include/io/mpqemu-link.h +1 -0
index 4cf81f5..7462968 100644
@@ -39,6 +39,7 @@ typedef enum {
     BAR_WRITE,
     BAR_READ,
     SET_IRQFD,
+    DEVICE_RESET,
     MPQEMU_CMD_MAX,
 } MPQemuCmd;
 
Signature · 1 line
1.8.3.1
#27 lore raw

PATCH v11 18/19 multi-process: add the concept description to docs/devel/qemu-multiprocess

In reply to Jagannathan Raman

From: John G Johnson <[email protected]>

---
 MAINTAINERS                  |   1 +
 docs/devel/index.rst         |   1 +
 docs/devel/multi-process.rst | 966 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 968 insertions(+)
 create mode 100644 docs/devel/multi-process.rst
MAINTAINERS +1 -0
index f33de17..9a911e0 100644
@@ -3117,6 +3117,7 @@ F: hw/pci/memory-sync.c
 F: include/hw/pci/memory-sync.h
 F: hw/i386/remote-iohub.c
 F: include/hw/i386/remote-iohub.h
+F: docs/devel/multi-process.rst
 
 Build and test automation
 -------------------------
docs/devel/index.rst +1 -0
index 77baae5..06480ce 100644
@@ -34,3 +34,4 @@ Contents:
    clocks
    qom
    block-coroutine-wrapper
+   multi-process
docs/devel/multi-process.rst new file +966 -0

966 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#28 lore raw

PATCH v11 19/19 multi-process: add configure and usage information

In reply to Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

---
 MAINTAINERS                |  2 ++
 docs/multi-process.rst     | 67 ++++++++++++++++++++++++++++++++++++++++++++++
 scripts/mpqemu-launcher.py | 49 +++++++++++++++++++++++++++++++++
 3 files changed, 118 insertions(+)
 create mode 100644 docs/multi-process.rst
 create mode 100755 scripts/mpqemu-launcher.py
MAINTAINERS +2 -0
index 9a911e0..d12aba7 100644
@@ -3118,6 +3118,8 @@ F: include/hw/pci/memory-sync.h
 F: hw/i386/remote-iohub.c
 F: include/hw/i386/remote-iohub.h
 F: docs/devel/multi-process.rst
+F: scripts/mpqemu-launcher.py
+F: scripts/mpqemu-launcher-perf-mode.py
 
 Build and test automation
 -------------------------
docs/multi-process.rst new file +67 -0

67 lines not on this page · raw message

scripts/mpqemu-launcher.py new file +49 -0

49 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#29 lore raw

Re: [PATCH v11 19/19] multi-process: add configure and usage information

Hi Jagannathan,

Quoted 1 line · Jagannathan Raman

From: Elena Ufimtseva <[email protected]>

Documentation is scarce ;)

Quoted 18 lines
---
 MAINTAINERS                |  2 ++
 docs/multi-process.rst     | 67 ++++++++++++++++++++++++++++++++++++++++++++++
 scripts/mpqemu-launcher.py | 49 +++++++++++++++++++++++++++++++++
 3 files changed, 118 insertions(+)
 create mode 100644 docs/multi-process.rst
 create mode 100755 scripts/mpqemu-launcher.py
MAINTAINERS +0 -0
index 9a911e0..d12aba7 100644

MAINTAINERS F: include/hw/pci/memory-sync.h · line 3122

Quoted 5 lines
MAINTAINERS +2 -0
@@ -3118,6 +3118,8 @@ F: include/hw/pci/memory-sync.h
 F: hw/i386/remote-iohub.c
 F: include/hw/i386/remote-iohub.h
 F: docs/devel/multi-process.rst
+F: scripts/mpqemu-launcher.py
+F: scripts/mpqemu-launcher-perf-mode.py

This one was in v7, Stefan asked about it, then the script disappeared in v8 =)
https://www.mail-archive.com/[email protected]/msg718984.html

Quoted 82 lines
MAINTAINERS +0 -0
 Build and test automation
 -------------------------
docs/multi-process.rst new file +67 -0
new file mode 100644
index 0000000..c4b022c
@@ -0,0 +1,67 @@
+Multi-process QEMU
+==================
+
+This document describes how to configure and use multi-process qemu.
+For the design document refer to docs/devel/qemu-multiprocess.
+
+1) Configuration
+----------------
+
+To enable support for multi-process add --enable-mpqemu
+to the list of options for the "configure" script.
+
+
+2) Usage
+--------
+
+Multi-process QEMU requires an orchestrator to launch. Please refer to a
+light-weight python based orchestrator for mpqemu in
+scripts/mpqemu-launcher.py to lauch QEMU in multi-process mode.
+
+Following is a description of command-line used to launch mpqemu.
+
+* Orchestrator:
+
+  - The Orchestrator creates a unix socketpair
+
+  - It launches the remote process and passes one of the
+    sockets to it via command-line.
+
+  - It then launches QEMU and specifies the other socket as an option
+    to the Proxy device object
+
+* Remote Process:
+
+  - QEMU can enter remote process mode by using the "remote" machine
+    option.
+
+  - The orchestrator creates a "remote-object" with details about
+    the device and the file descriptor for the device
+
+  - The remaining options are no different from how one launches QEMU with
+    devices.
+
+  - Example command-line for the remote process is as follows:
+
+      /usr/bin/qemu-system-x86_64                                        \
+      -machine x-remote                                                  \
+      -device lsi53c895a,id=lsi0                                         \
+      -drive id=drive_image2,file=/build/ol7-nvme-test-1.qcow2           \
+      -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0  \
+      -object x-remote-object,id=robj1,devid=lsi1,fd=4,
+
+* QEMU:
+
+  - Since parts of the RAM are shared between QEMU & remote process, a
+    memory-backend-memfd is required to facilitate this, as follows:
+
+    -object memory-backend-memfd,id=mem,size=2G
+
+  - A "x-pci-proxy-dev" device is created for each of the PCI devices emulated
+    in the remote process. A "socket" sub-option specifies the other end of
+    unix channel created by orchestrator. The "id" sub-option must be specified
+    and should be the same as the "id" specified for the remote PCI device
+
+  - Example commandline for QEMU is as follows:
+
+      -device x-pci-proxy-dev,id=lsi0,socket=3
scripts/mpqemu-launcher.py new file +0 -0
new file mode 100755
index 0000000..6e0ef22

scripts/mpqemu-launcher.py line 7

Quoted 7 lines
scripts/mpqemu-launcher.py new file +7 -0
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+import socket
+import os
+import subprocess
+import time
+
+PROC_QEMU='/usr/bin/qemu-system-x86_64'

If this is a (multiarch) test, then ...

scripts/mpqemu-launcher.py line 13

Quoted 6 lines
scripts/mpqemu-launcher.py new file +6 -0
+
+proxy, remote = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
+
+remote_cmd = [ PROC_QEMU,                                                      \
+               '-machine', 'x-remote',                                         \
+               '-device', 'lsi53c895a,id=lsi1',                                \

... I'd move it to tests/integration/multiproc-x86-lsi53c895a.py ...

scripts/mpqemu-launcher.py line 14

Quoted 1 line
scripts/mpqemu-launcher.py new file +1 -0
+               '-drive', 'id=drive_image1,file=/build/ol7-nvme-test-1.qcow2',  \

... use avocado.utils.vmimage (see tests/acceptance/boot_linux.py) to download a prebuilt image, ...

Quoted 26 lines
scripts/mpqemu-launcher.py new file +25 -0
+               '-device', 'scsi-hd,id=drive1,drive=drive_image1,bus=lsi1.0,'   \
+                              'scsi-id=0',                                     \
+               '-object',                                                      \
+               'x-remote-object,id=robj1,devid=lsi1,fd='+str(remote.fileno()), \
+               '-nographic',                                                   \
+             ]
+
+proxy_cmd = [ PROC_QEMU,                                                       \
+              '-name', 'OL7.4',                                                \
+              '-machine', 'q35,accel=kvm',                                     \
+              '-smp', 'sockets=1,cores=1,threads=1',                           \
+              '-m', '2048',                                                    \
+              '-object', 'memory-backend-memfd,id=sysmem-file,size=2G',        \
+              '-numa', 'node,memdev=sysmem-file',                              \
+              '-device', 'virtio-scsi-pci,id=virtio_scsi_pci0',                \
+              '-drive', 'id=drive_image1,if=none,format=qcow2,'                \
+                            'file=/home/ol7-hdd-1.qcow2',                      \
+              '-device', 'scsi-hd,id=image1,drive=drive_image1,'               \
+                             'bus=virtio_scsi_pci0.0',                         \
+              '-boot', 'd',                                                    \
+              '-vnc', ':0',                                                    \
+              '-device', 'x-pci-proxy-dev,id=lsi1,fd='+str(proxy.fileno()),    \
+            ]
+
+

scripts/mpqemu-launcher.py line 49

Quoted 10 lines
scripts/mpqemu-launcher.py new file +10 -0
+pid = os.fork();
+
+if pid:
+    # In Proxy
+    print('Launching QEMU with Proxy object');
+    process = subprocess.Popen(proxy_cmd, pass_fds=[proxy.fileno()])
+else:
+    # In remote
+    print('Launching Remote process');
+    process = subprocess.Popen(remote_cmd, pass_fds=[remote.fileno()])

... and do something within the guest to be sure MultiProc works :)

Regards,

Phil.

#30 lore raw
In reply to Jagannathan Raman
Quoted 11 lines · Jagannathan Raman

Hello,

This is the v11 of the patchset. Thank you very much for the review of the v10 of the series. We are glad to hear the patchset is getting in a better shape.

We made changes to the following patches in this series.

[PATCH v10 06/19] multi-process: define MPQemuMsg format and transmission functions [PATCH v10 08/19] multi-process: Associate fd of a PCIDevice with its object [PATCH v10 10/19] multi-process: introduce proxy object

I have reviewed these patches and posted a comment about a possible buffer overflow.

Stefan