Skip to contentSkip to conversation

hw/pvrdma: PVRDMA device implementation

PATCH Superseded Marcel Apfelbaum 4 participants 18 messages Jan 7, 2018 – Jan 10, 2018

Subsystems (2)

v6 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v6 cover hw/pvrdma: PVRDMA device implementation

V5 -> V6:
- Found a ppc machine and solved ppc compilation issues
- Tried to fix the s390x issue (still looking of a machine)

V4 -> V5:
- Fixed (at least tried to) compilation issues

V3 -> V4:
- Fixed documentation (added more impl details)
- Fixed compilation errors discovered by patchew.
- Addressed Michael S. Tsirkin comments:
- Removed unnecessary typedefs and replace them with macros in VMware header files, together with explanations.
- Moved more code from vmw specific to rdma generic code.
- Added page size limitations to the documentation.

V2 -> V3:
- Addressed Michael S. Tsirkin and Philippe Mathieu-Daudé comments:
- Moved the device to hw/rdma
- Addressed Michael S. Tsirkin comments:
- Split the code into generic (hw/rdma) and VMWare specific (hw/rdma/vmw)
- Added more details to documentation - VMware guest-host protocol.
- Remove mad processing
- limited the memory the Guest can pin.
- Addressed Philippe Mathieu-Daudé comment:
- s/roundup_pow_of_two/pow2roundup32 and move it to qemu/host-utils.h
- Added Shamit Rabinovici's review to documentation
- Rebased to latest master

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it and most of the code can be reused if we decide to continue to a Virtio based RDMA device.

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
BAR 2 - UAR, used to pass HW commands from driver.

- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/rdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

 MAINTAINERS                        |   8 +
 Makefile.objs                      |   2 +
 backends/hostmem-file.c            |  25 +-
 backends/hostmem-ram.c             |   4 +-
 backends/hostmem.c                 |  21 +
 configure                          |   9 +-
 default-configs/arm-softmmu.mak    |   1 +
 default-configs/i386-softmmu.mak   |   1 +
 default-configs/x86_64-softmmu.mak |   1 +
 docs/pvrdma.txt                    | 254 ++++++++++++
 exec.c                             |  26 +-
 hw/Makefile.objs                   |   1 +
 hw/pci/shpc.c                      |  13 +-
 hw/rdma/Makefile.objs              |   6 +
 hw/rdma/rdma_backend.c             | 815 +++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h             |  92 +++++
 hw/rdma/rdma_backend_defs.h        |  62 +++
 hw/rdma/rdma_rm.c                  | 619 ++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h                  |  69 ++++
 hw/rdma/rdma_rm_defs.h             | 106 +++++
 hw/rdma/rdma_utils.c               |  52 +++
 hw/rdma/rdma_utils.h               |  43 ++
 hw/rdma/trace-events               |   5 +
 hw/rdma/vmw/pvrdma.h               | 122 ++++++
 hw/rdma/vmw/pvrdma_cmd.c           | 679 ++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h       | 602 +++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c      | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h      |  42 ++
 hw/rdma/vmw/pvrdma_ib_verbs.h      | 433 ++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c          | 644 +++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c        | 212 ++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h        |  27 ++
 hw/rdma/vmw/pvrdma_ring.h          | 134 ++++++
 hw/rdma/vmw/trace-events           |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h       | 311 ++++++++++++++
 include/exec/memory.h              |  23 ++
 include/exec/ram_addr.h            |   3 +-
 include/hw/pci/pci_ids.h           |   3 +
 include/qemu/host-utils.h          |  10 +
 include/qemu/osdep.h               |   2 +-
 include/sysemu/hostmem.h           |   2 +-
 include/sysemu/kvm.h               |   2 +-
 memory.c                           |  16 +-
 target/s390x/kvm.c                 |   4 +-
 util/oslib-posix.c                 |   4 +-
 util/oslib-win32.c                 |   2 +-
 46 files changed, 5592 insertions(+), 64 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h
Signature · 1 line
2.13.5
#1 lore raw

PATCH v6 1/5 pci/shpc: Move function to generic header file

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
 hw/pci/shpc.c             | 13 ++-----------
 include/qemu/host-utils.h | 10 ++++++++++
 2 files changed, 12 insertions(+), 11 deletions(-)
hw/pci/shpc.c +2 -11
index 69fc14b218..a8462d48bb 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu-common.h"
+#include "qemu/host-utils.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
 #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
 #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
 static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
 {
     uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
@@ -656,7 +647,7 @@ int shpc_init(PCIDevice *d, PCIBus *sec_bus, MemoryRegion *bar,
 
 int shpc_bar_size(PCIDevice *d)
 {
-    return roundup_pow_of_two(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
+    return pow2roundup32(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
 }
 
 void shpc_cleanup(PCIDevice *d, MemoryRegion *bar)
include/qemu/host-utils.h +10 -0
index 5ac621cf1f..9fadb3f1ba 100644
@@ -400,6 +400,16 @@ static inline uint64_t pow2ceil(uint64_t value)
     return 0x8000000000000000ull >> (n - 1);
 }
 
+static inline int pow2roundup32(int x)
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);
+    return x + 1;
+}
+
 /**
  * urshift - 128-bit Unsigned Right Shift.
  * @plow: in/out - lower 64-bit integer.
Signature · 1 line
2.13.5
Reviewed-by on v6 1/5
#2 lore raw

Re: [Qemu-devel] [PATCH V6 1/5] pci/shpc: Move function to generic header file

Hi Marcel,

Quoted 55 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
 hw/pci/shpc.c             | 13 ++-----------
 include/qemu/host-utils.h | 10 ++++++++++
 2 files changed, 12 insertions(+), 11 deletions(-)
hw/pci/shpc.c +2 -11
index 69fc14b218..a8462d48bb 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu-common.h"
+#include "qemu/host-utils.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
 #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
 #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
 static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
 {
     uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
@@ -656,7 +647,7 @@ int shpc_init(PCIDevice *d, PCIBus *sec_bus, MemoryRegion *bar,
 
 int shpc_bar_size(PCIDevice *d)
 {
-    return roundup_pow_of_two(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
+    return pow2roundup32(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
 }
 
 void shpc_cleanup(PCIDevice *d, MemoryRegion *bar)
include/qemu/host-utils.h +0 -0
index 5ac621cf1f..9fadb3f1ba 100644

include/qemu/host-utils.h static inline uint64_t pow2ceil(uint64_t value) · line 403

Quoted 4 lines
include/qemu/host-utils.h +1 -0
@@ -400,6 +400,16 @@ static inline uint64_t pow2ceil(uint64_t value)
     return 0x8000000000000000ull >> (n - 1);
 }
 
+static inline int pow2roundup32(int x)

Can you change the type?

static inline uint32_t pow2roundup32(uint32_t value)

Then:

Quoted 13 lines
include/qemu/host-utils.h +9 -0
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);
+    return x + 1;
+}
+
 /**
  * urshift - 128-bit Unsigned Right Shift.
  * @plow: in/out - lower 64-bit integer.
#3 lore raw

Re: [Qemu-devel] [PATCH V6 1/5] pci/shpc: Move function to generic header file

Quoted 2 lines · Philippe Mathieu-Daudé

Hi Marcel,

Hi Philippe,

Quoted 59 lines · Marcel Apfelbaum
Quoted 58 lines

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
  hw/pci/shpc.c             | 13 ++-----------
  include/qemu/host-utils.h | 10 ++++++++++
  2 files changed, 12 insertions(+), 11 deletions(-)
hw/pci/shpc.c +2 -11
index 69fc14b218..a8462d48bb 100644
@@ -1,6 +1,7 @@
  #include "qemu/osdep.h"
  #include "qapi/error.h"
  #include "qemu-common.h"
+#include "qemu/host-utils.h"
  #include "qemu/range.h"
  #include "qemu/error-report.h"
  #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
  #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
  #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
  static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
  {
      uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
@@ -656,7 +647,7 @@ int shpc_init(PCIDevice *d, PCIBus *sec_bus, MemoryRegion *bar,
 
  int shpc_bar_size(PCIDevice *d)
  {
-    return roundup_pow_of_two(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
+    return pow2roundup32(SHPC_SLOT_REG(SHPC_MAX_SLOTS));
  }
 
  void shpc_cleanup(PCIDevice *d, MemoryRegion *bar)
include/qemu/host-utils.h +0 -0
index 5ac621cf1f..9fadb3f1ba 100644
@@ -400,6 +400,16 @@ static inline uint64_t pow2ceil(uint64_t value)
      return 0x8000000000000000ull >> (n - 1);
  }
 
Quoted 6 lines

include/qemu/host-utils.h static inline uint64_t pow2ceil(uint64_t value) · line 403

Quoted 1 line
include/qemu/host-utils.h +1 -0
+static inline int pow2roundup32(int x)

Can you change the type?

static inline uint32_t pow2roundup32(uint32_t value)

Sure.

Quoted 2 lines

Then:

Appreciated,
Marcel

Quoted 13 lines
Quoted 13 lines
include/qemu/host-utils.h +9 -0
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);
+    return x + 1;
+}
+
  /**
   * urshift - 128-bit Unsigned Right Shift.
   * @plow: in/out - lower 64-bit integer.
#4 lore raw

PATCH v6 2/5 mem: add share parameter to memory-backend-ram

In reply to Marcel Apfelbaum

Currently only file backed memory backend can be created with a "share" flag in order to allow sharing guest RAM with other processes in the host.

Add the "share" flag also to RAM Memory Backend
in order to allow remapping parts of the guest RAM to different host virtual addresses. This is needed by the RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.

Moved the "share" flag to the Host Memory base class, modified phys_mem_alloc to include the new parameter and a new interface memory_region_init_ram_shared_nomigrate.

There are no functional changes if the new flag is not used.

---
 backends/hostmem-file.c  | 25 +------------------------
 backends/hostmem-ram.c   |  4 ++--
 backends/hostmem.c       | 21 +++++++++++++++++++++
 exec.c                   | 26 +++++++++++++++-----------
 include/exec/memory.h    | 23 +++++++++++++++++++++++
 include/exec/ram_addr.h  |  3 ++-
 include/qemu/osdep.h     |  2 +-
 include/sysemu/hostmem.h |  2 +-
 include/sysemu/kvm.h     |  2 +-
 memory.c                 | 16 +++++++++++++---
 target/s390x/kvm.c       |  4 ++--
 util/oslib-posix.c       |  4 ++--
 util/oslib-win32.c       |  2 +-
 13 files changed, 85 insertions(+), 49 deletions(-)
backends/hostmem-file.c +1 -24
index e44c319915..bc95022a68 100644
@@ -31,7 +31,6 @@ typedef struct HostMemoryBackendFile HostMemoryBackendFile;
 struct HostMemoryBackendFile {
     HostMemoryBackend parent_obj;
 
-    bool share;
     bool discard_data;
     char *mem_path;
 };
@@ -58,7 +57,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
         path = object_get_canonical_path(OBJECT(backend));
         memory_region_init_ram_from_file(&backend->mr, OBJECT(backend),
                                  path,
-                                 backend->size, fb->share,
+                                 backend->size, backend->share,
                                  fb->mem_path, errp);
         g_free(path);
     }
@@ -85,25 +84,6 @@ static void set_mem_path(Object *o, const char *str, Error **errp)
     fb->mem_path = g_strdup(str);
 }
 
-static bool file_memory_backend_get_share(Object *o, Error **errp)
-{
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    return fb->share;
-}
-
-static void file_memory_backend_set_share(Object *o, bool value, Error **errp)
-{
-    HostMemoryBackend *backend = MEMORY_BACKEND(o);
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    if (host_memory_backend_mr_inited(backend)) {
-        error_setg(errp, "cannot change property value");
-        return;
-    }
-    fb->share = value;
-}
-
 static bool file_memory_backend_get_discard_data(Object *o, Error **errp)
 {
     return MEMORY_BACKEND_FILE(o)->discard_data;
@@ -136,9 +116,6 @@ file_backend_class_init(ObjectClass *oc, void *data)
     bc->alloc = file_backend_memory_alloc;
     oc->unparent = file_backend_unparent;
 
-    object_class_property_add_bool(oc, "share",
-        file_memory_backend_get_share, file_memory_backend_set_share,
-        &error_abort);
     object_class_property_add_bool(oc, "discard-data",
         file_memory_backend_get_discard_data, file_memory_backend_set_discard_data,
         &error_abort);
backends/hostmem-ram.c +2 -2
index 38977be73e..7ddd08d370 100644
@@ -28,8 +28,8 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     }
 
     path = object_get_canonical_path_component(OBJECT(backend));
-    memory_region_init_ram_nomigrate(&backend->mr, OBJECT(backend), path,
-                           backend->size, errp);
+    memory_region_init_ram_shared_nomigrate(&backend->mr, OBJECT(backend), path,
+                           backend->size, backend->share, errp);
     g_free(path);
 }
 
backends/hostmem.c +21 -0
index ee2c2d5bfd..1daf13bd2e 100644
@@ -369,6 +369,24 @@ static void set_id(Object *o, const char *str, Error **errp)
     backend->id = g_strdup(str);
 }
 
+static bool host_memory_backend_get_share(Object *o, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    return backend->share;
+}
+
+static void host_memory_backend_set_share(Object *o, bool value, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    if (host_memory_backend_mr_inited(backend)) {
+        error_setg(errp, "cannot change property value");
+        return;
+    }
+    backend->share = value;
+}
+
 static void
 host_memory_backend_class_init(ObjectClass *oc, void *data)
 {
@@ -399,6 +417,9 @@ host_memory_backend_class_init(ObjectClass *oc, void *data)
         host_memory_backend_get_policy,
         host_memory_backend_set_policy, &error_abort);
     object_class_property_add_str(oc, "id", get_id, set_id, &error_abort);
+    object_class_property_add_bool(oc, "share",
+        host_memory_backend_get_share, host_memory_backend_set_share,
+        &error_abort);
 }
 
 static void host_memory_backend_finalize(Object *o)
exec.c +15 -11
index 4722e521d4..247f8bd0c0 100644
@@ -1278,7 +1278,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end,
                              uint16_t section);
 static subpage_t *subpage_init(FlatView *fv, hwaddr base);
 
-static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
+static void *(*phys_mem_alloc)(size_t size, uint64_t *align, bool shared) =
                                qemu_anon_ram_alloc;
 
 /*
@@ -1286,7 +1286,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
  * Accelerators with unusual needs may need this.  Hopefully, we can
  * get rid of it eventually.
  */
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align))
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared))
 {
     phys_mem_alloc = alloc;
 }
@@ -1889,7 +1889,7 @@ static void dirty_memory_extend(ram_addr_t old_ram_size,
     }
 }
 
-static void ram_block_add(RAMBlock *new_block, Error **errp)
+static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)
 {
     RAMBlock *block;
     RAMBlock *last_block = NULL;
@@ -1912,7 +1912,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
             }
         } else {
             new_block->host = phys_mem_alloc(new_block->max_length,
-                                             &new_block->mr->align);
+                                             &new_block->mr->align, shared);
             if (!new_block->host) {
                 error_setg_errno(errp, errno,
                                  "cannot set up guest memory '%s'",
@@ -2017,7 +2017,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
         return NULL;
     }
 
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2059,7 +2059,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
                                   void (*resized)(const char*,
                                                   uint64_t length,
                                                   void *host),
-                                  void *host, bool resizeable,
+                                  void *host, bool resizeable, bool share,
                                   MemoryRegion *mr, Error **errp)
 {
     RAMBlock *new_block;
@@ -2082,7 +2082,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     if (resizeable) {
         new_block->flags |= RAM_RESIZEABLE;
     }
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2094,12 +2094,15 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                    MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, host, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, host, false,
+                                   false, mr, errp);
 }
 
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp)
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share,
+                         MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, NULL, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, NULL, false,
+                                   share, mr, errp);
 }
 
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
@@ -2108,7 +2111,8 @@ RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
                                                      void *host),
                                      MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, mr, errp);
+    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true,
+                                   false, mr, errp);
 }
 
 static void reclaim_ramblock(RAMBlock *block)
include/exec/memory.h +23 -0
index a4cabdf44c..dd28eaba68 100644
@@ -428,6 +428,29 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       Error **errp);
 
 /**
+ * memory_region_init_ram_shared_nomigrate:  Initialize RAM memory region.
+ *                                           Accesses into the region will
+ *                                           modify memory directly.
+ *
+ * @mr: the #MemoryRegion to be initialized.
+ * @owner: the object that tracks the region's reference count
+ * @name: Region name, becomes part of RAMBlock name used in migration stream
+ *        must be unique within any device
+ * @size: size of the region.
+ * @share: allow remapping RAM to different addresses
+ * @errp: pointer to Error*, to store an error if it happens.
+ *
+ * Note that this function is similar to memory_region_init_ram_nomigrate.
+ * The only difference is part of the RAM region can be remapped.
+ */
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             struct Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp);
+
+/**
  * memory_region_init_resizeable_ram:  Initialize memory region with resizeable
  *                                     RAM.  Accesses into the region will
  *                                     modify memory directly.  Only an initial
include/exec/ram_addr.h +2 -1
index 6cbc02aa0f..7d980572c0 100644
@@ -80,7 +80,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  Error **errp);
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                   MemoryRegion *mr, Error **errp);
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp);
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share, MemoryRegion *mr,
+                         Error **errp);
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t max_size,
                                     void (*resized)(const char*,
                                                     uint64_t length,
include/qemu/osdep.h +1 -1
index adb3758275..41658060a7 100644
@@ -255,7 +255,7 @@ extern int daemon(int, int);
 int qemu_daemon(int nochdir, int noclose);
 void *qemu_try_memalign(size_t alignment, size_t size);
 void *qemu_memalign(size_t alignment, size_t size);
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align);
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared);
 void qemu_vfree(void *ptr);
 void qemu_anon_ram_free(void *ptr, size_t size);
 
include/sysemu/hostmem.h +1 -1
index ed6a437f4d..4d8f859f03 100644
@@ -55,7 +55,7 @@ struct HostMemoryBackend {
     char *id;
     uint64_t size;
     bool merge, dump;
-    bool prealloc, force_prealloc, is_mapped;
+    bool prealloc, force_prealloc, is_mapped, share;
     DECLARE_BITMAP(host_nodes, MAX_NODES + 1);
     HostMemPolicy policy;
 
include/sysemu/kvm.h +1 -1
index bbf12a1723..85002ac49a 100644
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);
 
 /* interface with exec.c */
 
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align));
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared));
 
 /* internal API */
 
memory.c +13 -3
index 4b41fb837b..cb4fe1a55a 100644
@@ -1538,11 +1538,21 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       uint64_t size,
                                       Error **errp)
 {
+    memory_region_init_ram_shared_nomigrate(mr, owner, name, size, false, errp);
+}
+
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp)
+{
     memory_region_init(mr, owner, name, size);
     mr->ram = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, share, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1651,7 +1661,7 @@ void memory_region_init_rom_nomigrate(MemoryRegion *mr,
     mr->readonly = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1670,7 +1680,7 @@ void memory_region_init_rom_device_nomigrate(MemoryRegion *mr,
     mr->terminates = true;
     mr->rom_device = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false,  mr, errp);
 }
 
 void memory_region_init_iommu(void *_iommu_mr,
target/s390x/kvm.c +2 -2
index 9b8b59f2a2..6c0fc2f89c 100644
@@ -144,7 +144,7 @@ static int cap_gs;
 
 static int active_cmma;
 
-static void *legacy_s390_alloc(size_t size, uint64_t *align);
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared);
 
 static int kvm_s390_query_mem_limit(uint64_t *memory_limit)
 {
@@ -743,7 +743,7 @@ int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf,
  * to grow. We also have to use MAP parameters that avoid
  * read-only mapping of guest pages.
  */
-static void *legacy_s390_alloc(size_t size, uint64_t *align)
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *mem;
 
util/oslib-posix.c +2 -2
index 77369c92ce..0cf3548778 100644
@@ -127,10 +127,10 @@ void *qemu_memalign(size_t alignment, size_t size)
 }
 
 /* alloc shared memory pages */
-void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment, bool shared)
 {
     size_t align = QEMU_VMALLOC_ALIGN;
-    void *ptr = qemu_ram_mmap(-1, size, align, false);
+    void *ptr = qemu_ram_mmap(-1, size, align, shared);
 
     if (ptr == MAP_FAILED) {
         return NULL;
util/oslib-win32.c +1 -1
index 69a6286d50..bb5ad28bd3 100644
@@ -67,7 +67,7 @@ void *qemu_memalign(size_t alignment, size_t size)
     return qemu_oom_check(qemu_try_memalign(alignment, size));
 }
 
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *ptr;
 
Signature · 1 line
2.13.5
#5 lore raw

[Qemu-devel] Getting rid of phys_mem_set_alloc (was: Re: [PATCH V6 2/5] mem: add share parameter to memory-backend-ram)

On Sun, 7 Jan 2018 14:32:21 +0200
Marcel Apfelbaum <[email protected]> wrote:

Looking at the churn below...

Quoted 51 lines
exec.c +2 -2
index 4722e521d4..247f8bd0c0 100644
@@ -1278,7 +1278,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end,
                              uint16_t section);
 static subpage_t *subpage_init(FlatView *fv, hwaddr base);
 
-static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
+static void *(*phys_mem_alloc)(size_t size, uint64_t *align, bool shared) =
                                qemu_anon_ram_alloc;
 
 /*
@@ -1286,7 +1286,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
  * Accelerators with unusual needs may need this.  Hopefully, we can
  * get rid of it eventually.
  */
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align))
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared))
 {
     phys_mem_alloc = alloc;
 }
include/sysemu/kvm.h +1 -1
index bbf12a1723..85002ac49a 100644
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);
 
 /* interface with exec.c */
 
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align));
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared));
 
 /* internal API */
 
target/s390x/kvm.c +1 -1
index 9b8b59f2a2..6c0fc2f89c 100644
@@ -144,7 +144,7 @@ static int cap_gs;
 
 static int active_cmma;
 
-static void *legacy_s390_alloc(size_t size, uint64_t *align);
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared);
 
 static int kvm_s390_query_mem_limit(uint64_t *memory_limit)
 {

target/s390x/kvm.c int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf, · line 749

Quoted 8 lines
target/s390x/kvm.c +1 -1
@@ -743,7 +743,7 @@ int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf,
  * to grow. We also have to use MAP parameters that avoid
  * read-only mapping of guest pages.
  */
-static void *legacy_s390_alloc(size_t size, uint64_t *align)
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *mem;
 

...I'm wondering whether we have any chance to get rid of this in the future?

s390x/kvm is the only user:

    if (!kvm_check_extension(s, KVM_CAP_S390_GMAP)
        || !kvm_check_extension(s, KVM_CAP_S390_COW)) {
        phys_mem_set_alloc(legacy_s390_alloc);
    }

In practice, this means depending on ESOP in the host. Are there still any machines/hypervisors without ESOP that we can reasonably expect people to run kvm on?

#6 lore raw

Re: [Qemu-devel] Getting rid of phys_mem_set_alloc

Quoted 5 lines · Cornelia Huck

On Sun, 7 Jan 2018 14:32:21 +0200
Marcel Apfelbaum <[email protected]> wrote:

Looking at the churn below...

Hi Cornelia,
Thanks for looking at the patch.

Quoted 48 lines
Quoted 48 lines
exec.c +2 -2
index 4722e521d4..247f8bd0c0 100644
@@ -1278,7 +1278,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end,
                               uint16_t section);
  static subpage_t *subpage_init(FlatView *fv, hwaddr base);
 
-static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
+static void *(*phys_mem_alloc)(size_t size, uint64_t *align, bool shared) =
                                 qemu_anon_ram_alloc;
 
  /*
@@ -1286,7 +1286,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
   * Accelerators with unusual needs may need this.  Hopefully, we can
   * get rid of it eventually.
   */
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align))
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared))
  {
      phys_mem_alloc = alloc;
  }
include/sysemu/kvm.h +1 -1
index bbf12a1723..85002ac49a 100644
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);
 
  /* interface with exec.c */
 
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align));
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared));
 
  /* internal API */
 
target/s390x/kvm.c +1 -1
index 9b8b59f2a2..6c0fc2f89c 100644
@@ -144,7 +144,7 @@ static int cap_gs;
 
  static int active_cmma;
 
-static void *legacy_s390_alloc(size_t size, uint64_t *align);
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared);
 
Quoted 15 lines

target/s390x/kvm.c int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf, · line 749

Quoted 11 lines
target/s390x/kvm.c +1 -1
  static int kvm_s390_query_mem_limit(uint64_t *memory_limit)
  {
@@ -743,7 +743,7 @@ int kvm_s390_mem_op(S390CPU *cpu, vaddr addr, uint8_t ar, void *hostbuf,
   * to grow. We also have to use MAP parameters that avoid
   * read-only mapping of guest pages.
   */
-static void *legacy_s390_alloc(size_t size, uint64_t *align)
+static void *legacy_s390_alloc(size_t size, uint64_t *align, bool shared)
  {
      void *mem;
 

...I'm wondering whether we have any chance to get rid of this in the future?

As far as I know, we don't usually stop supporting very ancient archs/systems; if there is at least a system needing the support (e.g. s390x/kvm), it will probably remain in the code. But is s390's maintainers decision, of course.

Thanks,
Marcel

Quoted 11 lines

s390x/kvm is the only user:

     if (!kvm_check_extension(s, KVM_CAP_S390_GMAP)
         || !kvm_check_extension(s, KVM_CAP_S390_COW)) {
         phys_mem_set_alloc(legacy_s390_alloc);
     }

In practice, this means depending on ESOP in the host. Are there still any machines/hypervisors without ESOP that we can reasonably expect people to run kvm on?

#7 lore raw

PATCH v6 3/5 docs: add pvrdma device documentation.

In reply to Marcel Apfelbaum
---
 docs/pvrdma.txt | 254 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 254 insertions(+)
 create mode 100644 docs/pvrdma.txt
docs/pvrdma.txt new file +254 -0
new file mode 100644
index 0000000000..8d004b79a1
@@ -0,0 +1,254 @@
+Paravirtualized RDMA Device (PVRDMA)
+====================================
+
+
+1. Description
+===============
+PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
+It works with its Linux Kernel driver AS IS, no need for any special guest
+modifications.
+
+While it complies with the VMware device, it can also communicate with bare
+metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
+can work with Soft-RoCE (rxe).
+
+It does not require the whole guest RAM to be pinned allowing memory
+over-commit and, even if not implemented yet, migration support will be
+possible with some HW assistance.
+
+A project presentation accompany this document:
+- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf
+
+
+
+2. Setup
+========
+
+
+2.1 Guest setup
+===============
+Fedora 27+ kernels work out of the box, older distributions
+require updating the kernel to 4.14 to include the pvrdma driver.
+
+However the libpvrdma library needed by User Level Software is still
+not available as part of the distributions, so the rdma-core library
+needs to be compiled and optionally installed.
+
+Please follow the instructions at:
+  https://github.com/linux-rdma/rdma-core.git
+
+
+2.2 Host Setup
+==============
+The pvrdma backend is an ibdevice interface that can be exposed
+either by a Soft-RoCE(rxe) device on machines with no RDMA device,
+or an HCA SRIOV function(VF/PF).
+Note that ibdevice interfaces can't be shared between pvrdma devices,
+each one requiring a separate instance (rxe or SRIOV VF).
+
+
+2.2.1 Soft-RoCE backend(rxe)
+===========================
+A stable version of rxe is required, Fedora 27+ or a Linux
+Kernel 4.14+ is preferred.
+
+The rdma_rxe module is part of the Linux Kernel but not loaded by default.
+Install the User Level library (librxe) following the instructions from:
+https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
+
+Associate an ETH interface with rxe by running:
+   rxe_cfg add eth0
+An rxe0 ibdevice interface will be created and can be used as pvrdma backend.
+
+
+2.2.2 RDMA device Virtual Function backend
+==========================================
+Nothing special is required, the pvrdma device can work not only with
+Ethernet Links, but also Infinibands Links.
+All is needed is an ibdevice with an active port, for Mellanox cards
+will be something like mlx5_6 which can be the backend.
+
+
+2.2.3 QEMU setup
+================
+Configure QEMU with --enable-rdma flag, installing
+the required RDMA libraries.
+
+
+
+3. Usage
+========
+Currently the device is working only with memory backed RAM
+and it must be mark as "shared":
+   -m 1G \
+   -object memory-backend-ram,id=mb1,size=1G,share \
+   -numa node,memdev=mb1 \
+
+The pvrdma device is composed of two functions:
+ - Function 0 is a vmxnet Ethernet Device which is redundant in Guest
+   but is required to pass the ibdevice GID using its MAC.
+   Examples:
+     For an rxe backend using eth0 interface it will use its mac:
+       -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
+     For an SRIOV VF, we take the Ethernet Interface exposed by it:
+       -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
+ - Function 1 is the actual device:
+       -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>
+   where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
+ Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.
+ The rules of conversion are part of the RoCE spec, but since manual conversion
+ is not required, spotting problems is not hard:
+    Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
+             MAC: 7c:fe:90:cb:74:3a
+    Note the difference between the first byte of the MAC and the GID.
+
+
+
+4. Implementation details
+=========================
+
+
+4.1 Overview
+============
+The device acts like a proxy between the Guest Driver and the host
+ibdevice interface.
+On configuration path:
+ - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
+   a resource from the backend interface, maintaining a 1-1 mapping
+   between the guest and host.
+On data path:
+ - Every post_send/receive received from the guest will be converted into
+   a post_send/receive for the backend. The buffers data will not be touched
+   or copied resulting in near bare-metal performance for large enough buffers.
+ - Completions from the backend interface will result in completions for
+   the pvrdma device.
+
+
+4.2 PCI BARs
+============
+PCI Bars:
+	BAR 0 - MSI-X
+        MSI-X vectors:
+		(0) Command - used when execution of a command is completed.
+		(1) Async - not in use.
+		(2) Completion - used when a completion event is placed in
+		  device's CQ ring.
+	BAR 1 - Registers
+        --------------------------------------------------------
+        | VERSION |  DSR | CTL | REQ | ERR |  ICR | IMR  | MAC |
+        --------------------------------------------------------
+		DSR - Address of driver/device shared memory used
+              for the command channel, used for passing:
+			    - General info such as driver version
+			    - Address of 'command' and 'response'
+			    - Address of async ring
+			    - Address of device's CQ ring
+			    - Device capabilities
+		CTL - Device control operations (activate, reset etc)
+		IMG - Set interrupt mask
+		REQ - Command execution register
+		ERR - Operation status
+
+	BAR 2 - UAR
+        ---------------------------------------------------------
+        | QP_NUM  | SEND/RECV Flag ||  CQ_NUM |   ARM/POLL Flag |
+        ---------------------------------------------------------
+		- Offset 0 used for QP operations (send and recv)
+		- Offset 4 used for CQ operations (arm and poll)
+
+
+4.3 Major flows
+===============
+
+4.3.1 Create CQ
+===============
+    - Guest driver
+        - Allocates pages for CQ ring
+        - Creates page directory (pdir) to hold CQ ring's pages
+        - Initializes CQ ring
+        - Initializes 'Create CQ' command object (cqe, pdir etc)
+        - Copies the command to 'command' address
+        - Writes 0 into REQ register
+    - Device
+        - Reads the request object from the 'command' address
+        - Allocates CQ object and initialize CQ ring based on pdir
+        - Creates the backend CQ
+        - Writes operation status to ERR register
+        - Posts command-interrupt to guest
+    - Guest driver
+        - Reads the HW response code from ERR register
+
+4.3.2 Create QP
+===============
+    - Guest driver
+        - Allocates pages for send and receive rings
+        - Creates page directory(pdir) to hold the ring's pages
+        - Initializes 'Create QP' command object (max_send_wr,
+          send_cq_handle, recv_cq_handle, pdir etc)
+        - Copies the object to 'command' address
+        - Write 0 into REQ register
+    - Device
+        - Reads the request object from 'command' address
+        - Allocates the QP object and initialize
+            - Send and recv rings based on pdir
+            - Send and recv ring state
+        - Creates the backend QP
+        - Writes the operation status to ERR register
+        - Posts command-interrupt to guest
+    - Guest driver
+        - Reads the HW response code from ERR register
+
+4.3.3 Post receive
+==================
+    - Guest driver
+        - Initializes a wqe and place it on recv ring
+        - Write to qpn|qp_recv_bit (31) to QP offset in UAR
+    - Device
+        - Extracts qpn from UAR
+        - Walks through the ring and does the following for each wqe
+            - Prepares the backend CQE context to be used when
+              receiving completion from backend (wr_id, op_code, emu_cq_num)
+            - For each sge prepares backend sge
+            - Calls backend's post_recv
+
+4.3.4 Process backend events
+============================
+    - Done by a dedicated thread used to process backend events;
+      at initialization is attached to the device and creates
+      the communication channel.
+    - Thread main loop:
+        - Polls for completions
+        - Extracts QEMU _cq_num, wr_id and op_code from context
+        - Writes CQE to CQ ring
+        - Writes CQ number to device CQ
+        - Sends completion-interrupt to guest
+        - Deallocates context
+        - Acks the event to backend
+
+
+
+5. Limitations
+==============
+- The device obviously is limited by the Guest Linux Driver features implementation
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much.
+- The device requires target page size to be the same as the host page size.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge
+  pages available, QEMU will use them.
+- As previously stated, migration is not supported yet, however with some hardware
+  support can be done.
+
+
+
+6. Performance
+==============
+By design the pvrdma device exits on each post-send/receive, so for small buffers
+the performance is affected; however for medium buffers it will became close to
+bare metal and from 1MB buffers and  up it reaches bare metal performance.
+(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)
+
+All the above assumes no memory registration is done on data path.
Signature · 1 line
2.13.5
#8 lore raw

Re: [Qemu-devel] [PATCH V6 3/5] docs: add pvrdma device documentation.

Quoted 11 lines · On Sun, 7 Jan 2018 14:32:22 +0200 Marcel Apfelbaum <[email protected]> wrote:
---
 docs/pvrdma.txt | 254 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 254 insertions(+)
 create mode 100644 docs/pvrdma.txt

+5. Limitations

+==============
+- The device obviously is limited by the Guest Linux Driver features implementation

docs/pvrdma.txt line 240

Quoted 8 lines
docs/pvrdma.txt new file +8 -0
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much.
+- The device requires target page size to be the same as the host page size.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge

s/Gust/Guest/

Quoted 3 lines
docs/pvrdma.txt new file +3 -0
+  pages available, QEMU will use them.
+- As previously stated, migration is not supported yet, however with some hardware
+  support can be done.
#9 lore raw

Re: [Qemu-devel] [PATCH V6 3/5] docs: add pvrdma device documentation.

Quoted 23 lines · Cornelia Huck
Quoted 11 lines · On Sun, 7 Jan 2018 14:32:22 +0200 Marcel Apfelbaum <[email protected]> wrote:
---
  docs/pvrdma.txt | 254 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 254 insertions(+)

create mode 100644 docs/pvrdma.txt

+5. Limitations

+==============
+- The device obviously is limited by the Guest Linux Driver features implementation

docs/pvrdma.txt line 240

Quoted 8 lines
docs/pvrdma.txt new file +8 -0
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much.
+- The device requires target page size to be the same as the host page size.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge
Quoted 1 line

s/Gust/Guest/

Fixed (for V7)

Thanks!
Marcel

Quoted 4 lines
Quoted 4 lines
docs/pvrdma.txt new file +3 -0
+  pages available, QEMU will use them.
+- As previously stated, migration is not supported yet, however with some hardware
+  support can be done.
#10 lore raw

PATCH v6 4/5 pvrdma: initial implementation

In reply to Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---
 Makefile.objs                      |   2 +
 configure                          |   9 +-
 default-configs/arm-softmmu.mak    |   1 +
 default-configs/i386-softmmu.mak   |   1 +
 default-configs/x86_64-softmmu.mak |   1 +
 hw/Makefile.objs                   |   1 +
 hw/rdma/Makefile.objs              |   6 +
 hw/rdma/rdma_backend.c             | 815 +++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h             |  92 +++++
 hw/rdma/rdma_backend_defs.h        |  62 +++
 hw/rdma/rdma_rm.c                  | 619 ++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h                  |  69 ++++
 hw/rdma/rdma_rm_defs.h             | 106 +++++
 hw/rdma/rdma_utils.c               |  52 +++
 hw/rdma/rdma_utils.h               |  43 ++
 hw/rdma/trace-events               |   5 +
 hw/rdma/vmw/pvrdma.h               | 122 ++++++
 hw/rdma/vmw/pvrdma_cmd.c           | 679 ++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h       | 602 +++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c      | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h      |  42 ++
 hw/rdma/vmw/pvrdma_ib_verbs.h      | 433 ++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c          | 644 +++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c        | 212 ++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h        |  27 ++
 hw/rdma/vmw/pvrdma_ring.h          | 134 ++++++
 hw/rdma/vmw/trace-events           |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h       | 311 ++++++++++++++
 include/hw/pci/pci_ids.h           |   3 +
 29 files changed, 5233 insertions(+), 4 deletions(-)
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h
Makefile.objs +2 -0
index 285c6f3c15..4acaa1352d 100644
@@ -129,6 +129,8 @@ trace-events-subdirs += hw/block/dataplane
 trace-events-subdirs += hw/char
 trace-events-subdirs += hw/intc
 trace-events-subdirs += hw/net
+trace-events-subdirs += hw/rdma
+trace-events-subdirs += hw/rdma/vmw
 trace-events-subdirs += hw/virtio
 trace-events-subdirs += hw/audio
 trace-events-subdirs += hw/misc
configure +5 -4
index 100309c33f..65a1a76f39 100755
@@ -1529,7 +1529,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   bluez           bluez stack connectivity
   kvm             KVM acceleration support
   hax             HAX acceleration support
-  rdma            RDMA-based migration support
+  rdma            Enable RDMA-based migration support and PVRDMA
   vde             support for vde network
   netmap          support for netmap network
   linux-aio       Linux AIO support
@@ -2856,15 +2856,16 @@ if test "$rdma" != "no" ; then
 #include <rdma/rdma_cma.h>
 int main(void) { return 0; }
 EOF
-  rdma_libs="-lrdmacm -libverbs"
+  rdma_libs="-lrdmacm -libverbs -libumad"
   if compile_prog "" "$rdma_libs" ; then
     rdma="yes"
+    libs_softmmu="$libs_softmmu $rdma_libs"
   else
     if test "$rdma" = "yes" ; then
         error_exit \
-            " OpenFabrics librdmacm/libibverbs not present." \
+            " OpenFabrics librdmacm/libibverbs/libibumad not present." \
             " Your options:" \
-            "  (1) Fast: Install infiniband packages from your distro." \
+            "  (1) Fast: Install infiniband packages (devel) from your distro." \
             "  (2) Cleanest: Install libraries from www.openfabrics.org" \
             "  (3) Also: Install softiwarp if you don't have RDMA hardware"
     fi
default-configs/arm-softmmu.mak +1 -0
index b0d6e65038..0e7a3c1700 100644
@@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y
 CONFIG_MSF2=y
 CONFIG_FW_CFG_DMA=y
 CONFIG_XILINX_AXI=y
+CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
index 95ac4b464a..88298e4ef5 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +1 -0
index 0221236825..f571da36eb 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
hw/Makefile.objs +1 -0
index cf4cb2010b..6a0ffe0afd 100644
@@ -18,6 +18,7 @@ devices-dirs-$(CONFIG_IPMI) += ipmi/
 devices-dirs-$(CONFIG_SOFTMMU) += isa/
 devices-dirs-$(CONFIG_SOFTMMU) += misc/
 devices-dirs-$(CONFIG_SOFTMMU) += net/
+devices-dirs-$(CONFIG_SOFTMMU) += rdma/
 devices-dirs-$(CONFIG_SOFTMMU) += nvram/
 devices-dirs-$(CONFIG_SOFTMMU) += pci/
 devices-dirs-$(CONFIG_PCI) += pci-bridge/ pci-host/
hw/rdma/Makefile.objs new file +6 -0
new file mode 100644
index 0000000000..05af0b7bec
@@ -0,0 +1,6 @@
+ifeq ($(CONFIG_RDMA),y)
+obj-$(CONFIG_PVRDMA) += rdma_utils.o rdma_backend.o rdma_rm.o
+obj-$(CONFIG_PVRDMA) += vmw/pvrdma_dev_ring.o vmw/pvrdma_cmd.o \
+                        vmw/pvrdma_qp_ops.o vmw/pvrdma_main.o
+endif
+
hw/rdma/rdma_backend.c new file +815 -0

815 lines not on this page · raw message

hw/rdma/rdma_backend.h new file +92 -0

92 lines not on this page · raw message

hw/rdma/rdma_backend_defs.h new file +62 -0

62 lines not on this page · raw message

hw/rdma/rdma_rm.c new file +619 -0

619 lines not on this page · raw message

hw/rdma/rdma_rm.h new file +69 -0

69 lines not on this page · raw message

hw/rdma/rdma_rm_defs.h new file +106 -0

106 lines not on this page · raw message

hw/rdma/rdma_utils.c new file +52 -0

52 lines not on this page · raw message

hw/rdma/rdma_utils.h new file +43 -0

43 lines not on this page · raw message

hw/rdma/trace-events new file +5 -0
new file mode 100644
index 0000000000..421986adbe
@@ -0,0 +1,5 @@
+# See docs/tracing.txt for syntax documentation.
+
+#hw/rdma/rdma_backend.c
+create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
+create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
hw/rdma/vmw/pvrdma.h new file +122 -0

122 lines not on this page · raw message

hw/rdma/vmw/pvrdma_cmd.c new file +679 -0

679 lines not on this page · raw message

hw/rdma/vmw/pvrdma_dev_api.h new file +602 -0

276 lines not on this page · raw message

hw/rdma/vmw/pvrdma_dev_ring.c new file +139 -0

139 lines not on this page · raw message

hw/rdma/vmw/pvrdma_dev_ring.h new file +42 -0

42 lines not on this page · raw message

hw/rdma/vmw/pvrdma_ib_verbs.h new file +433 -0

433 lines not on this page · raw message

hw/rdma/vmw/pvrdma_main.c new file +644 -0

644 lines not on this page · raw message

hw/rdma/vmw/pvrdma_qp_ops.c new file +212 -0

212 lines not on this page · raw message

hw/rdma/vmw/pvrdma_qp_ops.h new file +27 -0
new file mode 100644
index 0000000000..61ec036835

Lines of this file are not shown. View the raw message.

hw/rdma/vmw/pvrdma_ring.h new file +134 -0

134 lines not on this page · raw message

hw/rdma/vmw/trace-events new file +5 -0
new file mode 100644
index 0000000000..dd4dde311e

Lines of this file are not shown. View the raw message.

hw/rdma/vmw/vmw_pvrdma-abi.h new file +311 -0

311 lines not on this page · raw message

include/hw/pci/pci_ids.h +3 -0
index 35df1874a9..1dbf53627c 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.13.5

Part of the diff is not shown here. View the raw message.

#11 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

Quoted 69 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---
 Makefile.objs                      |   2 +
 configure                          |   9 +-
 default-configs/arm-softmmu.mak    |   1 +
 default-configs/i386-softmmu.mak   |   1 +
 default-configs/x86_64-softmmu.mak |   1 +
 hw/Makefile.objs                   |   1 +
 hw/rdma/Makefile.objs              |   6 +
 hw/rdma/rdma_backend.c             | 815 +++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h             |  92 +++++
 hw/rdma/rdma_backend_defs.h        |  62 +++
 hw/rdma/rdma_rm.c                  | 619 ++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h                  |  69 ++++
 hw/rdma/rdma_rm_defs.h             | 106 +++++
 hw/rdma/rdma_utils.c               |  52 +++
 hw/rdma/rdma_utils.h               |  43 ++
 hw/rdma/trace-events               |   5 +
 hw/rdma/vmw/pvrdma.h               | 122 ++++++
 hw/rdma/vmw/pvrdma_cmd.c           | 679 ++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h       | 602 +++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c      | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h      |  42 ++
 hw/rdma/vmw/pvrdma_ib_verbs.h      | 433 ++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c          | 644 +++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c        | 212 ++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h        |  27 ++
 hw/rdma/vmw/pvrdma_ring.h          | 134 ++++++
 hw/rdma/vmw/trace-events           |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h       | 311 ++++++++++++++
 include/hw/pci/pci_ids.h           |   3 +
 29 files changed, 5233 insertions(+), 4 deletions(-)
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h

(...)

Quoted 23 lines
default-configs/arm-softmmu.mak +1 -0
index b0d6e65038..0e7a3c1700 100644
@@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y
 CONFIG_MSF2=y
 CONFIG_FW_CFG_DMA=y
 CONFIG_XILINX_AXI=y
+CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
index 95ac4b464a..88298e4ef5 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +0 -0
index 0221236825..f571da36eb 100644

default-configs/x86_64-softmmu.mak CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) · line 64

Quoted 4 lines
default-configs/x86_64-softmmu.mak +1 -0
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y

Any reason you did not add this to other architectures?

I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at least builds (did not try to actually get it to work, although I don't see any immediate blocker for that).

(...)

Quoted 5 lines
hw/rdma/rdma_backend.c new file +0 -0
new file mode 100644
index 0000000000..dcb799f49b

(...)

Quoted 15 lines
hw/rdma/rdma_backend.c new file +15 -0
+static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq,
+                    bool one_poll)
+{
+    int i, ne;
+    BackendCtx *bctx;
+    struct ibv_wc wc[2];
+
+    pr_dbg("Entering poll_cq loop on cq %p\n", ibcq);
+    do {
+        ne = ibv_poll_cq(ibcq, 2, wc);
+        if (ne == 0 && one_poll) {
+            pr_dbg("CQ is empty\n");
+            return;
+        }
+    } while (ne < 0);

hw/rdma/rdma_backend.c line 74

Quoted 10 lines
hw/rdma/rdma_backend.c new file +10 -0
+
+    pr_dbg("Got %d completion(s) from cq %p\n", ne, ibcq);
+
+    for (i = 0; i < ne; i++) {
+        pr_dbg("wr_id=0x%lx\n", wc[i].wr_id);
+        pr_dbg("status=%d\n", wc[i].status);
+
+        bctx = rdma_rm_get_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Fail to find ctx for req %ld\n", wc[i].wr_id);

s/Fail/Failed/

(A lot of these through out the various files. Just thought I'd point that out; but I don't really have time to do a real review.)

Quoted 10 lines
hw/rdma/rdma_backend.c new file +10 -0
+            continue;
+        }
+        pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv");
+
+        comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx);
+
+        rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        free(bctx);
+    }
+}

(...)

Quoted 19 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +13 -0
new file mode 100644
index 0000000000..bf1986a976
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *

hw/rdma/vmw/pvrdma_dev_api.h line 23

Quoted 10 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +10 -0
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h

Could that file be exported as UAPI in the kernel and added to the linux-headers script?

(...)

Quoted 25 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +19 -0
new file mode 100644
index 0000000000..cf1430024b
@@ -0,0 +1,433 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_IB_VERBS_H
+#define PVRDMA_IB_VERBS_H
+
+/*

hw/rdma/vmw/pvrdma_ib_verbs.h line 29

Quoted 10 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +10 -0
+ * VMWARE headers we got from Linux kernel do not fully comply QEMU coding
+ * standards in sense of types and defines used.
+ * Since we didn't want to change VMWARE code, following set of typedefs
+ * and defines needed to compile these headers with QEMU introduced.
+ */
+
+#define u8     uint8_t
+#define u16    unsigned short
+#define u32    uint32_t
+#define u64    uint64_t

I think the headers update already takes care of some conversions. Otherwise, same comment as for the header above.

Quoted 7 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +7 -0
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h
+ */

(...)

Quoted 20 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +14 -0
new file mode 100644
index 0000000000..8cfb9d7745
@@ -0,0 +1,311 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */

hw/rdma/vmw/vmw_pvrdma-abi.h line 24

Quoted 10 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +10 -0
+
+#ifndef VMW_PVRDMA_ABI_H
+#define VMW_PVRDMA_ABI_H
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * include/uapi/rdma/vmw_pvrdma-abi.h
+ */

This one is already exported.

#12 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

Quoted 147 lines · Cornelia Huck
Quoted 69 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---
 Makefile.objs                      |   2 +
 configure                          |   9 +-
 default-configs/arm-softmmu.mak    |   1 +
 default-configs/i386-softmmu.mak   |   1 +
 default-configs/x86_64-softmmu.mak |   1 +
 hw/Makefile.objs                   |   1 +
 hw/rdma/Makefile.objs              |   6 +
 hw/rdma/rdma_backend.c             | 815 +++++++++++++++++++++++++++++++++++++
 hw/rdma/rdma_backend.h             |  92 +++++
 hw/rdma/rdma_backend_defs.h        |  62 +++
 hw/rdma/rdma_rm.c                  | 619 ++++++++++++++++++++++++++++
 hw/rdma/rdma_rm.h                  |  69 ++++
 hw/rdma/rdma_rm_defs.h             | 106 +++++
 hw/rdma/rdma_utils.c               |  52 +++
 hw/rdma/rdma_utils.h               |  43 ++
 hw/rdma/trace-events               |   5 +
 hw/rdma/vmw/pvrdma.h               | 122 ++++++
 hw/rdma/vmw/pvrdma_cmd.c           | 679 ++++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_api.h       | 602 +++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_dev_ring.c      | 139 +++++++
 hw/rdma/vmw/pvrdma_dev_ring.h      |  42 ++
 hw/rdma/vmw/pvrdma_ib_verbs.h      | 433 ++++++++++++++++++++
 hw/rdma/vmw/pvrdma_main.c          | 644 +++++++++++++++++++++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.c        | 212 ++++++++++
 hw/rdma/vmw/pvrdma_qp_ops.h        |  27 ++
 hw/rdma/vmw/pvrdma_ring.h          | 134 ++++++
 hw/rdma/vmw/trace-events           |   5 +
 hw/rdma/vmw/vmw_pvrdma-abi.h       | 311 ++++++++++++++
 include/hw/pci/pci_ids.h           |   3 +
 29 files changed, 5233 insertions(+), 4 deletions(-)
 create mode 100644 hw/rdma/Makefile.objs
 create mode 100644 hw/rdma/rdma_backend.c
 create mode 100644 hw/rdma/rdma_backend.h
 create mode 100644 hw/rdma/rdma_backend_defs.h
 create mode 100644 hw/rdma/rdma_rm.c
 create mode 100644 hw/rdma/rdma_rm.h
 create mode 100644 hw/rdma/rdma_rm_defs.h
 create mode 100644 hw/rdma/rdma_utils.c
 create mode 100644 hw/rdma/rdma_utils.h
 create mode 100644 hw/rdma/trace-events
 create mode 100644 hw/rdma/vmw/pvrdma.h
 create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
 create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
 create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
 create mode 100644 hw/rdma/vmw/pvrdma_main.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
 create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
 create mode 100644 hw/rdma/vmw/pvrdma_ring.h
 create mode 100644 hw/rdma/vmw/trace-events
 create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h

(...)

Quoted 23 lines
default-configs/arm-softmmu.mak +1 -0
index b0d6e65038..0e7a3c1700 100644
@@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y
 CONFIG_MSF2=y
 CONFIG_FW_CFG_DMA=y
 CONFIG_XILINX_AXI=y
+CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
index 95ac4b464a..88298e4ef5 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +0 -0
index 0221236825..f571da36eb 100644

default-configs/x86_64-softmmu.mak CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) · line 64

Quoted 4 lines
default-configs/x86_64-softmmu.mak +1 -0
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y

Any reason you did not add this to other architectures?

I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at least builds (did not try to actually get it to work, although I don't see any immediate blocker for that).

(...)

Quoted 5 lines
hw/rdma/rdma_backend.c new file +0 -0
new file mode 100644
index 0000000000..dcb799f49b

(...)

Quoted 15 lines
hw/rdma/rdma_backend.c new file +15 -0
+static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq,
+                    bool one_poll)
+{
+    int i, ne;
+    BackendCtx *bctx;
+    struct ibv_wc wc[2];
+
+    pr_dbg("Entering poll_cq loop on cq %p\n", ibcq);
+    do {
+        ne = ibv_poll_cq(ibcq, 2, wc);
+        if (ne == 0 && one_poll) {
+            pr_dbg("CQ is empty\n");
+            return;
+        }
+    } while (ne < 0);

hw/rdma/rdma_backend.c line 74

Quoted 10 lines
hw/rdma/rdma_backend.c new file +10 -0
+
+    pr_dbg("Got %d completion(s) from cq %p\n", ne, ibcq);
+
+    for (i = 0; i < ne; i++) {
+        pr_dbg("wr_id=0x%lx\n", wc[i].wr_id);
+        pr_dbg("status=%d\n", wc[i].status);
+
+        bctx = rdma_rm_get_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Fail to find ctx for req %ld\n", wc[i].wr_id);

s/Fail/Failed/

Quoted 2 lines

(A lot of these through out the various files. Just thought I'd point that out; but I don't really have time to do a real review.)

Thanks!

Quoted 38 lines
Quoted 11 lines
hw/rdma/rdma_backend.c new file +10 -0
+            continue;
+        }
+        pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv");
+
+        comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx);
+
+        rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id);
+        free(bctx);
+    }
+}

(...)

Quoted 24 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +18 -0
new file mode 100644
index 0000000000..bf1986a976
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
Quoted 8 lines

hw/rdma/vmw/pvrdma_dev_api.h line 23

Quoted 5 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +5 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h

Could that file be exported as UAPI in the kernel and added to the linux-headers script?

We took this approach as apposed to kernel-headers with the following on our mind:
(1) This is the convention used in vmxnet3.
(2) vmw_pvrdma was introduced only lately, taking the kernel-headers approach will force specific kernel on a host in order to compile QEMU.
(3) To support VMWare's pvrdma device we took a snapshot of existing driver/device settings and breezed there. This is driver/device API and we can't allow our self to chase VMWare's tail whenever they are changing the API. Just consider a case where they will change for example the ARM bit.

Just IMHO.

Quoted 33 lines

(...)

Quoted 30 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +24 -0
new file mode 100644
index 0000000000..cf1430024b
@@ -0,0 +1,433 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_IB_VERBS_H
+#define PVRDMA_IB_VERBS_H
+
+/*
+ * VMWARE headers we got from Linux kernel do not fully comply QEMU coding
+ * standards in sense of types and defines used.
+ * Since we didn't want to change VMWARE code, following set of typedefs
+ * and defines needed to compile these headers with QEMU introduced.
+ */
Quoted 8 lines

hw/rdma/vmw/pvrdma_ib_verbs.h line 29

Quoted 5 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +5 -0
+
+#define u8     uint8_t
+#define u16    unsigned short
+#define u32    uint32_t
+#define u64    uint64_t

I think the headers update already takes care of some conversions. Otherwise, same comment as for the header above.

Sorry, i'm not following, can you elaborate on that?

Quoted 35 lines
Quoted 8 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +7 -0
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h
+ */

(...)

Quoted 24 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +18 -0
new file mode 100644
index 0000000000..8cfb9d7745
@@ -0,0 +1,311 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef VMW_PVRDMA_ABI_H
+#define VMW_PVRDMA_ABI_H
+
Quoted 8 lines

hw/rdma/vmw/vmw_pvrdma-abi.h line 24

Quoted 6 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +6 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * include/uapi/rdma/vmw_pvrdma-abi.h
+ */

This one is already exported.

Same argument as above.

Yuval

#13 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

Quoted 37 lines · On Tue, 9 Jan 2018 13:08:33 +0200 Yuval Shaia <[email protected]> wrote:
Quoted 27 lines · Cornelia Huck
Quoted 24 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:
hw/rdma/vmw/pvrdma_dev_api.h new file +18 -0
new file mode 100644
index 0000000000..bf1986a976
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
Quoted 8 lines

hw/rdma/vmw/pvrdma_dev_api.h line 23

Quoted 5 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +5 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h  

Could that file be exported as UAPI in the kernel and added to the linux-headers script?

Quoted 5 lines

We took this approach as apposed to kernel-headers with the following on our mind:
(1) This is the convention used in vmxnet3.
(2) vmw_pvrdma was introduced only lately, taking the kernel-headers approach will force specific kernel on a host in order to compile QEMU.

qemu will get the kernel headers once from the upstream kernel and then will be able to be built everywhere.

Quoted 4 lines

(3) To support VMWare's pvrdma device we took a snapshot of existing driver/device settings and breezed there. This is driver/device API and we can't allow our self to chase VMWare's tail whenever they are changing the API. Just consider a case where they will change for example the ARM bit.

But as want to enable the existing device driver, you'll want to be able to produce a compatible device anyway, don't you? Also, wouldn't VMWare break older kernels if they suddenly changed the api?

[Also, is there a canonical reference for this API?]

Quoted 45 lines

Just IMHO.

Quoted 33 lines

(...)

Quoted 30 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +24 -0
new file mode 100644
index 0000000000..cf1430024b
@@ -0,0 +1,433 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_IB_VERBS_H
+#define PVRDMA_IB_VERBS_H
+
+/*
+ * VMWARE headers we got from Linux kernel do not fully comply QEMU coding
+ * standards in sense of types and defines used.
+ * Since we didn't want to change VMWARE code, following set of typedefs
+ * and defines needed to compile these headers with QEMU introduced.
+ */
Quoted 8 lines

hw/rdma/vmw/pvrdma_ib_verbs.h line 29

Quoted 5 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +5 -0
+
+#define u8     uint8_t
+#define u16    unsigned short
+#define u32    uint32_t
+#define u64    uint64_t  

I think the headers update already takes care of some conversions. Otherwise, same comment as for the header above.

Quoted 1 line

Sorry, i'm not following, can you elaborate on that?

There's some post processing done by the headers update. But I think you'd need to use __u8 and friends for an exported header anyway.

Quoted 45 lines
Quoted 36 lines
Quoted 9 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +7 -0
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h
+ */  

(...)

Quoted 24 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +18 -0
new file mode 100644
index 0000000000..8cfb9d7745
@@ -0,0 +1,311 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef VMW_PVRDMA_ABI_H
+#define VMW_PVRDMA_ABI_H
+
Quoted 8 lines

hw/rdma/vmw/vmw_pvrdma-abi.h line 24

Quoted 6 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +6 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * include/uapi/rdma/vmw_pvrdma-abi.h
+ */  

This one is already exported.

Quoted 1 line

Same argument as above.

Well, I sure hope VMWare is not planning to change an exported ABI in an incompatible way...

#14 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

Quoted 49 lines · Cornelia Huck
Quoted 37 lines · On Tue, 9 Jan 2018 13:08:33 +0200 Yuval Shaia <[email protected]> wrote:
Quoted 27 lines · Cornelia Huck
Quoted 24 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:
hw/rdma/vmw/pvrdma_dev_api.h new file +18 -0
new file mode 100644
index 0000000000..bf1986a976
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
Quoted 8 lines

hw/rdma/vmw/pvrdma_dev_api.h line 23

Quoted 5 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +5 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h

Could that file be exported as UAPI in the kernel and added to the linux-headers script?

Quoted 5 lines

We took this approach as apposed to kernel-headers with the following on our mind:
(1) This is the convention used in vmxnet3.
(2) vmw_pvrdma was introduced only lately, taking the kernel-headers approach will force specific kernel on a host in order to compile QEMU.

qemu will get the kernel headers once from the upstream kernel and then will be able to be built everywhere.

Quoted 9 lines
Quoted 4 lines

(3) To support VMWare's pvrdma device we took a snapshot of existing driver/device settings and breezed there. This is driver/device API and we can't allow our self to chase VMWare's tail whenever they are changing the API. Just consider a case where they will change for example the ARM bit.

But as want to enable the existing device driver, you'll want to be able to produce a compatible device anyway, don't you? Also, wouldn't VMWare break older kernels if they suddenly changed the api?

I think it was a missunderstanding here :)

We don't actually need the VMware headers here in order to compile the device. Requesting the pvrdma headers to be present really limits the hosts we could compile QEMU without an actual benefit.

The headers describe the data structures passed by the Guest driver to the PVRDMA device. Having them on the host should not be a requirement, we could simply define our own structs but it would be harder to maintain. (diffing with linux headers to see what's new)

Note we don't care about ABI changes, we emulate a PCI device, not
a para-virt one, the driver has the contract to pass the same data definitions always. If the data changes, there are 2 options:
1. Pass another "version nr" on the command channel (DSR) and we declare as not supported by this device and fail gracefully.
2. Use another device id.

We looked for what are the options for *not requiring* the headers to be present and we saw two in QEMU:
1. Adding it to include/standard-headers/linux/
But it seemed to much, only our device need them.
2. Adding it to the hw/rdma/vwm directory
Please look at: (hw/net/vmxnet3.h)

The second option seemed better, but we could go with "1." if I got it right and you think is cleaner.
But again, the first option kind of requires updating the linux headers always, but we want quite the opposite, to not be connected to the changes.
I hope is more clear now.

Quoted 2 lines

[Also, is there a canonical reference for this API?]

Linux Kernel PVRDMA driver implementation...

Quoted 3 lines
Quoted 3 lines

Just IMHO.

Thanks a lot for your inputs, much appreciated!
Marcel

Quoted 97 lines
Quoted 42 lines
Quoted 33 lines

(...)

Quoted 30 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +24 -0
new file mode 100644
index 0000000000..cf1430024b
@@ -0,0 +1,433 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_IB_VERBS_H
+#define PVRDMA_IB_VERBS_H
+
+/*
+ * VMWARE headers we got from Linux kernel do not fully comply QEMU coding
+ * standards in sense of types and defines used.
+ * Since we didn't want to change VMWARE code, following set of typedefs
+ * and defines needed to compile these headers with QEMU introduced.
+ */
Quoted 8 lines

hw/rdma/vmw/pvrdma_ib_verbs.h line 29

Quoted 5 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +5 -0
+
+#define u8     uint8_t
+#define u16    unsigned short
+#define u32    uint32_t
+#define u64    uint64_t

I think the headers update already takes care of some conversions. Otherwise, same comment as for the header above.

Quoted 1 line

Sorry, i'm not following, can you elaborate on that?

There's some post processing done by the headers update. But I think you'd need to use __u8 and friends for an exported header anyway.

Quoted 45 lines
Quoted 36 lines
Quoted 9 lines
hw/rdma/vmw/pvrdma_ib_verbs.h new file +7 -0
+
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h
+ */

(...)

Quoted 24 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +18 -0
new file mode 100644
index 0000000000..8cfb9d7745
@@ -0,0 +1,311 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef VMW_PVRDMA_ABI_H
+#define VMW_PVRDMA_ABI_H
+
Quoted 8 lines

hw/rdma/vmw/vmw_pvrdma-abi.h line 24

Quoted 6 lines
hw/rdma/vmw/vmw_pvrdma-abi.h new file +6 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * include/uapi/rdma/vmw_pvrdma-abi.h
+ */

This one is already exported.

Quoted 1 line

Same argument as above.

Well, I sure hope VMWare is not planning to change an exported ABI in an incompatible way...

#15 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

Quoted 70 lines · On Wed, 10 Jan 2018 11:28:59 +0200 Marcel Apfelbaum <[email protected]> wrote:
Quoted 49 lines · Cornelia Huck
Quoted 37 lines · On Tue, 9 Jan 2018 13:08:33 +0200 Yuval Shaia <[email protected]> wrote:
Quoted 27 lines · Cornelia Huck
Quoted 24 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:
hw/rdma/vmw/pvrdma_dev_api.h new file +18 -0
new file mode 100644
index 0000000000..bf1986a976
@@ -0,0 +1,602 @@
+/*
+ * QEMU VMWARE paravirtual RDMA device definitions
+ *
+ * Copyright (C) 2018 Oracle
+ * Copyright (C) 2018 Red Hat Inc
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_DEV_API_H
+#define PVRDMA_DEV_API_H
+
Quoted 8 lines

hw/rdma/vmw/pvrdma_dev_api.h line 23

Quoted 5 lines
hw/rdma/vmw/pvrdma_dev_api.h new file +5 -0
+/*
+ * Following is an interface definition for PVRDMA device as provided by
+ * VMWARE.
+ * See original copyright from Linux kernel v4.14.5 header file
+ * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h  

Could that file be exported as UAPI in the kernel and added to the linux-headers script?

Quoted 5 lines

We took this approach as apposed to kernel-headers with the following on our mind:
(1) This is the convention used in vmxnet3.
(2) vmw_pvrdma was introduced only lately, taking the kernel-headers approach will force specific kernel on a host in order to compile QEMU.

qemu will get the kernel headers once from the upstream kernel and then will be able to be built everywhere.

Quoted 9 lines
Quoted 4 lines

(3) To support VMWare's pvrdma device we took a snapshot of existing driver/device settings and breezed there. This is driver/device API and we can't allow our self to chase VMWare's tail whenever they are changing the API. Just consider a case where they will change for example the ARM bit.

But as want to enable the existing device driver, you'll want to be able to produce a compatible device anyway, don't you? Also, wouldn't VMWare break older kernels if they suddenly changed the api?

I think it was a missunderstanding here :)

We don't actually need the VMware headers here in order to compile the device. Requesting the pvrdma headers to be present really limits the hosts we could compile QEMU without an actual benefit.

The headers describe the data structures passed by the Guest driver to the PVRDMA device. Having them on the host should not be a requirement, we could simply define our own structs but it would be harder to maintain. (diffing with linux headers to see what's new)

Quoted 6 lines

Note we don't care about ABI changes, we emulate a PCI device, not
a para-virt one, the driver has the contract to pass the same data definitions always. If the data changes, there are 2 options:
1. Pass another "version nr" on the command channel (DSR) and we declare as not supported by this device and fail gracefully.
2. Use another device id.

OK, thanks for the clarification, that makes sense.

Quoted 8 lines

We looked for what are the options for *not requiring* the headers to be present and we saw two in QEMU:
1. Adding it to include/standard-headers/linux/
But it seemed to much, only our device need them.
2. Adding it to the hw/rdma/vwm directory
Please look at: (hw/net/vmxnet3.h)

Quoted 6 lines

The second option seemed better, but we could go with "1." if I got it right and you think is cleaner.
But again, the first option kind of requires updating the linux headers always, but we want quite the opposite, to not be connected to the changes.
I hope is more clear now.

OK, that makes sense as well, if you intend this to be a one-time effort. "Copy a header from the Linux kernel" just raised kind of a flag for me.

Quoted 6 lines
Quoted 4 lines

[Also, is there a canonical reference for this API?]

Linux Kernel PVRDMA driver implementation...

:)

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Just IMHO.

Thanks a lot for your inputs, much appreciated!

NP. I'd like to do more, as this is interesting stuff, but you know how it goes...

#16 lore raw

Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation

In reply to Cornelia Huck

Hi Cornelia,

Quoted 96 lines · Cornelia Huck
Quoted 69 lines · On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum <[email protected]> wrote:

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---
  Makefile.objs                      |   2 +
  configure                          |   9 +-
  default-configs/arm-softmmu.mak    |   1 +
  default-configs/i386-softmmu.mak   |   1 +
  default-configs/x86_64-softmmu.mak |   1 +
  hw/Makefile.objs                   |   1 +
  hw/rdma/Makefile.objs              |   6 +
  hw/rdma/rdma_backend.c             | 815 +++++++++++++++++++++++++++++++++++++
  hw/rdma/rdma_backend.h             |  92 +++++
  hw/rdma/rdma_backend_defs.h        |  62 +++
  hw/rdma/rdma_rm.c                  | 619 ++++++++++++++++++++++++++++
  hw/rdma/rdma_rm.h                  |  69 ++++
  hw/rdma/rdma_rm_defs.h             | 106 +++++
  hw/rdma/rdma_utils.c               |  52 +++
  hw/rdma/rdma_utils.h               |  43 ++
  hw/rdma/trace-events               |   5 +
  hw/rdma/vmw/pvrdma.h               | 122 ++++++
  hw/rdma/vmw/pvrdma_cmd.c           | 679 ++++++++++++++++++++++++++++++
  hw/rdma/vmw/pvrdma_dev_api.h       | 602 +++++++++++++++++++++++++++
  hw/rdma/vmw/pvrdma_dev_ring.c      | 139 +++++++
  hw/rdma/vmw/pvrdma_dev_ring.h      |  42 ++
  hw/rdma/vmw/pvrdma_ib_verbs.h      | 433 ++++++++++++++++++++
  hw/rdma/vmw/pvrdma_main.c          | 644 +++++++++++++++++++++++++++++
  hw/rdma/vmw/pvrdma_qp_ops.c        | 212 ++++++++++
  hw/rdma/vmw/pvrdma_qp_ops.h        |  27 ++
  hw/rdma/vmw/pvrdma_ring.h          | 134 ++++++
  hw/rdma/vmw/trace-events           |   5 +
  hw/rdma/vmw/vmw_pvrdma-abi.h       | 311 ++++++++++++++
  include/hw/pci/pci_ids.h           |   3 +
  29 files changed, 5233 insertions(+), 4 deletions(-)

create mode 100644 hw/rdma/Makefile.objs create mode 100644 hw/rdma/rdma_backend.c create mode 100644 hw/rdma/rdma_backend.h create mode 100644 hw/rdma/rdma_backend_defs.h create mode 100644 hw/rdma/rdma_rm.c create mode 100644 hw/rdma/rdma_rm.h create mode 100644 hw/rdma/rdma_rm_defs.h create mode 100644 hw/rdma/rdma_utils.c create mode 100644 hw/rdma/rdma_utils.h create mode 100644 hw/rdma/trace-events create mode 100644 hw/rdma/vmw/pvrdma.h create mode 100644 hw/rdma/vmw/pvrdma_cmd.c create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h create mode 100644 hw/rdma/vmw/pvrdma_main.c create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h create mode 100644 hw/rdma/vmw/pvrdma_ring.h create mode 100644 hw/rdma/vmw/trace-events create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h

(...)

Quoted 21 lines
default-configs/arm-softmmu.mak +1 -0
index b0d6e65038..0e7a3c1700 100644
@@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y
  CONFIG_MSF2=y
  CONFIG_FW_CFG_DMA=y
  CONFIG_XILINX_AXI=y
+CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
index 95ac4b464a..88298e4ef5 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
  CONFIG_PXB=y
  CONFIG_ACPI_VMGENID=y
  CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +0 -1
index 0221236825..f571da36eb 100644
--- a/default-configs/x86_64-softmmu.mak
Quoted 9 lines

default-configs/x86_64-softmmu.mak CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) · line 64

Quoted 6 lines

+++ b/default-configs/x86_64-softmmu.mak

default-configs/x86_64-softmmu.mak +1 -0
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
  CONFIG_PXB=y
  CONFIG_ACPI_VMGENID=y
  CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y

Any reason you did not add this to other architectures?

We simply didn't test other architectures, but it could work.

Quoted 4 lines

I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at least builds (did not try to actually get it to work, although I don't see any immediate blocker for that).

We will add it to s390, sure.

Thanks,
Marcel

Quoted 1 line

(...)

[...]

#17 lore raw

PATCH v6 5/5 MAINTAINERS: add entry for hw/rdma

In reply to Marcel Apfelbaum
---
 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)
MAINTAINERS +8 -0
index 73a5555735..256a284753 100644
@@ -1979,6 +1979,14 @@ F: block/replication.c
 F: tests/test-replication.c
 F: docs/block-replication.txt
 
+PVRDMA
+M: Yuval Shaia <[email protected]>
+M: Marcel Apfelbaum <[email protected]>
+S: Maintained
+F: hw/rdma/*
+F: hw/rdma/vmw/*
+F: docs/pvrdma.txt
+
 Build and test automation
 -------------------------
 Build and test automation
Signature · 1 line
2.13.5