Skip to contentSkip to conversation

hw/pvrdma: PVRDMA device implementation

PATCH Superseded Marcel Apfelbaum 4 participants 30 messages Dec 17, 2017 – Dec 22, 2017

Subsystems (2)

v2 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v2 cover hw/pvrdma: PVRDMA device implementation

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it and most of the code can be reused if we decide to continue to a Virtio based RDMA device.

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
BAR 2 - UAR, used to pass HW commands from driver.

- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

 MAINTAINERS                         |   7 +
 Makefile.objs                       |   1 +
 backends/hostmem-file.c             |  25 +-
 backends/hostmem-ram.c              |   4 +-
 backends/hostmem.c                  |  21 +
 configure                           |   9 +-
 default-configs/arm-softmmu.mak     |   2 +
 default-configs/i386-softmmu.mak    |   1 +
 default-configs/x86_64-softmmu.mak  |   1 +
 docs/pvrdma.txt                     | 145 ++++++
 exec.c                              |  26 +-
 hw/net/Makefile.objs                |   7 +
 hw/net/pvrdma/pvrdma.h              | 179 +++++++
 hw/net/pvrdma/pvrdma_backend.c      | 986 ++++++++++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_backend.h      |  74 +++
 hw/net/pvrdma/pvrdma_backend_defs.h |  68 +++
 hw/net/pvrdma/pvrdma_cmd.c          | 338 ++++++++++++
 hw/net/pvrdma/pvrdma_defs.h         | 121 +++++
 hw/net/pvrdma/pvrdma_dev_api.h      | 580 +++++++++++++++++++++
 hw/net/pvrdma/pvrdma_dev_ring.c     | 138 +++++
 hw/net/pvrdma/pvrdma_dev_ring.h     |  42 ++
 hw/net/pvrdma/pvrdma_ib_verbs.h     | 399 +++++++++++++++
 hw/net/pvrdma/pvrdma_main.c         | 664 ++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_qp_ops.c       | 187 +++++++
 hw/net/pvrdma/pvrdma_qp_ops.h       |  26 +
 hw/net/pvrdma/pvrdma_ring.h         | 134 +++++
 hw/net/pvrdma/pvrdma_rm.c           | 791 +++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_rm.h           |  54 ++
 hw/net/pvrdma/pvrdma_rm_defs.h      | 111 ++++
 hw/net/pvrdma/pvrdma_types.h        |  37 ++
 hw/net/pvrdma/pvrdma_utils.c        | 133 +++++
 hw/net/pvrdma/pvrdma_utils.h        |  41 ++
 hw/net/pvrdma/trace-events          |   9 +
 hw/pci/shpc.c                       |  11 +-
 include/exec/memory.h               |  23 +
 include/exec/ram_addr.h             |   3 +-
 include/hw/pci/pci_ids.h            |   3 +
 include/qemu/cutils.h               |  10 +
 include/qemu/osdep.h                |   2 +-
 include/sysemu/hostmem.h            |   2 +-
 include/sysemu/kvm.h                |   2 +-
 memory.c                            |  16 +-
 util/oslib-posix.c                  |   4 +-
 util/oslib-win32.c                  |   2 +-
 44 files changed, 5378 insertions(+), 61 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Signature · 1 line
2.13.5
#1 lore raw

PATCH v2 1/5 pci/shpc: Move function to generic header file

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
 hw/pci/shpc.c         | 11 +----------
 include/qemu/cutils.h | 10 ++++++++++
 2 files changed, 11 insertions(+), 10 deletions(-)
hw/pci/shpc.c +1 -10
index 69fc14b218..3d22424fd2 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu-common.h"
+#include "qemu/cutils.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
 #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
 #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
 static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
 {
     uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
include/qemu/cutils.h +10 -0
index f0878eaafa..4895334645 100644
@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);
 int uleb128_encode_small(uint8_t *out, uint32_t n);
 int uleb128_decode_small(const uint8_t *in, uint32_t *n);
 
+static inline int roundup_pow_of_two(int x)
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);
+    return x + 1;
+}
+
 #endif
Signature · 1 line
2.13.5
Reviewed-by on v2 1/5
#2 lore raw

Re: [Qemu-devel] [PATCH V2 1/5] pci/shpc: Move function to generic header file

Hi Marcel, Yuval,

include/qemu/cutils.h

Quoted 45 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
 hw/pci/shpc.c         | 11 +----------
 include/qemu/cutils.h | 10 ++++++++++
 2 files changed, 11 insertions(+), 10 deletions(-)
hw/pci/shpc.c +1 -10
index 69fc14b218..3d22424fd2 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu-common.h"
+#include "qemu/cutils.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
 #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
 #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
 static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
 {
     uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
include/qemu/cutils.h +0 -0
index f0878eaafa..4895334645 100644

I'd rather move this function below pow2ceil() in "qemu/host-utils.h" and rename it pow2roundup().

include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 173

Quoted 11 lines
include/qemu/cutils.h +7 -0
@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);
 int uleb128_encode_small(uint8_t *out, uint32_t n);
 int uleb128_decode_small(const uint8_t *in, uint32_t *n);
 
+static inline int roundup_pow_of_two(w x)
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);

So this would be pow2roundup32(uint32_t value)...

Naming it pow2roundup() without specifying the integer size, I'd directly use a uint64_t argument, and:

       x |= (x >> 32);

include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 177

Quoted 4 lines
include/qemu/cutils.h +3 -0
+    return x + 1;
+}
+
 #endif

Naming it pow2roundup() in "qemu/host-utils.h" (regardless the arg size):

Regards,

Phil.

#3 lore raw

Re: [Qemu-devel] [PATCH V2 1/5] pci/shpc: Move function to generic header file

Quoted 66 lines · Philippe Mathieu-Daudé

Hi Marcel, Yuval,

include/qemu/cutils.h

Quoted 45 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

This function should be declared in generic header file so we can utilize it.

---
 hw/pci/shpc.c         | 11 +----------
 include/qemu/cutils.h | 10 ++++++++++
 2 files changed, 11 insertions(+), 10 deletions(-)
hw/pci/shpc.c +1 -10
index 69fc14b218..3d22424fd2 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu-common.h"
+#include "qemu/cutils.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
 #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@
 #define SHPC_PCI_TO_IDX(pci_slot) ((pci_slot) - 1)
 #define SHPC_IDX_TO_PHYSICAL(slot) ((slot) + 1)
 
-static int roundup_pow_of_two(int x)
-{
-    x |= (x >> 1);
-    x |= (x >> 2);
-    x |= (x >> 4);
-    x |= (x >> 8);
-    x |= (x >> 16);
-    return x + 1;
-}
-
 static uint16_t shpc_get_status(SHPCDevice *shpc, int slot, uint16_t msk)
 {
     uint8_t *status = shpc->config + SHPC_SLOT_STATUS(slot);
include/qemu/cutils.h +0 -0
index f0878eaafa..4895334645 100644

I'd rather move this function below pow2ceil() in "qemu/host-utils.h" and rename it pow2roundup().

include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 173

Quoted 11 lines
include/qemu/cutils.h +7 -0
@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);
 int uleb128_encode_small(uint8_t *out, uint32_t n);
 int uleb128_decode_small(const uint8_t *in, uint32_t *n);
 
+static inline int roundup_pow_of_two(w x)
+{
+    x |= (x >> 1);
+    x |= (x >> 2);
+    x |= (x >> 4);
+    x |= (x >> 8);
+    x |= (x >> 16);

So this would be pow2roundup32(uint32_t value)...

Quoted 4 lines

Naming it pow2roundup() without specifying the integer size, I'd directly use a uint64_t argument, and:

       x |= (x >> 32);

Make sense, will do.

Quoted 8 lines

include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 177

Quoted 5 lines
include/qemu/cutils.h +3 -0
+    return x + 1;
+}
+
 #endif

Naming it pow2roundup() in "qemu/host-utils.h" (regardless the arg size):

Thanks.

Quoted 4 lines

Regards,

Phil.

#4 lore raw

PATCH v2 2/5 mem: add share parameter to memory-backend-ram

In reply to Marcel Apfelbaum

Currently only file backed memory backend can be created with a "share" flag in order to allow sharing guest RAM with other processes in the host.

Add the "share" flag also to RAM Memory Backend
in order to allow remapping parts of the guest RAM to different host virtual addresses. This is needed by the RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.

Moved the "share" flag to the Host Memory base class, modified phys_mem_alloc to include the new parameter and a new interface memory_region_init_ram_shared_nomigrate.

There are no functional changes if the new flag is not used.

---
 backends/hostmem-file.c  | 25 +------------------------
 backends/hostmem-ram.c   |  4 ++--
 backends/hostmem.c       | 21 +++++++++++++++++++++
 exec.c                   | 26 +++++++++++++++-----------
 include/exec/memory.h    | 23 +++++++++++++++++++++++
 include/exec/ram_addr.h  |  3 ++-
 include/qemu/osdep.h     |  2 +-
 include/sysemu/hostmem.h |  2 +-
 include/sysemu/kvm.h     |  2 +-
 memory.c                 | 16 +++++++++++++---
 util/oslib-posix.c       |  4 ++--
 util/oslib-win32.c       |  2 +-
 12 files changed, 83 insertions(+), 47 deletions(-)
backends/hostmem-file.c +1 -24
index e44c319915..bc95022a68 100644
@@ -31,7 +31,6 @@ typedef struct HostMemoryBackendFile HostMemoryBackendFile;
 struct HostMemoryBackendFile {
     HostMemoryBackend parent_obj;
 
-    bool share;
     bool discard_data;
     char *mem_path;
 };
@@ -58,7 +57,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
         path = object_get_canonical_path(OBJECT(backend));
         memory_region_init_ram_from_file(&backend->mr, OBJECT(backend),
                                  path,
-                                 backend->size, fb->share,
+                                 backend->size, backend->share,
                                  fb->mem_path, errp);
         g_free(path);
     }
@@ -85,25 +84,6 @@ static void set_mem_path(Object *o, const char *str, Error **errp)
     fb->mem_path = g_strdup(str);
 }
 
-static bool file_memory_backend_get_share(Object *o, Error **errp)
-{
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    return fb->share;
-}
-
-static void file_memory_backend_set_share(Object *o, bool value, Error **errp)
-{
-    HostMemoryBackend *backend = MEMORY_BACKEND(o);
-    HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(o);
-
-    if (host_memory_backend_mr_inited(backend)) {
-        error_setg(errp, "cannot change property value");
-        return;
-    }
-    fb->share = value;
-}
-
 static bool file_memory_backend_get_discard_data(Object *o, Error **errp)
 {
     return MEMORY_BACKEND_FILE(o)->discard_data;
@@ -136,9 +116,6 @@ file_backend_class_init(ObjectClass *oc, void *data)
     bc->alloc = file_backend_memory_alloc;
     oc->unparent = file_backend_unparent;
 
-    object_class_property_add_bool(oc, "share",
-        file_memory_backend_get_share, file_memory_backend_set_share,
-        &error_abort);
     object_class_property_add_bool(oc, "discard-data",
         file_memory_backend_get_discard_data, file_memory_backend_set_discard_data,
         &error_abort);
backends/hostmem-ram.c +2 -2
index 38977be73e..7ddd08d370 100644
@@ -28,8 +28,8 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     }
 
     path = object_get_canonical_path_component(OBJECT(backend));
-    memory_region_init_ram_nomigrate(&backend->mr, OBJECT(backend), path,
-                           backend->size, errp);
+    memory_region_init_ram_shared_nomigrate(&backend->mr, OBJECT(backend), path,
+                           backend->size, backend->share, errp);
     g_free(path);
 }
 
backends/hostmem.c +21 -0
index ee2c2d5bfd..1daf13bd2e 100644
@@ -369,6 +369,24 @@ static void set_id(Object *o, const char *str, Error **errp)
     backend->id = g_strdup(str);
 }
 
+static bool host_memory_backend_get_share(Object *o, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    return backend->share;
+}
+
+static void host_memory_backend_set_share(Object *o, bool value, Error **errp)
+{
+    HostMemoryBackend *backend = MEMORY_BACKEND(o);
+
+    if (host_memory_backend_mr_inited(backend)) {
+        error_setg(errp, "cannot change property value");
+        return;
+    }
+    backend->share = value;
+}
+
 static void
 host_memory_backend_class_init(ObjectClass *oc, void *data)
 {
@@ -399,6 +417,9 @@ host_memory_backend_class_init(ObjectClass *oc, void *data)
         host_memory_backend_get_policy,
         host_memory_backend_set_policy, &error_abort);
     object_class_property_add_str(oc, "id", get_id, set_id, &error_abort);
+    object_class_property_add_bool(oc, "share",
+        host_memory_backend_get_share, host_memory_backend_set_share,
+        &error_abort);
 }
 
 static void host_memory_backend_finalize(Object *o)
exec.c +15 -11
index 2202f2d731..1bc77dde0b 100644
@@ -1273,7 +1273,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end,
                              uint16_t section);
 static subpage_t *subpage_init(FlatView *fv, hwaddr base);
 
-static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
+static void *(*phys_mem_alloc)(size_t size, uint64_t *align, bool shared) =
                                qemu_anon_ram_alloc;
 
 /*
@@ -1281,7 +1281,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =
  * Accelerators with unusual needs may need this.  Hopefully, we can
  * get rid of it eventually.
  */
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align))
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared))
 {
     phys_mem_alloc = alloc;
 }
@@ -1884,7 +1884,7 @@ static void dirty_memory_extend(ram_addr_t old_ram_size,
     }
 }
 
-static void ram_block_add(RAMBlock *new_block, Error **errp)
+static void ram_block_add(RAMBlock *new_block, Error **errp, bool shared)
 {
     RAMBlock *block;
     RAMBlock *last_block = NULL;
@@ -1907,7 +1907,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp)
             }
         } else {
             new_block->host = phys_mem_alloc(new_block->max_length,
-                                             &new_block->mr->align);
+                                             &new_block->mr->align, shared);
             if (!new_block->host) {
                 error_setg_errno(errp, errno,
                                  "cannot set up guest memory '%s'",
@@ -2012,7 +2012,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
         return NULL;
     }
 
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2054,7 +2054,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
                                   void (*resized)(const char*,
                                                   uint64_t length,
                                                   void *host),
-                                  void *host, bool resizeable,
+                                  void *host, bool resizeable, bool share,
                                   MemoryRegion *mr, Error **errp)
 {
     RAMBlock *new_block;
@@ -2077,7 +2077,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
     if (resizeable) {
         new_block->flags |= RAM_RESIZEABLE;
     }
-    ram_block_add(new_block, &local_err);
+    ram_block_add(new_block, &local_err, share);
     if (local_err) {
         g_free(new_block);
         error_propagate(errp, local_err);
@@ -2089,12 +2089,15 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                    MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, host, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, host, false,
+                                   false, mr, errp);
 }
 
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp)
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share,
+                         MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, size, NULL, NULL, false, mr, errp);
+    return qemu_ram_alloc_internal(size, size, NULL, NULL, false,
+                                   share, mr, errp);
 }
 
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
@@ -2103,7 +2106,8 @@ RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
                                                      void *host),
                                      MemoryRegion *mr, Error **errp)
 {
-    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, mr, errp);
+    return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true,
+                                   false, mr, errp);
 }
 
 static void reclaim_ramblock(RAMBlock *block)
include/exec/memory.h +23 -0
index 5ed4042f87..791a26c45b 100644
@@ -428,6 +428,29 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       Error **errp);
 
 /**
+ * memory_region_init_ram_shared_nomigrate:  Initialize RAM memory region.
+ *                                           Accesses into the region will
+ *                                           modify memory directly.
+ *
+ * @mr: the #MemoryRegion to be initialized.
+ * @owner: the object that tracks the region's reference count
+ * @name: Region name, becomes part of RAMBlock name used in migration stream
+ *        must be unique within any device
+ * @size: size of the region.
+ * @share: allow remapping RAM to different addresses
+ * @errp: pointer to Error*, to store an error if it happens.
+ *
+ * Note that this function is similar to memory_region_init_ram_nomigrate.
+ * The only difference is part of the RAM region can be remapped.
+ */
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             struct Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp);
+
+/**
  * memory_region_init_resizeable_ram:  Initialize memory region with resizeable
  *                                     RAM.  Accesses into the region will
  *                                     modify memory directly.  Only an initial
include/exec/ram_addr.h +2 -1
index 6cbc02aa0f..7d980572c0 100644
@@ -80,7 +80,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
                                  Error **errp);
 RAMBlock *qemu_ram_alloc_from_ptr(ram_addr_t size, void *host,
                                   MemoryRegion *mr, Error **errp);
-RAMBlock *qemu_ram_alloc(ram_addr_t size, MemoryRegion *mr, Error **errp);
+RAMBlock *qemu_ram_alloc(ram_addr_t size, bool share, MemoryRegion *mr,
+                         Error **errp);
 RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t max_size,
                                     void (*resized)(const char*,
                                                     uint64_t length,
include/qemu/osdep.h +1 -1
index 281782d526..bd0141f709 100644
@@ -228,7 +228,7 @@ extern int daemon(int, int);
 int qemu_daemon(int nochdir, int noclose);
 void *qemu_try_memalign(size_t alignment, size_t size);
 void *qemu_memalign(size_t alignment, size_t size);
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align);
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared);
 void qemu_vfree(void *ptr);
 void qemu_anon_ram_free(void *ptr, size_t size);
 
include/sysemu/hostmem.h +1 -1
index ed6a437f4d..4d8f859f03 100644
@@ -55,7 +55,7 @@ struct HostMemoryBackend {
     char *id;
     uint64_t size;
     bool merge, dump;
-    bool prealloc, force_prealloc, is_mapped;
+    bool prealloc, force_prealloc, is_mapped, share;
     DECLARE_BITMAP(host_nodes, MAX_NODES + 1);
     HostMemPolicy policy;
 
include/sysemu/kvm.h +1 -1
index bbf12a1723..85002ac49a 100644
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);
 
 /* interface with exec.c */
 
-void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align));
+void phys_mem_set_alloc(void *(*alloc)(size_t, uint64_t *align, bool shared));
 
 /* internal API */
 
memory.c +13 -3
index e26e5a3b1d..be5b58e6ae 100644
@@ -1538,11 +1538,21 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,
                                       uint64_t size,
                                       Error **errp)
 {
+    memory_region_init_ram_shared_nomigrate(mr, owner, name, size, false, errp);
+}
+
+void memory_region_init_ram_shared_nomigrate(MemoryRegion *mr,
+                                             Object *owner,
+                                             const char *name,
+                                             uint64_t size,
+                                             bool share,
+                                             Error **errp)
+{
     memory_region_init(mr, owner, name, size);
     mr->ram = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, share, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1651,7 +1661,7 @@ void memory_region_init_rom_nomigrate(MemoryRegion *mr,
     mr->readonly = true;
     mr->terminates = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false, mr, errp);
     mr->dirty_log_mask = tcg_enabled() ? (1 << DIRTY_MEMORY_CODE) : 0;
 }
 
@@ -1670,7 +1680,7 @@ void memory_region_init_rom_device_nomigrate(MemoryRegion *mr,
     mr->terminates = true;
     mr->rom_device = true;
     mr->destructor = memory_region_destructor_ram;
-    mr->ram_block = qemu_ram_alloc(size, mr, errp);
+    mr->ram_block = qemu_ram_alloc(size, false,  mr, errp);
 }
 
 void memory_region_init_iommu(void *_iommu_mr,
util/oslib-posix.c +2 -2
index 77369c92ce..0cf3548778 100644
@@ -127,10 +127,10 @@ void *qemu_memalign(size_t alignment, size_t size)
 }
 
 /* alloc shared memory pages */
-void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *alignment, bool shared)
 {
     size_t align = QEMU_VMALLOC_ALIGN;
-    void *ptr = qemu_ram_mmap(-1, size, align, false);
+    void *ptr = qemu_ram_mmap(-1, size, align, shared);
 
     if (ptr == MAP_FAILED) {
         return NULL;
util/oslib-win32.c +1 -1
index 69a6286d50..bb5ad28bd3 100644
@@ -67,7 +67,7 @@ void *qemu_memalign(size_t alignment, size_t size)
     return qemu_oom_check(qemu_try_memalign(alignment, size));
 }
 
-void *qemu_anon_ram_alloc(size_t size, uint64_t *align)
+void *qemu_anon_ram_alloc(size_t size, uint64_t *align, bool shared)
 {
     void *ptr;
 
Signature · 1 line
2.13.5
#5 lore raw

PATCH v2 3/5 docs: add pvrdma device documentation

In reply to Marcel Apfelbaum
---
 docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 145 insertions(+)
 create mode 100644 docs/pvrdma.txt
docs/pvrdma.txt new file +145 -0
new file mode 100644
index 0000000000..74c5cf2495
@@ -0,0 +1,145 @@
+Paravirtualized RDMA Device (PVRDMA)
+====================================
+
+
+1. Description
+===============
+PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
+It works with its Linux Kernel driver AS IS, no need for any special guest
+modifications.
+
+While it complies with the VMware device, it can also communicate with bare
+metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
+can work with Soft-RoCE (rxe).
+
+It does not require the whole guest RAM to be pinned allowing memory
+over-commit and, even if not implemented yet, migration support will be
+possible with some HW assistance.
+
+A project presentation accompany this document:
+- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf
+
+
+
+2. Setup
+========
+
+
+2.1 Guest setup
+===============
+Fedora 27+ kernels work out of the box, older distributions
+require updating the kernel to 4.14 to include the pvrdma driver.
+
+However the libpvrdma library needed by User Level Software is still
+not available as part of the distributions, so the rdma-core library
+needs to be compiled and optionally installed.
+
+Please follow the instructions at:
+  https://github.com/linux-rdma/rdma-core.git
+
+
+2.2 Host Setup
+==============
+The pvrdma backend is an ibdevice interface that can be exposed
+either by a Soft-RoCE(rxe) device on machines with no RDMA device,
+or an HCA SRIOV function(VF/PF).
+Note that ibdevice interfaces can't be shared between pvrdma devices,
+each one requiring a separate instance (rxe or SRIOV VF).
+
+
+2.2.1 Soft-RoCE backend(rxe)
+===========================
+A stable version of rxe is required, Fedora 27+ or a Linux
+Kernel 4.14+ is preferred.
+
+The rdma_rxe module is part of the Linux Kernel but not loaded by default.
+Install the User Level library (librxe) following the instructions from:
+https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
+
+Associate an ETH interface with rxe by running:
+   rxe_cfg add eth0
+An rxe0 ibdevice interface will be created and can be used as pvrdma backend.
+
+
+2.2.2 RDMA device Virtual Function backend
+==========================================
+Nothing special is required, the pvrdma device can work not only with
+Ethernet Links, but also Infinibands Links.
+All is needed is an ibdevice with an active port, for Mellanox cards
+will be something like mlx5_6 which can be the backend.
+
+
+2.2.3 QEMU setup
+================
+Configure QEMU with --enable-rdma flag, installing
+the required RDMA libraries.
+
+
+3. Usage
+========
+Currently the device is working only with memory backed RAM
+and it must be mark as "shared":
+   -m 1G \
+   -object memory-backend-ram,id=mb1,size=1G,share \
+   -numa node,memdev=mb1 \
+
+The pvrdma device is composed of two functions:
+ - Function 0 is a vmxnet Ethernet Device which is redundant in Guest
+   but is required to pass the ibdevice GID using its MAC.
+   Examples:
+     For an rxe backend using eth0 interface it will use its mac:
+       -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
+     For an SRIOV VF, we take the Ethernet Interface exposed by it:
+       -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
+ - Function 1 is the actual device:
+       -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>
+   where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
+ Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.
+ The rules of conversion are part of the RoCE spec, but since manual conversion
+ is not required, spotting problems is not hard:
+    Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
+             MAC: 7c:fe:90:cb:74:3a
+    Note the difference between the first byte of the MAC and the GID.
+
+
+4. Implementation details
+=========================
+The device acts like a proxy between the Guest Driver and the host
+ibdevice interface.
+On configuration path:
+ - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
+   a resource from the backend interface, maintaining a 1-1 mapping
+   between the guest and host.
+On data path:
+ - Every post_send/receive received from the guest will be converted into
+   a post_send/receive for the backend. The buffers data will not be touched
+   or copied resulting in near bare-metal performance for large enough buffers.
+ - Completions from the backend interface will result in completions for
+   the pvrdma device.
+
+
+
+5. Limitations
+==============
+- The device obviously is limited by the Guest Linux Driver features implementation
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge
+  pages available, QEMU will use them.
+- As previously stated, migration is not supported yet, however with some hardware
+  support can be done.
+
+
+
+6. Performance
+==============
+By design the pvrdma device exits on each post-send/receive, so for small buffers
+the performance is affected; however for medium buffers it will became close to
+bare metal and from 1MB buffers and  up it reaches bare metal performance.
+(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)
+
+All the above assumes no memory registration is done on data path.
Signature · 1 line
2.13.5
#6 lore raw

Re: [Qemu-devel] [PATCH V2 3/5] docs: add pvrdma device documentation

Quoted 121 lines · Marcel Apfelbaum
---
 docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 145 insertions(+)
 create mode 100644 docs/pvrdma.txt
docs/pvrdma.txt new file +108 -0
new file mode 100644
index 0000000000..74c5cf2495
@@ -0,0 +1,145 @@
+Paravirtualized RDMA Device (PVRDMA)
+====================================
+
+
+1. Description
+===============
+PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
+It works with its Linux Kernel driver AS IS, no need for any special guest
+modifications.
+
+While it complies with the VMware device, it can also communicate with bare
+metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
+can work with Soft-RoCE (rxe).
+
+It does not require the whole guest RAM to be pinned allowing memory
+over-commit and, even if not implemented yet, migration support will be
+possible with some HW assistance.
+
+A project presentation accompany this document:
+- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf
+
+
+
+2. Setup
+========
+
+
+2.1 Guest setup
+===============
+Fedora 27+ kernels work out of the box, older distributions
+require updating the kernel to 4.14 to include the pvrdma driver.
+
+However the libpvrdma library needed by User Level Software is still
+not available as part of the distributions, so the rdma-core library
+needs to be compiled and optionally installed.
+
+Please follow the instructions at:
+  https://github.com/linux-rdma/rdma-core.git
+
+
+2.2 Host Setup
+==============
+The pvrdma backend is an ibdevice interface that can be exposed
+either by a Soft-RoCE(rxe) device on machines with no RDMA device,
+or an HCA SRIOV function(VF/PF).
+Note that ibdevice interfaces can't be shared between pvrdma devices,
+each one requiring a separate instance (rxe or SRIOV VF).
+
+
+2.2.1 Soft-RoCE backend(rxe)
+===========================
+A stable version of rxe is required, Fedora 27+ or a Linux
+Kernel 4.14+ is preferred.
+
+The rdma_rxe module is part of the Linux Kernel but not loaded by default.
+Install the User Level library (librxe) following the instructions from:
+https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
+
+Associate an ETH interface with rxe by running:
+   rxe_cfg add eth0
+An rxe0 ibdevice interface will be created and can be used as pvrdma backend.
+
+
+2.2.2 RDMA device Virtual Function backend
+==========================================
+Nothing special is required, the pvrdma device can work not only with
+Ethernet Links, but also Infinibands Links.
+All is needed is an ibdevice with an active port, for Mellanox cards
+will be something like mlx5_6 which can be the backend.
+
+
+2.2.3 QEMU setup
+================
+Configure QEMU with --enable-rdma flag, installing
+the required RDMA libraries.
+
+
+3. Usage
+========
+Currently the device is working only with memory backed RAM
+and it must be mark as "shared":
+   -m 1G \
+   -object memory-backend-ram,id=mb1,size=1G,share \
+   -numa node,memdev=mb1 \
+
+The pvrdma device is composed of two functions:
+ - Function 0 is a vmxnet Ethernet Device which is redundant in Guest
+   but is required to pass the ibdevice GID using its MAC.
+   Examples:
+     For an rxe backend using eth0 interface it will use its mac:
+       -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>
+     For an SRIOV VF, we take the Ethernet Interface exposed by it:
+       -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>
+ - Function 1 is the actual device:
+       -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>
+   where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)
+ Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.
+ The rules of conversion are part of the RoCE spec, but since manual conversion
+ is not required, spotting problems is not hard:
+    Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a
+             MAC: 7c:fe:90:cb:74:3a
+    Note the difference between the first byte of the MAC and the GID.
+
+
+4. Implementation details
+=========================
+The device acts like a proxy between the Guest Driver and the host
+ibdevice interface.

docs/pvrdma.txt line 118

Quoted 10 lines
docs/pvrdma.txt new file +10 -0
+On configuration path:
+ - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
+   a resource from the backend interface, maintaining a 1-1 mapping
+   between the guest and host.
+On data path:
+ - Every post_send/receive received from the guest will be converted into
+   a post_send/receive for the backend. The buffers data will not be touched
+   or copied resulting in near bare-metal performance for large enough buffers.
+ - Completions from the backend interface will result in completions for
+   the pvrdma device.

Where's the host/guest interface documented?

Quoted 28 lines
docs/pvrdma.txt new file +26 -0
+
+
+5. Limitations
+==============
+- The device obviously is limited by the Guest Linux Driver features implementation
+  of the VMware device API.
+- Memory registration mechanism requires mremap for every page in the buffer in order
+  to map it to a contiguous virtual address range. Since this is not the data path
+  it should not matter much.
+- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
+  so it can't work with huge pages. The limitation will be addressed in the future,
+  however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge
+  pages available, QEMU will use them.
+- As previously stated, migration is not supported yet, however with some hardware
+  support can be done.
+
+
+
+6. Performance
+==============
+By design the pvrdma device exits on each post-send/receive, so for small buffers
+the performance is affected; however for medium buffers it will became close to
+bare metal and from 1MB buffers and  up it reaches bare metal performance.
+(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)
+
+All the above assumes no memory registration is done on data path.
Signature · 1 line
2.13.5
#7 lore raw

Re: [Qemu-devel] [PATCH V2 3/5] docs: add pvrdma device documentation

Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Marcel Apfelbaum
---
  docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 145 insertions(+)

create mode 100644 docs/pvrdma.txt

docs/pvrdma.txt new file +3 -0
new file mode 100644
index 0000000000..74c5cf2495
@@ -0,0 +1,145 @@
+Paravirtualized RDMA Device (PVRDMA)
+====================================
+

[...]

Quoted 7 lines
Quoted 7 lines
docs/pvrdma.txt new file +7 -0
+
+4. Implementation details
+=========================
+The device acts like a proxy between the Guest Driver and the host
+ibdevice interface.
+On configuration path:
+ - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
Quoted 9 lines

docs/pvrdma.txt line 118

Quoted 8 lines
docs/pvrdma.txt new file +8 -0
+   a resource from the backend interface, maintaining a 1-1 mapping
+   between the guest and host.
+On data path:
+ - Every post_send/receive received from the guest will be converted into
+   a post_send/receive for the backend. The buffers data will not be touched
+   or copied resulting in near bare-metal performance for large enough buffers.
+ - Completions from the backend interface will result in completions for
+   the pvrdma device.

Hi Michael,

Quoted 3 lines

Where's the host/guest interface documented?

It is the VMware PVRDMA spec, I am not sure is publicly available, we kind of reverse-engineered it.
We will add some info from the linked presentation on the PCI BARs and how are they used.

Thanks,
Marcel

Quoted 2 lines
Quoted 2 lines
docs/pvrdma.txt new file +2 -0
+
+

[...]

#8 lore raw

PATCH v2 4/5 pvrdma: initial implementation

In reply to Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---
 Makefile.objs                       |   1 +
 configure                           |   9 +-
 default-configs/arm-softmmu.mak     |   2 +
 default-configs/i386-softmmu.mak    |   1 +
 default-configs/x86_64-softmmu.mak  |   1 +
 hw/net/Makefile.objs                |   7 +
 hw/net/pvrdma/pvrdma.h              | 179 +++++++
 hw/net/pvrdma/pvrdma_backend.c      | 986 ++++++++++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_backend.h      |  74 +++
 hw/net/pvrdma/pvrdma_backend_defs.h |  68 +++
 hw/net/pvrdma/pvrdma_cmd.c          | 338 ++++++++++++
 hw/net/pvrdma/pvrdma_defs.h         | 121 +++++
 hw/net/pvrdma/pvrdma_dev_api.h      | 580 +++++++++++++++++++++
 hw/net/pvrdma/pvrdma_dev_ring.c     | 138 +++++
 hw/net/pvrdma/pvrdma_dev_ring.h     |  42 ++
 hw/net/pvrdma/pvrdma_ib_verbs.h     | 399 +++++++++++++++
 hw/net/pvrdma/pvrdma_main.c         | 664 ++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_qp_ops.c       | 187 +++++++
 hw/net/pvrdma/pvrdma_qp_ops.h       |  26 +
 hw/net/pvrdma/pvrdma_ring.h         | 134 +++++
 hw/net/pvrdma/pvrdma_rm.c           | 791 +++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_rm.h           |  54 ++
 hw/net/pvrdma/pvrdma_rm_defs.h      | 111 ++++
 hw/net/pvrdma/pvrdma_types.h        |  37 ++
 hw/net/pvrdma/pvrdma_utils.c        | 133 +++++
 hw/net/pvrdma/pvrdma_utils.h        |  41 ++
 hw/net/pvrdma/trace-events          |   9 +
 include/hw/pci/pci_ids.h            |   3 +
 28 files changed, 5132 insertions(+), 4 deletions(-)
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Makefile.objs +1 -0
index 285c6f3c15..728981be30 100644
@@ -129,6 +129,7 @@ trace-events-subdirs += hw/block/dataplane
 trace-events-subdirs += hw/char
 trace-events-subdirs += hw/intc
 trace-events-subdirs += hw/net
+trace-events-subdirs += hw/net/pvrdma
 trace-events-subdirs += hw/virtio
 trace-events-subdirs += hw/audio
 trace-events-subdirs += hw/misc
configure +5 -4
index 0e856bbc04..83068c5d65 100755
@@ -1523,7 +1523,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   bluez           bluez stack connectivity
   kvm             KVM acceleration support
   hax             HAX acceleration support
-  rdma            RDMA-based migration support
+  rdma            Enable RDMA-based migration support and PVRDMA
   vde             support for vde network
   netmap          support for netmap network
   linux-aio       Linux AIO support
@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then
 #include <rdma/rdma_cma.h>
 int main(void) { return 0; }
 EOF
-  rdma_libs="-lrdmacm -libverbs"
+  rdma_libs="-lrdmacm -libverbs -libumad"
   if compile_prog "" "$rdma_libs" ; then
     rdma="yes"
   else
+    libs_softmmu="$libs_softmmu $rdma_libs"
     if test "$rdma" = "yes" ; then
         error_exit \
-            " OpenFabrics librdmacm/libibverbs not present." \
+            " OpenFabrics librdmacm/libibverbs/libibumad not present." \
             " Your options:" \
-            "  (1) Fast: Install infiniband packages from your distro." \
+	    "  (1) Fast: Install infiniband packages (devel) from your distro." \
             "  (2) Cleanest: Install libraries from www.openfabrics.org" \
             "  (3) Also: Install softiwarp if you don't have RDMA hardware"
     fi
default-configs/arm-softmmu.mak +2 -0
index d37edc4312..51b2052514 100644
@@ -132,3 +132,5 @@ CONFIG_GPIO_KEY=y
 CONFIG_MSF2=y
 
 CONFIG_FW_CFG_DMA=y
+
+CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
index 95ac4b464a..88298e4ef5 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +1 -0
index 0221236825..f571da36eb 100644
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
 CONFIG_PXB=y
 CONFIG_ACPI_VMGENID=y
 CONFIG_FW_CFG_DMA=y
+CONFIG_PVRDMA=y
hw/net/Makefile.objs +7 -0
index 4171af0b5d..6645495574 100644
@@ -46,3 +46,10 @@ common-obj-$(CONFIG_ROCKER) += rocker/rocker.o rocker/rocker_fp.o \
                                rocker/rocker_desc.o rocker/rocker_world.o \
                                rocker/rocker_of_dpa.o
 obj-$(call lnot,$(CONFIG_ROCKER)) += rocker/qmp-norocker.o
+
+ifeq ($(CONFIG_RDMA),y)
+obj-$(CONFIG_PVRDMA) += pvrdma/pvrdma_dev_ring.o pvrdma/pvrdma_rm.o \
+                        pvrdma/pvrdma_utils.o pvrdma/pvrdma_qp_ops.o \
+                        pvrdma/pvrdma_backend.o pvrdma/pvrdma_cmd.o \
+                        pvrdma/pvrdma_main.o
+endif
hw/net/pvrdma/pvrdma.h new file +179 -0

179 lines not on this page · raw message

hw/net/pvrdma/pvrdma_backend.c new file +986 -0

986 lines not on this page · raw message

hw/net/pvrdma/pvrdma_backend.h new file +74 -0

74 lines not on this page · raw message

hw/net/pvrdma/pvrdma_backend_defs.h new file +68 -0

68 lines not on this page · raw message

hw/net/pvrdma/pvrdma_cmd.c new file +338 -0

338 lines not on this page · raw message

hw/net/pvrdma/pvrdma_defs.h new file +121 -0

121 lines not on this page · raw message

hw/net/pvrdma/pvrdma_dev_api.h new file +580 -0

580 lines not on this page · raw message

hw/net/pvrdma/pvrdma_dev_ring.c new file +138 -0

138 lines not on this page · raw message

hw/net/pvrdma/pvrdma_dev_ring.h new file +42 -0

42 lines not on this page · raw message

hw/net/pvrdma/pvrdma_ib_verbs.h new file +399 -0

399 lines not on this page · raw message

hw/net/pvrdma/pvrdma_main.c new file +664 -0

18 lines not on this page · raw message

hw/net/pvrdma/pvrdma_qp_ops.c new file +187 -0

187 lines not on this page · raw message

hw/net/pvrdma/pvrdma_qp_ops.h new file +26 -0
new file mode 100644
index 0000000000..d4b84eb842

Lines of this file are not shown. View the raw message.

hw/net/pvrdma/pvrdma_ring.h new file +134 -0

134 lines not on this page · raw message

hw/net/pvrdma/pvrdma_rm.c new file +791 -0

791 lines not on this page · raw message

hw/net/pvrdma/pvrdma_rm.h new file +54 -0

54 lines not on this page · raw message

hw/net/pvrdma/pvrdma_rm_defs.h new file +111 -0

111 lines not on this page · raw message

hw/net/pvrdma/pvrdma_types.h new file +37 -0
new file mode 100644
index 0000000000..22c78e4733

Lines of this file are not shown. View the raw message.

hw/net/pvrdma/pvrdma_utils.c new file +133 -0

133 lines not on this page · raw message

hw/net/pvrdma/pvrdma_utils.h new file +41 -0

41 lines not on this page · raw message

hw/net/pvrdma/trace-events new file +9 -0
new file mode 100644
index 0000000000..bbc52286bc

Lines of this file are not shown. View the raw message.

include/hw/pci/pci_ids.h +3 -0
index 35df1874a9..1dbf53627c 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.13.5

Part of the diff is not shown here. View the raw message.

#9 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 10 lines · Marcel Apfelbaum
configure +2 -1
@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then
 #include <rdma/rdma_cma.h>
 int main(void) { return 0; }
 EOF
-  rdma_libs="-lrdmacm -libverbs"
+  rdma_libs="-lrdmacm -libverbs -libumad"
   if compile_prog "" "$rdma_libs" ; then
     rdma="yes"
   else
+    libs_softmmu="$libs_softmmu $rdma_libs"

configure if test "$rdma" != "no" ; then · line 2862

Quoted 10 lines
configure +2 -2
     if test "$rdma" = "yes" ; then
         error_exit \
-            " OpenFabrics librdmacm/libibverbs not present." \
+            " OpenFabrics librdmacm/libibverbs/libibumad not present." \
             " Your options:" \
-            "  (1) Fast: Install infiniband packages from your distro." \
+	    "  (1) Fast: Install infiniband packages (devel) from your distro." \
             "  (2) Cleanest: Install libraries from www.openfabrics.org" \
             "  (3) Also: Install softiwarp if you don't have RDMA hardware"
     fi

Some whitespace damage here.

#10 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Marcel Apfelbaum
configure +3 -2
@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then
  #include <rdma/rdma_cma.h>
  int main(void) { return 0; }
  EOF
-  rdma_libs="-lrdmacm -libverbs"
+  rdma_libs="-lrdmacm -libverbs -libumad"
    if compile_prog "" "$rdma_libs" ; then
      rdma="yes"
    else
+    libs_softmmu="$libs_softmmu $rdma_libs"
      if test "$rdma" = "yes" ; then
          error_exit \
-            " OpenFabrics librdmacm/libibverbs not present." \
+            " OpenFabrics librdmacm/libibverbs/libibumad not present." \
Quoted 9 lines

configure if test "$rdma" != "no" ; then · line 2862

Quoted 6 lines
configure +1 -1
              " Your options:" \
-            "  (1) Fast: Install infiniband packages from your distro." \
+	    "  (1) Fast: Install infiniband packages (devel) from your distro." \
              "  (2) Cleanest: Install libraries from www.openfabrics.org" \
              "  (3) Also: Install softiwarp if you don't have RDMA hardware"
      fi

Some whitespace damage here.

Thanks Michael, we will take care of that and go over the code again to spot similar issues.

Marcel

#11 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

In reply to Marcel Apfelbaum

I won't have time to review before the next year.
Results of a quick peek.

Quoted 12 lines · Marcel Apfelbaum
hw/net/pvrdma/pvrdma_backend.c new file +12 -0
+static void *mad_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    QObject *o_ctx_id;
+    unsigned long cqe_ctx_id;
+    BackendCtx *bctx;
+    /*
+    int len;
+    void *mad;
+    */
+

hw/net/pvrdma/pvrdma_backend.c line 86

Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10 -0
+    pr_dbg("Starting\n");
+
+    dev->backend_dev.mad_thread.run = false;
+
+    while (dev->backend_dev.mad_thread.run) {
+        /* Get next buffer to pust MAD into */
+        o_ctx_id = qlist_pop(dev->backend_dev.mad_agent.recv_mads_list);
+        if (!o_ctx_id) {
+            /* pr_dbg("Error: No more free MADs buffers\n"); */
+            sleep(5);

Looks suspicious. What is above doing?

Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c new file +8 -0
+            continue;
+        }
+        cqe_ctx_id = qnum_get_uint(qobject_to_qnum(o_ctx_id));
+        bctx = rm_get_cqe_ctx(dev, cqe_ctx_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id);
+            continue;
+        }

hw/net/pvrdma/pvrdma_backend.c line 104

Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10 -0
+
+        pr_dbg("Calling umad_recv\n");
+        /*
+        mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr,
+                                 bctx->req.sge[0].length);
+
+        len = bctx->req.sge[0].length;
+
+        do {
+            rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);

That's a huge timeout.

Quoted 160 lines
hw/net/pvrdma/pvrdma_backend.c new file +160 -0
+        } while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run);
+        pr_dbg("umad_recv, rc=%d\n", rc);
+
+        pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length);
+        */
+        rc = -1;
+
+        /* rc is used as vendor_err */
+        comp_handler(rc > 0 ? IB_WC_SUCCESS : IB_WC_GENERAL_ERR, rc,
+                     bctx->up_ctx);
+
+        rm_dealloc_cqe_ctx(dev, cqe_ctx_id);
+        free(bctx);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining MADs in list */
+
+    qlist_destroy_obj(QOBJECT(dev->backend_dev.mad_agent.recv_mads_list));
+
+    return NULL;
+}
+
+static void *comp_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    struct ibv_cq *ev_cq;
+    void *ev_ctx;
+
+    pr_dbg("Starting\n");
+
+    while (dev->backend_dev.comp_thread.run) {
+        pr_dbg("Waiting for completion on channel %p\n",
+               dev->backend_dev.channel);
+        rc = ibv_get_cq_event(dev->backend_dev.channel, &ev_cq, &ev_ctx);
+        pr_dbg("ibv_get_cq_event=%d\n", rc);
+        if (unlikely(rc)) {
+            pr_dbg("---> ibv_get_cq_event (%d)\n", rc);
+            continue;
+        }
+
+        if (unlikely(ibv_req_notify_cq(ev_cq, 0))) {
+            pr_dbg("---> ibv_req_notify_cq\n");
+        }
+
+        poll_cq(dev, ev_cq, false);
+
+        ibv_ack_cq_events(ev_cq, 1);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining buffs that were posted */
+
+    return NULL;
+}
+
+void backend_register_comp_handler(void (*handler)(int status,
+                                   unsigned int vendor_err, void *ctx))
+{
+    comp_handler = handler;
+}
+
+int backend_query_port(BackendDevice *dev, struct pvrdma_port_attr *attrs)
+{
+    int rc;
+    struct ibv_port_attr port_attr;
+
+    rc = ibv_query_port(dev->context, dev->port_num, &port_attr);
+    if (rc) {
+        pr_dbg("Error %d from ibv_query_port\n", rc);
+        return -EIO;
+    }
+
+    attrs->state = port_attr.state;
+    attrs->max_mtu = port_attr.max_mtu;
+    attrs->active_mtu = port_attr.active_mtu;
+    attrs->gid_tbl_len = port_attr.gid_tbl_len;
+    attrs->pkey_tbl_len = port_attr.pkey_tbl_len;
+    attrs->phys_state = port_attr.phys_state;
+
+    return 0;
+}
+
+void backend_poll_cq(PVRDMADev *dev, BackendCQ *cq)
+{
+    poll_cq(dev, cq->ibcq, true);
+}
+
+static GHashTable *ah_hash;
+
+static struct ibv_ah *create_ah(BackendDevice *dev, struct ibv_pd *pd,
+                                union ibv_gid *dgid, uint8_t sgid_idx)
+{
+    GBytes *ah_key = g_bytes_new(dgid, sizeof(*dgid));
+    struct ibv_ah *ah = g_hash_table_lookup(ah_hash, ah_key);
+
+    if (ah) {
+        trace_create_ah_cache_hit(be64_to_cpu(dgid->global.subnet_prefix),
+                                  be64_to_cpu(dgid->global.interface_id));
+    } else {
+        struct ibv_ah_attr ah_attr = {
+            .is_global     = 1,
+            .port_num      = dev->port_num,
+            .grh.hop_limit = 1,
+        };
+
+        ah_attr.grh.dgid = *dgid;
+        ah_attr.grh.sgid_index = sgid_idx;
+
+        ah = ibv_create_ah(pd, &ah_attr);
+        if (ah) {
+            g_hash_table_insert(ah_hash, ah_key, ah);
+        } else {
+            pr_dbg("ibv_create_ah failed for gid <%lx %lx>\n",
+                    be64_to_cpu(dgid->global.subnet_prefix),
+                    be64_to_cpu(dgid->global.interface_id));
+        }
+
+        trace_create_ah_cache_miss(be64_to_cpu(dgid->global.subnet_prefix),
+                                   be64_to_cpu(dgid->global.interface_id));
+    }
+
+    return ah;
+}
+
+static void destroy_ah(gpointer data)
+{
+    struct ibv_ah *ah = data;
+    ibv_destroy_ah(ah);
+}
+
+static void ah_cache_init(void)
+{
+    ah_hash = g_hash_table_new_full(g_bytes_hash, g_bytes_equal,
+                                    NULL, destroy_ah);
+}
+
+static int send_mad(PVRDMADev *dev, struct pvrdma_sge *sge, u32 num_sge)
+{
+    int ret = -1;
+
+    /*
+     * TODO: Currently QP1 is not supported
+     *
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+    char mad_msg[1024];
+    void *hdr, *msg;
+    struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg;
+
+    umad->length = sge[0].length + sge[1].length;
+
+    if (num_sge != 2)
+        return -EINVAL;
+
+    pr_dbg("msg_len=%d\n", umad->length);
+
+    hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length);
+    msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length);
+

hw/net/pvrdma/pvrdma_backend.c line 274

Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10 -0
+    memcpy(&mad_msg[64], hdr, sge[0].length);
+    memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length);
+
+    pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length);
+    pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length);
+
+    ret = umad_send(dev->backend_dev.mad_agent.port_id,
+                    dev->backend_dev.mad_agent.agent_id,
+                    mad_msg, umad->length, 10, 10);
+    */

Then what is above code doing here?

Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?

How does guest know it's missing?

...

Quoted 37 lines
hw/net/pvrdma/pvrdma_utils.h new file +31 -0
new file mode 100644
index 0000000000..a09e39946c
@@ -0,0 +1,41 @@
+/*
+ * QEMU VMWARE paravirtual RDMA interface definitions
+ *
+ * Developed by Oracle & Redhat
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_UTILS_H
+#define PVRDMA_UTILS_H
+
+#include <include/hw/pci/pci.h>
+
+#define pr_info(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma",  __func__, __LINE__,\
+           ## __VA_ARGS__)
+
+#define pr_err(fmt, ...) \
+    fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \
+        __LINE__, ## __VA_ARGS__)
+
+#ifdef PVRDMA_DEBUG
+#define pr_dbg(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\
+           ## __VA_ARGS__)

hw/net/pvrdma/pvrdma_utils.h line 41

Quoted 10 lines
hw/net/pvrdma/pvrdma_utils.h new file +10 -0
+#else
+#define pr_dbg(fmt, ...)
+#endif
+
+void pvrdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len);
+void *pvrdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen);
+void *map_to_pdir(PCIDevice *pdev, uint64_t pdir_dma, uint32_t nchunks,
+                  size_t length);
+
+#endif

Can you make sure all prefixes are pvrdma_?

Quoted 29 lines
hw/net/pvrdma/trace-events new file +9 -0
new file mode 100644
index 0000000000..bbc52286bc
@@ -0,0 +1,9 @@
+# See docs/tracing.txt for syntax documentation.
+
+# hw/net/pvrdma/pvrdma_main.c
+pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+
+#hw/net/pvrdma/pvrdma_backend.c
+create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
+create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
include/hw/pci/pci_ids.h +3 -0
index 35df1874a9..1dbf53627c 100644
@@ -266,4 +266,7 @@
 #define PCI_VENDOR_ID_TEWS               0x1498
 #define PCI_DEVICE_ID_TEWS_TPCI200       0x30C8
 
+#define PCI_VENDOR_ID_VMWARE             0x15ad
+#define PCI_DEVICE_ID_VMWARE_PVRDMA      0x0820
+
 #endif
Signature · 1 line
2.13.5
#12 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 2 lines · Michael S. Tsirkin

I won't have time to review before the next year.
Results of a quick peek.

Thanks for the parts you found the time to review.

Quoted 18 lines · Marcel Apfelbaum
Quoted 16 lines
hw/net/pvrdma/pvrdma_backend.c new file +16 -0
+static void *mad_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    QObject *o_ctx_id;
+    unsigned long cqe_ctx_id;
+    BackendCtx *bctx;
+    /*
+    int len;
+    void *mad;
+    */
+
+    pr_dbg("Starting\n");
+
+    dev->backend_dev.mad_thread.run = false;
+
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 86

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+    while (dev->backend_dev.mad_thread.run) {
+        /* Get next buffer to pust MAD into */
+        o_ctx_id = qlist_pop(dev->backend_dev.mad_agent.recv_mads_list);
+        if (!o_ctx_id) {
+            /* pr_dbg("Error: No more free MADs buffers\n"); */
+            sleep(5);

Looks suspicious. What is above doing?

This function is responsible to process incoming MAD messages.

Usual (good) flow is that guest driver prepares some buffers to be used for that purpose and gives it to device with the usual post_recv mechanism. Upon receiving a MAD message from the device the driver should pass a new buffer to device.

But what if the device didn't do it.

This section handle such case, as we have nothing to do - let's sleep and hope for the best.

Quoted 13 lines
Quoted 13 lines
hw/net/pvrdma/pvrdma_backend.c new file +12 -0
+            continue;
+        }
+        cqe_ctx_id = qnum_get_uint(qobject_to_qnum(o_ctx_id));
+        bctx = rm_get_cqe_ctx(dev, cqe_ctx_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id);
+            continue;
+        }
+
+        pr_dbg("Calling umad_recv\n");
+        /*
+        mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr,
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 104

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+                                 bctx->req.sge[0].length);
+
+        len = bctx->req.sge[0].length;
+
+        do {
+            rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);

That's a huge timeout.

This is the maximum timeout.
On low MAD traffic we don't want to take much of the CPU for that purpose so 5 seconds looks good to me.

Anyway, just because it is not obvious i will make it configurable.

Quoted 165 lines
Quoted 165 lines
hw/net/pvrdma/pvrdma_backend.c new file +164 -0
+        } while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run);
+        pr_dbg("umad_recv, rc=%d\n", rc);
+
+        pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length);
+        */
+        rc = -1;
+
+        /* rc is used as vendor_err */
+        comp_handler(rc > 0 ? IB_WC_SUCCESS : IB_WC_GENERAL_ERR, rc,
+                     bctx->up_ctx);
+
+        rm_dealloc_cqe_ctx(dev, cqe_ctx_id);
+        free(bctx);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining MADs in list */
+
+    qlist_destroy_obj(QOBJECT(dev->backend_dev.mad_agent.recv_mads_list));
+
+    return NULL;
+}
+
+static void *comp_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    struct ibv_cq *ev_cq;
+    void *ev_ctx;
+
+    pr_dbg("Starting\n");
+
+    while (dev->backend_dev.comp_thread.run) {
+        pr_dbg("Waiting for completion on channel %p\n",
+               dev->backend_dev.channel);
+        rc = ibv_get_cq_event(dev->backend_dev.channel, &ev_cq, &ev_ctx);
+        pr_dbg("ibv_get_cq_event=%d\n", rc);
+        if (unlikely(rc)) {
+            pr_dbg("---> ibv_get_cq_event (%d)\n", rc);
+            continue;
+        }
+
+        if (unlikely(ibv_req_notify_cq(ev_cq, 0))) {
+            pr_dbg("---> ibv_req_notify_cq\n");
+        }
+
+        poll_cq(dev, ev_cq, false);
+
+        ibv_ack_cq_events(ev_cq, 1);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining buffs that were posted */
+
+    return NULL;
+}
+
+void backend_register_comp_handler(void (*handler)(int status,
+                                   unsigned int vendor_err, void *ctx))
+{
+    comp_handler = handler;
+}
+
+int backend_query_port(BackendDevice *dev, struct pvrdma_port_attr *attrs)
+{
+    int rc;
+    struct ibv_port_attr port_attr;
+
+    rc = ibv_query_port(dev->context, dev->port_num, &port_attr);
+    if (rc) {
+        pr_dbg("Error %d from ibv_query_port\n", rc);
+        return -EIO;
+    }
+
+    attrs->state = port_attr.state;
+    attrs->max_mtu = port_attr.max_mtu;
+    attrs->active_mtu = port_attr.active_mtu;
+    attrs->gid_tbl_len = port_attr.gid_tbl_len;
+    attrs->pkey_tbl_len = port_attr.pkey_tbl_len;
+    attrs->phys_state = port_attr.phys_state;
+
+    return 0;
+}
+
+void backend_poll_cq(PVRDMADev *dev, BackendCQ *cq)
+{
+    poll_cq(dev, cq->ibcq, true);
+}
+
+static GHashTable *ah_hash;
+
+static struct ibv_ah *create_ah(BackendDevice *dev, struct ibv_pd *pd,
+                                union ibv_gid *dgid, uint8_t sgid_idx)
+{
+    GBytes *ah_key = g_bytes_new(dgid, sizeof(*dgid));
+    struct ibv_ah *ah = g_hash_table_lookup(ah_hash, ah_key);
+
+    if (ah) {
+        trace_create_ah_cache_hit(be64_to_cpu(dgid->global.subnet_prefix),
+                                  be64_to_cpu(dgid->global.interface_id));
+    } else {
+        struct ibv_ah_attr ah_attr = {
+            .is_global     = 1,
+            .port_num      = dev->port_num,
+            .grh.hop_limit = 1,
+        };
+
+        ah_attr.grh.dgid = *dgid;
+        ah_attr.grh.sgid_index = sgid_idx;
+
+        ah = ibv_create_ah(pd, &ah_attr);
+        if (ah) {
+            g_hash_table_insert(ah_hash, ah_key, ah);
+        } else {
+            pr_dbg("ibv_create_ah failed for gid <%lx %lx>\n",
+                    be64_to_cpu(dgid->global.subnet_prefix),
+                    be64_to_cpu(dgid->global.interface_id));
+        }
+
+        trace_create_ah_cache_miss(be64_to_cpu(dgid->global.subnet_prefix),
+                                   be64_to_cpu(dgid->global.interface_id));
+    }
+
+    return ah;
+}
+
+static void destroy_ah(gpointer data)
+{
+    struct ibv_ah *ah = data;
+    ibv_destroy_ah(ah);
+}
+
+static void ah_cache_init(void)
+{
+    ah_hash = g_hash_table_new_full(g_bytes_hash, g_bytes_equal,
+                                    NULL, destroy_ah);
+}
+
+static int send_mad(PVRDMADev *dev, struct pvrdma_sge *sge, u32 num_sge)
+{
+    int ret = -1;
+
+    /*
+     * TODO: Currently QP1 is not supported
+     *
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+    char mad_msg[1024];
+    void *hdr, *msg;
+    struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg;
+
+    umad->length = sge[0].length + sge[1].length;
+
+    if (num_sge != 2)
+        return -EINVAL;
+
+    pr_dbg("msg_len=%d\n", umad->length);
+
+    hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length);
+    msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length);
+
+    memcpy(&mad_msg[64], hdr, sge[0].length);
+    memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length);
+
+    pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length);
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 274

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+    pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length);
+
+    ret = umad_send(dev->backend_dev.mad_agent.port_id,
+                    dev->backend_dev.mad_agent.agent_id,
+                    mad_msg, umad->length, 10, 10);
+    */

Then what is above code doing here?

Support for QP1 is a work in progress.
In the meantime i can take this entire code out to a separate private branch.

Quoted 3 lines

Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?

Even without QP1 the device can still be used for usual RDMA send and receive operations, but yes, no rdma_cm and multicast for now. Please note that vmware support for QP1 is proprietary and can be used only between two ESX guests so we give more or less the same.

Quoted 2 lines

How does guest know it's missing?

I had two options, one is to reject the creation of QP1 via the control interface and second is to post CQE with error. Since the former causes some driver loading errors in guest i choose the second.

Quoted 46 lines

...

Quoted 41 lines
hw/net/pvrdma/pvrdma_utils.h new file +35 -0
new file mode 100644
index 0000000000..a09e39946c
@@ -0,0 +1,41 @@
+/*
+ * QEMU VMWARE paravirtual RDMA interface definitions
+ *
+ * Developed by Oracle & Redhat
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_UTILS_H
+#define PVRDMA_UTILS_H
+
+#include <include/hw/pci/pci.h>
+
+#define pr_info(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma",  __func__, __LINE__,\
+           ## __VA_ARGS__)
+
+#define pr_err(fmt, ...) \
+    fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \
+        __LINE__, ## __VA_ARGS__)
+
+#ifdef PVRDMA_DEBUG
+#define pr_dbg(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\
+           ## __VA_ARGS__)
+#else
+#define pr_dbg(fmt, ...)
+#endif
+
Quoted 8 lines

hw/net/pvrdma/pvrdma_utils.h line 41

Quoted 6 lines
hw/net/pvrdma/pvrdma_utils.h new file +6 -0
+void pvrdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len);
+void *pvrdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen);
+void *map_to_pdir(PCIDevice *pdev, uint64_t pdir_dma, uint32_t nchunks,
+                  size_t length);
+
+#endif

Can you make sure all prefixes are pvrdma_?

Is it a general practice or just for non-static functions?

Quoted 30 lines
Quoted 30 lines
hw/net/pvrdma/trace-events new file +9 -0
new file mode 100644
index 0000000000..bbc52286bc
@@ -0,0 +1,9 @@
+# See docs/tracing.txt for syntax documentation.
+
+# hw/net/pvrdma/pvrdma_main.c
+pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+
+#hw/net/pvrdma/pvrdma_backend.c
+create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
+create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
include/hw/pci/pci_ids.h +3 -0
index 35df1874a9..1dbf53627c 100644
@@ -266,4 +266,7 @@
 #define PCI_VENDOR_ID_TEWS               0x1498
 #define PCI_DEVICE_ID_TEWS_TPCI200       0x30C8
 
+#define PCI_VENDOR_ID_VMWARE             0x15ad
+#define PCI_DEVICE_ID_VMWARE_PVRDMA      0x0820
+
 #endif
Signature · 1 line
2.13.5
#13 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 40 lines · Yuval Shaia
Quoted 2 lines · Michael S. Tsirkin

I won't have time to review before the next year.
Results of a quick peek.

Thanks for the parts you found the time to review.

Quoted 18 lines · Marcel Apfelbaum
Quoted 16 lines
hw/net/pvrdma/pvrdma_backend.c new file +16 -0
+static void *mad_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    QObject *o_ctx_id;
+    unsigned long cqe_ctx_id;
+    BackendCtx *bctx;
+    /*
+    int len;
+    void *mad;
+    */
+
+    pr_dbg("Starting\n");
+
+    dev->backend_dev.mad_thread.run = false;
+
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 86

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+    while (dev->backend_dev.mad_thread.run) {
+        /* Get next buffer to pust MAD into */
+        o_ctx_id = qlist_pop(dev->backend_dev.mad_agent.recv_mads_list);
+        if (!o_ctx_id) {
+            /* pr_dbg("Error: No more free MADs buffers\n"); */
+            sleep(5);

Looks suspicious. What is above doing?

This function is responsible to process incoming MAD messages.

Usual (good) flow is that guest driver prepares some buffers to be used for that purpose and gives it to device with the usual post_recv mechanism. Upon receiving a MAD message from the device the driver should pass a new buffer to device.

Quoted 4 lines

But what if the device didn't do it.

This section handle such case, as we have nothing to do - let's sleep and hope for the best.

So is this broken hardware then? I would say print an error and disable the device.

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
hw/net/pvrdma/pvrdma_backend.c new file +12 -0
+            continue;
+        }
+        cqe_ctx_id = qnum_get_uint(qobject_to_qnum(o_ctx_id));
+        bctx = rm_get_cqe_ctx(dev, cqe_ctx_id);
+        if (unlikely(!bctx)) {
+            pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id);
+            continue;
+        }
+
+        pr_dbg("Calling umad_recv\n");
+        /*
+        mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr,
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 104

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+                                 bctx->req.sge[0].length);
+
+        len = bctx->req.sge[0].length;
+
+        do {
+            rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);

That's a huge timeout.

Quoted 5 lines

This is the maximum timeout.
On low MAD traffic we don't want to take much of the CPU for that purpose so 5 seconds looks good to me.

Anyway, just because it is not obvious i will make it configurable.

I'm not sure why not an infinite timeout then?

Quoted 179 lines
Quoted 165 lines
Quoted 165 lines
hw/net/pvrdma/pvrdma_backend.c new file +164 -0
+        } while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run);
+        pr_dbg("umad_recv, rc=%d\n", rc);
+
+        pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length);
+        */
+        rc = -1;
+
+        /* rc is used as vendor_err */
+        comp_handler(rc > 0 ? IB_WC_SUCCESS : IB_WC_GENERAL_ERR, rc,
+                     bctx->up_ctx);
+
+        rm_dealloc_cqe_ctx(dev, cqe_ctx_id);
+        free(bctx);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining MADs in list */
+
+    qlist_destroy_obj(QOBJECT(dev->backend_dev.mad_agent.recv_mads_list));
+
+    return NULL;
+}
+
+static void *comp_handler_thread(void *arg)
+{
+    PVRDMADev *dev = (PVRDMADev *)arg;
+    int rc;
+    struct ibv_cq *ev_cq;
+    void *ev_ctx;
+
+    pr_dbg("Starting\n");
+
+    while (dev->backend_dev.comp_thread.run) {
+        pr_dbg("Waiting for completion on channel %p\n",
+               dev->backend_dev.channel);
+        rc = ibv_get_cq_event(dev->backend_dev.channel, &ev_cq, &ev_ctx);
+        pr_dbg("ibv_get_cq_event=%d\n", rc);
+        if (unlikely(rc)) {
+            pr_dbg("---> ibv_get_cq_event (%d)\n", rc);
+            continue;
+        }
+
+        if (unlikely(ibv_req_notify_cq(ev_cq, 0))) {
+            pr_dbg("---> ibv_req_notify_cq\n");
+        }
+
+        poll_cq(dev, ev_cq, false);
+
+        ibv_ack_cq_events(ev_cq, 1);
+    }
+
+    pr_dbg("Going down\n");
+    /* TODO: Post cqe for all remaining buffs that were posted */
+
+    return NULL;
+}
+
+void backend_register_comp_handler(void (*handler)(int status,
+                                   unsigned int vendor_err, void *ctx))
+{
+    comp_handler = handler;
+}
+
+int backend_query_port(BackendDevice *dev, struct pvrdma_port_attr *attrs)
+{
+    int rc;
+    struct ibv_port_attr port_attr;
+
+    rc = ibv_query_port(dev->context, dev->port_num, &port_attr);
+    if (rc) {
+        pr_dbg("Error %d from ibv_query_port\n", rc);
+        return -EIO;
+    }
+
+    attrs->state = port_attr.state;
+    attrs->max_mtu = port_attr.max_mtu;
+    attrs->active_mtu = port_attr.active_mtu;
+    attrs->gid_tbl_len = port_attr.gid_tbl_len;
+    attrs->pkey_tbl_len = port_attr.pkey_tbl_len;
+    attrs->phys_state = port_attr.phys_state;
+
+    return 0;
+}
+
+void backend_poll_cq(PVRDMADev *dev, BackendCQ *cq)
+{
+    poll_cq(dev, cq->ibcq, true);
+}
+
+static GHashTable *ah_hash;
+
+static struct ibv_ah *create_ah(BackendDevice *dev, struct ibv_pd *pd,
+                                union ibv_gid *dgid, uint8_t sgid_idx)
+{
+    GBytes *ah_key = g_bytes_new(dgid, sizeof(*dgid));
+    struct ibv_ah *ah = g_hash_table_lookup(ah_hash, ah_key);
+
+    if (ah) {
+        trace_create_ah_cache_hit(be64_to_cpu(dgid->global.subnet_prefix),
+                                  be64_to_cpu(dgid->global.interface_id));
+    } else {
+        struct ibv_ah_attr ah_attr = {
+            .is_global     = 1,
+            .port_num      = dev->port_num,
+            .grh.hop_limit = 1,
+        };
+
+        ah_attr.grh.dgid = *dgid;
+        ah_attr.grh.sgid_index = sgid_idx;
+
+        ah = ibv_create_ah(pd, &ah_attr);
+        if (ah) {
+            g_hash_table_insert(ah_hash, ah_key, ah);
+        } else {
+            pr_dbg("ibv_create_ah failed for gid <%lx %lx>\n",
+                    be64_to_cpu(dgid->global.subnet_prefix),
+                    be64_to_cpu(dgid->global.interface_id));
+        }
+
+        trace_create_ah_cache_miss(be64_to_cpu(dgid->global.subnet_prefix),
+                                   be64_to_cpu(dgid->global.interface_id));
+    }
+
+    return ah;
+}
+
+static void destroy_ah(gpointer data)
+{
+    struct ibv_ah *ah = data;
+    ibv_destroy_ah(ah);
+}
+
+static void ah_cache_init(void)
+{
+    ah_hash = g_hash_table_new_full(g_bytes_hash, g_bytes_equal,
+                                    NULL, destroy_ah);
+}
+
+static int send_mad(PVRDMADev *dev, struct pvrdma_sge *sge, u32 num_sge)
+{
+    int ret = -1;
+
+    /*
+     * TODO: Currently QP1 is not supported
+     *
+    PCIDevice *pci_dev = PCI_DEVICE(dev);
+    char mad_msg[1024];
+    void *hdr, *msg;
+    struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg;
+
+    umad->length = sge[0].length + sge[1].length;
+
+    if (num_sge != 2)
+        return -EINVAL;
+
+    pr_dbg("msg_len=%d\n", umad->length);
+
+    hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length);
+    msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length);
+
+    memcpy(&mad_msg[64], hdr, sge[0].length);
+    memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length);
+
+    pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length);
Quoted 8 lines

hw/net/pvrdma/pvrdma_backend.c line 274

Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6 -0
+    pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length);
+
+    ret = umad_send(dev->backend_dev.mad_agent.port_id,
+                    dev->backend_dev.mad_agent.agent_id,
+                    mad_msg, umad->length, 10, 10);
+    */

Then what is above code doing here?

Support for QP1 is a work in progress.
In the meantime i can take this entire code out to a separate private branch.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?

Even without QP1 the device can still be used for usual RDMA send and receive operations, but yes, no rdma_cm and multicast for now. Please note that vmware support for QP1 is proprietary and can be used only between two ESX guests so we give more or less the same.

You will want to document this stuff.

Quoted 62 lines
Quoted 2 lines

How does guest know it's missing?

I had two options, one is to reject the creation of QP1 via the control interface and second is to post CQE with error. Since the former causes some driver loading errors in guest i choose the second.

Quoted 46 lines

...

Quoted 41 lines
hw/net/pvrdma/pvrdma_utils.h new file +35 -0
new file mode 100644
index 0000000000..a09e39946c
@@ -0,0 +1,41 @@
+/*
+ * QEMU VMWARE paravirtual RDMA interface definitions
+ *
+ * Developed by Oracle & Redhat
+ *
+ * Authors:
+ *     Yuval Shaia <[email protected]>
+ *     Marcel Apfelbaum <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef PVRDMA_UTILS_H
+#define PVRDMA_UTILS_H
+
+#include <include/hw/pci/pci.h>
+
+#define pr_info(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma",  __func__, __LINE__,\
+           ## __VA_ARGS__)
+
+#define pr_err(fmt, ...) \
+    fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \
+        __LINE__, ## __VA_ARGS__)
+
+#ifdef PVRDMA_DEBUG
+#define pr_dbg(fmt, ...) \
+    fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\
+           ## __VA_ARGS__)
+#else
+#define pr_dbg(fmt, ...)
+#endif
+
Quoted 8 lines

hw/net/pvrdma/pvrdma_utils.h line 41

Quoted 6 lines
hw/net/pvrdma/pvrdma_utils.h new file +6 -0
+void pvrdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len);
+void *pvrdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen);
+void *map_to_pdir(PCIDevice *pdev, uint64_t pdir_dma, uint32_t nchunks,
+                  size_t length);
+
+#endif

Can you make sure all prefixes are pvrdma_?

Quoted 1 line

Is it a general practice or just for non-static functions?

Doing this generally is preferred. A must for non-static.

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
hw/net/pvrdma/trace-events new file +9 -0
new file mode 100644
index 0000000000..bbc52286bc
@@ -0,0 +1,9 @@
+# See docs/tracing.txt for syntax documentation.
+
+# hw/net/pvrdma/pvrdma_main.c
+pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx"
+
+#hw/net/pvrdma/pvrdma_backend.c
+create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
+create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"
include/hw/pci/pci_ids.h +3 -0
index 35df1874a9..1dbf53627c 100644
@@ -266,4 +266,7 @@
 #define PCI_VENDOR_ID_TEWS               0x1498
 #define PCI_DEVICE_ID_TEWS_TPCI200       0x30C8
 
+#define PCI_VENDOR_ID_VMWARE             0x15ad
+#define PCI_DEVICE_ID_VMWARE_PVRDMA      0x0820
+
 #endif
Signature · 1 line
2.13.5
#14 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

In reply to Marcel Apfelbaum

Hi Marcel, Yuval,

Quoted 17 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---

[...]

Quoted 14 lines
 28 files changed, 5132 insertions(+), 4 deletions(-)
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
Quoted 8 lines
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events

[...]

Since we already have a hw/xenpv/ directory, can we place these files into hw/vmwarepv/ rather than hw/net/pvrdma/?

A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.

Regards,

Phil.

#15 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 43 lines · Philippe Mathieu-Daudé

Hi Marcel, Yuval,

Quoted 17 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---

[...]

Quoted 14 lines
 28 files changed, 5132 insertions(+), 4 deletions(-)
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
Quoted 8 lines
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Quoted 3 lines

[...]

Since we already have a hw/xenpv/ directory,

But e.g. xen nic is under hw/net/

Quoted 8 lines

can we place these files
into hw/vmwarepv/ rather than hw/net/pvrdma/?

A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.

Regards,

Phil.

That's not how we layout things. We group them by function not by interface. Thus I think that hw/rdma/ is better.

Signature · 1 line
MST
#16 lore raw

Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation

Quoted 53 lines · Michael S. Tsirkin
Quoted 43 lines · Philippe Mathieu-Daudé

Hi Marcel, Yuval,

Quoted 17 lines · Marcel Apfelbaum

From: Yuval Shaia <[email protected]>

PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.

---

[...]

Quoted 14 lines
  28 files changed, 5132 insertions(+), 4 deletions(-)

create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h

Quoted 8 lines

create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-events

Quoted 3 lines

[...]

Since we already have a hw/xenpv/ directory,

But e.g. xen nic is under hw/net/

Quoted 3 lines

can we place these files
into hw/vmwarepv/ rather than hw/net/pvrdma/?

Quoted 9 lines
Quoted 5 lines

A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.

Regards,

Phil.

That's not how we layout things. We group them by function not by interface. Thus I think that hw/rdma/ is better.

Hi Philippe and Michael,

Thanks for the suggestion, we will move the code to hw/rdma. Marcel

#17 lore raw

PATCH v2 5/5 MAINTAINERS: add entry for hw/net/pvrdma

In reply to Marcel Apfelbaum
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
index ffd77b461c..d24401a4d0 100644
@@ -1960,6 +1960,13 @@ F: block/replication.c
 F: tests/test-replication.c
 F: docs/block-replication.txt
 
+PVRDMA
+M: Yuval Shaia <[email protected]>
+M: Marcel Apfelbaum <[email protected]>
+S: Maintained
+F: hw/net/pvrdma/*
+F: docs/pvrdma.txt
+
 Build and test automation
 -------------------------
 Build and test automation
Signature · 1 line
2.13.5
#18 lore raw

Re: [Qemu-devel] [PATCH V2 5/5] MAINTAINERS: add entry for hw/net/pvrdma

Quoted 14 lines · Marcel Apfelbaum
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)
MAINTAINERS +0 -0
index ffd77b461c..d24401a4d0 100644
@@ -1960,6 +1960,13 @@ F: block/replication.c
 F: tests/test-replication.c
 F: docs/block-replication.txt
 

MAINTAINERS F: block/replication.c · line 1972

Quoted 10 lines
MAINTAINERS +7 -0
+PVRDMA
+M: Yuval Shaia <[email protected]>
+M: Marcel Apfelbaum <[email protected]>
+S: Maintained
+F: hw/net/pvrdma/*
+F: docs/pvrdma.txt
+
 Build and test automation
 -------------------------
 Build and test automation

Not really a network device, I think it needs its own directory.

Quoted 2 lines
Signature · 1 line
2.13.5
#19 lore raw
In reply to Marcel Apfelbaum
Quoted 10 lines · Marcel Apfelbaum

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

Quoted 5 lines

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned

What happens if guest attempts to register all its memory?

Quoted 4 lines

allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.

What does "HW assistance" mean here?
Can it work with any existing hardware?

Quoted 5 lines

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it

Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?

Quoted 2 lines

and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.

I suspect that without virtio we won't be able to do any future extensions.

Quoted 4 lines

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers

What does this mean?

Quoted 1 line

BAR 2 - UAR, used to pass HW commands from driver.

A detailed description of above belongs in documentation.

Quoted 3 lines

- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.

I am wondering how do you make connections? QP#s are exposed on the wire during connection management.

Quoted 5 lines

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

So what's the advantage of this over pass-through then?

Quoted 95 lines

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

 MAINTAINERS                         |   7 +
 Makefile.objs                       |   1 +
 backends/hostmem-file.c             |  25 +-
 backends/hostmem-ram.c              |   4 +-
 backends/hostmem.c                  |  21 +
 configure                           |   9 +-
 default-configs/arm-softmmu.mak     |   2 +
 default-configs/i386-softmmu.mak    |   1 +
 default-configs/x86_64-softmmu.mak  |   1 +
 docs/pvrdma.txt                     | 145 ++++++
 exec.c                              |  26 +-
 hw/net/Makefile.objs                |   7 +
 hw/net/pvrdma/pvrdma.h              | 179 +++++++
 hw/net/pvrdma/pvrdma_backend.c      | 986 ++++++++++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_backend.h      |  74 +++
 hw/net/pvrdma/pvrdma_backend_defs.h |  68 +++
 hw/net/pvrdma/pvrdma_cmd.c          | 338 ++++++++++++
 hw/net/pvrdma/pvrdma_defs.h         | 121 +++++
 hw/net/pvrdma/pvrdma_dev_api.h      | 580 +++++++++++++++++++++
 hw/net/pvrdma/pvrdma_dev_ring.c     | 138 +++++
 hw/net/pvrdma/pvrdma_dev_ring.h     |  42 ++
 hw/net/pvrdma/pvrdma_ib_verbs.h     | 399 +++++++++++++++
 hw/net/pvrdma/pvrdma_main.c         | 664 ++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_qp_ops.c       | 187 +++++++
 hw/net/pvrdma/pvrdma_qp_ops.h       |  26 +
 hw/net/pvrdma/pvrdma_ring.h         | 134 +++++
 hw/net/pvrdma/pvrdma_rm.c           | 791 +++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_rm.h           |  54 ++
 hw/net/pvrdma/pvrdma_rm_defs.h      | 111 ++++
 hw/net/pvrdma/pvrdma_types.h        |  37 ++
 hw/net/pvrdma/pvrdma_utils.c        | 133 +++++
 hw/net/pvrdma/pvrdma_utils.h        |  41 ++
 hw/net/pvrdma/trace-events          |   9 +
 hw/pci/shpc.c                       |  11 +-
 include/exec/memory.h               |  23 +
 include/exec/ram_addr.h             |   3 +-
 include/hw/pci/pci_ids.h            |   3 +
 include/qemu/cutils.h               |  10 +
 include/qemu/osdep.h                |   2 +-
 include/sysemu/hostmem.h            |   2 +-
 include/sysemu/kvm.h                |   2 +-
 memory.c                            |  16 +-
 util/oslib-posix.c                  |   4 +-
 util/oslib-win32.c                  |   2 +-
 44 files changed, 5378 insertions(+), 61 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Signature · 1 line
2.13.5
#20 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

Quoted 6 lines
Quoted 5 lines

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned

Hi Michael,

Quoted 2 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM. However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

Quoted 6 lines
Quoted 4 lines

allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.

What does "HW assistance" mean here?

Several things:
1. We need to be able to pass resource numbers when we create them on the destination machine.
2. We also need a way to stall prev connections while starting the new ones.
3. Last, we need the HW to pass resources states.

Quoted 2 lines

Can it work with any existing hardware?

Sadly no, however we talked with Mellanox at the last year Plumbers Conference and all the above are on their plans. We hope this submission will help, since now we will have a fast way to test and use it.

For Soft-RoCE backend is doable, but is best to wait first to see how HCAs are going to expose the changes.

Quoted 6 lines
Quoted 5 lines

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it

Quoted 9 lines

Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?

Quoted 2 lines

and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.

I suspect that without virtio we won't be able to do any future extensions.

While I do agree is harder to work with a 3rd party spec, their Linux driver is open source and we may be able to do sane modifications.

Quoted 4 lines
Quoted 4 lines

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers

[...]

Quoted 8 lines
Quoted 5 lines

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

So what's the advantage of this over pass-through then?

1. We can work also with the same ibdevice for multiple pvrdma devices using multiple GIDs; it works (tested).
The problem begins when we think about migration, the way HCAs work today is one resource namespace per ibdevice, not per GID. I emphasize that this can be changed, however we don't have a timeline for it.

2. We do have advantages:
- Guest agnostic device (we can change host HCA)
- Memory over commit (unless the guest registers all the memory)
- Future migration support
- A friendly migration of RDMA VMWare guests to QEMU.

3. In case when live migration is not a must we can use multiple GIDs of the same port, so we do not depend on SRIOV.

4. We support Soft RoCE backend, people can test their software on guest without RDMA hw.

Thanks,
Marcel

Quoted 26 lines
Quoted 26 lines

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

[...]

#21 lore raw
Quoted 19 lines · Marcel Apfelbaum
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

Quoted 6 lines
Quoted 5 lines

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned

Quoted 6 lines

Hi Michael,

Quoted 2 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

Quoted 2 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.

What does "HW assistance" mean here?

Several things:
1. We need to be able to pass resource numbers when we create them on the destination machine.

These resources are mostly managed by software.

Quoted 1 line

2. We also need a way to stall prev connections while starting the new ones.

Look at what hardware can do.

Quoted 1 line

3. Last, we need the HW to pass resources states.

Look at the spec, some of this can be done.

Quoted 4 lines
Quoted 2 lines

Can it work with any existing hardware?

Sadly no,

Above can be done. What's needed is host kernel work to support it.

Quoted 4 lines

however we talked with Mellanox at the last year Plumbers Conference and all the above are on their plans. We hope this submission will help, since now we will have a fast way to test and use it.

I'm doubtful it'll help.

Quoted 15 lines

For Soft-RoCE backend is doable, but is best to wait first to see how HCAs are going to expose the changes.

Quoted 12 lines
Quoted 5 lines

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it

Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?

Quoted 2 lines

and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.

Quoted 7 lines
Quoted 3 lines

I suspect that without virtio we won't be able to do any future extensions.

While I do agree is harder to work with a 3rd party spec, their Linux driver is open source and we may be able to do sane modifications.

I am sceptical. ARM guys did not want to add a single bit in their IOMMU spec. You want an open spec that everyone can contribute to.

Quoted 23 lines
Quoted 4 lines
Quoted 4 lines

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers

[...]

Quoted 8 lines
Quoted 5 lines

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

So what's the advantage of this over pass-through then?

1. We can work also with the same ibdevice for multiple pvrdma devices using multiple GIDs; it works (tested).
The problem begins when we think about migration, the way HCAs work today is one resource namespace per ibdevice, not per GID. I emphasize that this can be changed, however we don't have a timeline for it.

Quoted 3 lines

2. We do have advantages:
- Guest agnostic device (we can change host HCA)
- Memory over commit (unless the guest registers all the memory)

Not just all. You trust guest and this is a problem. If you do try to overcommit, at any point guest can try to register too much and host will stall.

Quoted 1 line

- Future migration support

So there are lots of difficult problems to solve for this. E.g. any MR that is hardware writeable can be changed and hypervisor won't know. All this can be solvable but it might also be solveable with passthrough too.

Quoted 1 line

- A friendly migration of RDMA VMWare guests to QEMU.

Why do we need to emulate their device for this? Reboot is required anyway, you can switch to a passthrough easily.

Quoted 10 lines

3. In case when live migration is not a must we can use multiple GIDs of the same port, so we do not depend on SRIOV.

4. We support Soft RoCE backend, people can test their software on guest without RDMA hw.

Thanks,
Marcel

These two are nice, if very niche, features.

Quoted 28 lines
Quoted 26 lines
Quoted 26 lines

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

[...]

#22 lore raw
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#23 lore raw
Quoted 4 lines · Yuval Shaia
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Isn't that still leaving the option for guest to register all memory, just in chunks?

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#24 lore raw
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

Quoted 7 lines
Quoted 3 lines

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Isn't that still leaving the option for guest to register all memory, just in chunks?

We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.

Thanks,
Marcel

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#25 lore raw
Quoted 19 lines · Marcel Apfelbaum
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

Quoted 7 lines
Quoted 3 lines

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Isn't that still leaving the option for guest to register all memory, just in chunks?

Quoted 5 lines

We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.

Thanks,
Marcel

You might want to limit the default values then.

Right now:

+#define MAX_MR_SIZE (1UL << 32)
+#define MAX_MR 2048

Which is IIUC 8TB.

That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#26 lore raw
Quoted 30 lines · Michael S. Tsirkin
Quoted 19 lines · Marcel Apfelbaum
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

Quoted 7 lines
Quoted 3 lines

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Isn't that still leaving the option for guest to register all memory, just in chunks?

Quoted 5 lines

We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.

Thanks,
Marcel

You might want to limit the default values then.

Right now:

Review

Quoted 2 lines
(quoted diff) +2 -0
+#define MAX_MR_SIZE           (1UL << 32)
+#define MAX_MR                2048

Maybe limiting by constant number is not a good approach, it looks odd if one guest with 16G ram and second with 32G ram will have the same settings, right?
So how about limiting by a specific percentage of total memory? In that case, what would be this percentage? 100%? 80%?

Quoted 13 lines

Which is IIUC 8TB.

That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#27 lore raw
Quoted 21 lines · Yuval Shaia
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Marcel Apfelbaum
Quoted 17 lines · Michael S. Tsirkin
Quoted 4 lines · Yuval Shaia
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

What happens if guest attempts to register all its memory?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Then we loose, is not different from bare metal, reg_mr will pin all the RAM.

We need to find a way to communicate to guests about amount of memory they can pin.

dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).

Isn't that still leaving the option for guest to register all memory, just in chunks?

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.

Thanks,
Marcel

You might want to limit the default values then.

Hi Yuval,

Quoted 11 lines

Review

Quoted 4 lines

Right now:

(quoted diff) +2 -0
+#define MAX_MR_SIZE           (1UL << 32)
+#define MAX_MR                2048

Maybe limiting by constant number is not a good approach, it looks odd if one guest with 16G ram and second with 32G ram will have the same settings, right?
So how about limiting by a specific percentage of total memory? In that case, what would be this percentage? 100%? 80%?

I think is too complicated. Maybe we can limit the max pined memory to 2G assuming the RDMA guests have a lot of RAM and let the users fine-tune the parameters.

Thanks,
Marcel

Quoted 13 lines
Quoted 13 lines

Which is IIUC 8TB.

That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).

SRP does it too AFAIK.

#28 lore raw
In reply to Michael S. Tsirkin
Quoted 28 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

Quoted 5 lines

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned

What happens if guest attempts to register all its memory?

Quoted 4 lines

allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.

What does "HW assistance" mean here?
Can it work with any existing hardware?

Quoted 1 line
Quoted 10 lines
Quoted 4 lines

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it

Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?

Quoted 2 lines

and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.

The current design takes into account a future code reuse with virtio-rdma device although not sure it is 100%.

We divided it to four software layers:
- Front-end interface with PCI:
- pvrdma_main.c
- Front-end interface with pvrdma driver:
- pvrdma_cmd.c
- pvrdma_qp_ops.c
- pvrdma_dev_ring.c
- pvrdma_utils.c
- Device emulation:
- pvrdma_rm.c
- Back-end interface:
- pvrdma_backend.c

So in the future, when starting to work on virtio-rdma device we will move the generic code to generic directory.

Any reason why we want to split it now, when we have only one device?

Quoted 3 lines

I suspect that without virtio we won't be able to do any future extensions.

As i see it these are two different issues, virtio RDMA device is on our plate but the contribution of VMWare pvrdma device to QEMU is no doubt a real advantage that will allow customers that runs ESX to easy move to QEMU.

Quoted 7 lines
Quoted 5 lines

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers

What does this mean?

Device control operations:
- Setting of interrupt mask.
- Setup of Device/Driver shared configuration area.
- Reset device, activate device etc.
- Device commands such as create QP, create MR etc.

Quoted 4 lines
Quoted 2 lines

BAR 2 - UAR, used to pass HW commands from driver.

A detailed description of above belongs in documentation.

Will do.

Quoted 7 lines
Quoted 4 lines

- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.

I am wondering how do you make connections? QP#s are exposed on the wire during connection management.

QP#s that guest sees are the QP#s that are used on the wire. The meaning of "internal management of the RDMA resources" is that we keep context of internal QP in device (ex rings).

Quoted 105 lines
Quoted 6 lines

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

So what's the advantage of this over pass-through then?

Quoted 95 lines

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

 MAINTAINERS                         |   7 +
 Makefile.objs                       |   1 +
 backends/hostmem-file.c             |  25 +-
 backends/hostmem-ram.c              |   4 +-
 backends/hostmem.c                  |  21 +
 configure                           |   9 +-
 default-configs/arm-softmmu.mak     |   2 +
 default-configs/i386-softmmu.mak    |   1 +
 default-configs/x86_64-softmmu.mak  |   1 +
 docs/pvrdma.txt                     | 145 ++++++
 exec.c                              |  26 +-
 hw/net/Makefile.objs                |   7 +
 hw/net/pvrdma/pvrdma.h              | 179 +++++++
 hw/net/pvrdma/pvrdma_backend.c      | 986 ++++++++++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_backend.h      |  74 +++
 hw/net/pvrdma/pvrdma_backend_defs.h |  68 +++
 hw/net/pvrdma/pvrdma_cmd.c          | 338 ++++++++++++
 hw/net/pvrdma/pvrdma_defs.h         | 121 +++++
 hw/net/pvrdma/pvrdma_dev_api.h      | 580 +++++++++++++++++++++
 hw/net/pvrdma/pvrdma_dev_ring.c     | 138 +++++
 hw/net/pvrdma/pvrdma_dev_ring.h     |  42 ++
 hw/net/pvrdma/pvrdma_ib_verbs.h     | 399 +++++++++++++++
 hw/net/pvrdma/pvrdma_main.c         | 664 ++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_qp_ops.c       | 187 +++++++
 hw/net/pvrdma/pvrdma_qp_ops.h       |  26 +
 hw/net/pvrdma/pvrdma_ring.h         | 134 +++++
 hw/net/pvrdma/pvrdma_rm.c           | 791 +++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_rm.h           |  54 ++
 hw/net/pvrdma/pvrdma_rm_defs.h      | 111 ++++
 hw/net/pvrdma/pvrdma_types.h        |  37 ++
 hw/net/pvrdma/pvrdma_utils.c        | 133 +++++
 hw/net/pvrdma/pvrdma_utils.h        |  41 ++
 hw/net/pvrdma/trace-events          |   9 +
 hw/pci/shpc.c                       |  11 +-
 include/exec/memory.h               |  23 +
 include/exec/ram_addr.h             |   3 +-
 include/hw/pci/pci_ids.h            |   3 +
 include/qemu/cutils.h               |  10 +
 include/qemu/osdep.h                |   2 +-
 include/sysemu/hostmem.h            |   2 +-
 include/sysemu/kvm.h                |   2 +-
 memory.c                            |  16 +-
 util/oslib-posix.c                  |   4 +-
 util/oslib-win32.c                  |   2 +-
 44 files changed, 5378 insertions(+), 61 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Signature · 1 line
2.13.5
#29 lore raw
Quoted 56 lines · Yuval Shaia
Quoted 28 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum

RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module

General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.

Quoted 5 lines

While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned

What happens if guest attempts to register all its memory?

Quoted 4 lines

allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.

What does "HW assistance" mean here?
Can it work with any existing hardware?

Quoted 1 line
Quoted 10 lines
Quoted 4 lines

Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it

Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?

Quoted 2 lines

and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.

The current design takes into account a future code reuse with virtio-rdma device although not sure it is 100%.

We divided it to four software layers:
- Front-end interface with PCI:
- pvrdma_main.c
- Front-end interface with pvrdma driver:
- pvrdma_cmd.c
- pvrdma_qp_ops.c
- pvrdma_dev_ring.c
- pvrdma_utils.c
- Device emulation:
- pvrdma_rm.c
- Back-end interface:
- pvrdma_backend.c

Quoted 4 lines

So in the future, when starting to work on virtio-rdma device we will move the generic code to generic directory.

Any reason why we want to split it now, when we have only one device?

To make it easier for me to ignore pvrdma stuff and review the generic stuff.

Quoted 7 lines
Quoted 3 lines

I suspect that without virtio we won't be able to do any future extensions.

As i see it these are two different issues, virtio RDMA device is on our plate but the contribution of VMWare pvrdma device to QEMU is no doubt a real advantage that will allow customers that runs ESX to easy move to QEMU.

I don't have anything against it but I'm not really interested in reviewing it either.

Quoted 26 lines
Quoted 7 lines
Quoted 5 lines

- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers

What does this mean?

Device control operations:
- Setting of interrupt mask.
- Setup of Device/Driver shared configuration area.
- Reset device, activate device etc.
- Device commands such as create QP, create MR etc.

Quoted 4 lines
Quoted 2 lines

BAR 2 - UAR, used to pass HW commands from driver.

A detailed description of above belongs in documentation.

Will do.

Quoted 5 lines
Quoted 4 lines

- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.

Quoted 6 lines
Quoted 2 lines

I am wondering how do you make connections? QP#s are exposed on the wire during connection management.

QP#s that guest sees are the QP#s that are used on the wire. The meaning of "internal management of the RDMA resources" is that we keep context of internal QP in device (ex rings).

The question was that you need to parse CM/MAD etc messages if you need to change QP#s on the fly, and that code does not seem to be there. I guess the answer is that
a bunch of this stuff is just broken or non-spec compliant.

Quoted 105 lines
Quoted 105 lines
Quoted 6 lines

The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).

So what's the advantage of this over pass-through then?

Quoted 95 lines

Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.

The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.

Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma

Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation

 MAINTAINERS                         |   7 +
 Makefile.objs                       |   1 +
 backends/hostmem-file.c             |  25 +-
 backends/hostmem-ram.c              |   4 +-
 backends/hostmem.c                  |  21 +
 configure                           |   9 +-
 default-configs/arm-softmmu.mak     |   2 +
 default-configs/i386-softmmu.mak    |   1 +
 default-configs/x86_64-softmmu.mak  |   1 +
 docs/pvrdma.txt                     | 145 ++++++
 exec.c                              |  26 +-
 hw/net/Makefile.objs                |   7 +
 hw/net/pvrdma/pvrdma.h              | 179 +++++++
 hw/net/pvrdma/pvrdma_backend.c      | 986 ++++++++++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_backend.h      |  74 +++
 hw/net/pvrdma/pvrdma_backend_defs.h |  68 +++
 hw/net/pvrdma/pvrdma_cmd.c          | 338 ++++++++++++
 hw/net/pvrdma/pvrdma_defs.h         | 121 +++++
 hw/net/pvrdma/pvrdma_dev_api.h      | 580 +++++++++++++++++++++
 hw/net/pvrdma/pvrdma_dev_ring.c     | 138 +++++
 hw/net/pvrdma/pvrdma_dev_ring.h     |  42 ++
 hw/net/pvrdma/pvrdma_ib_verbs.h     | 399 +++++++++++++++
 hw/net/pvrdma/pvrdma_main.c         | 664 ++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_qp_ops.c       | 187 +++++++
 hw/net/pvrdma/pvrdma_qp_ops.h       |  26 +
 hw/net/pvrdma/pvrdma_ring.h         | 134 +++++
 hw/net/pvrdma/pvrdma_rm.c           | 791 +++++++++++++++++++++++++++++
 hw/net/pvrdma/pvrdma_rm.h           |  54 ++
 hw/net/pvrdma/pvrdma_rm_defs.h      | 111 ++++
 hw/net/pvrdma/pvrdma_types.h        |  37 ++
 hw/net/pvrdma/pvrdma_utils.c        | 133 +++++
 hw/net/pvrdma/pvrdma_utils.h        |  41 ++
 hw/net/pvrdma/trace-events          |   9 +
 hw/pci/shpc.c                       |  11 +-
 include/exec/memory.h               |  23 +
 include/exec/ram_addr.h             |   3 +-
 include/hw/pci/pci_ids.h            |   3 +
 include/qemu/cutils.h               |  10 +
 include/qemu/osdep.h                |   2 +-
 include/sysemu/hostmem.h            |   2 +-
 include/sysemu/kvm.h                |   2 +-
 memory.c                            |  16 +-
 util/oslib-posix.c                  |   4 +-
 util/oslib-win32.c                  |   2 +-
 44 files changed, 5378 insertions(+), 61 deletions(-)
 create mode 100644 docs/pvrdma.txt
 create mode 100644 hw/net/pvrdma/pvrdma.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend.c
 create mode 100644 hw/net/pvrdma/pvrdma_backend.h
 create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_cmd.c
 create mode 100644 hw/net/pvrdma/pvrdma_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c
 create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h
 create mode 100644 hw/net/pvrdma/pvrdma_main.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c
 create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
 create mode 100644 hw/net/pvrdma/pvrdma_ring.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm.c
 create mode 100644 hw/net/pvrdma/pvrdma_rm.h
 create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h
 create mode 100644 hw/net/pvrdma/pvrdma_types.h
 create mode 100644 hw/net/pvrdma/pvrdma_utils.c
 create mode 100644 hw/net/pvrdma/pvrdma_utils.h
 create mode 100644 hw/net/pvrdma/trace-events
Signature · 1 line
2.13.5