hw/pvrdma: PVRDMA device implementation
v2 cover hw/pvrdma: PVRDMA device implementation
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module
General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it and most of the code can be reused if we decide to continue to a Virtio based RDMA device.
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
BAR 2 - UAR, used to pass HW commands from driver.
- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.
The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.
Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdma
Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation
MAINTAINERS | 7 + Makefile.objs | 1 + backends/hostmem-file.c | 25 +- backends/hostmem-ram.c | 4 +- backends/hostmem.c | 21 + configure | 9 +- default-configs/arm-softmmu.mak | 2 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + docs/pvrdma.txt | 145 ++++++ exec.c | 26 +- hw/net/Makefile.objs | 7 + hw/net/pvrdma/pvrdma.h | 179 +++++++ hw/net/pvrdma/pvrdma_backend.c | 986 ++++++++++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_backend.h | 74 +++ hw/net/pvrdma/pvrdma_backend_defs.h | 68 +++ hw/net/pvrdma/pvrdma_cmd.c | 338 ++++++++++++ hw/net/pvrdma/pvrdma_defs.h | 121 +++++ hw/net/pvrdma/pvrdma_dev_api.h | 580 +++++++++++++++++++++ hw/net/pvrdma/pvrdma_dev_ring.c | 138 +++++ hw/net/pvrdma/pvrdma_dev_ring.h | 42 ++ hw/net/pvrdma/pvrdma_ib_verbs.h | 399 +++++++++++++++ hw/net/pvrdma/pvrdma_main.c | 664 ++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_qp_ops.c | 187 +++++++ hw/net/pvrdma/pvrdma_qp_ops.h | 26 + hw/net/pvrdma/pvrdma_ring.h | 134 +++++ hw/net/pvrdma/pvrdma_rm.c | 791 +++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_rm.h | 54 ++ hw/net/pvrdma/pvrdma_rm_defs.h | 111 ++++ hw/net/pvrdma/pvrdma_types.h | 37 ++ hw/net/pvrdma/pvrdma_utils.c | 133 +++++ hw/net/pvrdma/pvrdma_utils.h | 41 ++ hw/net/pvrdma/trace-events | 9 + hw/pci/shpc.c | 11 +- include/exec/memory.h | 23 + include/exec/ram_addr.h | 3 +- include/hw/pci/pci_ids.h | 3 + include/qemu/cutils.h | 10 + include/qemu/osdep.h | 2 +- include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 2 +- memory.c | 16 +- util/oslib-posix.c | 4 +- util/oslib-win32.c | 2 +- 44 files changed, 5378 insertions(+), 61 deletions(-) create mode 100644 docs/pvrdma.txt create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-events
Signature · 1 line
2.13.5
PATCH v2 1/5 pci/shpc: Move function to generic header file
From: Yuval Shaia <[email protected]>
This function should be declared in generic header file so we can utilize it.
--- hw/pci/shpc.c | 11 +---------- include/qemu/cutils.h | 10 ++++++++++ 2 files changed, 11 insertions(+), 10 deletions(-)
hw/pci/shpc.c +1 -10
@@ -1,6 +1,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu-common.h" +#include "qemu/cutils.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "hw/pci/shpc.h"
@@ -122,16 +123,6 @@#define SHPC_PCI_TO_IDX (pci_slot) ((pci_slot) - 1)#define SHPC_IDX_TO_PHYSICAL (slot) ((slot) + 1) -static int roundup_pow_of_two (int x) -{ - x |= (x >>1 ); - x |= (x >>2 ); - x |= (x >>4 ); - x |= (x >>8 ); - x |= (x >>16 ); -return x +1 ; -} -static uint16_t shpc_get_status (SHPCDevice *shpc,int slot,uint16_t msk) {uint8_t *status = shpc->config +SHPC_SLOT_STATUS (slot);
include/qemu/cutils.h +10 -0
@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);int uleb128_encode_small (uint8_t *out,uint32_t n);int uleb128_decode_small (const uint8_t *in,uint32_t *n); +static inline int roundup_pow_of_two (int x) +{ + x |= (x >>1 ); + x |= (x >>2 ); + x |= (x >>4 ); + x |= (x >>8 ); + x |= (x >>16 ); +return x +1 ; +} +#endif
Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 1/5] pci/shpc: Move function to generic header file
Hi Marcel, Yuval,
include/qemu/cutils.h
Quoted 45 lines · Marcel Apfelbaum
From: Yuval Shaia <[email protected]>
This function should be declared in generic header file so we can utilize it.
Signed-off-by Yuval Shaia <[email protected]>Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/shpc.c | 11 +---------- include/qemu/cutils.h | 10 ++++++++++ 2 files changed, 11 insertions(+), 10 deletions(-)hw/pci/shpc.c +1
-10@@ -1,6 +1,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu-common.h" +#include "qemu/cutils.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "hw/pci/shpc.h" @@ -122,16 +123,6 @@#define SHPC_PCI_TO_IDX (pci_slot) ((pci_slot) - 1)#define SHPC_IDX_TO_PHYSICAL (slot) ((slot) + 1) -static int roundup_pow_of_two (int x) -{ - x |= (x >>1 ); - x |= (x >>2 ); - x |= (x >>4 ); - x |= (x >>8 ); - x |= (x >>16 ); -return x +1 ; -} -static uint16_t shpc_get_status (SHPCDevice *shpc,int slot,uint16_t msk) {uint8_t *status = shpc->config +SHPC_SLOT_STATUS (slot);include/qemu/cutils.h +0
-0
I'd rather move this function below pow2ceil() in "qemu/host-utils.h" and rename it pow2roundup().
include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 173
Quoted 11 lines
include/qemu/cutils.h +7
-0@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);int uleb128_encode_small (uint8_t *out,uint32_t n);int uleb128_decode_small (const uint8_t *in,uint32_t *n); +static inline int roundup_pow_of_two (w x) +{ + x |= (x >>1 ); + x |= (x >>2 ); + x |= (x >>4 ); + x |= (x >>8 ); + x |= (x >>16 );
So this would be pow2roundup32(uint32_t value)...
Naming it pow2roundup() without specifying the integer size, I'd directly use a uint64_t argument, and:
x |= (x >> 32);
include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 177
Quoted 4 lines
include/qemu/cutils.h +3
-0+ return x + 1; +} +#endif
Naming it pow2roundup() in "qemu/host-utils.h" (regardless the arg size):
Regards,
Phil.
Re: [Qemu-devel] [PATCH V2 1/5] pci/shpc: Move function to generic header file
Quoted 66 lines · Philippe Mathieu-Daudé
Hi Marcel, Yuval,
include/qemu/cutils.h
Quoted 45 lines · Marcel Apfelbaum
From: Yuval Shaia <[email protected]>
This function should be declared in generic header file so we can utilize it.
Signed-off-by Yuval Shaia <[email protected]>Signed-off-by Marcel Apfelbaum <[email protected]>--- hw/pci/shpc.c | 11 +---------- include/qemu/cutils.h | 10 ++++++++++ 2 files changed, 11 insertions(+), 10 deletions(-)hw/pci/shpc.c +1
-10@@ -1,6 +1,7 @@#include "qemu/osdep.h" #include "qapi/error.h" #include "qemu-common.h" +#include "qemu/cutils.h" #include "qemu/range.h" #include "qemu/error-report.h" #include "hw/pci/shpc.h" @@ -122,16 +123,6 @@#define SHPC_PCI_TO_IDX (pci_slot) ((pci_slot) - 1)#define SHPC_IDX_TO_PHYSICAL (slot) ((slot) + 1) -static int roundup_pow_of_two (int x) -{ - x |= (x >>1 ); - x |= (x >>2 ); - x |= (x >>4 ); - x |= (x >>8 ); - x |= (x >>16 ); -return x +1 ; -} -static uint16_t shpc_get_status (SHPCDevice *shpc,int slot,uint16_t msk) {uint8_t *status = shpc->config +SHPC_SLOT_STATUS (slot);include/qemu/cutils.h +0
-0I'd rather move this function below
pow2ceil()in "qemu/host-utils.h" and rename itpow2roundup().include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 173
Quoted 11 lines
include/qemu/cutils.h +7
-0@@ -164,4 +164,14 @@ bool test_buffer_is_zero_next_accel(void);int uleb128_encode_small (uint8_t *out,uint32_t n);int uleb128_decode_small (const uint8_t *in,uint32_t *n); +static inline int roundup_pow_of_two (w x) +{ + x |= (x >>1 ); + x |= (x >>2 ); + x |= (x >>4 ); + x |= (x >>8 ); + x |= (x >>16 );So this would be
pow2roundup32(uint32_t value)...
Quoted 4 lines
Naming it
pow2roundup()without specifying the integer size, I'd directly use auint64_targument, and:x |= (x >> 32);
Make sense, will do.
Quoted 8 lines
include/qemu/cutils.h bool test_buffer_is_zero_next_accel(void); · line 177
Quoted 5 lines
include/qemu/cutils.h +3
-0+ return x + 1; +} +#endif Naming it
pow2roundup()in "qemu/host-utils.h" (regardless the arg size):Reviewed-by Philippe Mathieu-Daudé <[email protected]>
Thanks.
Quoted 4 lines
Regards,
Phil.
PATCH v2 2/5 mem: add share parameter to memory-backend-ram
In reply to Marcel ApfelbaumCurrently only file backed memory backend can be created with a "share" flag in order to allow sharing guest RAM with other processes in the host.
Add the "share" flag also to RAM Memory Backend
in order to allow remapping parts of the guest RAM to different host virtual addresses. This is needed by the RDMA devices in order to remap non-contiguous QEMU virtual addresses to a contiguous virtual address range.
Moved the "share" flag to the Host Memory base class, modified phys_mem_alloc to include the new parameter and a new interface memory_region_init_ram_shared_nomigrate.
There are no functional changes if the new flag is not used.
--- backends/hostmem-file.c | 25 +------------------------ backends/hostmem-ram.c | 4 ++-- backends/hostmem.c | 21 +++++++++++++++++++++ exec.c | 26 +++++++++++++++----------- include/exec/memory.h | 23 +++++++++++++++++++++++ include/exec/ram_addr.h | 3 ++- include/qemu/osdep.h | 2 +- include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 2 +- memory.c | 16 +++++++++++++--- util/oslib-posix.c | 4 ++-- util/oslib-win32.c | 2 +- 12 files changed, 83 insertions(+), 47 deletions(-)
backends/hostmem-file.c +1 -24
@@ -31,7 +31,6 @@ typedef struct HostMemoryBackendFile HostMemoryBackendFile;struct HostMemoryBackendFile {HostMemoryBackend parent_obj; -bool share;bool discard_data;char *mem_path; };
@@ -58,7 +57,7 @@ file_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) path =object_get_canonical_path (OBJECT (backend));memory_region_init_ram_from_file (&backend->mr,OBJECT (backend), path, - backend->size, fb->share, + backend->size, backend->share, fb->mem_path, errp);g_free (path); }
@@ -85,25 +84,6 @@ static void set_mem_path(Object *o, const char *str, Error **errp) fb->mem_path =g_strdup (str); } -static bool file_memory_backend_get_share (Object *o,Error **errp) -{ -HostMemoryBackendFile *fb =MEMORY_BACKEND_FILE (o); - -return fb->share; -} - -static void file_memory_backend_set_share (Object *o,bool value,Error **errp) -{ -HostMemoryBackend *backend =MEMORY_BACKEND (o); -HostMemoryBackendFile *fb =MEMORY_BACKEND_FILE (o); - -if (host_memory_backend_mr_inited (backend)) { -error_setg (errp,"cannot change property value" ); -return ; - } - fb->share = value; -} -static bool file_memory_backend_get_discard_data (Object *o,Error **errp) {return MEMORY_BACKEND_FILE (o)->discard_data;
@@ -136,9 +116,6 @@ file_backend_class_init(ObjectClass *oc, void *data) bc->alloc = file_backend_memory_alloc; oc->unparent = file_backend_unparent; -object_class_property_add_bool (oc,"share" , - file_memory_backend_get_share, file_memory_backend_set_share, - &error_abort);object_class_property_add_bool (oc,"discard-data" , file_memory_backend_get_discard_data, file_memory_backend_set_discard_data, &error_abort);
backends/hostmem-ram.c +2 -2
@@ -28,8 +28,8 @@ ram_backend_memory_alloc(HostMemoryBackend *backend, Error **errp) } path =object_get_canonical_path_component (OBJECT (backend)); -memory_region_init_ram_nomigrate (&backend->mr,OBJECT (backend), path, - backend->size, errp); +memory_region_init_ram_shared_nomigrate (&backend->mr,OBJECT (backend), path, + backend->size, backend->share, errp);g_free (path); }
backends/hostmem.c +21 -0
@@ -369,6 +369,24 @@ static void set_id(Object *o, const char *str, Error **errp) backend->id =g_strdup (str); } +static bool host_memory_backend_get_share (Object *o,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + +return backend->share; +} + +static void host_memory_backend_set_share (Object *o,bool value,Error **errp) +{ +HostMemoryBackend *backend =MEMORY_BACKEND (o); + +if (host_memory_backend_mr_inited (backend)) { +error_setg (errp,"cannot change property value" ); +return ; + } + backend->share = value; +} +static void host_memory_backend_class_init (ObjectClass *oc,void *data) {
@@ -399,6 +417,9 @@ host_memory_backend_class_init(ObjectClass *oc, void *data) host_memory_backend_get_policy, host_memory_backend_set_policy, &error_abort);object_class_property_add_str (oc,"id" , get_id, set_id, &error_abort); +object_class_property_add_bool (oc,"share" , + host_memory_backend_get_share, host_memory_backend_set_share, + &error_abort); }static void host_memory_backend_finalize (Object *o)
exec.c +15 -11
@@ -1273,7 +1273,7 @@ static int subpage_register (subpage_t *mmio, uint32_t start, uint32_t end, uint16_t section);static subpage_t *subpage_init (FlatView *fv,hwaddr base); -static void *(*phys_mem_alloc)(size_t size,uint64_t *align) = +static void *(*phys_mem_alloc)(size_t size,uint64_t *align,bool shared) = qemu_anon_ram_alloc;/*
@@ -1281,7 +1281,7 @@ static void *(*phys_mem_alloc)(size_t size, uint64_t *align) =* Accelerators with unusual needs may need this. Hopefully, we can * get rid of it eventually. */ -void phys_mem_set_alloc (void *(*alloc)(size_t ,uint64_t *align)) +void phys_mem_set_alloc (void *(*alloc)(size_t ,uint64_t *align,bool shared)) { phys_mem_alloc = alloc; }
@@ -1884,7 +1884,7 @@ static void dirty_memory_extend(ram_addr_t old_ram_size, } } -static void ram_block_add (RAMBlock *new_block,Error **errp) +static void ram_block_add (RAMBlock *new_block,Error **errp,bool shared) {RAMBlock *block;RAMBlock *last_block =NULL ;
@@ -1907,7 +1907,7 @@ static void ram_block_add(RAMBlock *new_block, Error **errp) } }else { new_block->host =phys_mem_alloc (new_block->max_length, - &new_block->mr->align); + &new_block->mr->align, shared);if (!new_block->host) { error_setg_errno(errp, errno, "cannot set up guest memory '%s'",
@@ -2012,7 +2012,7 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,return NULL ; } -ram_block_add (new_block, &local_err); +ram_block_add (new_block, &local_err, share);if (local_err) {g_free (new_block);error_propagate (errp, local_err);
@@ -2054,7 +2054,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,void (*resized)(const char *,uint64_t length,void *host), - void *host, bool resizeable, + void *host, bool resizeable, bool share, MemoryRegion *mr, Error **errp) { RAMBlock *new_block;
@@ -2077,7 +2077,7 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,if (resizeable) { new_block->flags |=RAM_RESIZEABLE ; } -ram_block_add (new_block, &local_err); +ram_block_add (new_block, &local_err, share);if (local_err) {g_free (new_block);error_propagate (errp, local_err);
@@ -2089,12 +2089,15 @@ RAMBlock *qemu_ram_alloc_internal(ram_addr_t size, ram_addr_t max_size,RAMBlock *qemu_ram_alloc_from_ptr (ram_addr_t size,void *host,MemoryRegion *mr,Error **errp) { -return qemu_ram_alloc_internal (size, size,NULL , host, false, mr, errp); +return qemu_ram_alloc_internal (size, size,NULL , host, false, + false, mr, errp); } -RAMBlock *qemu_ram_alloc (ram_addr_t size,MemoryRegion *mr,Error **errp) +RAMBlock *qemu_ram_alloc (ram_addr_t size,bool share, +MemoryRegion *mr,Error **errp) { -return qemu_ram_alloc_internal (size, size,NULL ,NULL , false, mr, errp); +return qemu_ram_alloc_internal (size, size,NULL ,NULL , false, + share, mr, errp); } RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz,
@@ -2103,7 +2106,8 @@ RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t maxsz, void *host), MemoryRegion *mr, Error **errp) { - return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, mr, errp); + return qemu_ram_alloc_internal(size, maxsz, resized, NULL, true, +false ,mr ,errp ); }static void reclaim_ramblock (RAMBlock *block)
include/exec/memory.h +23 -0
@@ -428,6 +428,29 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr, Error **errp);/** +* memory_region_init_ram_shared_nomigrate: Initialize RAM memory region. +* Accesses into the region will +* modify memory directly. +* +* @mr: the #MemoryRegion to be initialized. +* @owner: the object that tracks the region's reference count +* @name: Region name, becomes part of RAMBlock name used in migration stream +* must be unique within any device +* @size: size of the region. +* @share: allow remapping RAM to different addresses +* @errp: pointer to Error*, to store an error if it happens. +* +* Note that this function is similar to memory_region_init_ram_nomigrate. +* The only difference is part of the RAM region can be remapped. +*/ +void memory_region_init_ram_shared_nomigrate (MemoryRegion *mr, +struct Object *owner, +const char *name, +uint64_t size, +bool share, +Error **errp); + +/** * memory_region_init_resizeable_ram: Initialize memory region with resizeable * RAM. Accesses into the region will * modify memory directly. Only an initial
include/exec/ram_addr.h +2 -1
@@ -80,7 +80,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr, Error **errp);RAMBlock *qemu_ram_alloc_from_ptr (ram_addr_t size,void *host,MemoryRegion *mr,Error **errp); -RAMBlock *qemu_ram_alloc (ram_addr_t size,MemoryRegion *mr,Error **errp); +RAMBlock *qemu_ram_alloc (ram_addr_t size,bool share,MemoryRegion *mr, +Error **errp); RAMBlock *qemu_ram_alloc_resizeable(ram_addr_t size, ram_addr_t max_size, void (*resized)(const char*, uint64_t length,
include/qemu/osdep.h +1 -1
@@ -228,7 +228,7 @@ extern int daemon(int, int);int qemu_daemon (int nochdir,int noclose);void *qemu_try_memalign (size_t alignment,size_t size);void *qemu_memalign (size_t alignment,size_t size); -void *qemu_anon_ram_alloc (size_t size,uint64_t *align); +void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared);void qemu_vfree (void *ptr);void qemu_anon_ram_free (void *ptr,size_t size);
include/sysemu/hostmem.h +1 -1
@@ -55,7 +55,7 @@ struct HostMemoryBackend {char *id;uint64_t size;bool merge, dump; -bool prealloc, force_prealloc, is_mapped; +bool prealloc, force_prealloc, is_mapped, share;DECLARE_BITMAP (host_nodes,MAX_NODES +1 );HostMemPolicy policy;
include/sysemu/kvm.h +1 -1
@@ -248,7 +248,7 @@ int kvm_on_sigbus(int code, void *addr);/* interface with exec.c */ -void phys_mem_set_alloc (void *(*alloc)(size_t ,uint64_t *align)); +void phys_mem_set_alloc (void *(*alloc)(size_t ,uint64_t *align,bool shared));/* internal API */
memory.c +13 -3
@@ -1538,11 +1538,21 @@ void memory_region_init_ram_nomigrate(MemoryRegion *mr,uint64_t size, Error **errp) { +memory_region_init_ram_shared_nomigrate (mr ,owner ,name ,size ,false ,errp ); +} + +void memory_region_init_ram_shared_nomigrate (MemoryRegion *mr, +Object *owner, +const char *name, +uint64_t size, +bool share, +Error **errp) +{memory_region_init (mr, owner, name, size); mr->ram = true; mr->terminates = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc (size, mr, errp); + mr->ram_block =qemu_ram_alloc (size, share, mr, errp); mr->dirty_log_mask =tcg_enabled () ? (1 <<DIRTY_MEMORY_CODE ) :0 ; }
@@ -1651,7 +1661,7 @@ void memory_region_init_rom_nomigrate(MemoryRegion *mr, mr->readonly = true; mr->terminates = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc (size, mr, errp); + mr->ram_block =qemu_ram_alloc (size, false, mr, errp); mr->dirty_log_mask =tcg_enabled () ? (1 <<DIRTY_MEMORY_CODE ) :0 ; }
@@ -1670,7 +1680,7 @@ void memory_region_init_rom_device_nomigrate(MemoryRegion *mr, mr->terminates = true; mr->rom_device = true; mr->destructor = memory_region_destructor_ram; - mr->ram_block =qemu_ram_alloc (size, mr, errp); + mr->ram_block =qemu_ram_alloc (size, false, mr, errp); } void memory_region_init_iommu(void *_iommu_mr,
util/oslib-posix.c +2 -2
@@ -127,10 +127,10 @@ void *qemu_memalign(size_t alignment, size_t size) }/* alloc shared memory pages */ -void *qemu_anon_ram_alloc (size_t size,uint64_t *alignment) +void *qemu_anon_ram_alloc (size_t size,uint64_t *alignment,bool shared) {size_t align =QEMU_VMALLOC_ALIGN ; -void *ptr =qemu_ram_mmap (-1 , size, align, false); +void *ptr =qemu_ram_mmap (-1 , size, align, shared);if (ptr ==MAP_FAILED ) {return NULL ;
util/oslib-win32.c +1 -1
@@ -67,7 +67,7 @@ void *qemu_memalign(size_t alignment, size_t size)return qemu_oom_check (qemu_try_memalign (alignment, size)); } -void *qemu_anon_ram_alloc (size_t size,uint64_t *align) +void *qemu_anon_ram_alloc (size_t size,uint64_t *align,bool shared) {void *ptr;
Signature · 1 line
2.13.5
PATCH v2 3/5 docs: add pvrdma device documentation
In reply to Marcel Apfelbaum--- docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 docs/pvrdma.txt
docs/pvrdma.txt new file +145 -0
@@ -0,0 +1,145 @@ +Paravirtualized RDMA Device (PVRDMA) +==================================== + + +1. Description +=============== +PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. +It works with its Linux Kernel driver AS IS, no need for any special guest +modifications. + +While it complies with the VMware device, it can also communicate with bare +metal RDMA-enabled machines and does not require an RDMA HCA in the host, it +can work with Soft-RoCE (rxe). + +It does not require the whole guest RAM to be pinned allowing memory +over-commit and, even if not implemented yet, migration support will be +possible with some HW assistance. + +A project presentation accompany this document: +- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf + + + +2. Setup +======== + + +2.1 Guest setup +=============== +Fedora 27+ kernels work out of the box, older distributions +require updating the kernel to 4.14 to include the pvrdma driver. + +However the libpvrdma library needed by User Level Software is still +not available as part of the distributions, so the rdma-core library +needs to be compiled and optionally installed. + +Please follow the instructions at: + https://github.com/linux-rdma/rdma-core.git + + +2.2 Host Setup +============== +The pvrdma backend is an ibdevice interface that can be exposed +either by a Soft-RoCE(rxe) device on machines with no RDMA device, +or an HCA SRIOV function(VF/PF). +Note that ibdevice interfaces can't be shared between pvrdma devices, +each one requiring a separate instance (rxe or SRIOV VF). + + +2.2.1 Soft-RoCE backend(rxe) +=========================== +A stable version of rxe is required, Fedora 27+ or a Linux +Kernel 4.14+ is preferred. + +The rdma_rxe module is part of the Linux Kernel but not loaded by default. +Install the User Level library (librxe) following the instructions from: +https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home + +Associate an ETH interface with rxe by running: + rxe_cfg add eth0 +An rxe0 ibdevice interface will be created and can be used as pvrdma backend. + + +2.2.2 RDMA device Virtual Function backend +========================================== +Nothing special is required, the pvrdma device can work not only with +Ethernet Links, but also Infinibands Links. +All is needed is an ibdevice with an active port, for Mellanox cards +will be something like mlx5_6 which can be the backend. + + +2.2.3 QEMU setup +================ +Configure QEMU with --enable-rdma flag, installing +the required RDMA libraries. + + +3. Usage +======== +Currently the device is working only with memory backed RAM +and it must be mark as "shared": + -m 1G \ + -object memory-backend-ram,id=mb1,size=1G,share \ + -numa node,memdev=mb1 \ + +The pvrdma device is composed of two functions: + - Function 0 is a vmxnet Ethernet Device which is redundant in Guest + but is required to pass the ibdevice GID using its MAC. + Examples: + For an rxe backend using eth0 interface it will use its mac: + -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC> + For an SRIOV VF, we take the Ethernet Interface exposed by it: + -device vmxnet3,multifunction=on,mac=<RoCE eth MAC> + - Function 1 is the actual device: + -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port> + where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) + Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC. + The rules of conversion are part of the RoCE spec, but since manual conversion + is not required, spotting problems is not hard: + Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a + MAC: 7c:fe:90:cb:74:3a + Note the difference between the first byte of the MAC and the GID. + + +4. Implementation details +========================= +The device acts like a proxy between the Guest Driver and the host +ibdevice interface. +On configuration path: + - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request + a resource from the backend interface, maintaining a 1-1 mapping + between the guest and host. +On data path: + - Every post_send/receive received from the guest will be converted into + a post_send/receive for the backend. The buffers data will not be touched + or copied resulting in near bare-metal performance for large enough buffers. + - Completions from the backend interface will result in completions for + the pvrdma device. + + + +5. Limitations +============== +- The device obviously is limited by the Guest Linux Driver features implementation + of the VMware device API. +- Memory registration mechanism requires mremap for every page in the buffer in order + to map it to a contiguous virtual address range. Since this is not the data path + it should not matter much. +- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached, + so it can't work with huge pages. The limitation will be addressed in the future, + however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge + pages available, QEMU will use them. +- As previously stated, migration is not supported yet, however with some hardware + support can be done. + + + +6. Performance +============== +By design the pvrdma device exits on each post-send/receive, so for small buffers +the performance is affected; however for medium buffers it will became close to +bare metal and from 1MB buffers and up it reaches bare metal performance. +(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device) + +All the above assumes no memory registration is done on data path.
Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 3/5] docs: add pvrdma device documentation
Quoted 121 lines · Marcel Apfelbaum
Signed-off-by Marcel Apfelbaum <[email protected]>Signed-off-by Yuval Shaia <[email protected]>--- docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 docs/pvrdma.txtdocs/pvrdma.txt new file +108
-0@@ -0,0 +1,145 @@ +Paravirtualized RDMA Device (PVRDMA) +==================================== + + +1. Description +=============== +PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. +It works with its Linux Kernel driver AS IS, no need for any special guest +modifications. + +While it complies with the VMware device, it can also communicate with bare +metal RDMA-enabled machines and does not require an RDMA HCA in the host, it +can work with Soft-RoCE (rxe). + +It does not require the whole guest RAM to be pinned allowing memory +over-commit and, even if not implemented yet, migration support will be +possible with some HW assistance. + +A project presentation accompany this document: +- http://events.linuxfoundation.org/sites/events/files/slides/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf + + + +2. Setup +======== + + +2.1 Guest setup +=============== +Fedora 27+ kernels work out of the box, older distributions +require updating the kernel to 4.14 to include the pvrdma driver. + +However the libpvrdma library needed by User Level Software is still +not available as part of the distributions, so the rdma-core library +needs to be compiled and optionally installed. + +Please follow the instructions at: + https://github.com/linux-rdma/rdma-core.git + + +2.2 Host Setup +============== +The pvrdma backend is an ibdevice interface that can be exposed +either by a Soft-RoCE(rxe) device on machines with no RDMA device, +or an HCA SRIOV function(VF/PF). +Note that ibdevice interfaces can't be shared between pvrdma devices, +each one requiring a separate instance (rxe or SRIOV VF). + + +2.2.1 Soft-RoCE backend(rxe) +=========================== +A stable version of rxe is required, Fedora 27+ or a Linux +Kernel 4.14+ is preferred. + +The rdma_rxe module is part of the Linux Kernel but not loaded by default. +Install the User Level library (librxe) following the instructions from: +https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home + +Associate an ETH interface with rxe by running: + rxe_cfg add eth0 +An rxe0 ibdevice interface will be created and can be used as pvrdma backend. + + +2.2.2 RDMA device Virtual Function backend +========================================== +Nothing special is required, the pvrdma device can work not only with +Ethernet Links, but also Infinibands Links. +All is needed is an ibdevice with an active port, for Mellanox cards +will be something like mlx5_6 which can be the backend. + + +2.2.3 QEMU setup +================ +Configure QEMU with --enable-rdma flag, installing +the required RDMA libraries. + + +3. Usage +======== +Currently the device is working only with memory backed RAM +and it must be mark as "shared": + -m 1G \ + -object memory-backend-ram,id=mb1,size=1G,share \ + -numa node,memdev=mb1 \ + +The pvrdma device is composed of two functions: + - Function 0 is a vmxnet Ethernet Device which is redundant in Guest + but is required to pass the ibdevice GID using its MAC. + Examples: + For an rxe backend using eth0 interface it will use its mac: + -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC> + For an SRIOV VF, we take the Ethernet Interface exposed by it: + -device vmxnet3,multifunction=on,mac=<RoCE eth MAC> + - Function 1 is the actual device: + -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port> + where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) + Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC. + The rules of conversion are part of the RoCE spec, but since manual conversion + is not required, spotting problems is not hard: + Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a + MAC: 7c:fe:90:cb:74:3a + Note the difference between the first byte of the MAC and the GID. + + +4. Implementation details +========================= +The device acts like a proxy between the Guest Driver and the host +ibdevice interface.
docs/pvrdma.txt line 118
Quoted 10 lines
docs/pvrdma.txt new file +10
-0+On configuration path: + - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request + a resource from the backend interface, maintaining a 1-1 mapping + between the guest and host. +On data path: + - Every post_send/receive received from the guest will be converted into + a post_send/receive for the backend. The buffers data will not be touched + or copied resulting in near bare-metal performance for large enough buffers. + - Completions from the backend interface will result in completions for + the pvrdma device.
Where's the host/guest interface documented?
Quoted 28 lines
docs/pvrdma.txt new file +26
-0+ + +5. Limitations +============== +- The device obviously is limited by the Guest Linux Driver features implementation + of the VMware device API. +- Memory registration mechanism requires mremap for every page in the buffer in order + to map it to a contiguous virtual address range. Since this is not the data path + it should not matter much. +- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached, + so it can't work with huge pages. The limitation will be addressed in the future, + however QEMU allocates Gust RAM with MADV_HUGEPAGE so if there are enough huge + pages available, QEMU will use them. +- As previously stated, migration is not supported yet, however with some hardware + support can be done. + + + +6. Performance +============== +By design the pvrdma device exits on each post-send/receive, so for small buffers +the performance is affected; however for medium buffers it will became close to +bare metal and from 1MB buffers and up it reaches bare metal performance. +(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device) + +All the above assumes no memory registration is done on data path.Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 3/5] docs: add pvrdma device documentation
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Marcel Apfelbaum
Signed-off-by Marcel Apfelbaum <[email protected]>Signed-off-by Yuval Shaia <[email protected]>--- docs/pvrdma.txt | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+)create mode 100644
docs/pvrdma.txtdocs/pvrdma.txt new file +3
-0@@ -0,0 +1,145 @@ +Paravirtualized RDMA Device (PVRDMA) +==================================== +
[...]
Quoted 7 lines
Quoted 7 lines
docs/pvrdma.txt new file +7
-0+ +4. Implementation details +========================= +The device acts like a proxy between the Guest Driver and the host +ibdevice interface. +On configuration path: + - For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
Quoted 9 lines
docs/pvrdma.txt line 118
Quoted 8 lines
docs/pvrdma.txt new file +8
-0+ a resource from the backend interface, maintaining a 1-1 mapping + between the guest and host. +On data path: + - Every post_send/receive received from the guest will be converted into + a post_send/receive for the backend. The buffers data will not be touched + or copied resulting in near bare-metal performance for large enough buffers. + - Completions from the backend interface will result in completions for + the pvrdma device.
Hi Michael,
Quoted 3 lines
Where's the host/guest interface documented?
It is the VMware PVRDMA spec, I am not sure is publicly available, we kind of reverse-engineered it.
We will add some info from the linked presentation on the PCI BARs and how are they used.
Thanks,
Marcel
Quoted 2 lines
Quoted 2 lines
docs/pvrdma.txt new file +2
-0+ +
[...]
PATCH v2 4/5 pvrdma: initial implementation
In reply to Marcel ApfelbaumFrom: Yuval Shaia <[email protected]>
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.
--- Makefile.objs | 1 + configure | 9 +- default-configs/arm-softmmu.mak | 2 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/net/Makefile.objs | 7 + hw/net/pvrdma/pvrdma.h | 179 +++++++ hw/net/pvrdma/pvrdma_backend.c | 986 ++++++++++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_backend.h | 74 +++ hw/net/pvrdma/pvrdma_backend_defs.h | 68 +++ hw/net/pvrdma/pvrdma_cmd.c | 338 ++++++++++++ hw/net/pvrdma/pvrdma_defs.h | 121 +++++ hw/net/pvrdma/pvrdma_dev_api.h | 580 +++++++++++++++++++++ hw/net/pvrdma/pvrdma_dev_ring.c | 138 +++++ hw/net/pvrdma/pvrdma_dev_ring.h | 42 ++ hw/net/pvrdma/pvrdma_ib_verbs.h | 399 +++++++++++++++ hw/net/pvrdma/pvrdma_main.c | 664 ++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_qp_ops.c | 187 +++++++ hw/net/pvrdma/pvrdma_qp_ops.h | 26 + hw/net/pvrdma/pvrdma_ring.h | 134 +++++ hw/net/pvrdma/pvrdma_rm.c | 791 +++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_rm.h | 54 ++ hw/net/pvrdma/pvrdma_rm_defs.h | 111 ++++ hw/net/pvrdma/pvrdma_types.h | 37 ++ hw/net/pvrdma/pvrdma_utils.c | 133 +++++ hw/net/pvrdma/pvrdma_utils.h | 41 ++ hw/net/pvrdma/trace-events | 9 + include/hw/pci/pci_ids.h | 3 + 28 files changed, 5132 insertions(+), 4 deletions(-) create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-events
Makefile.objs +1 -0
@@ -129,6 +129,7 @@ trace-events-subdirs += hw/block/dataplane trace-events-subdirs += hw/char trace-events-subdirs += hw/intc trace-events-subdirs += hw/net +trace-events-subdirs += hw/net/pvrdma trace-events-subdirs += hw/virtio trace-events-subdirs += hw/audio trace-events-subdirs += hw/misc
configure +5 -4
@@ -1523,7 +1523,7 @@ disabled with --disable-FEATURE, default is enabled if available: bluez bluez stack connectivity kvm KVM acceleration support hax HAX acceleration support - rdma RDMA-based migration support + rdma Enable RDMA-based migration support and PVRDMA vde support for vde network netmap support for netmap network linux-aio Linux AIO support
@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then #include <rdma/rdma_cma.h> int main(void) { return 0; } EOF - rdma_libs="-lrdmacm -libverbs" + rdma_libs="-lrdmacm -libverbs -libumad" if compile_prog "" "$rdma_libs" ; then rdma="yes" else + libs_softmmu="$libs_softmmu $rdma_libs" if test "$rdma" = "yes" ; then error_exit \ - " OpenFabrics librdmacm/libibverbs not present." \ + " OpenFabrics librdmacm/libibverbs/libibumad not present." \ " Your options:" \ - " (1) Fast: Install infiniband packages from your distro." \ + " (1) Fast: Install infiniband packages (devel) from your distro." \ " (2) Cleanest: Install libraries from www.openfabrics.org" \ " (3) Also: Install softiwarp if you don't have RDMA hardware" fi
default-configs/arm-softmmu.mak +2 -0
@@ -132,3 +132,5 @@ CONFIG_GPIO_KEY=y CONFIG_MSF2=y CONFIG_FW_CFG_DMA=y + +CONFIG_PVRDMA=y
default-configs/i386-softmmu.mak +1 -0
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) CONFIG_PXB=y CONFIG_ACPI_VMGENID=y CONFIG_FW_CFG_DMA=y +CONFIG_PVRDMA=y
default-configs/x86_64-softmmu.mak +1 -0
@@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) CONFIG_PXB=y CONFIG_ACPI_VMGENID=y CONFIG_FW_CFG_DMA=y +CONFIG_PVRDMA=y
hw/net/Makefile.objs +7 -0
@@ -46,3 +46,10 @@ common-obj-$(CONFIG_ROCKER) += rocker/rocker.o rocker/rocker_fp.o \ rocker/rocker_desc.o rocker/rocker_world.o \ rocker/rocker_of_dpa.o obj-$(call lnot,$(CONFIG_ROCKER)) += rocker/qmp-norocker.o + +ifeq ($(CONFIG_RDMA),y) +obj-$(CONFIG_PVRDMA) += pvrdma/pvrdma_dev_ring.o pvrdma/pvrdma_rm.o \ + pvrdma/pvrdma_utils.o pvrdma/pvrdma_qp_ops.o \ + pvrdma/pvrdma_backend.o pvrdma/pvrdma_cmd.o \ + pvrdma/pvrdma_main.o +endif
hw/net/pvrdma/pvrdma.h new file +179 -0
179 lines not on this page · raw message
hw/net/pvrdma/pvrdma_backend.c new file +986 -0
986 lines not on this page · raw message
hw/net/pvrdma/pvrdma_backend.h new file +74 -0
74 lines not on this page · raw message
hw/net/pvrdma/pvrdma_backend_defs.h new file +68 -0
68 lines not on this page · raw message
hw/net/pvrdma/pvrdma_cmd.c new file +338 -0
338 lines not on this page · raw message
hw/net/pvrdma/pvrdma_defs.h new file +121 -0
121 lines not on this page · raw message
hw/net/pvrdma/pvrdma_dev_api.h new file +580 -0
580 lines not on this page · raw message
hw/net/pvrdma/pvrdma_dev_ring.c new file +138 -0
138 lines not on this page · raw message
hw/net/pvrdma/pvrdma_dev_ring.h new file +42 -0
42 lines not on this page · raw message
hw/net/pvrdma/pvrdma_ib_verbs.h new file +399 -0
399 lines not on this page · raw message
hw/net/pvrdma/pvrdma_main.c new file +664 -0
18 lines not on this page · raw message
hw/net/pvrdma/pvrdma_qp_ops.c new file +187 -0
187 lines not on this page · raw message
hw/net/pvrdma/pvrdma_qp_ops.h new file +26 -0
Lines of this file are not shown. View the raw message.
hw/net/pvrdma/pvrdma_ring.h new file +134 -0
134 lines not on this page · raw message
hw/net/pvrdma/pvrdma_rm.c new file +791 -0
791 lines not on this page · raw message
hw/net/pvrdma/pvrdma_rm.h new file +54 -0
54 lines not on this page · raw message
hw/net/pvrdma/pvrdma_rm_defs.h new file +111 -0
111 lines not on this page · raw message
hw/net/pvrdma/pvrdma_utils.c new file +133 -0
133 lines not on this page · raw message
hw/net/pvrdma/pvrdma_utils.h new file +41 -0
41 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.13.5
Part of the diff is not shown here. View the raw message.
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 10 lines · Marcel Apfelbaum
configure +2
-1@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then #include <rdma/rdma_cma.h> int main(void) { return 0; } EOF - rdma_libs="-lrdmacm -libverbs" + rdma_libs="-lrdmacm -libverbs -libumad" if compile_prog "" "$rdma_libs" ; then rdma="yes" else + libs_softmmu="$libs_softmmu $rdma_libs"
configure if test "$rdma" != "no" ; then · line 2862
Quoted 10 lines
configure +2
-2if test "$rdma" = "yes" ; then error_exit \ - " OpenFabrics librdmacm/libibverbs not present." \ + " OpenFabrics librdmacm/libibverbs/libibumad not present." \ " Your options:" \ - " (1) Fast: Install infiniband packages from your distro." \ + " (1) Fast: Install infiniband packages (devel) from your distro." \ " (2) Cleanest: Install libraries from www.openfabrics.org" \ " (3) Also: Install softiwarp if you don't have RDMA hardware" fi
Some whitespace damage here.
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Marcel Apfelbaum
configure +3
-2@@ -2847,15 +2847,16 @@ if test "$rdma" != "no" ; then #include <rdma/rdma_cma.h> int main(void) { return 0; } EOF - rdma_libs="-lrdmacm -libverbs" + rdma_libs="-lrdmacm -libverbs -libumad" if compile_prog "" "$rdma_libs" ; then rdma="yes" else + libs_softmmu="$libs_softmmu $rdma_libs" if test "$rdma" = "yes" ; then error_exit \ - " OpenFabrics librdmacm/libibverbs not present." \ + " OpenFabrics librdmacm/libibverbs/libibumad not present." \
Quoted 9 lines
configure if test "$rdma" != "no" ; then · line 2862
Quoted 6 lines
configure +1
-1" Your options:" \ - " (1) Fast: Install infiniband packages from your distro." \ + " (1) Fast: Install infiniband packages (devel) from your distro." \ " (2) Cleanest: Install libraries from www.openfabrics.org" \ " (3) Also: Install softiwarp if you don't have RDMA hardware" fiSome whitespace damage here.
Thanks Michael, we will take care of that and go over the code again to spot similar issues.
Marcel
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
In reply to Marcel ApfelbaumI won't have time to review before the next year.
Results of a quick peek.
Quoted 12 lines · Marcel Apfelbaum
hw/net/pvrdma/pvrdma_backend.c new file +12
-0+static void *mad_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +QObject *o_ctx_id; +unsigned long cqe_ctx_id; +BackendCtx *bctx; +/* +int len; +void *mad; +*/ +
hw/net/pvrdma/pvrdma_backend.c line 86
Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10
-0+pr_dbg ("Starting\n" ); + + dev->backend_dev.mad_thread.run = false; + +while (dev->backend_dev.mad_thread.run) { +/* Get next buffer to pust MAD into */ + o_ctx_id =qlist_pop (dev->backend_dev.mad_agent.recv_mads_list); +if (!o_ctx_id) { +/* pr_dbg("Error: No more free MADs buffers\n"); */ +sleep (5 );
Looks suspicious. What is above doing?
Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c new file +8
-0+continue ; + } + cqe_ctx_id =qnum_get_uint (qobject_to_qnum (o_ctx_id)); + bctx =rm_get_cqe_ctx (dev, cqe_ctx_id); +if (unlikely (!bctx)) { +pr_dbg ("Error: Fail to find ctx for %ld\n" , cqe_ctx_id); +continue ; + }
hw/net/pvrdma/pvrdma_backend.c line 104
Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10
-0+ +pr_dbg ("Calling umad_recv\n" ); +/* +mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr, +bctx->req.sge[0].length); + +len = bctx->req.sge[0].length; + +do { +rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);
That's a huge timeout.
Quoted 160 lines
hw/net/pvrdma/pvrdma_backend.c new file +160
-0+} while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run); +pr_dbg("umad_recv, rc=%d\n", rc); + +pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length); +*/ + rc =-1 ; + +/* rc is used as vendor_err */ +comp_handler (rc >0 ?IB_WC_SUCCESS :IB_WC_GENERAL_ERR , rc, + bctx->up_ctx); + +rm_dealloc_cqe_ctx (dev, cqe_ctx_id); +free (bctx); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining MADs in list */ + +qlist_destroy_obj (QOBJECT (dev->backend_dev.mad_agent.recv_mads_list)); + +return NULL ; +} + +static void *comp_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +struct ibv_cq *ev_cq; +void *ev_ctx; + +pr_dbg ("Starting\n" ); + +while (dev->backend_dev.comp_thread.run) { +pr_dbg ("Waiting for completion on channel %p\n" , + dev->backend_dev.channel); + rc =ibv_get_cq_event (dev->backend_dev.channel, &ev_cq, &ev_ctx); +pr_dbg ("ibv_get_cq_event=%d\n" , rc); +if (unlikely (rc)) { +pr_dbg ("---> ibv_get_cq_event (%d)\n" , rc); +continue ; + } + +if (unlikely (ibv_req_notify_cq (ev_cq,0 ))) { +pr_dbg ("---> ibv_req_notify_cq\n" ); + } + +poll_cq (dev, ev_cq, false); + +ibv_ack_cq_events (ev_cq,1 ); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining buffs that were posted */ + +return NULL ; +} + +void backend_register_comp_handler (void (*handler)(int status, +unsigned int vendor_err,void *ctx)) +{ + comp_handler = handler; +} + +int backend_query_port (BackendDevice *dev,struct pvrdma_port_attr *attrs) +{ +int rc; +struct ibv_port_attr port_attr; + + rc =ibv_query_port (dev->context, dev->port_num, &port_attr); +if (rc) { +pr_dbg ("Error %d from ibv_query_port\n" , rc); +return -EIO ; + } + + attrs->state = port_attr.state; + attrs->max_mtu = port_attr.max_mtu; + attrs->active_mtu = port_attr.active_mtu; + attrs->gid_tbl_len = port_attr.gid_tbl_len; + attrs->pkey_tbl_len = port_attr.pkey_tbl_len; + attrs->phys_state = port_attr.phys_state; + +return 0 ; +} + +void backend_poll_cq (PVRDMADev *dev,BackendCQ *cq) +{ +poll_cq (dev, cq->ibcq, true); +} + +static GHashTable *ah_hash; + +static struct ibv_ah *create_ah (BackendDevice *dev,struct ibv_pd *pd, +union ibv_gid *dgid,uint8_t sgid_idx) +{ +GBytes *ah_key =g_bytes_new (dgid,sizeof (*dgid)); +struct ibv_ah *ah =g_hash_table_lookup (ah_hash, ah_key); + +if (ah) { +trace_create_ah_cache_hit (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + }else { +struct ibv_ah_attr ah_attr = { + .is_global =1 , + .port_num = dev->port_num, + .grh.hop_limit =1 , + }; + + ah_attr.grh.dgid = *dgid; + ah_attr.grh.sgid_index = sgid_idx; + + ah =ibv_create_ah (pd, &ah_attr); +if (ah) { +g_hash_table_insert (ah_hash, ah_key, ah); + }else { +pr_dbg ("ibv_create_ah failed for gid <%lx %lx>\n" , +be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +trace_create_ah_cache_miss (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +return ah; +} + +static void destroy_ah (gpointer data) +{ +struct ibv_ah *ah = data; +ibv_destroy_ah (ah); +} + +static void ah_cache_init (void ) +{ + ah_hash =g_hash_table_new_full (g_bytes_hash, g_bytes_equal, +NULL , destroy_ah); +} + +static int send_mad (PVRDMADev *dev,struct pvrdma_sge *sge,u32 num_sge) +{ +int ret =-1 ; + +/* +* TODO: Currently QP1 is not supported +* +PCIDevice *pci_dev = PCI_DEVICE(dev); +char mad_msg[1024]; +void *hdr, *msg; +struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg; + +umad->length = sge[0].length + sge[1].length; + +if (num_sge != 2) +return -EINVAL; + +pr_dbg("msg_len=%d\n", umad->length); + +hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length); +msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length); +
hw/net/pvrdma/pvrdma_backend.c line 274
Quoted 10 lines
hw/net/pvrdma/pvrdma_backend.c new file +10
-0+memcpy(&mad_msg[64], hdr, sge[0].length); +memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length); + +pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length); +pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length); + +ret = umad_send(dev->backend_dev.mad_agent.port_id, +dev->backend_dev.mad_agent.agent_id, +mad_msg, umad->length, 10, 10); +*/
Then what is above code doing here?
Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?
How does guest know it's missing?
...
Quoted 37 lines
hw/net/pvrdma/pvrdma_utils.h new file +31
-0@@ -0,0 +1,41 @@ +/* +* QEMU VMWARE paravirtual RDMA interface definitions +* +* Developed by Oracle & Redhat +* +* Authors: +* Yuval Shaia <[email protected]> +* Marcel Apfelbaum <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef PVRDMA_UTILS_H +#define PVRDMA_UTILS_H + +#include <include/hw/pci/pci.h> + +#define pr_info (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__) + +#define pr_err (fmt, ...) \ + fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \ + __LINE__, ## __VA_ARGS__) + +#ifdef PVRDMA_DEBUG +#define pr_dbg (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__)
hw/net/pvrdma/pvrdma_utils.h line 41
Quoted 10 lines
hw/net/pvrdma/pvrdma_utils.h new file +10
-0+#else +#define pr_dbg (fmt, ...) +#endif + +void pvrdma_pci_dma_unmap (PCIDevice *dev,void *buffer,dma_addr_t len); +void *pvrdma_pci_dma_map (PCIDevice *dev,dma_addr_t addr,dma_addr_t plen); +void *map_to_pdir (PCIDevice *pdev,uint64_t pdir_dma,uint32_t nchunks, +size_t length); + +#endif
Can you make sure all prefixes are pvrdma_?
Quoted 29 lines
hw/net/pvrdma/trace-events new file +9
-0@@ -0,0 +1,9 @@ +# See docs/tracing.txt for syntax documentation. + +# hw/net/pvrdma/pvrdma_main.c +pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" +pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" + +#hw/net/pvrdma/pvrdma_backend.c +create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx" +create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"include/hw/pci/pci_ids.h +3
-0@@ -266,4 +266,7 @@#define PCI_VENDOR_ID_TEWS 0x1498#define PCI_DEVICE_ID_TEWS_TPCI200 0x30C8 +#define PCI_VENDOR_ID_VMWARE 0x15ad +#define PCI_DEVICE_ID_VMWARE_PVRDMA 0x0820 +#endif Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 2 lines · Michael S. Tsirkin
I won't have time to review before the next year.
Results of a quick peek.
Thanks for the parts you found the time to review.
Quoted 18 lines · Marcel Apfelbaum
Quoted 16 lines
hw/net/pvrdma/pvrdma_backend.c new file +16
-0+static void *mad_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +QObject *o_ctx_id; +unsigned long cqe_ctx_id; +BackendCtx *bctx; +/* +int len; +void *mad; +*/ + +pr_dbg ("Starting\n" ); + + dev->backend_dev.mad_thread.run = false; +
Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 86
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+while (dev->backend_dev.mad_thread.run) { +/* Get next buffer to pust MAD into */ + o_ctx_id =qlist_pop (dev->backend_dev.mad_agent.recv_mads_list); +if (!o_ctx_id) { +/* pr_dbg("Error: No more free MADs buffers\n"); */ +sleep (5 );Looks suspicious. What is above doing?
This function is responsible to process incoming MAD messages.
Usual (good) flow is that guest driver prepares some buffers to be used for that purpose and gives it to device with the usual post_recv mechanism. Upon receiving a MAD message from the device the driver should pass a new buffer to device.
But what if the device didn't do it.
This section handle such case, as we have nothing to do - let's sleep and hope for the best.
Quoted 13 lines
Quoted 13 lines
hw/net/pvrdma/pvrdma_backend.c new file +12
-0+continue ; + } + cqe_ctx_id =qnum_get_uint (qobject_to_qnum (o_ctx_id)); + bctx =rm_get_cqe_ctx (dev, cqe_ctx_id); +if (unlikely (!bctx)) { +pr_dbg ("Error: Fail to find ctx for %ld\n" , cqe_ctx_id); +continue ; + } + +pr_dbg ("Calling umad_recv\n" ); +/* +mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr,
Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 104
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+ bctx->req.sge[0].length); + + len = bctx->req.sge[0 ].length; + + do { + rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);That's a huge timeout.
This is the maximum timeout.
On low MAD traffic we don't want to take much of the CPU for that purpose so 5 seconds looks good to me.
Anyway, just because it is not obvious i will make it configurable.
Quoted 165 lines
Quoted 165 lines
hw/net/pvrdma/pvrdma_backend.c new file +164
-0+} while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run); +pr_dbg("umad_recv, rc=%d\n", rc); + +pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length); +*/ + rc =-1 ; + +/* rc is used as vendor_err */ +comp_handler (rc >0 ?IB_WC_SUCCESS :IB_WC_GENERAL_ERR , rc, + bctx->up_ctx); + +rm_dealloc_cqe_ctx (dev, cqe_ctx_id); +free (bctx); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining MADs in list */ + +qlist_destroy_obj (QOBJECT (dev->backend_dev.mad_agent.recv_mads_list)); + +return NULL ; +} + +static void *comp_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +struct ibv_cq *ev_cq; +void *ev_ctx; + +pr_dbg ("Starting\n" ); + +while (dev->backend_dev.comp_thread.run) { +pr_dbg ("Waiting for completion on channel %p\n" , + dev->backend_dev.channel); + rc =ibv_get_cq_event (dev->backend_dev.channel, &ev_cq, &ev_ctx); +pr_dbg ("ibv_get_cq_event=%d\n" , rc); +if (unlikely (rc)) { +pr_dbg ("---> ibv_get_cq_event (%d)\n" , rc); +continue ; + } + +if (unlikely (ibv_req_notify_cq (ev_cq,0 ))) { +pr_dbg ("---> ibv_req_notify_cq\n" ); + } + +poll_cq (dev, ev_cq, false); + +ibv_ack_cq_events (ev_cq,1 ); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining buffs that were posted */ + +return NULL ; +} + +void backend_register_comp_handler (void (*handler)(int status, +unsigned int vendor_err,void *ctx)) +{ + comp_handler = handler; +} + +int backend_query_port (BackendDevice *dev,struct pvrdma_port_attr *attrs) +{ +int rc; +struct ibv_port_attr port_attr; + + rc =ibv_query_port (dev->context, dev->port_num, &port_attr); +if (rc) { +pr_dbg ("Error %d from ibv_query_port\n" , rc); +return -EIO ; + } + + attrs->state = port_attr.state; + attrs->max_mtu = port_attr.max_mtu; + attrs->active_mtu = port_attr.active_mtu; + attrs->gid_tbl_len = port_attr.gid_tbl_len; + attrs->pkey_tbl_len = port_attr.pkey_tbl_len; + attrs->phys_state = port_attr.phys_state; + +return 0 ; +} + +void backend_poll_cq (PVRDMADev *dev,BackendCQ *cq) +{ +poll_cq (dev, cq->ibcq, true); +} + +static GHashTable *ah_hash; + +static struct ibv_ah *create_ah (BackendDevice *dev,struct ibv_pd *pd, +union ibv_gid *dgid,uint8_t sgid_idx) +{ +GBytes *ah_key =g_bytes_new (dgid,sizeof (*dgid)); +struct ibv_ah *ah =g_hash_table_lookup (ah_hash, ah_key); + +if (ah) { +trace_create_ah_cache_hit (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + }else { +struct ibv_ah_attr ah_attr = { + .is_global =1 , + .port_num = dev->port_num, + .grh.hop_limit =1 , + }; + + ah_attr.grh.dgid = *dgid; + ah_attr.grh.sgid_index = sgid_idx; + + ah =ibv_create_ah (pd, &ah_attr); +if (ah) { +g_hash_table_insert (ah_hash, ah_key, ah); + }else { +pr_dbg ("ibv_create_ah failed for gid <%lx %lx>\n" , +be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +trace_create_ah_cache_miss (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +return ah; +} + +static void destroy_ah (gpointer data) +{ +struct ibv_ah *ah = data; +ibv_destroy_ah (ah); +} + +static void ah_cache_init (void ) +{ + ah_hash =g_hash_table_new_full (g_bytes_hash, g_bytes_equal, +NULL , destroy_ah); +} + +static int send_mad (PVRDMADev *dev,struct pvrdma_sge *sge,u32 num_sge) +{ +int ret =-1 ; + +/* +* TODO: Currently QP1 is not supported +* +PCIDevice *pci_dev = PCI_DEVICE(dev); +char mad_msg[1024]; +void *hdr, *msg; +struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg; + +umad->length = sge[0].length + sge[1].length; + +if (num_sge != 2) +return -EINVAL; + +pr_dbg("msg_len=%d\n", umad->length); + +hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length); +msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length); + +memcpy(&mad_msg[64], hdr, sge[0].length); +memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length); + +pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length);
Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 274
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length); + +ret = umad_send(dev->backend_dev.mad_agent.port_id, +dev->backend_dev.mad_agent.agent_id, +mad_msg, umad->length, 10, 10); +*/ Then what is above code doing here?
Support for QP1 is a work in progress.
In the meantime i can take this entire code out to a separate private branch.
Quoted 3 lines
Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?
Even without QP1 the device can still be used for usual RDMA send and receive operations, but yes, no rdma_cm and multicast for now. Please note that vmware support for QP1 is proprietary and can be used only between two ESX guests so we give more or less the same.
Quoted 2 lines
How does guest know it's missing?
I had two options, one is to reject the creation of QP1 via the control interface and second is to post CQE with error. Since the former causes some driver loading errors in guest i choose the second.
Quoted 46 lines
...
Quoted 41 lines
hw/net/pvrdma/pvrdma_utils.h new file +35
-0@@ -0,0 +1,41 @@ +/* +* QEMU VMWARE paravirtual RDMA interface definitions +* +* Developed by Oracle & Redhat +* +* Authors: +* Yuval Shaia <[email protected]> +* Marcel Apfelbaum <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef PVRDMA_UTILS_H +#define PVRDMA_UTILS_H + +#include <include/hw/pci/pci.h> + +#define pr_info (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__) + +#define pr_err (fmt, ...) \ + fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \ + __LINE__, ## __VA_ARGS__) + +#ifdef PVRDMA_DEBUG +#define pr_dbg (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__) +#else +#define pr_dbg (fmt, ...) +#endif +
Quoted 8 lines
hw/net/pvrdma/pvrdma_utils.h line 41
Quoted 6 lines
hw/net/pvrdma/pvrdma_utils.h new file +6
-0+void pvrdma_pci_dma_unmap (PCIDevice *dev,void *buffer,dma_addr_t len); +void *pvrdma_pci_dma_map (PCIDevice *dev,dma_addr_t addr,dma_addr_t plen); +void *map_to_pdir (PCIDevice *pdev,uint64_t pdir_dma,uint32_t nchunks, +size_t length); + +#endif Can you make sure all prefixes are
pvrdma_?
Is it a general practice or just for non-static functions?
Quoted 30 lines
Quoted 30 lines
hw/net/pvrdma/trace-events new file +9
-0@@ -0,0 +1,9 @@ +# See docs/tracing.txt for syntax documentation. + +# hw/net/pvrdma/pvrdma_main.c +pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" +pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" + +#hw/net/pvrdma/pvrdma_backend.c +create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx" +create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"include/hw/pci/pci_ids.h +3
-0@@ -266,4 +266,7 @@#define PCI_VENDOR_ID_TEWS 0x1498#define PCI_DEVICE_ID_TEWS_TPCI200 0x30C8 +#define PCI_VENDOR_ID_VMWARE 0x15ad +#define PCI_DEVICE_ID_VMWARE_PVRDMA 0x0820 +#endif Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 40 lines · Yuval Shaia
Quoted 2 lines · Michael S. Tsirkin
I won't have time to review before the next year.
Results of a quick peek.Thanks for the parts you found the time to review.
Quoted 18 lines · Marcel Apfelbaum
Quoted 16 lines
hw/net/pvrdma/pvrdma_backend.c new file +16
-0+static void *mad_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +QObject *o_ctx_id; +unsigned long cqe_ctx_id; +BackendCtx *bctx; +/* +int len; +void *mad; +*/ + +pr_dbg ("Starting\n" ); + + dev->backend_dev.mad_thread.run = false; +Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 86
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+while (dev->backend_dev.mad_thread.run) { +/* Get next buffer to pust MAD into */ + o_ctx_id =qlist_pop (dev->backend_dev.mad_agent.recv_mads_list); +if (!o_ctx_id) { +/* pr_dbg("Error: No more free MADs buffers\n"); */ +sleep (5 );Looks suspicious. What is above doing?
This function is responsible to process incoming MAD messages.
Usual (good) flow is that guest driver prepares some buffers to be used for that purpose and gives it to device with the usual
post_recvmechanism. Upon receiving a MAD message from the device the driver should pass a new buffer to device.
Quoted 4 lines
But what if the device didn't do it.
This section handle such case, as we have nothing to do - let's sleep and hope for the best.
So is this broken hardware then? I would say print an error and disable the device.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
hw/net/pvrdma/pvrdma_backend.c new file +12
-0+continue ; + } + cqe_ctx_id =qnum_get_uint (qobject_to_qnum (o_ctx_id)); + bctx =rm_get_cqe_ctx (dev, cqe_ctx_id); +if (unlikely (!bctx)) { +pr_dbg ("Error: Fail to find ctx for %ld\n" , cqe_ctx_id); +continue ; + } + +pr_dbg ("Calling umad_recv\n" ); +/* +mad = pvrdma_pci_dma_map(PCI_DEVICE(dev), bctx->req.sge[0].addr, Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 104
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+ bctx->req.sge[0].length); + + len = bctx->req.sge[0 ].length; + + do { + rc = umad_recv(dev->backend_dev.mad_agent.port_id, mad, &len, 5000);That's a huge timeout.
Quoted 5 lines
This is the maximum timeout.
On low MAD traffic we don't want to take much of the CPU for that purpose so 5 seconds looks good to me.Anyway, just because it is not obvious i will make it configurable.
I'm not sure why not an infinite timeout then?
Quoted 179 lines
Quoted 165 lines
Quoted 165 lines
hw/net/pvrdma/pvrdma_backend.c new file +164
-0+} while ( (rc != ETIMEDOUT) && dev->backend_dev.mad_thread.run); +pr_dbg("umad_recv, rc=%d\n", rc); + +pvrdma_pci_dma_unmap(PCI_DEVICE(dev), mad, bctx->req.sge[0].length); +*/ + rc =-1 ; + +/* rc is used as vendor_err */ +comp_handler (rc >0 ?IB_WC_SUCCESS :IB_WC_GENERAL_ERR , rc, + bctx->up_ctx); + +rm_dealloc_cqe_ctx (dev, cqe_ctx_id); +free (bctx); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining MADs in list */ + +qlist_destroy_obj (QOBJECT (dev->backend_dev.mad_agent.recv_mads_list)); + +return NULL ; +} + +static void *comp_handler_thread (void *arg) +{ +PVRDMADev *dev = (PVRDMADev *)arg; +int rc; +struct ibv_cq *ev_cq; +void *ev_ctx; + +pr_dbg ("Starting\n" ); + +while (dev->backend_dev.comp_thread.run) { +pr_dbg ("Waiting for completion on channel %p\n" , + dev->backend_dev.channel); + rc =ibv_get_cq_event (dev->backend_dev.channel, &ev_cq, &ev_ctx); +pr_dbg ("ibv_get_cq_event=%d\n" , rc); +if (unlikely (rc)) { +pr_dbg ("---> ibv_get_cq_event (%d)\n" , rc); +continue ; + } + +if (unlikely (ibv_req_notify_cq (ev_cq,0 ))) { +pr_dbg ("---> ibv_req_notify_cq\n" ); + } + +poll_cq (dev, ev_cq, false); + +ibv_ack_cq_events (ev_cq,1 ); + } + +pr_dbg ("Going down\n" ); +/* TODO: Post cqe for all remaining buffs that were posted */ + +return NULL ; +} + +void backend_register_comp_handler (void (*handler)(int status, +unsigned int vendor_err,void *ctx)) +{ + comp_handler = handler; +} + +int backend_query_port (BackendDevice *dev,struct pvrdma_port_attr *attrs) +{ +int rc; +struct ibv_port_attr port_attr; + + rc =ibv_query_port (dev->context, dev->port_num, &port_attr); +if (rc) { +pr_dbg ("Error %d from ibv_query_port\n" , rc); +return -EIO ; + } + + attrs->state = port_attr.state; + attrs->max_mtu = port_attr.max_mtu; + attrs->active_mtu = port_attr.active_mtu; + attrs->gid_tbl_len = port_attr.gid_tbl_len; + attrs->pkey_tbl_len = port_attr.pkey_tbl_len; + attrs->phys_state = port_attr.phys_state; + +return 0 ; +} + +void backend_poll_cq (PVRDMADev *dev,BackendCQ *cq) +{ +poll_cq (dev, cq->ibcq, true); +} + +static GHashTable *ah_hash; + +static struct ibv_ah *create_ah (BackendDevice *dev,struct ibv_pd *pd, +union ibv_gid *dgid,uint8_t sgid_idx) +{ +GBytes *ah_key =g_bytes_new (dgid,sizeof (*dgid)); +struct ibv_ah *ah =g_hash_table_lookup (ah_hash, ah_key); + +if (ah) { +trace_create_ah_cache_hit (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + }else { +struct ibv_ah_attr ah_attr = { + .is_global =1 , + .port_num = dev->port_num, + .grh.hop_limit =1 , + }; + + ah_attr.grh.dgid = *dgid; + ah_attr.grh.sgid_index = sgid_idx; + + ah =ibv_create_ah (pd, &ah_attr); +if (ah) { +g_hash_table_insert (ah_hash, ah_key, ah); + }else { +pr_dbg ("ibv_create_ah failed for gid <%lx %lx>\n" , +be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +trace_create_ah_cache_miss (be64_to_cpu (dgid->global.subnet_prefix), +be64_to_cpu (dgid->global.interface_id)); + } + +return ah; +} + +static void destroy_ah (gpointer data) +{ +struct ibv_ah *ah = data; +ibv_destroy_ah (ah); +} + +static void ah_cache_init (void ) +{ + ah_hash =g_hash_table_new_full (g_bytes_hash, g_bytes_equal, +NULL , destroy_ah); +} + +static int send_mad (PVRDMADev *dev,struct pvrdma_sge *sge,u32 num_sge) +{ +int ret =-1 ; + +/* +* TODO: Currently QP1 is not supported +* +PCIDevice *pci_dev = PCI_DEVICE(dev); +char mad_msg[1024]; +void *hdr, *msg; +struct ib_user_mad *umad = (struct ib_user_mad *)&mad_msg; + +umad->length = sge[0].length + sge[1].length; + +if (num_sge != 2) +return -EINVAL; + +pr_dbg("msg_len=%d\n", umad->length); + +hdr = pvrdma_pci_dma_map(pci_dev, sge[0].addr, sge[0].length); +msg = pvrdma_pci_dma_map(pci_dev, sge[1].addr, sge[1].length); + +memcpy(&mad_msg[64], hdr, sge[0].length); +memcpy(&mad_msg[sge[0].length+64], msg, sge[1].length); + +pvrdma_pci_dma_unmap(pci_dev, msg, sge[1].length); Quoted 8 lines
hw/net/pvrdma/pvrdma_backend.c line 274
Quoted 6 lines
hw/net/pvrdma/pvrdma_backend.c new file +6
-0+pvrdma_pci_dma_unmap(pci_dev, hdr, sge[0].length); + +ret = umad_send(dev->backend_dev.mad_agent.port_id, +dev->backend_dev.mad_agent.agent_id, +mad_msg, umad->length, 10, 10); +*/ Then what is above code doing here?
Support for QP1 is a work in progress.
In the meantime i can take this entire code out to a separate private branch.Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Also, isn't QP1 a big deal? If it's missing then how do you do multicast etc?
Even without QP1 the device can still be used for usual RDMA send and receive operations, but yes, no
rdma_cmand multicast for now. Please note that vmware support for QP1 is proprietary and can be used only between two ESX guests so we give more or less the same.
You will want to document this stuff.
Quoted 62 lines
Quoted 2 lines
How does guest know it's missing?
I had two options, one is to reject the creation of QP1 via the control interface and second is to post CQE with error. Since the former causes some driver loading errors in guest i choose the second.
Quoted 46 lines
...
Quoted 41 lines
hw/net/pvrdma/pvrdma_utils.h new file +35
-0@@ -0,0 +1,41 @@ +/* +* QEMU VMWARE paravirtual RDMA interface definitions +* +* Developed by Oracle & Redhat +* +* Authors: +* Yuval Shaia <[email protected]> +* Marcel Apfelbaum <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2. +* See the COPYING file in the top-level directory. +* +*/ + +#ifndef PVRDMA_UTILS_H +#define PVRDMA_UTILS_H + +#include <include/hw/pci/pci.h> + +#define pr_info (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__) + +#define pr_err (fmt, ...) \ + fprintf(stderr, "%s: Error at %-20s (%3d): " fmt, "pvrdma", __func__, \ + __LINE__, ## __VA_ARGS__) + +#ifdef PVRDMA_DEBUG +#define pr_dbg (fmt, ...) \ + fprintf(stdout, "%s: %-20s (%3d): " fmt, "pvrdma", __func__, __LINE__,\ + ## __VA_ARGS__) +#else +#define pr_dbg (fmt, ...) +#endif +Quoted 8 lines
hw/net/pvrdma/pvrdma_utils.h line 41
Quoted 6 lines
hw/net/pvrdma/pvrdma_utils.h new file +6
-0+void pvrdma_pci_dma_unmap (PCIDevice *dev,void *buffer,dma_addr_t len); +void *pvrdma_pci_dma_map (PCIDevice *dev,dma_addr_t addr,dma_addr_t plen); +void *map_to_pdir (PCIDevice *pdev,uint64_t pdir_dma,uint32_t nchunks, +size_t length); + +#endif Can you make sure all prefixes are
pvrdma_?
Quoted 1 line
Is it a general practice or just for non-static functions?
Doing this generally is preferred. A must for non-static.
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
hw/net/pvrdma/trace-events new file +9
-0@@ -0,0 +1,9 @@ +# See docs/tracing.txt for syntax documentation. + +# hw/net/pvrdma/pvrdma_main.c +pvrdma_regs_read(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" +pvrdma_regs_write(uint64_t addr, uint64_t val) "regs[0x%lx] = 0x%lx" + +#hw/net/pvrdma/pvrdma_backend.c +create_ah_cache_hit(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx" +create_ah_cache_miss(uint64_t subnet, uint64_t net_id) "subnet = 0x%lx net_id = 0x%lx"include/hw/pci/pci_ids.h +3
-0@@ -266,4 +266,7 @@#define PCI_VENDOR_ID_TEWS 0x1498#define PCI_DEVICE_ID_TEWS_TPCI200 0x30C8 +#define PCI_VENDOR_ID_VMWARE 0x15ad +#define PCI_DEVICE_ID_VMWARE_PVRDMA 0x0820 +#endif Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
In reply to Marcel ApfelbaumHi Marcel, Yuval,
Quoted 17 lines · Marcel Apfelbaum
From: Yuval Shaia <[email protected]>
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.
Signed-off-by Yuval Shaia <[email protected]>Signed-off-by Marcel Apfelbaum <[email protected]>---
[...]
Quoted 14 lines
28 files changed, 5132 insertions(+), 4 deletions(-) create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h
Quoted 8 lines
create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-events
[...]
Since we already have a hw/xenpv/ directory, can we place these files into hw/vmwarepv/ rather than hw/net/pvrdma/?
A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.
Regards,
Phil.
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 43 lines · Philippe Mathieu-Daudé
Hi Marcel, Yuval,
Quoted 17 lines · Marcel Apfelbaum
From: Yuval Shaia <[email protected]>
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.
Signed-off-by Yuval Shaia <[email protected]>Signed-off-by Marcel Apfelbaum <[email protected]>---
[...]
Quoted 14 lines
28 files changed, 5132 insertions(+), 4 deletions(-) create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.hQuoted 8 lines
create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-events
Quoted 3 lines
[...]
Since we already have a hw/xenpv/ directory,
But e.g. xen nic is under hw/net/
Quoted 8 lines
can we place these files
into hw/vmwarepv/ rather than hw/net/pvrdma/?A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.
Regards,
Phil.
That's not how we layout things. We group them by function not by interface. Thus I think that hw/rdma/ is better.
Signature · 1 line
MST
Re: [Qemu-devel] [PATCH V2 4/5] pvrdma: initial implementation
Quoted 53 lines · Michael S. Tsirkin
Quoted 43 lines · Philippe Mathieu-Daudé
Hi Marcel, Yuval,
Quoted 17 lines · Marcel Apfelbaum
From: Yuval Shaia <[email protected]>
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance.
Signed-off-by Yuval Shaia <[email protected]>Signed-off-by Marcel Apfelbaum <[email protected]>---
[...]
Quoted 14 lines
28 files changed, 5132 insertions(+), 4 deletions(-)create mode 100644
hw/net/pvrdma/pvrdma.hcreate mode 100644hw/net/pvrdma/pvrdma_backend.ccreate mode 100644hw/net/pvrdma/pvrdma_backend.hcreate mode 100644hw/net/pvrdma/pvrdma_backend_defs.hcreate mode 100644hw/net/pvrdma/pvrdma_cmd.ccreate mode 100644hw/net/pvrdma/pvrdma_defs.hcreate mode 100644hw/net/pvrdma/pvrdma_dev_api.hcreate mode 100644hw/net/pvrdma/pvrdma_dev_ring.ccreate mode 100644hw/net/pvrdma/pvrdma_dev_ring.hcreate mode 100644hw/net/pvrdma/pvrdma_ib_verbs.hcreate mode 100644hw/net/pvrdma/pvrdma_main.ccreate mode 100644hw/net/pvrdma/pvrdma_qp_ops.ccreate mode 100644hw/net/pvrdma/pvrdma_qp_ops.hQuoted 8 lines
create mode 100644
hw/net/pvrdma/pvrdma_ring.hcreate mode 100644hw/net/pvrdma/pvrdma_rm.ccreate mode 100644hw/net/pvrdma/pvrdma_rm.hcreate mode 100644hw/net/pvrdma/pvrdma_rm_defs.hcreate mode 100644hw/net/pvrdma/pvrdma_types.hcreate mode 100644hw/net/pvrdma/pvrdma_utils.ccreate mode 100644hw/net/pvrdma/pvrdma_utils.hcreate mode 100644 hw/net/pvrdma/trace-eventsQuoted 3 lines
[...]
Since we already have a hw/xenpv/ directory,
But e.g. xen nic is under hw/net/
Quoted 3 lines
can we place these files
into hw/vmwarepv/ rather than hw/net/pvrdma/?
Quoted 9 lines
Quoted 5 lines
A smarter move might be to create a hw/pv/ dir and have hw/pv/{xen,vmware}.
Regards,
Phil.
That's not how we layout things. We group them by function not by interface. Thus I think that hw/rdma/ is better.
Hi Philippe and Michael,
Thanks for the suggestion, we will move the code to hw/rdma. Marcel
PATCH v2 5/5 MAINTAINERS: add entry for hw/net/pvrdma
In reply to Marcel Apfelbaum--- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
@@ -1960,6 +1960,13 @@ F: block/replication.c F: tests/test-replication.c F: docs/block-replication.txt +PVRDMA +M: Yuval Shaia <[email protected]> +M: Marcel Apfelbaum <[email protected]> +S: Maintained +F: hw/net/pvrdma/* +F: docs/pvrdma.txt + Build and test automation ------------------------- Build and test automation
Signature · 1 line
2.13.5
Re: [Qemu-devel] [PATCH V2 5/5] MAINTAINERS: add entry for hw/net/pvrdma
Quoted 14 lines · Marcel Apfelbaum
Signed-off-by Marcel Apfelbaum <[email protected]>Signed-off-by Yuval Shaia <[email protected]>--- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+)MAINTAINERS +0
-0@@ -1960,6 +1960,13 @@ F: block/replication.c F: tests/test-replication.c F: docs/block-replication.txt
MAINTAINERS F: block/replication.c · line 1972
Quoted 10 lines
MAINTAINERS +7
-0+PVRDMA +M: Yuval Shaia <[email protected]> +M: Marcel Apfelbaum <[email protected]> +S: Maintained +F: hw/net/pvrdma/* +F: docs/pvrdma.txt + Build and test automation ------------------------- Build and test automation
Not really a network device, I think it needs its own directory.
Quoted 2 lines
Signature · 1 line
2.13.5
Quoted 10 lines · Marcel Apfelbaum
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR moduleGeneral description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
Quoted 5 lines
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned
What happens if guest attempts to register all its memory?
Quoted 4 lines
allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.
What does "HW assistance" mean here?
Can it work with any existing hardware?
Quoted 5 lines
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it
Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?
Quoted 2 lines
and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.
I suspect that without virtio we won't be able to do any future extensions.
Quoted 4 lines
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
What does this mean?
Quoted 1 line
BAR 2 - UAR, used to pass HW commands from driver.
A detailed description of above belongs in documentation.
Quoted 3 lines
- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.
I am wondering how do you make connections? QP#s are exposed on the wire during connection management.
Quoted 5 lines
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).
So what's the advantage of this over pass-through then?
Quoted 95 lines
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdmaYuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementationMAINTAINERS | 7 + Makefile.objs | 1 + backends/hostmem-file.c | 25 +- backends/hostmem-ram.c | 4 +- backends/hostmem.c | 21 + configure | 9 +- default-configs/arm-softmmu.mak | 2 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + docs/pvrdma.txt | 145 ++++++ exec.c | 26 +- hw/net/Makefile.objs | 7 + hw/net/pvrdma/pvrdma.h | 179 +++++++ hw/net/pvrdma/pvrdma_backend.c | 986 ++++++++++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_backend.h | 74 +++ hw/net/pvrdma/pvrdma_backend_defs.h | 68 +++ hw/net/pvrdma/pvrdma_cmd.c | 338 ++++++++++++ hw/net/pvrdma/pvrdma_defs.h | 121 +++++ hw/net/pvrdma/pvrdma_dev_api.h | 580 +++++++++++++++++++++ hw/net/pvrdma/pvrdma_dev_ring.c | 138 +++++ hw/net/pvrdma/pvrdma_dev_ring.h | 42 ++ hw/net/pvrdma/pvrdma_ib_verbs.h | 399 +++++++++++++++ hw/net/pvrdma/pvrdma_main.c | 664 ++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_qp_ops.c | 187 +++++++ hw/net/pvrdma/pvrdma_qp_ops.h | 26 + hw/net/pvrdma/pvrdma_ring.h | 134 +++++ hw/net/pvrdma/pvrdma_rm.c | 791 +++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_rm.h | 54 ++ hw/net/pvrdma/pvrdma_rm_defs.h | 111 ++++ hw/net/pvrdma/pvrdma_types.h | 37 ++ hw/net/pvrdma/pvrdma_utils.c | 133 +++++ hw/net/pvrdma/pvrdma_utils.h | 41 ++ hw/net/pvrdma/trace-events | 9 + hw/pci/shpc.c | 11 +- include/exec/memory.h | 23 + include/exec/ram_addr.h | 3 +- include/hw/pci/pci_ids.h | 3 + include/qemu/cutils.h | 10 + include/qemu/osdep.h | 2 +- include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 2 +- memory.c | 16 +- util/oslib-posix.c | 4 +- util/oslib-win32.c | 2 +- 44 files changed, 5378 insertions(+), 61 deletions(-) create mode 100644 docs/pvrdma.txt create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-eventsSignature · 1 line
2.13.5
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR moduleGeneral description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.
Quoted 6 lines
Quoted 5 lines
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned
Hi Michael,
Quoted 2 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal, reg_mr will pin all the RAM. However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).
Quoted 6 lines
Quoted 4 lines
allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.What does "HW assistance" mean here?
Several things:
1. We need to be able to pass resource numbers when we create them on the destination machine.
2. We also need a way to stall prev connections while starting the new ones.
3. Last, we need the HW to pass resources states.
Quoted 2 lines
Can it work with any existing hardware?
Sadly no, however we talked with Mellanox at the last year Plumbers Conference and all the above are on their plans. We hope this submission will help, since now we will have a fast way to test and use it.
For Soft-RoCE backend is doable, but is best to wait first to see how HCAs are going to expose the changes.
Quoted 6 lines
Quoted 5 lines
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with it
Quoted 9 lines
Everything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?
Quoted 2 lines
and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.I suspect that without virtio we won't be able to do any future extensions.
While I do agree is harder to work with a 3rd party spec, their Linux driver is open source and we may be able to do sane modifications.
Quoted 4 lines
Quoted 4 lines
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
[...]
Quoted 8 lines
Quoted 5 lines
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).So what's the advantage of this over pass-through then?
1. We can work also with the same ibdevice for multiple pvrdma devices using multiple GIDs; it works (tested).
The problem begins when we think about migration, the way HCAs work today is one resource namespace per ibdevice, not per GID. I emphasize that this can be changed, however we don't have a timeline for it.
2. We do have advantages:
- Guest agnostic device (we can change host HCA)
- Memory over commit (unless the guest registers all the memory)
- Future migration support
- A friendly migration of RDMA VMWare guests to QEMU.
3. In case when live migration is not a must we can use multiple GIDs of the same port, so we do not depend on SRIOV.
4. We support Soft RoCE backend, people can test their software on guest without RDMA hw.
Thanks,
Marcel
Quoted 26 lines
Quoted 26 lines
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdmaYuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation
[...]
Quoted 19 lines · Marcel Apfelbaum
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR moduleGeneral description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.Quoted 6 lines
Quoted 5 lines
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned
Quoted 6 lines
Hi Michael,
Quoted 2 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.
We need to find a way to communicate to guests about amount of memory they can pin.
Quoted 2 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).
SRP does it too AFAIK.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.What does "HW assistance" mean here?
Several things:
1. We need to be able to pass resource numbers when we create them on the destination machine.
These resources are mostly managed by software.
Quoted 1 line
2. We also need a way to stall prev connections while starting the new ones.
Look at what hardware can do.
Quoted 1 line
3. Last, we need the HW to pass resources states.
Look at the spec, some of this can be done.
Quoted 4 lines
Quoted 2 lines
Can it work with any existing hardware?
Sadly no,
Above can be done. What's needed is host kernel work to support it.
Quoted 4 lines
however we talked with Mellanox at the last year Plumbers Conference and all the above are on their plans. We hope this submission will help, since now we will have a fast way to test and use it.
I'm doubtful it'll help.
Quoted 15 lines
For Soft-RoCE backend is doable, but is best to wait first to see how HCAs are going to expose the changes.
Quoted 12 lines
Quoted 5 lines
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with itEverything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?
Quoted 2 lines
and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.
Quoted 7 lines
Quoted 3 lines
I suspect that without virtio we won't be able to do any future extensions.
While I do agree is harder to work with a 3rd party spec, their Linux driver is open source and we may be able to do sane modifications.
I am sceptical. ARM guys did not want to add a single bit in their IOMMU spec. You want an open spec that everyone can contribute to.
Quoted 23 lines
Quoted 4 lines
Quoted 4 lines
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers[...]
Quoted 8 lines
Quoted 5 lines
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).So what's the advantage of this over pass-through then?
1. We can work also with the same ibdevice for multiple pvrdma devices using multiple GIDs; it works (tested).
The problem begins when we think about migration, the way HCAs work today is one resource namespace per ibdevice, not per GID. I emphasize that this can be changed, however we don't have a timeline for it.
Quoted 3 lines
2. We do have advantages:
- Guest agnostic device (we can change host HCA)
- Memory over commit (unless the guest registers all the memory)
Not just all. You trust guest and this is a problem. If you do try to overcommit, at any point guest can try to register too much and host will stall.
Quoted 1 line
- Future migration support
So there are lots of difficult problems to solve for this. E.g. any MR that is hardware writeable can be changed and hypervisor won't know. All this can be solvable but it might also be solveable with passthrough too.
Quoted 1 line
- A friendly migration of RDMA VMWare guests to QEMU.
Why do we need to emulate their device for this? Reboot is required anyway, you can switch to a passthrough easily.
Quoted 10 lines
3. In case when live migration is not a must we can use multiple GIDs of the same port, so we do not depend on SRIOV.
4. We support Soft RoCE backend, people can test their software on guest without RDMA hw.
Thanks,
Marcel
These two are nice, if very niche, features.
Quoted 28 lines
Quoted 26 lines
Quoted 26 lines
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdmaYuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation[...]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 4 lines · Yuval Shaia
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Isn't that still leaving the option for guest to register all memory, just in chunks?
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
Quoted 7 lines
Quoted 3 lines
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Isn't that still leaving the option for guest to register all memory, just in chunks?
We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.
Thanks,
Marcel
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 19 lines · Marcel Apfelbaum
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
Quoted 7 lines
Quoted 3 lines
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Isn't that still leaving the option for guest to register all memory, just in chunks?
Quoted 5 lines
We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.
Thanks,
Marcel
You might want to limit the default values then.
Right now:
+#define MAX_MR_SIZE (1UL << 32)
+#define MAX_MR 2048
Which is IIUC 8TB.
That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 30 lines · Michael S. Tsirkin
Quoted 19 lines · Marcel Apfelbaum
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Yuval Shaia
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
Quoted 7 lines
Quoted 3 lines
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Isn't that still leaving the option for guest to register all memory, just in chunks?
Quoted 5 lines
We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.
Thanks,
MarcelYou might want to limit the default values then.
Right now:
Review
Quoted 2 lines
(quoted diff) +2
-0+#define MAX_MR_SIZE (1UL << 32) +#define MAX_MR 2048
Maybe limiting by constant number is not a good approach, it looks odd if one guest with 16G ram and second with 32G ram will have the same settings, right?
So how about limiting by a specific percentage of total memory? In that case, what would be this percentage? 100%? 80%?
Quoted 13 lines
Which is IIUC 8TB.
That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 21 lines · Yuval Shaia
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Marcel Apfelbaum
Quoted 17 lines · Michael S. Tsirkin
Quoted 4 lines · Yuval Shaia
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What happens if guest attempts to register all its memory?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Then we loose, is not different from bare metal,
reg_mrwill pin all the RAM.We need to find a way to communicate to guests about amount of memory they can pin.
dev_caps.max_mr_size is the way device limits guest driver. This value is controlled by the command line argument dev-caps-max-mr-size so we should be fine (btw, default value is 1<<32).
Isn't that still leaving the option for guest to register all memory, just in chunks?
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
We also have a parameter limiting the number of mrs (dev-caps-max-mr), together with dev-caps-max-mr-size we can limit the memory the guests can pin.
Thanks,
MarcelYou might want to limit the default values then.
Hi Yuval,
Quoted 11 lines
Review
Quoted 4 lines
Right now:
(quoted diff) +2
-0+#define MAX_MR_SIZE (1UL << 32) +#define MAX_MR 2048Maybe limiting by constant number is not a good approach, it looks odd if one guest with 16G ram and second with 32G ram will have the same settings, right?
So how about limiting by a specific percentage of total memory? In that case, what would be this percentage? 100%? 80%?
I think is too complicated. Maybe we can limit the max pined memory to 2G assuming the RDMA guests have a lot of RAM and let the users fine-tune the parameters.
Thanks,
Marcel
Quoted 13 lines
Quoted 13 lines
Which is IIUC 8TB.
That's pretty close to unlimited, and so far overcommit seems to be the main feature for users.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
However this is only one scenario, and hopefully not much used
for RoCE. (I know IPoIB does that, but it doesn't make sense to use it with RoCE).SRP does it too AFAIK.
Quoted 28 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR moduleGeneral description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.Quoted 5 lines
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned
What happens if guest attempts to register all its memory?
Quoted 4 lines
allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.What does "HW assistance" mean here?
Can it work with any existing hardware?Quoted 1 line
Quoted 10 lines
Quoted 4 lines
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with itEverything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?
Quoted 2 lines
and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.
The current design takes into account a future code reuse with virtio-rdma device although not sure it is 100%.
We divided it to four software layers:
- Front-end interface with PCI:
- pvrdma_main.c
- Front-end interface with pvrdma driver:
- pvrdma_cmd.c
- pvrdma_qp_ops.c
- pvrdma_dev_ring.c
- pvrdma_utils.c
- Device emulation:
- pvrdma_rm.c
- Back-end interface:
- pvrdma_backend.c
So in the future, when starting to work on virtio-rdma device we will move the generic code to generic directory.
Any reason why we want to split it now, when we have only one device?
Quoted 3 lines
I suspect that without virtio we won't be able to do any future extensions.
As i see it these are two different issues, virtio RDMA device is on our plate but the contribution of VMWare pvrdma device to QEMU is no doubt a real advantage that will allow customers that runs ESX to easy move to QEMU.
Quoted 7 lines
Quoted 5 lines
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registersWhat does this mean?
Device control operations:
- Setting of interrupt mask.
- Setup of Device/Driver shared configuration area.
- Reset device, activate device etc.
- Device commands such as create QP, create MR etc.
Quoted 4 lines
Quoted 2 lines
BAR 2 - UAR, used to pass HW commands from driver.
A detailed description of above belongs in documentation.
Will do.
Quoted 7 lines
Quoted 4 lines
- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.
I am wondering how do you make connections? QP#s are exposed on the wire during connection management.
QP#s that guest sees are the QP#s that are used on the wire. The meaning of "internal management of the RDMA resources" is that we keep context of internal QP in device (ex rings).
Quoted 105 lines
Quoted 6 lines
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).So what's the advantage of this over pass-through then?
Quoted 95 lines
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdmaYuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementationMAINTAINERS | 7 + Makefile.objs | 1 + backends/hostmem-file.c | 25 +- backends/hostmem-ram.c | 4 +- backends/hostmem.c | 21 + configure | 9 +- default-configs/arm-softmmu.mak | 2 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + docs/pvrdma.txt | 145 ++++++ exec.c | 26 +- hw/net/Makefile.objs | 7 + hw/net/pvrdma/pvrdma.h | 179 +++++++ hw/net/pvrdma/pvrdma_backend.c | 986 ++++++++++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_backend.h | 74 +++ hw/net/pvrdma/pvrdma_backend_defs.h | 68 +++ hw/net/pvrdma/pvrdma_cmd.c | 338 ++++++++++++ hw/net/pvrdma/pvrdma_defs.h | 121 +++++ hw/net/pvrdma/pvrdma_dev_api.h | 580 +++++++++++++++++++++ hw/net/pvrdma/pvrdma_dev_ring.c | 138 +++++ hw/net/pvrdma/pvrdma_dev_ring.h | 42 ++ hw/net/pvrdma/pvrdma_ib_verbs.h | 399 +++++++++++++++ hw/net/pvrdma/pvrdma_main.c | 664 ++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_qp_ops.c | 187 +++++++ hw/net/pvrdma/pvrdma_qp_ops.h | 26 + hw/net/pvrdma/pvrdma_ring.h | 134 +++++ hw/net/pvrdma/pvrdma_rm.c | 791 +++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_rm.h | 54 ++ hw/net/pvrdma/pvrdma_rm_defs.h | 111 ++++ hw/net/pvrdma/pvrdma_types.h | 37 ++ hw/net/pvrdma/pvrdma_utils.c | 133 +++++ hw/net/pvrdma/pvrdma_utils.h | 41 ++ hw/net/pvrdma/trace-events | 9 + hw/pci/shpc.c | 11 +- include/exec/memory.h | 23 + include/exec/ram_addr.h | 3 +- include/hw/pci/pci_ids.h | 3 + include/qemu/cutils.h | 10 + include/qemu/osdep.h | 2 +- include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 2 +- memory.c | 16 +- util/oslib-posix.c | 4 +- util/oslib-win32.c | 2 +- 44 files changed, 5378 insertions(+), 61 deletions(-) create mode 100644 docs/pvrdma.txt create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-eventsSignature · 1 line
2.13.5
Quoted 56 lines · Yuval Shaia
Quoted 28 lines · Michael S. Tsirkin
Quoted 10 lines · Marcel Apfelbaum
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR moduleGeneral description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications.Quoted 5 lines
While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned
What happens if guest attempts to register all its memory?
Quoted 4 lines
allowing memory
over-commit
and, even if not implemented yet, migration support will be possible with some HW assistance.What does "HW assistance" mean here?
Can it work with any existing hardware?Quoted 1 line
Quoted 10 lines
Quoted 4 lines
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly coupled with itEverything seems to be in pvrdma. Since it's not coupled, could you split code to pvrdma specific and generic parts?
Quoted 2 lines
and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.The current design takes into account a future code reuse with virtio-rdma device although not sure it is 100%.
We divided it to four software layers:
- Front-end interface with PCI:
-pvrdma_main.c
- Front-end interface with pvrdma driver:
-pvrdma_cmd.c
-pvrdma_qp_ops.c
-pvrdma_dev_ring.c
-pvrdma_utils.c
- Device emulation:
-pvrdma_rm.c
- Back-end interface:
-pvrdma_backend.c
Quoted 4 lines
So in the future, when starting to work on virtio-rdma device we will move the generic code to generic directory.
Any reason why we want to split it now, when we have only one device?
To make it easier for me to ignore pvrdma stuff and review the generic stuff.
Quoted 7 lines
Quoted 3 lines
I suspect that without virtio we won't be able to do any future extensions.
As i see it these are two different issues, virtio RDMA device is on our plate but the contribution of VMWare pvrdma device to QEMU is no doubt a real advantage that will allow customers that runs ESX to easy move to QEMU.
I don't have anything against it but I'm not really interested in reviewing it either.
Quoted 26 lines
Quoted 7 lines
Quoted 5 lines
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registersWhat does this mean?
Device control operations:
- Setting of interrupt mask.
- Setup of Device/Driver shared configuration area.
- Reset device, activate device etc.
- Device commands such as create QP, create MR etc.Quoted 4 lines
Quoted 2 lines
BAR 2 - UAR, used to pass HW commands from driver.
A detailed description of above belongs in documentation.
Will do.
Quoted 5 lines
Quoted 4 lines
- The device performs internal management of the RDMA resources (PDs, CQs, QPs, ...), meaning the objects are not directly coupled to a physical RDMA device resources.
Quoted 6 lines
Quoted 2 lines
I am wondering how do you make connections? QP#s are exposed on the wire during connection management.
QP#s that guest sees are the QP#s that are used on the wire. The meaning of "internal management of the RDMA resources" is that we keep context of internal QP in device (ex rings).
The question was that you need to parse CM/MAD etc messages if you need to change QP#s on the fly, and that code does not seem to be there. I guess the answer is that
a bunch of this stuff is just broken or non-spec compliant.
Quoted 105 lines
Quoted 105 lines
Quoted 6 lines
The pvrdma backend is an ibdevice interface that can be exposed either by a Soft-RoCE(rxe) device on machines with no RDMA device, or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices, each one requiring a separate instance (rxe or SRIOV VF).So what's the advantage of this over pass-through then?
Quoted 95 lines
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3, and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.The best performance achieved with ConnectX HCAs and buffer size bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no actual performance penalties compared to bare metal for big enough buffers (which is quite common when using RDMA), while allowing memory overcommit.Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/net/pvrdmaYuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementationMAINTAINERS | 7 + Makefile.objs | 1 + backends/hostmem-file.c | 25 +- backends/hostmem-ram.c | 4 +- backends/hostmem.c | 21 + configure | 9 +- default-configs/arm-softmmu.mak | 2 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + docs/pvrdma.txt | 145 ++++++ exec.c | 26 +- hw/net/Makefile.objs | 7 + hw/net/pvrdma/pvrdma.h | 179 +++++++ hw/net/pvrdma/pvrdma_backend.c | 986 ++++++++++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_backend.h | 74 +++ hw/net/pvrdma/pvrdma_backend_defs.h | 68 +++ hw/net/pvrdma/pvrdma_cmd.c | 338 ++++++++++++ hw/net/pvrdma/pvrdma_defs.h | 121 +++++ hw/net/pvrdma/pvrdma_dev_api.h | 580 +++++++++++++++++++++ hw/net/pvrdma/pvrdma_dev_ring.c | 138 +++++ hw/net/pvrdma/pvrdma_dev_ring.h | 42 ++ hw/net/pvrdma/pvrdma_ib_verbs.h | 399 +++++++++++++++ hw/net/pvrdma/pvrdma_main.c | 664 ++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_qp_ops.c | 187 +++++++ hw/net/pvrdma/pvrdma_qp_ops.h | 26 + hw/net/pvrdma/pvrdma_ring.h | 134 +++++ hw/net/pvrdma/pvrdma_rm.c | 791 +++++++++++++++++++++++++++++ hw/net/pvrdma/pvrdma_rm.h | 54 ++ hw/net/pvrdma/pvrdma_rm_defs.h | 111 ++++ hw/net/pvrdma/pvrdma_types.h | 37 ++ hw/net/pvrdma/pvrdma_utils.c | 133 +++++ hw/net/pvrdma/pvrdma_utils.h | 41 ++ hw/net/pvrdma/trace-events | 9 + hw/pci/shpc.c | 11 +- include/exec/memory.h | 23 + include/exec/ram_addr.h | 3 +- include/hw/pci/pci_ids.h | 3 + include/qemu/cutils.h | 10 + include/qemu/osdep.h | 2 +- include/sysemu/hostmem.h | 2 +- include/sysemu/kvm.h | 2 +- memory.c | 16 +- util/oslib-posix.c | 4 +- util/oslib-win32.c | 2 +- 44 files changed, 5378 insertions(+), 61 deletions(-) create mode 100644 docs/pvrdma.txt create mode 100644 hw/net/pvrdma/pvrdma.h create mode 100644 hw/net/pvrdma/pvrdma_backend.c create mode 100644 hw/net/pvrdma/pvrdma_backend.h create mode 100644 hw/net/pvrdma/pvrdma_backend_defs.h create mode 100644 hw/net/pvrdma/pvrdma_cmd.c create mode 100644 hw/net/pvrdma/pvrdma_defs.h create mode 100644 hw/net/pvrdma/pvrdma_dev_api.h create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.c create mode 100644 hw/net/pvrdma/pvrdma_dev_ring.h create mode 100644 hw/net/pvrdma/pvrdma_ib_verbs.h create mode 100644 hw/net/pvrdma/pvrdma_main.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.c create mode 100644 hw/net/pvrdma/pvrdma_qp_ops.h create mode 100644 hw/net/pvrdma/pvrdma_ring.h create mode 100644 hw/net/pvrdma/pvrdma_rm.c create mode 100644 hw/net/pvrdma/pvrdma_rm.h create mode 100644 hw/net/pvrdma/pvrdma_rm_defs.h create mode 100644 hw/net/pvrdma/pvrdma_types.h create mode 100644 hw/net/pvrdma/pvrdma_utils.c create mode 100644 hw/net/pvrdma/pvrdma_utils.h create mode 100644 hw/net/pvrdma/trace-eventsSignature · 1 line
2.13.5