RDMA/vmw_pvrdma: Use resource ids from physical device if available
Subsystems (2)
PATCH 1/1 RDMA/vmw_pvrdma: Use resource ids from physical device if available
From: Bryan Tan <[email protected]>
From: Bryan Tan <[email protected]>
This change allows the RDMA stack to use physical resource numbers if they are passed up from the device. This is accomplished by separating the concept of the QP number from the QP handle. Previously, the two were the same, as the QP number was exposed to the guest and also used to reference a virtual QP in the device backend.
With physical resource numbers exposed, the QP number given to the guest is the number assigned from the physical HCA's QP, while the QP handle is still the internal handle used to reference a virtual QP. Regardless of whether the device is exposing physical ids, the driver will still try to pick up the QP handle from the backend if possible. The MR keys exposed to the guest will also be the MR keys created by the physical HCA, instead of virtual MR keys. The distinction between handle and keys is already present for MRs so there is no need to do anything special here.
A new version of the create QP response has been added to the device API to pass up the QP number and handle. The driver will also report these to userspace in the udata response if userspace supports it or not create the queuepair if not. I also had to do a refactor of the destroy qp code to reuse it if we fail to copy to userspace.
---
Github PR for rdma-core:
- https://github.com/linux-rdma/rdma-core/pull/581
Changes:
v0 -> v1:
- Dropped the ABI version bump (suggested by Jason)
- Added a new create qp flag to indicate userspace support
- Refactor of destroy/free qp
- Updated commit description (suggested by Yuval)
drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h | 15 ++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c | 118 +++++++++++++++++----- include/uapi/rdma/vmw_pvrdma-abi.h | 13 +++ 3 files changed, 117 insertions(+), 29 deletions(-)
drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h +14 -1
@@ -58,7 +58,8 @@#define PVRDMA_ROCEV1_VERSION 17#define PVRDMA_ROCEV2_VERSION 18#define PVRDMA_PPN64_VERSION 19 -#define PVRDMA_VERSION PVRDMA_PPN64_VERSION +#define PVRDMA_QPHANDLE_VERSION 20 +#define PVRDMA_VERSION PVRDMA_QPHANDLE_VERSION#define PVRDMA_BOARD_ID 1#define PVRDMA_REV_ID 1
@@ -581,6 +582,17 @@ struct pvrdma_cmd_create_qp_resp {u32 max_inline_data; }; +struct pvrdma_cmd_create_qp_resp_v2 { +struct pvrdma_cmd_resp_hdr hdr; +u32 qpn; +u32 qp_handle; +u32 max_send_wr; +u32 max_recv_wr; +u32 max_send_sge; +u32 max_recv_sge; +u32 max_inline_data; +}; +struct pvrdma_cmd_modify_qp {struct pvrdma_cmd_hdr hdr;u32 qp_handle;
@@ -663,6 +675,7 @@ struct pvrdma_cmd_destroy_bind {struct pvrdma_cmd_create_cq_resp create_cq_resp;struct pvrdma_cmd_resize_cq_resp resize_cq_resp;struct pvrdma_cmd_create_qp_resp create_qp_resp; +struct pvrdma_cmd_create_qp_resp_v2 create_qp_resp_v2;struct pvrdma_cmd_query_qp_resp query_qp_resp;struct pvrdma_cmd_destroy_qp_resp destroy_qp_resp;struct pvrdma_cmd_create_srq_resp create_srq_resp;
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +90 -28
183 lines not on this page · raw message
include/uapi/rdma/vmw_pvrdma-abi.h +13 -0
@@ -133,6 +133,10 @@ enum pvrdma_wc_flags { PVRDMA_WC_FLAGS_MAX = PVRDMA_WC_WITH_NETWORK_HDR_TYPE, }; +enum pvrdma_user_qp_create_flags { +PVRDMA_USER_QP_CREATE_USE_RESP =1 <<0 , +}; +struct pvrdma_alloc_ucontext_resp {__u32 qp_tab_size;__u32 reserved;
@@ -177,6 +181,15 @@ struct pvrdma_create_qp {__u32 rbuf_size;__u32 sbuf_size;__aligned_u64 qp_addr; +__u32 flags; +__u32 reserved; +}; + +struct pvrdma_create_qp_resp { +__u32 qpn; +__u32 qp_handle; +__u32 qpn_valid; +__u32 reserved; };/* PVRDMA masked atomic compare and swap */
Signature · 1 line
1.8.3.1
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 411
Quoted 3 lines · Adit Ranadive
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +2
-0+if (!qp->is_kernel) { +if (ucmd.flags ==PVRDMA_USER_QP_CREATE_USE_RESP ) {
Why does this flag exist? Don't old userspaces pass in a 0 length? Just use the length to signal new userspace?
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 417
Quoted 6 lines
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +6
-0+ qp_resp.qpn = qp->ibqp.qp_num; + qp_resp.qp_handle = qp->qp_handle; + qp_resp.qpn_valid =PVRDMA_USER_QP_CREATE_USE_RESP ; + +if (ib_copy_to_udata (udata, &qp_resp, +sizeof (qp_resp))) {
This should limit the copy size to the length of the user buffer
Jason
Quoted 9 lines · Jason Gunthorpe
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 411
Quoted 3 lines · Adit Ranadive
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +2
-0+if (!qp->is_kernel) { +if (ucmd.flags ==PVRDMA_USER_QP_CREATE_USE_RESP ) {Why does this flag exist? Don't old userspaces pass in a 0 length? Just use the length to signal new userspace?
I did have that in an earlier version but we decided it to make it more explicit. It would be easier to add another flag later on if required than to check the length (which might be same).
Quoted 9 lines
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 417
Quoted 7 lines
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +6
-0+ qp_resp.qpn = qp->ibqp.qp_num; + qp_resp.qp_handle = qp->qp_handle; + qp_resp.qpn_valid =PVRDMA_USER_QP_CREATE_USE_RESP ; + +if (ib_copy_to_udata (udata, &qp_resp, +sizeof (qp_resp))) {This should limit the copy size to the length of the user buffer
Probably should be min(sizeof(qp_resp), udata->outlen)?
Quoted 11 lines · Adit Ranadive
Quoted 9 lines · Jason Gunthorpe
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 411
Quoted 3 lines · Adit Ranadive
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +2
-0+if (!qp->is_kernel) { +if (ucmd.flags ==PVRDMA_USER_QP_CREATE_USE_RESP ) {Why does this flag exist? Don't old userspaces pass in a 0 length? Just use the length to signal new userspace?
Quoted 3 lines
I did have that in an earlier version but we decided it to make it more explicit. It would be easier to add another flag later on if required than to check the length (which might be same).
You should add a flag if either the length or detecting value == 0 is not sufficient to detect the new support. No reason to add things until this situation comes up
Jason
Quoted 16 lines · Jason Gunthorpe
Quoted 11 lines · Adit Ranadive
Quoted 9 lines · Jason Gunthorpe
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, · line 411
Quoted 3 lines · Adit Ranadive
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +2
-0+if (!qp->is_kernel) { +if (ucmd.flags ==PVRDMA_USER_QP_CREATE_USE_RESP ) {Why does this flag exist? Don't old userspaces pass in a 0 length? Just use the length to signal new userspace?
Quoted 3 lines
I did have that in an earlier version but we decided it to make it more explicit. It would be easier to add another flag later on if required than to check the length (which might be same).
Quoted 6 lines
You should add a flag if either the length or detecting value == 0 is not sufficient to detect the new support. No reason to add things until this situation comes up
Jason
It seemed a bit hacky to me but if that's better for now, then sure. Will send a v2 for both.
Thanks,
Adit