Skip to contentSkip to conversation

IB/core: Guarantee that a local_dma_lkey is available

PATCH In review Jason Gunthorpe 2 participants 2 messages Aug 5, 2015 – Aug 15, 2015 1 message not in the archive InfiniBand

lore
Share & tools
lore

Jason Gunthorpe

author
#0 lore raw

PATCH v4 1/12 IB/core: Guarantee that a local_dma_lkey is available

Every single ULP requires a local_dma_lkey to do anything with a QP, so let us ensure one exists for every PD created.

If the driver can supply a global local_dma_lkey then use that, otherwise ask the driver to create a local use all physical memory MR associated with the new PD.

Signed-off-by Jason Gunthorpe <[email protected]>
Reviewed-by Sagi Grimberg <[email protected]>
Acked-by Christoph Hellwig <[email protected]>
Reviewed-by Steve Wise <[email protected]>
Reviewed-by Ira Weiny <[email protected]>
Tested-by Ira Weiny <[email protected]>
---
 drivers/infiniband/core/uverbs_cmd.c |  1 +
 drivers/infiniband/core/verbs.c      | 47 ++++++++++++++++++++++++++++++++----
 include/rdma/ib_verbs.h              |  9 ++-----
 3 files changed, 45 insertions(+), 12 deletions(-)

v4 change
- The order of the atomic_read and mr destruction is swapped. The old order was intended to not destroy anything if we still have users, but the mr consitutes a user, so that just doesn't work. Since nothing should touch the mr pointer we can safely destory it.

This bug means mr using drivers (ie mthca) would silently never tear down a pd because we don't check for errors..

Tested again by faking no-local_dma_lkey support and using my new error clean up patch.

drivers/infiniband/core/uverbs_cmd.c +1 -0
index bbb02ffe87df..258485ee46b2 100644
@@ -562,6 +562,7 @@ ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file,
 
 	pd->device  = file->device->ib_dev;
 	pd->uobject = uobj;
+	pd->local_mr = NULL;
 	atomic_set(&pd->usecnt, 0);
 
 	uobj->object = pd;
drivers/infiniband/core/verbs.c +42 -5

66 lines not on this page · raw message

include/rdma/ib_verbs.h +2 -7
index b0f898e3b2e7..eaec3081fb87 100644
@@ -1252,9 +1252,11 @@ struct ib_udata {
 };
 
 struct ib_pd {
+	u32			local_dma_lkey;
 	struct ib_device       *device;
 	struct ib_uobject      *uobject;
 	atomic_t          	usecnt; /* count all resources */
+	struct ib_mr	       *local_mr;
 };
 
 struct ib_xrcd {

@@ -2135,13 +2137,6 @@ int ib_find_gid(struct ib_device *device, union ib_gid *gid,
 int ib_find_pkey(struct ib_device *device,
 		 u8 port_num, u16 pkey, u16 *index);
 
-/**
- * ib_alloc_pd - Allocates an unused protection domain.
- * @device: The device on which to allocate the protection domain.
- *
- * A protection domain object provides an association between QPs, shared
- * receive queues, address handles, memory regions, and memory windows.
- */
 struct ib_pd *ib_alloc_pd(struct ib_device *device);
 
 /**
Signature · 1 line
2.1.4

A message that is not in the archive (sent off-list or not yet received): 20150805201445.GA30271-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#2 lore raw
Quoted 7 lines · Jason Gunthorpe

Every single ULP requires a local_dma_lkey to do anything with a QP, so let us ensure one exists for every PD created.

If the driver can supply a global local_dma_lkey then use that, otherwise ask the driver to create a local use all physical memory MR associated with the new PD.

Quoted 6 lines
Signed-off-by Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Reviewed-by Sagi Grimberg <[email protected]>
Acked-by Christoph Hellwig <[email protected]>
Reviewed-by Steve Wise <[email protected]>
Reviewed-by Ira Weiny <[email protected]>
Tested-by Ira Weiny <[email protected]>

...and then I switched it up for this one ;-)

Signature · 2 lines
Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD