Date:   Wed, 24 Jun 2020 11:00:47 -0300
From:   Jason Gunthorpe <jgg@ziepe.ca>
To:     Maor Gottlieb <maorg@mellanox.com>
Cc:     Leon Romanovsky <leon@kernel.org>,
        Doug Ledford <dledford@redhat.com>, linux-rdma@vger.kernel.org
Subject: Re: [PATCH rdma-next v1 2/2] RDMA/core: Optimize XRC target lookup
Message-ID: <20200624140047.GG6578@ziepe.ca>
References: <20200623111531.1227013-1-leon@kernel.org>
 <20200623111531.1227013-3-leon@kernel.org>
 <20200623175200.GA3096958@mellanox.com>
 <20200623181506.GC184720@unreal>
 <20200623184940.GN2874652@mellanox.com>
 <d5206962-69ae-48e7-261b-485db71d2a41@mellanox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <d5206962-69ae-48e7-261b-485db71d2a41@mellanox.com>
Sender: linux-rdma-owner@vger.kernel.org
Precedence: bulk

On Wed, Jun 24, 2020 at 01:42:49PM +0300, Maor Gottlieb wrote:
> 
> On 6/23/2020 9:49 PM, Jason Gunthorpe wrote:
> > On Tue, Jun 23, 2020 at 09:15:06PM +0300, Leon Romanovsky wrote:
> > > On Tue, Jun 23, 2020 at 02:52:00PM -0300, Jason Gunthorpe wrote:
> > > > On Tue, Jun 23, 2020 at 02:15:31PM +0300, Leon Romanovsky wrote:
> > > > > From: Maor Gottlieb <maorg@mellanox.com>
> > > > > 
> > > > > Replace the mutex with read write semaphore and use xarray instead
> > > > > of linked list for XRC target QPs. This will give faster XRC target
> > > > > lookup. In addition, when QP is closed, don't insert it back to the
> > > > > xarray if the destroy command failed.
> > > > > 
> > > > > Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
> > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > >   drivers/infiniband/core/verbs.c | 57 ++++++++++++---------------------
> > > > >   include/rdma/ib_verbs.h         |  5 ++-
> > > > >   2 files changed, 23 insertions(+), 39 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
> > > > > index d66a0ad62077..1ccbe43e33cd 100644
> > > > > +++ b/drivers/infiniband/core/verbs.c
> > > > > @@ -1090,13 +1090,6 @@ static void __ib_shared_qp_event_handler(struct ib_event *event, void *context)
> > > > >   	spin_unlock_irqrestore(&qp->device->qp_open_list_lock, flags);
> > > > >   }
> > > > > 
> > > > > -static void __ib_insert_xrcd_qp(struct ib_xrcd *xrcd, struct ib_qp *qp)
> > > > > -{
> > > > > -	mutex_lock(&xrcd->tgt_qp_mutex);
> > > > > -	list_add(&qp->xrcd_list, &xrcd->tgt_qp_list);
> > > > > -	mutex_unlock(&xrcd->tgt_qp_mutex);
> > > > > -}
> > > > > -
> > > > >   static struct ib_qp *__ib_open_qp(struct ib_qp *real_qp,
> > > > >   				  void (*event_handler)(struct ib_event *, void *),
> > > > >   				  void *qp_context)
> > > > > @@ -1139,16 +1132,15 @@ struct ib_qp *ib_open_qp(struct ib_xrcd *xrcd,
> > > > >   	if (qp_open_attr->qp_type != IB_QPT_XRC_TGT)
> > > > >   		return ERR_PTR(-EINVAL);
> > > > > 
> > > > > -	qp = ERR_PTR(-EINVAL);
> > > > > -	mutex_lock(&xrcd->tgt_qp_mutex);
> > > > > -	list_for_each_entry(real_qp, &xrcd->tgt_qp_list, xrcd_list) {
> > > > > -		if (real_qp->qp_num == qp_open_attr->qp_num) {
> > > > > -			qp = __ib_open_qp(real_qp, qp_open_attr->event_handler,
> > > > > -					  qp_open_attr->qp_context);
> > > > > -			break;
> > > > > -		}
> > > > > +	down_read(&xrcd->tgt_qps_rwsem);
> > > > > +	real_qp = xa_load(&xrcd->tgt_qps, qp_open_attr->qp_num);
> > > > > +	if (!real_qp) {
> > > > Don't we already have a xarray indexed against qp_num in res_track?
> > > > Can we use it somehow?
> > > We don't have restrack for XRC, we will need somehow manage QP-to-XRC
> > > connection there.
> > It is not xrc, this is just looking up a qp and checking if it is part
> > of the xrcd
> > 
> > Jason
> 
> It's the XRC target  QP and it is not tracked.

Really? Something called 'real_qp' isn't stored in the restrack?
Doesn't that sound like a bug already?

Jason