Skip to content

Fix request completion holes

Steve Wise linux-nvme linux-rdma 1 version New

Get series with b4
b4 am '[email protected]'

Versions newest first

  1. v1 Latest0 of 3 patches 0 repliesNew

Patches in v1 No applies recorded yet

None of the 3 patches the cover letter announces are in this archive.

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/3] Fix request completion holes
Quoted 33 lines

-----Original Message-----
From: linux-rdma-owner at vger.kernel.org [mailto:linux-rdma- owner at vger.kernel.org] On Behalf Of Jason Gunthorpe
Sent: Thursday, November 02, 2017 10:13 AM
To: Sagi Grimberg
Cc: idanb; Max Gurtovoy; linux-rdma at vger.kernel.org; linux- nvme at lists.infradead.org; Christoph Hellwig
Subject: Re: [PATCH 0/3] Fix request completion holes

Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
Quoted 2 lines

the default mode utilize remote invalidation, so no, its not valid.

Well, usually the ULP design should allow some things to be reaped async, and latency senstive things to be sync.

Eg if you have a SEND using a local buffer with a RKEY, then you'd declare the RKEY data completed when SEND_WITH_INVALIDATE is returned.

Quoted 7 lines
Quoted 3 lines

Recycling the lkey command buffer is an async process and can wait unsignaled until something signalled comes to push its completion through.

Not when using inline data with the send, which is the main issue here. if we inline data to the command, we will use the local dma lkey, which does not even have a local invalidate following it.

Does nvme use inline data send and RKEY transfer in the same SEND? Then it would need to signal the SEND if remote invalidate is used, otherwise it only needs to signal the local invalidate for the RKEY..

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Local invalidate is defined to always be ordered by the spec, so it is required to guarentee that the SEND is completed.

So local invalidate completion is guaranteed to come after all the completions prior to it in the send queue?

IBA spec says so..

iWARP spec too. This is in regard to completion ordering though. The local
invalidate send WR must have the IB_SEND_FENCE flag set if you want it to only
be executed after all prior send WRs are executed. Either way, the
completions are always inserted into the cq in-sq-submission-order and a signaled completion implies completions of all prior unsignaled WRs.

Steve.

Read the cover letter in its thread