Skip to content

Fix request completion holes

Jason Gunthorpe linux-nvme linux-rdma 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest0 of 3 patches 0 repliesNew

Patches in v1 No applies recorded yet

None of the 3 patches the cover letter announces are in this archive.

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/3] Fix request completion holes
Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
Quoted 2 lines

the default mode utilize remote invalidation, so no, its not valid.

Well, usually the ULP design should allow some things to be reaped async, and latency senstive things to be sync.

Eg if you have a SEND using a local buffer with a RKEY, then you'd declare the RKEY data completed when SEND_WITH_INVALIDATE is returned.

Quoted 7 lines
Quoted 3 lines

Recycling the lkey command buffer is an async process and can wait unsignaled until something signalled comes to push its completion through.

Not when using inline data with the send, which is the main issue here. if we inline data to the command, we will use the local dma lkey, which does not even have a local invalidate following it.

Does nvme use inline data send and RKEY transfer in the same SEND? Then it would need to signal the SEND if remote invalidate is used, otherwise it only needs to signal the local invalidate for the RKEY..

Quoted 5 lines
Quoted 2 lines

Local invalidate is defined to always be ordered by the spec, so it is required to guarentee that the SEND is completed.

So local invalidate completion is guaranteed to come after all the completions prior to it in the send queue?

IBA spec says so..

Jason