Fix request completion holes

Quoted 2 lines

iWARP spec too. This is in regard to completion ordering though. The

Quoted 1 line

invalidate send WR must have the IB_SEND_FENCE flag set if you want it to

Quoted 1 line

be executed after all prior send WRs are executed.

Oh right, yes, you need to have local invalidate fence to guarentee that, but usually a local operation will not refer to a RKEY, so execution ordering wont matter.

But this is a good general point, doesn't a ULP need to set FENCE on SEND, eg:

RDMA READ (rkey)
RDMA READ (rkey)
SEND w/ FENCE (tell remote to invalidate rkey)

Otherwise IBA Table 79 says RDMA READ can pass SEND and we have a situation where the rkey has become invalidated when the remote is still trying to use it.

IBA actually explicitly calls this out (pg 538):

RDMA Read operations may be executed at the target after subse- quent Send and Invalidate operation already performed the invalida- tion at the target. That may cause the RDMA Read operation to fail. Setting the Fence Indicator on the subsequent operations guarantees that the RDMA Read will fully complete before the invalidation takes place.

None of our ULPs use IB_SEND_FENCE - so this is a bug!