RFC some progress on rpcrdma bug [was: Re: [PATCH v8 5/7] NFSD: issue READs using O_DIRECT even if IO is misaligned]
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest2 patches 15 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | NFSD: fix misaligned DIO READ to not use a start_extra_page, exposes rpcrdma bug?fs/ | +6 | 14 replies | |
| 2/2 | NFSD: use /end/ of rq_pages for front_pad page, simpler workaround for rpcrdma bugfs/ | +3 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/2] some progress on rpcrdma bug [was: Re: [PATCH v8 5/7] NFSD: issue READs using O_DIRECT even if IO is misaligned]
Hi Chuck,
[just including context from thread in cover-letter for these 2 RFC patches]
Quoted 4 lines · Mike Snitzer
Quoted 3 lines · Chuck Lever
Quoted 2 lines · Mike Snitzer
Quoted 1 line · Chuck Lever
Quoted 1 line
<snip>
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
- When the I/O is complete, adjust the offset in the first bvec entry forward by setting a non-zero page offset, and adjust the returned count downward to match the requested byte count from the client
Tried it long ago, such bvec manipulation only works when not using RDMA. When the memory is remote, twiddling a local bvec isn't going to ensure the correct pages have the correct data upon return to the client.
RDMA is why the pages must be used in-place, and RDMA is also why the extra page needed by this patch (for use as throwaway front-pad for expanded misaligned DIO READ) must either be allocated _or_ hopefully it can be from
rq_pages(after the end of the client requested READ payload).
<snip>
Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
There's nothing I can think of in the RDMA or RPC/RDMA protocols that mandates that the first page offset must always be zero. Moving data at one address on the server to an entirely different address and alignment on the client is exactly what RDMA is supposed to do.
It sounds like an implementation omission because the server's upper layers have never needed it before now. If TCP already handles it, I'm guessing it's going to be straightforward to fix.
I never said that first page offset must be zero. I said that I already did what you suggested and it didn't work with RDMA. This is recall of too many months ago now, but: the client will see the correct READ payload _except_ IIRC it is offset by whatever front-pad was added to expand the misaligned DIO; no matter whether
rqstp->rq_bvecupdated when IO completes.Quoted 5 lines
Quoted 1 line
But I'll revisit it again.
For the record, this email thread is the very first time I've heard that you tried the simple approach and that it worked with TCP and not with RDMA. I wish I had known that a while ago.
Likewise, but the story is all in the patch header and the code tells the story too. Hence your finding it with closer review (thanks for that BTW!). I agree something is off so I'm happy to work it further.
Quoted 7 lines
I have iterated on quite a few aspects to this patch 5. Christoph had suggestion for using memmove in
nfsd_complete_misaligned_read_dio. You had the feedback that required ensuring the lightest touch relative to branching so that buffered IO mode remain as fast as possible.Looking forward to tackling this RDMA-specific weirdness now.
Hopeful these 2 patches more clearly demonstrate what I'm finding needed when using RDMA with my NFSD misaligned DIO READ patch.
These patches build ontop of my v8 patchset. I've included quite a lot of context for the data mismatch seen by the NFS client, etc in the patch headers.
If I'm understanding you correctly, next step is to look closer at the rpcrdma code that would skip the throwaway front-pad page from being mapped to the start of the RDMA READ payload returned to the NFS client?
Such important adjustment code would need to know that the rq_bvec[] that reflects the READ payload doesn't include a bvec that points to the first page of rqstp->rq_pages (pointed to by rqstp->rq_next_page on entry to nfsd_iter_read) -- so it must skip past that memory in the READ payload's RDMA memory returned to NFS client?
Thanks,
Mike
Mike Snitzer (2):
NFSD: fix misaligned DIO READ to not use a start_extra_page, exposes rpcrdma bug?
NFSD: use /end/ of rq_pages for front_pad page, simpler workaround for rpcrdma bug
fs/nfsd/vfs.c | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-)
Signature · 1 line
2.44.0