Skip to content

RFC some progress on rpcrdma bug [was: Re: [PATCH v8 5/7] NFSD: issue READs using O_DIRECT even if IO is misaligned]

Mike Snitzer linux-nfs 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest2 patches 15 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2NFSD: fix misaligned DIO READ to not use a start_extra_page, exposes rpcrdma bug?fs/nfsd/vfs.c+6 -1914 replies
2/2NFSD: use /end/ of rq_pages for front_pad page, simpler workaround for rpcrdma bugfs/nfsd/vfs.c+3 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/2] some progress on rpcrdma bug [was: Re: [PATCH v8 5/7] NFSD: issue READs using O_DIRECT even if IO is misaligned]

Hi Chuck,

[just including context from thread in cover-letter for these 2 RFC patches]

Quoted 4 lines · Mike Snitzer
Quoted 3 lines · Chuck Lever
Quoted 2 lines · Mike Snitzer
Quoted 1 line · Chuck Lever
Quoted 1 line

<snip>

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines

- When the I/O is complete, adjust the offset in the first bvec entry forward by setting a non-zero page offset, and adjust the returned count downward to match the requested byte count from the client

Tried it long ago, such bvec manipulation only works when not using RDMA. When the memory is remote, twiddling a local bvec isn't going to ensure the correct pages have the correct data upon return to the client.

RDMA is why the pages must be used in-place, and RDMA is also why the extra page needed by this patch (for use as throwaway front-pad for expanded misaligned DIO READ) must either be allocated _or_ hopefully it can be from rq_pages (after the end of the client requested READ payload).

<snip>

Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines

There's nothing I can think of in the RDMA or RPC/RDMA protocols that mandates that the first page offset must always be zero. Moving data at one address on the server to an entirely different address and alignment on the client is exactly what RDMA is supposed to do.

It sounds like an implementation omission because the server's upper layers have never needed it before now. If TCP already handles it, I'm guessing it's going to be straightforward to fix.

I never said that first page offset must be zero. I said that I already did what you suggested and it didn't work with RDMA. This is recall of too many months ago now, but: the client will see the correct READ payload _except_ IIRC it is offset by whatever front-pad was added to expand the misaligned DIO; no matter whether rqstp->rq_bvec updated when IO completes.

Quoted 5 lines
Quoted 1 line

But I'll revisit it again.

For the record, this email thread is the very first time I've heard that you tried the simple approach and that it worked with TCP and not with RDMA. I wish I had known that a while ago.

Likewise, but the story is all in the patch header and the code tells the story too. Hence your finding it with closer review (thanks for that BTW!). I agree something is off so I'm happy to work it further.

Quoted 7 lines

I have iterated on quite a few aspects to this patch 5. Christoph had suggestion for using memmove in nfsd_complete_misaligned_read_dio. You had the feedback that required ensuring the lightest touch relative to branching so that buffered IO mode remain as fast as possible.

Looking forward to tackling this RDMA-specific weirdness now.

Hopeful these 2 patches more clearly demonstrate what I'm finding needed when using RDMA with my NFSD misaligned DIO READ patch.

These patches build ontop of my v8 patchset. I've included quite a lot of context for the data mismatch seen by the NFS client, etc in the patch headers.

If I'm understanding you correctly, next step is to look closer at the rpcrdma code that would skip the throwaway front-pad page from being mapped to the start of the RDMA READ payload returned to the NFS client?

Such important adjustment code would need to know that the rq_bvec[] that reflects the READ payload doesn't include a bvec that points to the first page of rqstp->rq_pages (pointed to by rqstp->rq_next_page on entry to nfsd_iter_read) -- so it must skip past that memory in the READ payload's RDMA memory returned to NFS client?

Thanks,
Mike

Mike Snitzer (2):
NFSD: fix misaligned DIO READ to not use a start_extra_page, exposes rpcrdma bug?
NFSD: use /end/ of rq_pages for front_pad page, simpler workaround for rpcrdma bug

 fs/nfsd/vfs.c | 27 ++++++++-------------------
 1 file changed, 8 insertions(+), 19 deletions(-)
Signature · 1 line
2.44.0