nfsd: Fix NFSv4 READ on RDMA when using readv
Subsystems (2)
RFC PATCH 1/1 nfsd: Fix NFSv4 READ on RDMA when using readv
svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "xdr->iov = NULL" forces xdr_reserve_space() to always use pages from xdr->buf->pages when calling nfsd_readv.
Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in xdr->buf->tail so that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.
---
Howdy Bruce-
I'm struggling with nfsd4_encode_readv().
- for NFS/RDMA Write chunks, the READ payload has to be in buf->pages. I've fixed that.
- xdr_reserve_space() calls don't need to explicitly align the @nbytes argument: xdr_reserve_space() already does this?
- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.
- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.
Do you have suggestions? Thanks in advance.
fs/nfsd/nfs4xdr.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)
fs/nfsd/nfs4xdr.c +7 -10
@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,u32 zzz =0 ;int pad; +/* Ensure xdr_reserve_space behaves itself */ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + len = maxcount; v =0 ; - - thislen =min_t (long, len, ((void *)xdr->end - (void *)xdr->p)); - p =xdr_reserve_space (xdr, (thislen+3 )&~3 ); -WARN_ON_ONCE (!p); - resp->rqstp->rq_vec[v].iov_base = p; - resp->rqstp->rq_vec[v].iov_len = thislen; - v++; - len -= thislen; -while (len) { thislen =min_t (long, len,PAGE_SIZE ); p =xdr_reserve_space (xdr, (thislen+3 )&~3 );
@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, read->rd_length = maxcount;if (nfserr)return nfserr; -xdr_truncate_encode (xdr, starting_len +8 + ((maxcount+3 )&~3 )); +xdr_truncate_encode (xdr, starting_len +8 + maxcount); tmp =htonl (eof);write_bytes_to_xdr_buf (xdr->buf, starting_len , &tmp,4 );
Quoted 28 lines · Chuck Lever
svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "
xdr->iov= NULL" forcesxdr_reserve_space()to always use pages fromxdr->buf->pageswhen callingnfsd_readv.Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in
xdr->buf->tailso that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.Fixes b04209806384 ("nfsd4: allow exotic read compounds")Signed-off-by Chuck Lever <[email protected]>---
Howdy Bruce-I'm struggling with
nfsd4_encode_readv().- for NFS/RDMA Write chunks, the READ payload has to be in
buf->pages. I've fixed that.-
xdr_reserve_space()calls don't need to explicitly align the @nbytes argument:xdr_reserve_space()already does this?
Quoted 8 lines
- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.
- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.
Do you have suggestions? Thanks in advance.
I'm experimenting with an idea I think has been mentioned on list a few times:
Having the RPC layer and transports deal with the padding of the xdr_buf->pages vector, and moving that responsibility out of the NFSD Reply encoder functions. xdr_buf->tail[0] then always begins on an XDR-aligned boundary.
This should be straightforward for NFSv3. The only two Reply encoders that are updated are READ and READLINK. I'm starting with that.
Not quite sure yet how krb5i/krb5p will deal with this. Obviously the page list pad needs to be inserted before each Reply is wrapped.
I'll post more as experimentation progresses.
Quoted 33 lines
fs/nfsd/nfs4xdr.c| 17 +++++++----------1 file changed, 7 insertions(+), 10 deletions(-)fs/nfsd/nfs4xdr.c +6
-9@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,u32 zzz =0 ;int pad; +/* Ensure xdr_reserve_space behaves itself */ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + len = maxcount; v =0 ; - - thislen =min_t (long, len, ((void *)xdr->end - (void *)xdr->p)); - p =xdr_reserve_space (xdr, (thislen+3 )&~3 ); -WARN_ON_ONCE (!p); - resp->rqstp->rq_vec[v].iov_base = p; - resp->rqstp->rq_vec[v].iov_len = thislen; - v++; - len -= thislen; -while (len) { thislen =min_t (long, len,PAGE_SIZE ); p =xdr_reserve_space (xdr, (thislen+3 )&~3 );
fs/nfsd/nfs4xdr.c static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, · line 3551
Quoted 8 lines
fs/nfsd/nfs4xdr.c +1
-1@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, read->rd_length = maxcount;if (nfserr)return nfserr; -xdr_truncate_encode (xdr, starting_len +8 + ((maxcount+3 )&~3 )); +xdr_truncate_encode (xdr, starting_len +8 + maxcount); tmp =htonl (eof);write_bytes_to_xdr_buf (xdr->buf, starting_len , &tmp,4 );
--
Chuck Lever
Quoted 21 lines · Chuck Lever
svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "
xdr->iov= NULL" forcesxdr_reserve_space()to always use pages fromxdr->buf->pageswhen callingnfsd_readv.Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in
xdr->buf->tailso that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.Fixes b04209806384 ("nfsd4: allow exotic read compounds")Signed-off-by Chuck Lever <[email protected]>---
Howdy Bruce-I'm struggling with
nfsd4_encode_readv().- for NFS/RDMA Write chunks, the READ payload has to be in
buf->pages. I've fixed that.
Quoted 6 lines
-
xdr_reserve_space()calls don't need to explicitly align the @nbytes argument:xdr_reserve_space()already does this?- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.
So the Solaris client sends multiple reads per compound in that case?
Quoted 2 lines
- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.
krb5i/p has to treat read data as padded regardless of the transport, doesn't it?
--b.
Quoted 43 lines
Do you have suggestions? Thanks in advance.
fs/nfsd/nfs4xdr.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)fs/nfsd/nfs4xdr.c +7
-10@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,u32 zzz =0 ;int pad; +/* Ensure xdr_reserve_space behaves itself */ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + len = maxcount; v =0 ; - - thislen =min_t (long, len, ((void *)xdr->end - (void *)xdr->p)); - p =xdr_reserve_space (xdr, (thislen+3 )&~3 ); -WARN_ON_ONCE (!p); - resp->rqstp->rq_vec[v].iov_base = p; - resp->rqstp->rq_vec[v].iov_len = thislen; - v++; - len -= thislen; -while (len) { thislen =min_t (long, len,PAGE_SIZE ); p =xdr_reserve_space (xdr, (thislen+3 )&~3 );@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, read->rd_length = maxcount;if (nfserr)return nfserr; -xdr_truncate_encode (xdr, starting_len +8 + ((maxcount+3 )&~3 )); +xdr_truncate_encode (xdr, starting_len +8 + maxcount); tmp =htonl (eof);write_bytes_to_xdr_buf (xdr->buf, starting_len , &tmp,4 );
Quoted 26 lines · J. Bruce Fields
Quoted 21 lines · Chuck Lever
svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "
xdr->iov= NULL" forcesxdr_reserve_space()to always use pages fromxdr->buf->pageswhen callingnfsd_readv.Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in
xdr->buf->tailso that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.Fixes b04209806384 ("nfsd4: allow exotic read compounds")Signed-off-by Chuck Lever <[email protected]>---
Howdy Bruce-I'm struggling with
nfsd4_encode_readv().- for NFS/RDMA Write chunks, the READ payload has to be in
buf->pages. I've fixed that.
Quoted 8 lines
Quoted 6 lines
-
xdr_reserve_space()calls don't need to explicitly align the @nbytes argument:xdr_reserve_space()already does this?- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.
So the Solaris client sends multiple reads per compound in that case?
If the application uses readv() and the mount is in forcedirectio mode, I'm told Solaris will send an NFSv4 COMPOUND with multiple READ operations in it.
Quoted 5 lines
Quoted 2 lines
- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.
krb5i/p has to treat read data as padded regardless of the transport, doesn't it?
Yes. See my reply from earlier today: I think we can get away with inserting the pad just before wrapping the Reply.
Quoted 43 lines
--b.
Quoted 41 lines
Do you have suggestions? Thanks in advance.
fs/nfsd/nfs4xdr.c| 17 +++++++----------1 file changed, 7 insertions(+), 10 deletions(-)fs/nfsd/nfs4xdr.c +7
-10@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,u32 zzz =0 ;int pad; +/* Ensure xdr_reserve_space behaves itself */ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + len = maxcount; v =0 ; - - thislen =min_t (long, len, ((void *)xdr->end - (void *)xdr->p)); - p =xdr_reserve_space (xdr, (thislen+3 )&~3 ); -WARN_ON_ONCE (!p); - resp->rqstp->rq_vec[v].iov_base = p; - resp->rqstp->rq_vec[v].iov_len = thislen; - v++; - len -= thislen; -while (len) { thislen =min_t (long, len,PAGE_SIZE ); p =xdr_reserve_space (xdr, (thislen+3 )&~3 );@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, read->rd_length = maxcount;if (nfserr)return nfserr; -xdr_truncate_encode (xdr, starting_len +8 + ((maxcount+3 )&~3 )); +xdr_truncate_encode (xdr, starting_len +8 + maxcount);
Quoted 2 lines
fs/nfsd/nfs4xdr.c static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, · line 3551
Quoted 2 lines
fs/nfsd/nfs4xdr.c +0
-0tmp =htonl (eof);write_bytes_to_xdr_buf (xdr->buf, starting_len , &tmp,4 );
--
Chuck Lever