Skip to contentSkip to conversation

nfsd: Fix NFSv4 READ on RDMA when using readv

RFC PATCH In review Chuck Lever 2 participants 4 messages Jan 15, 2020 – Jan 17, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 nfsd: Fix NFSv4 READ on RDMA when using readv

svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "xdr->iov = NULL" forces xdr_reserve_space() to always use pages from xdr->buf->pages when calling nfsd_readv.

Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in xdr->buf->tail so that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.

Fixes b04209806384 ("nfsd4: allow exotic read compounds")

---
Howdy Bruce-

I'm struggling with nfsd4_encode_readv().

- for NFS/RDMA Write chunks, the READ payload has to be in buf->pages. I've fixed that.

- xdr_reserve_space() calls don't need to explicitly align the @nbytes argument: xdr_reserve_space() already does this?

- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.

- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.

Do you have suggestions? Thanks in advance.

 fs/nfsd/nfs4xdr.c |   17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
fs/nfsd/nfs4xdr.c +7 -10
index d2dc4c0e22e8..14c68a136b4e 100644
@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	u32 zzz = 0;
 	int pad;
 
+	/* Ensure xdr_reserve_space behaves itself */
+	if (xdr->iov == xdr->buf->head) {
+		xdr->iov = NULL;
+		xdr->end = xdr->p;
+	}
+
 	len = maxcount;
 	v = 0;
-
-	thislen = min_t(long, len, ((void *)xdr->end - (void *)xdr->p));
-	p = xdr_reserve_space(xdr, (thislen+3)&~3);
-	WARN_ON_ONCE(!p);
-	resp->rqstp->rq_vec[v].iov_base = p;
-	resp->rqstp->rq_vec[v].iov_len = thislen;
-	v++;
-	len -= thislen;
-
 	while (len) {
 		thislen = min_t(long, len, PAGE_SIZE);
 		p = xdr_reserve_space(xdr, (thislen+3)&~3);
@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	read->rd_length = maxcount;
 	if (nfserr)
 		return nfserr;
-	xdr_truncate_encode(xdr, starting_len + 8 + ((maxcount+3)&~3));
+	xdr_truncate_encode(xdr, starting_len + 8 + maxcount);
 
 	tmp = htonl(eof);
 	write_bytes_to_xdr_buf(xdr->buf, starting_len    , &tmp, 4);
#1 lore raw
Quoted 28 lines · Chuck Lever

svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "xdr->iov = NULL" forces xdr_reserve_space() to always use pages from xdr->buf->pages when calling nfsd_readv.

Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in xdr->buf->tail so that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.

Fixes b04209806384 ("nfsd4: allow exotic read compounds")

---
Howdy Bruce-

I'm struggling with nfsd4_encode_readv().

- for NFS/RDMA Write chunks, the READ payload has to be in buf->pages. I've fixed that.

- xdr_reserve_space() calls don't need to explicitly align the @nbytes argument: xdr_reserve_space() already does this?

Quoted 8 lines

- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.

- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.

Do you have suggestions? Thanks in advance.

I'm experimenting with an idea I think has been mentioned on list a few times:

Having the RPC layer and transports deal with the padding of the xdr_buf->pages vector, and moving that responsibility out of the NFSD Reply encoder functions. xdr_buf->tail[0] then always begins on an XDR-aligned boundary.

This should be straightforward for NFSv3. The only two Reply encoders that are updated are READ and READLINK. I'm starting with that.

Not quite sure yet how krb5i/krb5p will deal with this. Obviously the page list pad needs to be inserted before each Reply is wrapped.

I'll post more as experimentation progresses.

Quoted 33 lines

fs/nfsd/nfs4xdr.c | 17 +++++++----------

1 file changed, 7 insertions(+), 10 deletions(-)
fs/nfsd/nfs4xdr.c +6 -9
index d2dc4c0e22e8..14c68a136b4e 100644
@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	u32 zzz = 0;
 	int pad;
 
+	/* Ensure xdr_reserve_space behaves itself */
+	if (xdr->iov == xdr->buf->head) {
+		xdr->iov = NULL;
+		xdr->end = xdr->p;
+	}
+
 	len = maxcount;
 	v = 0;
-
-	thislen = min_t(long, len, ((void *)xdr->end - (void *)xdr->p));
-	p = xdr_reserve_space(xdr, (thislen+3)&~3);
-	WARN_ON_ONCE(!p);
-	resp->rqstp->rq_vec[v].iov_base = p;
-	resp->rqstp->rq_vec[v].iov_len = thislen;
-	v++;
-	len -= thislen;
-
 	while (len) {
 		thislen = min_t(long, len, PAGE_SIZE);
 		p = xdr_reserve_space(xdr, (thislen+3)&~3);

fs/nfsd/nfs4xdr.c static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, · line 3551

Quoted 8 lines
fs/nfsd/nfs4xdr.c +1 -1
@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	read->rd_length = maxcount;
 	if (nfserr)
 		return nfserr;
-	xdr_truncate_encode(xdr, starting_len + 8 + ((maxcount+3)&~3));
+	xdr_truncate_encode(xdr, starting_len + 8 + maxcount);
 
 	tmp = htonl(eof);
 	write_bytes_to_xdr_buf(xdr->buf, starting_len    , &tmp, 4);

--
Chuck Lever

#2 lore raw
In reply to Chuck Lever
Quoted 21 lines · Chuck Lever

svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "xdr->iov = NULL" forces xdr_reserve_space() to always use pages from xdr->buf->pages when calling nfsd_readv.

Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in xdr->buf->tail so that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.

Fixes b04209806384 ("nfsd4: allow exotic read compounds")

---
Howdy Bruce-

I'm struggling with nfsd4_encode_readv().

- for NFS/RDMA Write chunks, the READ payload has to be in buf->pages. I've fixed that.

Quoted 6 lines

- xdr_reserve_space() calls don't need to explicitly align the @nbytes argument: xdr_reserve_space() already does this?

- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.

So the Solaris client sends multiple reads per compound in that case?

Quoted 2 lines

- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.

krb5i/p has to treat read data as padded regardless of the transport, doesn't it?

--b.

Quoted 43 lines

Do you have suggestions? Thanks in advance.

 fs/nfsd/nfs4xdr.c |   17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
fs/nfsd/nfs4xdr.c +7 -10
index d2dc4c0e22e8..14c68a136b4e 100644
@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	u32 zzz = 0;
 	int pad;
 
+	/* Ensure xdr_reserve_space behaves itself */
+	if (xdr->iov == xdr->buf->head) {
+		xdr->iov = NULL;
+		xdr->end = xdr->p;
+	}
+
 	len = maxcount;
 	v = 0;
-
-	thislen = min_t(long, len, ((void *)xdr->end - (void *)xdr->p));
-	p = xdr_reserve_space(xdr, (thislen+3)&~3);
-	WARN_ON_ONCE(!p);
-	resp->rqstp->rq_vec[v].iov_base = p;
-	resp->rqstp->rq_vec[v].iov_len = thislen;
-	v++;
-	len -= thislen;
-
 	while (len) {
 		thislen = min_t(long, len, PAGE_SIZE);
 		p = xdr_reserve_space(xdr, (thislen+3)&~3);
@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	read->rd_length = maxcount;
 	if (nfserr)
 		return nfserr;
-	xdr_truncate_encode(xdr, starting_len + 8 + ((maxcount+3)&~3));
+	xdr_truncate_encode(xdr, starting_len + 8 + maxcount);
 
 	tmp = htonl(eof);
 	write_bytes_to_xdr_buf(xdr->buf, starting_len    , &tmp, 4);
#3 lore raw
Quoted 26 lines · J. Bruce Fields
Quoted 21 lines · Chuck Lever

svcrdma expects that the READ payload falls precisely into the xdr_buf's page vector. Adding "xdr->iov = NULL" forces xdr_reserve_space() to always use pages from xdr->buf->pages when calling nfsd_readv.

Also, the XDR padding is problematic. For NFS/RDMA Write chunks, the padding needs to be in xdr->buf->tail so that the transport can skip over it. However for NFS/TCP and the NFS/RDMA Reply chunks, the padding has to be retained. Not yet sure how to add this.

Fixes b04209806384 ("nfsd4: allow exotic read compounds")

---
Howdy Bruce-

I'm struggling with nfsd4_encode_readv().

- for NFS/RDMA Write chunks, the READ payload has to be in buf->pages. I've fixed that.

Quoted 8 lines
Quoted 6 lines

- xdr_reserve_space() calls don't need to explicitly align the @nbytes argument: xdr_reserve_space() already does this?

- the while loop probably won't work if a later READ in the COMPOUND doesn't start on a page boundary. This isn't a problem until we run into a Solaris client in forcedirectio mode.

So the Solaris client sends multiple reads per compound in that case?

If the application uses readv() and the mount is in forcedirectio mode, I'm told Solaris will send an NFSv4 COMPOUND with multiple READ operations in it.

Quoted 5 lines
Quoted 2 lines

- the XDR padding doesn't work for NFS/RDMA Write chunks, which are supposed to skip padding altogether.

krb5i/p has to treat read data as padded regardless of the transport, doesn't it?

Yes. See my reply from earlier today: I think we can get away with inserting the pad just before wrapping the Reply.

Quoted 43 lines

--b.

Quoted 41 lines

Do you have suggestions? Thanks in advance.

fs/nfsd/nfs4xdr.c | 17 +++++++----------

1 file changed, 7 insertions(+), 10 deletions(-)
fs/nfsd/nfs4xdr.c +7 -10
index d2dc4c0e22e8..14c68a136b4e 100644
@@ -3519,17 +3519,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	u32 zzz = 0;
 	int pad;
 
+	/* Ensure xdr_reserve_space behaves itself */
+	if (xdr->iov == xdr->buf->head) {
+		xdr->iov = NULL;
+		xdr->end = xdr->p;
+	}
+
 	len = maxcount;
 	v = 0;
-
-	thislen = min_t(long, len, ((void *)xdr->end - (void *)xdr->p));
-	p = xdr_reserve_space(xdr, (thislen+3)&~3);
-	WARN_ON_ONCE(!p);
-	resp->rqstp->rq_vec[v].iov_base = p;
-	resp->rqstp->rq_vec[v].iov_len = thislen;
-	v++;
-	len -= thislen;
-
 	while (len) {
 		thislen = min_t(long, len, PAGE_SIZE);
 		p = xdr_reserve_space(xdr, (thislen+3)&~3);
@@ -3548,7 +3545,7 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp,
 	read->rd_length = maxcount;
 	if (nfserr)
 		return nfserr;
-	xdr_truncate_encode(xdr, starting_len + 8 + ((maxcount+3)&~3));
+	xdr_truncate_encode(xdr, starting_len + 8 + maxcount);
 
Quoted 2 lines

fs/nfsd/nfs4xdr.c static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, · line 3551

Quoted 2 lines
fs/nfsd/nfs4xdr.c +0 -0
 	tmp = htonl(eof);
 	write_bytes_to_xdr_buf(xdr->buf, starting_len    , &tmp, 4);

--
Chuck Lever