NFSD: Add support for the v4.2 READ_PLUS operation
Subsystems (3)
v3 cover NFSD: Add support for the v4.2 READ_PLUS operation
From: Anna Schumaker <[email protected]>
These patches add server support for the READ_PLUS operation, which breaks read requests into several "data" and "hole" segments when replying to the client.
Here are the results of some performance tests I ran on my own virtual machines, which does still have the slow SEEK_HOLE/SEEK_DATA behavior. A recent minimum-compiler-version patch makes installing newer kernels difficult on the lab machines I have access to, but I'll still try to collect that data since it seemed promising during my last posting
(https://www.spinics.net/lists/linux-nfs/msg76487.html).
I tested by reading various 2G files from a few different underlying filesystems and across several NFS versions. I used the vmtouch utility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h has 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.
| v3 | v4.0 | v4.1 | v4.2 | ext4 | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.697 : 1.365 | 2.539 : 1.482 | 2.246 : 1.595 | 2.516 : 1.716 | hole | 2.389 : 1.480 | 1.692 : 1.444 | 1.904 : 1.539 | 1.042 : 1.005 | mixed-4d | 2.426 : 1.448 | 2.468 : 1.480 | 2.258 : 1.731 | 2.408 : 1.674 | mixed-8d | 2.446 : 1.592 | 2.497 : 1.523 | 2.240 : 1.736 | 2.179 : 1.547 | mixed-16d | 2.608 : 1.575 | 2.446 : 1.582 | 2.330 : 1.670 | 2.192 : 1.488 | mixed-32d | 2.544 : 1.587 | 2.164 : 1.499 | 2.076 : 1.452 | 1.982 : 1.346 | mixed-4h | 2.591 : 1.486 | 2.233 : 1.503 | 2.190 : 1.520 | 3.679 : 1.815 | mixed-8h | 2.498 : 1.547 | 2.168 : 1.510 | 2.098 : 1.511 | 2.815 : 1.484 | mixed-16h | 2.480 : 1.664 | 2.152 : 1.597 | 2.096 : 1.537 | 2.288 : 1.580 | mixed-32h | 2.520 : 1.467 | 2.171 : 1.496 | 2.246 : 1.593 | 2.066 : 1.389 |
| v3 | v4.0 | v4.1 | v4.2 | xfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.074 : 1.353 | 2.129 : 1.489 | 2.198 : 1.564 | 2.579 : 1.719 | hole | 1.714 : 1.430 | 1.647 : 1.440 | 1.748 : 1.464 | 1.019 : 1.028 | mixed-4d | 2.699 : 1.561 | 2.782 : 1.657 | 2.800 : 1.619 | 2.848 : 2.166 | mixed-8d | 2.204 : 1.540 | 2.346 : 1.595 | 2.356 : 1.589 | 2.335 : 1.809 | mixed-16d | 2.034 : 1.445 | 2.212 : 1.561 | 2.172 : 1.546 | 2.127 : 1.658 | mixed-32d | 1.982 : 1.480 | 2.135 : 1.544 | 2.136 : 1.565 | 2.024 : 1.555 | mixed-4h | 2.600 : 1.549 | 2.790 : 1.660 | 2.745 : 1.634 | 8.949 : 2.529 | mixed-8h | 2.283 : 1.555 | 2.414 : 1.605 | 2.373 : 1.607 | 5.626 : 2.015 | mixed-16h | 2.115 : 1.512 | 2.247 : 1.593 | 2.217 : 1.608 | 3.740 : 1.803 | mixed-32h | 2.069 : 1.499 | 2.212 : 1.582 | 2.235 : 1.631 | 2.819 : 1.542 |
| v3 | v4.0 | v4.1 | v4.2 | btrfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.417 : 1.317 | 2.095 : 1.445 | 2.145 : 1.523 | 2.615 : 1.713 | hole | 2.107 : 1.483 | 2.121 : 1.496 | 2.106 : 1.461 | 1.011 : 1.061 | mixed-4d | 2.348 : 1.471 | 2.370 : 1.523 | 2.379 : 1.499 | 3.028 : 225.812 | mixed-8d | 2.227 : 1.476 | 2.231 : 1.467 | 2.272 : 1.529 | 2.723 : 36.179 | mixed-16d | 2.175 : 1.460 | 2.208 : 1.457 | 2.200 : 1.464 | 2.526 : 10.371 | mixed-32d | 2.148 : 1.501 | 2.191 : 1.468 | 2.167 : 1.471 | 2.455 : 3.367 | mixed-4h | 2.362 : 1.561 | 2.387 : 1.513 | 2.352 : 1.536 | 5.935 : 41.494 | mixed-8h | 2.241 : 1.477 | 2.251 : 1.503 | 2.256 : 1.496 | 3.672 : 10.261 | mixed-16h | 2.238 : 1.477 | 2.188 : 1.496 | 2.183 : 1.503 | 2.955 : 3.809 | mixed-32h | 2.146 : 1.490 | 2.135 : 1.523 | 2.157 : 1.557 | 2.327 : 2.088 |
Anna Schumaker (6):
SUNRPC: Implement xdr_reserve_space_vec()
NFSD: nfsd4_encode_readv() can use xdr_reserve_space_vec()
NFSD: Add READ_PLUS data support
NFSD: Add READ_PLUS hole segment encoding
NFSD: Return both a hole and a data segment
NFSD: Encode a full READ_PLUS reply
fs/nfsd/nfs4proc.c | 17 ++++ fs/nfsd/nfs4xdr.c | 169 +++++++++++++++++++++++++++++++------ include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 45 ++++++++++ 4 files changed, 206 insertions(+), 27 deletions(-)
Signature · 1 line
2.27.0
PATCH v3 1/6 SUNRPC: Implement xdr_reserve_space_vec()
From: Anna Schumaker <[email protected]>
Reserving space for a large READ payload requires special handling when reserving space in the xdr buffer pages. One problem we can have is use of the scratch buffer, which is used to get a pointer to a contiguous region of data up to PAGE_SIZE. When using the scratch buffer, calls to xdr_commit_encode() shift the data to it's proper alignment in the xdr buffer. If we've reserved several pages in a vector, then this could potentially invalidate earlier pointers and result in incorrect READ data being sent to the client.
I get around this by looking at the amount of space left in the current page, and never reserve more than that for each entry in the read vector. This lets us place data directly where it needs to go in the buffer pages.
--- include/linux/sunrpc/xdr.h | 2 ++ net/sunrpc/xdr.c | 45 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+)
include/linux/sunrpc/xdr.h +2 -0
@@ -234,6 +234,8 @@ typedef int (*kxdrdproc_t)(struct rpc_rqst *rqstp, struct xdr_stream *xdr,extern void xdr_init_encode (struct xdr_stream *xdr,struct xdr_buf *buf,__be32 *p,struct rpc_rqst *rqst);extern __be32 *xdr_reserve_space (struct xdr_stream *xdr,size_t nbytes); +extern int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec, +size_t nbytes);extern void xdr_commit_encode (struct xdr_stream *xdr);extern void xdr_truncate_encode (struct xdr_stream *xdr,size_t len);extern int xdr_restrict_buflen (struct xdr_stream *xdr,int newbuflen);
net/sunrpc/xdr.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH v3 1/6] SUNRPC: Implement xdr_reserve_space_vec()
Hi Anna-
Quoted 13 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
Reserving space for a large READ payload requires special handling when reserving space in the xdr buffer pages. One problem we can have is use of the scratch buffer, which is used to get a pointer to a contiguous region of data up to
PAGE_SIZE. When using the scratch buffer, calls toxdr_commit_encode()shift the data to it's proper alignment in the xdr buffer. If we've reserved several pages in a vector, then this could potentially invalidate earlier pointers and result in incorrect READ data being sent to the client.
Quoted 4 lines
I get around this by looking at the amount of space left in the current page, and never reserve more than that for each entry in the read vector. This lets us place data directly where it needs to go in the buffer pages.
Nit: This appears to be a refactoring change that should be squashed together with 2/6.
Quoted 78 lines
Signed-off-by Anna Schumaker <[email protected]>---
include/linux/sunrpc/xdr.h| 2 ++net/sunrpc/xdr.c| 45 ++++++++++++++++++++++++++++++++++++++2 files changed, 47 insertions(+)include/linux/sunrpc/xdr.h +2
-0@@ -234,6 +234,8 @@ typedef int (*kxdrdproc_t)(struct rpc_rqst *rqstp, struct xdr_stream *xdr,extern void xdr_init_encode (struct xdr_stream *xdr,struct xdr_buf *buf,__be32 *p,struct rpc_rqst *rqst);extern __be32 *xdr_reserve_space (struct xdr_stream *xdr,size_t nbytes); +extern int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec, +size_t nbytes);extern void xdr_commit_encode (struct xdr_stream *xdr);extern void xdr_truncate_encode (struct xdr_stream *xdr,size_t len);extern int xdr_restrict_buflen (struct xdr_stream *xdr,int newbuflen);net/sunrpc/xdr.c +45
-0@@ -648,6 +648,51 @@ __be32 * xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes) }EXPORT_SYMBOL_GPL (xdr_reserve_space); + +/** +* xdr_reserve_space_vec - Reserves a large amount of buffer space for sending +* @xdr: pointer to xdr_stream +* @vec: pointer to a kvec array +* @nbytes: number of bytes to reserve +* +* Reserves enough buffer space to encode 'nbytes' of data and stores the +* pointers in 'vec'. The size argument passed to xdr_reserve_space() is +* determined based on the number of bytes remaining in the current page to +* avoid invalidating iov_base pointers when xdr_commit_encode() is called. +*/ +int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec,size_t nbytes) +{ +int thislen; +int v =0 ; +__be32 *p; + +/* +* svcrdma requires every READ payload to start somewhere +* in xdr->pages. +*/ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + +while (nbytes) { + thislen = xdr->buf->page_len %PAGE_SIZE ; + thislen =min_t (size_t, nbytes,PAGE_SIZE - thislen); + + p =xdr_reserve_space (xdr, thislen); +if (!p) +return -EIO ; + + vec[v].iov_base = p; + vec[v].iov_len = thislen; + v++; + nbytes -= thislen; + } + +return v; +} +EXPORT_SYMBOL_GPL (xdr_reserve_space_vec); +/** * xdr_truncate_encode - truncate an encode buffer * @xdr: pointer to xdr_stream Signature · 1 line
2.27.0
--
Chuck Lever
Re: [PATCH v3 1/6] SUNRPC: Implement xdr_reserve_space_vec()
Hi Chuck,
Quoted 16 lines · Chuck Lever
Hi Anna-
Quoted 13 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
Reserving space for a large READ payload requires special handling when reserving space in the xdr buffer pages. One problem we can have is use of the scratch buffer, which is used to get a pointer to a contiguous region of data up to
PAGE_SIZE. When using the scratch buffer, calls toxdr_commit_encode()shift the data to it's proper alignment in the xdr buffer. If we've reserved several pages in a vector, then this could potentially invalidate earlier pointers and result in incorrect READ data being sent to the client.
Quoted 7 lines
Quoted 4 lines
I get around this by looking at the amount of space left in the current page, and never reserve more than that for each entry in the read vector. This lets us place data directly where it needs to go in the buffer pages.
Nit: This appears to be a refactoring change that should be squashed together with 2/6.
My default was to leave sunrpc and nfs changes as separate patches, but I can squash them together if you want me to!
Anna
Quoted 86 lines
Quoted 78 lines
Signed-off-by Anna Schumaker <[email protected]>---
include/linux/sunrpc/xdr.h| 2 ++net/sunrpc/xdr.c| 45 ++++++++++++++++++++++++++++++++++++++2 files changed, 47 insertions(+)include/linux/sunrpc/xdr.h +2
-0@@ -234,6 +234,8 @@ typedef int (*kxdrdproc_t)(struct rpc_rqst *rqstp, struct xdr_stream *xdr,extern void xdr_init_encode (struct xdr_stream *xdr,struct xdr_buf *buf,__be32 *p,struct rpc_rqst *rqst);extern __be32 *xdr_reserve_space (struct xdr_stream *xdr,size_t nbytes); +extern int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec, +size_t nbytes);extern void xdr_commit_encode (struct xdr_stream *xdr);extern void xdr_truncate_encode (struct xdr_stream *xdr,size_t len);extern int xdr_restrict_buflen (struct xdr_stream *xdr,int newbuflen);net/sunrpc/xdr.c +45
-1@@ -648,6 +648,51 @@ __be32 * xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes) }EXPORT_SYMBOL_GPL (xdr_reserve_space); + +/** +* xdr_reserve_space_vec - Reserves a large amount of buffer space for sending +* @xdr: pointer to xdr_stream +* @vec: pointer to a kvec array +* @nbytes: number of bytes to reserve +* +* Reserves enough buffer space to encode 'nbytes' of data and stores the +* pointers in 'vec'. The size argument passed to xdr_reserve_space() is +* determined based on the number of bytes remaining in the current page to +* avoid invalidating iov_base pointers when xdr_commit_encode() is called. +*/ +int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec,size_t nbytes) +{ +int thislen; +int v =0 ; +__be32 *p; + +/* +* svcrdma requires every READ payload to start somewhere +* in xdr->pages. +*/ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + +while (nbytes) { + thislen = xdr->buf->page_len %PAGE_SIZE ; + thislen =min_t (size_t, nbytes,PAGE_SIZE - thislen); + + p =xdr_reserve_space (xdr, thislen); +if (!p) +return -EIO ; + + vec[v].iov_base = p; + vec[v].iov_len = thislen; + v++; + nbytes -= thislen; + } + +return v; +} +EXPORT_SYMBOL_GPL (xdr_reserve_space_vec); +/** * xdr_truncate_encode - truncate an encode buffer * @xdr: pointer to xdr_stream -- Quoted 2 lines
2.27.0
--
Chuck Lever
Re: [PATCH v3 1/6] SUNRPC: Implement xdr_reserve_space_vec()
Quoted 28 lines · Anna Schumaker
Hi Chuck,
Quoted 21 lines · Chuck Lever
Hi Anna-
Quoted 17 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
Reserving space for a large READ payload requires special handling when reserving space in the xdr buffer pages. One problem we can have is use of the scratch buffer, which is used to get a pointer to a contiguous region of data up to
PAGE_SIZE. When using the scratch buffer, calls toxdr_commit_encode()shift the data to it's proper alignment in the xdr buffer. If we've reserved several pages in a vector, then this could potentially invalidate earlier pointers and result in incorrect READ data being sent to the client.I get around this by looking at the amount of space left in the current page, and never reserve more than that for each entry in the read vector. This lets us place data directly where it needs to go in the buffer pages.
Quoted 5 lines
Quoted 2 lines
Nit: This appears to be a refactoring change that should be squashed together with 2/6.
My default was to leave sunrpc and nfs changes as separate patches, but I can squash them together if you want me to!
IMO, in this case the rule about introducing and using a new helper in the same patch takes precedence.
Quoted 84 lines
Anna
Quoted 83 lines
Quoted 78 lines
Signed-off-by Anna Schumaker <[email protected]>---
include/linux/sunrpc/xdr.h| 2 ++net/sunrpc/xdr.c| 45 ++++++++++++++++++++++++++++++++++++++2 files changed, 47 insertions(+)include/linux/sunrpc/xdr.h +2
-0@@ -234,6 +234,8 @@ typedef int (*kxdrdproc_t)(struct rpc_rqst *rqstp, struct xdr_stream *xdr,extern void xdr_init_encode (struct xdr_stream *xdr,struct xdr_buf *buf,__be32 *p,struct rpc_rqst *rqst);extern __be32 *xdr_reserve_space (struct xdr_stream *xdr,size_t nbytes); +extern int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec, +size_t nbytes);extern void xdr_commit_encode (struct xdr_stream *xdr);extern void xdr_truncate_encode (struct xdr_stream *xdr,size_t len);extern int xdr_restrict_buflen (struct xdr_stream *xdr,int newbuflen);net/sunrpc/xdr.c +45
-1@@ -648,6 +648,51 @@ __be32 * xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes) }EXPORT_SYMBOL_GPL (xdr_reserve_space); + +/** +* xdr_reserve_space_vec - Reserves a large amount of buffer space for sending +* @xdr: pointer to xdr_stream +* @vec: pointer to a kvec array +* @nbytes: number of bytes to reserve +* +* Reserves enough buffer space to encode 'nbytes' of data and stores the +* pointers in 'vec'. The size argument passed to xdr_reserve_space() is +* determined based on the number of bytes remaining in the current page to +* avoid invalidating iov_base pointers when xdr_commit_encode() is called. +*/ +int xdr_reserve_space_vec (struct xdr_stream *xdr,struct kvec *vec,size_t nbytes) +{ +int thislen; +int v =0 ; +__be32 *p; + +/* +* svcrdma requires every READ payload to start somewhere +* in xdr->pages. +*/ +if (xdr->iov == xdr->buf->head) { + xdr->iov =NULL ; + xdr->end = xdr->p; + } + +while (nbytes) { + thislen = xdr->buf->page_len %PAGE_SIZE ; + thislen =min_t (size_t, nbytes,PAGE_SIZE - thislen); + + p =xdr_reserve_space (xdr, thislen); +if (!p) +return -EIO ; + + vec[v].iov_base = p; + vec[v].iov_len = thislen; + v++; + nbytes -= thislen; + } + +return v; +} +EXPORT_SYMBOL_GPL (xdr_reserve_space_vec); +/** * xdr_truncate_encode - truncate an encode buffer * @xdr: pointer to xdr_stream -- Quoted 2 lines
2.27.0
--
Chuck Lever
--
Chuck Lever
PATCH v3 2/6 NFSD: nfsd4_encode_readv() can use xdr_reserve_space_vec()
In reply to [email protected]From: Anna Schumaker <[email protected]>
Switch over to the new code so we don't need to reproduce it on the NFSD side.
--- fs/nfsd/nfs4xdr.c | 28 +++------------------------- 1 file changed, 3 insertions(+), 25 deletions(-)
fs/nfsd/nfs4xdr.c +3 -25
@@ -3584,36 +3584,14 @@ static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, {struct xdr_stream *xdr = &resp->xdr;u32 eof; -int v;int starting_len = xdr->buf->len -8 ; -long len; -int thislen;__be32 nfserr;__be32 tmp; -__be32 *p;int pad; -/* -* svcrdma requires every READ payload to start somewhere -* in xdr->pages. -*/ -if (xdr->iov == xdr->buf->head) { - xdr->iov =NULL ; - xdr->end = xdr->p; - } - - len = maxcount; - v =0 ; -while (len) { - thislen =min_t (long, len,PAGE_SIZE ); - p =xdr_reserve_space (xdr, thislen); -WARN_ON_ONCE (!p); - resp->rqstp->rq_vec[v].iov_base = p; - resp->rqstp->rq_vec[v].iov_len = thislen; - v++; - len -= thislen; - } - read->rd_vlen = v; + read->rd_vlen =xdr_reserve_space_vec (xdr, resp->rqstp->rq_vec, maxcount); +if (read->rd_vlen <0 ) +return nfserr_resource; nfserr = nfsd_readv(resp->rqstp, read->rd_fhp, file, read->rd_offset, resp->rqstp->rq_vec, read->rd_vlen, &maxcount,
Signature · 1 line
2.27.0
PATCH v3 3/6 NFSD: Add READ_PLUS data support
In reply to [email protected]From: Anna Schumaker <[email protected]>
This patch adds READ_PLUS support for returning a single NFS4_CONTENT_DATA segment to the client. This is basically the same as the READ operation, only with the extra information about data segments.
--- fs/nfsd/nfs4proc.c | 17 ++++++++++ fs/nfsd/nfs4xdr.c | 85 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 100 insertions(+), 2 deletions(-)
fs/nfsd/nfs4proc.c +17 -0
@@ -2523,6 +2523,16 @@ static inline u32 nfsd4_read_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)return (op_encode_hdr_size +2 +XDR_QUADLEN (rlen)) *sizeof (__be32); } +static inline u32 nfsd4_read_plus_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) +{ +u32 maxcount =svc_max_payload (rqstp); +u32 rlen =min (op->u.read.rd_length, maxcount); +/* enough extra xdr space for encoding either a hole or data segment. */ +u32 segments =1 +2 +2 ; + +return (op_encode_hdr_size +2 + segments +XDR_QUADLEN (rlen)) *sizeof (__be32); +} +static inline u32 nfsd4_readdir_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) {u32 maxcount =0 , rlen =0 ;
@@ -3059,6 +3069,13 @@ static const struct nfsd4_operation nfsd4_ops[] = { .op_name = "OP_COPY", .op_rsize_bop = nfsd4_copy_rsize, }, + [OP_READ_PLUS] = { + .op_func = nfsd4_read, + .op_release = nfsd4_read_release, + .op_name = "OP_READ_PLUS", + .op_rsize_bop = nfsd4_read_plus_rsize, + .op_get_currentstateid = nfsd4_get_readstateid, + }, [OP_SEEK] = { .op_func = nfsd4_seek, .op_name = "OP_SEEK",
fs/nfsd/nfs4xdr.c +83 -2
103 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH v3 3/6] NFSD: Add READ_PLUS data support
Hi Anna-
Quoted 82 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds
READ_PLUSsupport for returning a singleNFS4_CONTENT_DATAsegment to the client. This is basically the same as the READ operation, only with the extra information about data segments.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfsd/nfs4proc.c| 17 ++++++++++fs/nfsd/nfs4xdr.c| 85 ++++++++++++++++++++++++++++++++++++++++++++--2 files changed, 100 insertions(+), 2 deletions(-)fs/nfsd/nfs4proc.c +17
-0@@ -2523,6 +2523,16 @@ static inline u32 nfsd4_read_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)return (op_encode_hdr_size +2 +XDR_QUADLEN (rlen)) *sizeof (__be32); } +static inline u32 nfsd4_read_plus_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) +{ +u32 maxcount =svc_max_payload (rqstp); +u32 rlen =min (op->u.read.rd_length, maxcount); +/* enough extra xdr space for encoding either a hole or data segment. */ +u32 segments =1 +2 +2 ; + +return (op_encode_hdr_size +2 + segments +XDR_QUADLEN (rlen)) *sizeof (__be32); +} +static inline u32 nfsd4_readdir_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) {u32 maxcount =0 , rlen =0 ;@@ -3059,6 +3069,13 @@ static const struct nfsd4_operation nfsd4_ops[] = { .op_name = "OP_COPY", .op_rsize_bop = nfsd4_copy_rsize, }, + [OP_READ_PLUS] = { + .op_func = nfsd4_read, + .op_release = nfsd4_read_release, + .op_name = "OP_READ_PLUS", + .op_rsize_bop = nfsd4_read_plus_rsize, + .op_get_currentstateid = nfsd4_get_readstateid, + }, [OP_SEEK] = { .op_func = nfsd4_seek, .op_name = "OP_SEEK",fs/nfsd/nfs4xdr.c +17
-1@@ -1957,7 +1957,7 @@ static const nfsd4_dec nfsd4_dec_ops[] = { [OP_LAYOUTSTATS] = (nfsd4_dec)nfsd4_decode_notsupp, [OP_OFFLOAD_CANCEL] = (nfsd4_dec)nfsd4_decode_offload_status, [OP_OFFLOAD_STATUS] = (nfsd4_dec)nfsd4_decode_offload_status, - [OP_READ_PLUS] = (nfsd4_dec)nfsd4_decode_notsupp, + [OP_READ_PLUS] = (nfsd4_dec)nfsd4_decode_read, [OP_SEEK] = (nfsd4_dec)nfsd4_decode_seek, [OP_WRITE_SAME] = (nfsd4_dec)nfsd4_decode_notsupp, [OP_CLONE] = (nfsd4_dec)nfsd4_decode_clone,@@ -4367,6 +4367,87 @@ nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,return nfserr_resource; p =xdr_encode_hyper (p, os->count); *p++ =cpu_to_be32 (0 ); +return nfserr; +} + +static __be32 +nfsd4_encode_read_plus_data (struct nfsd4_compoundres *resp, +struct nfsd4_read *read, +unsigned long maxcount,u32 *eof) +{ +struct xdr_stream *xdr = &resp->xdr; +struct file *file = read->rd_nf->nf_file; +int starting_len = xdr->buf->len; +__be32 nfserr; +__be32 *p, tmp; +__be64 tmp64; + +/* Content type, offset, byte count */
fs/nfsd/nfs4xdr.c nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr, · line 4395
Quoted 10 lines
fs/nfsd/nfs4xdr.c +10
-0+ p =xdr_reserve_space (xdr,4 +8 +4 ); +if (!p) +return nfserr_resource; + + read->rd_vlen =xdr_reserve_space_vec (xdr, resp->rqstp->rq_vec, maxcount); +if (read->rd_vlen <0 ) +return nfserr_resource; + + nfserr =nfsd_readv (resp->rqstp, read->rd_fhp, file, read->rd_offset, + resp->rqstp->rq_vec, read->rd_vlen, &maxcount, eof);
nfsd4_encode_read() has this:
3904 if (file->f_op->splice_read &&
3905 test_bit(RQ_SPLICE_OK, &resp->rqstp->rq_flags))
3906 nfserr = nfsd4_encode_splice_read(resp, read, file, maxcount);
3907 else
3908 nfserr = nfsd4_encode_readv(resp, read, file, maxcount);
So READ_PLUS never uses ->splice_read.
readv is known to be less efficient than splice. Is there a measurable server performance impact (either latency or CPU utilization) when reading a file with no holes?
fs/nfsd/nfs4xdr.c nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr, · line 4399
Quoted 4 lines
fs/nfsd/nfs4xdr.c +4
-0+if (nfserr) +return nfserr; +if (svc_encode_read_payload (resp->rqstp, starting_len +16 , maxcount)) +return nfserr_io;
Not sure you want a read_payload call here. This is to notify the transport that there is a section of the message that can be sent via RDMA, but READ_PLUS has no DDP-eligible data items; especially not holes!
Although, the call is not likely to be much more than a no-op, since clients won't be providing any Write chunks for READ_PLUS.
Quoted 66 lines
fs/nfsd/nfs4xdr.c +52
-1+ + tmp =htonl (NFS4_CONTENT_DATA ); +write_bytes_to_xdr_buf (xdr->buf, starting_len, &tmp,4 ); + tmp64 =cpu_to_be64 (read->rd_offset); +write_bytes_to_xdr_buf (xdr->buf, starting_len +4 , &tmp64,8 ); + tmp =htonl (maxcount); +write_bytes_to_xdr_buf (xdr->buf, starting_len +12 , &tmp,4 ); +return nfs_ok; +} + +static __be32 +nfsd4_encode_read_plus (struct nfsd4_compoundres *resp,__be32 nfserr, +struct nfsd4_read *read) +{ +unsigned long maxcount; +struct xdr_stream *xdr = &resp->xdr; +struct file *file; +int starting_len = xdr->buf->len; +int segments =0 ; +__be32 *p, tmp; +u32 eof; + +if (nfserr) +return nfserr; + file = read->rd_nf->nf_file; + +/* eof flag, segment count */ + p =xdr_reserve_space (xdr,4 +4 ); +if (!p) +return nfserr_resource; +xdr_commit_encode (xdr); + + maxcount =svc_max_payload (resp->rqstp); + maxcount = min_t(unsigned long, maxcount, + (xdr->buf->buflen - xdr->buf->len)); + maxcount = min_t(unsigned long, maxcount, read->rd_length); + + eof = read->rd_offset >=i_size_read (file_inode (file)); +if (!eof) { + nfserr =nfsd4_encode_read_plus_data (resp, read, maxcount, &eof); + segments++; + } + +if (nfserr) +xdr_truncate_encode (xdr, starting_len); +else { + tmp =htonl (eof); +write_bytes_to_xdr_buf (xdr->buf, starting_len, &tmp,4 ); + tmp =htonl (segments); +write_bytes_to_xdr_buf (xdr->buf, starting_len +4 , &tmp,4 ); + }return nfserr; }@@ -4509,7 +4590,7 @@ static const nfsd4_enc nfsd4_enc_ops[] = { [OP_LAYOUTSTATS] = (nfsd4_enc)nfsd4_encode_noop, [OP_OFFLOAD_CANCEL] = (nfsd4_enc)nfsd4_encode_noop, [OP_OFFLOAD_STATUS] = (nfsd4_enc)nfsd4_encode_offload_status, - [OP_READ_PLUS] = (nfsd4_enc)nfsd4_encode_noop, + [OP_READ_PLUS] = (nfsd4_enc)nfsd4_encode_read_plus, [OP_SEEK] = (nfsd4_enc)nfsd4_encode_seek, [OP_WRITE_SAME] = (nfsd4_enc)nfsd4_encode_noop, [OP_CLONE] = (nfsd4_enc)nfsd4_encode_noop,Signature · 1 line
2.27.0
--
Chuck Lever
Re: [PATCH v3 3/6] NFSD: Add READ_PLUS data support
Hi Chuck,
Quoted 104 lines · Chuck Lever
Hi Anna-
Quoted 82 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds
READ_PLUSsupport for returning a singleNFS4_CONTENT_DATAsegment to the client. This is basically the same as the READ operation, only with the extra information about data segments.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfsd/nfs4proc.c| 17 ++++++++++fs/nfsd/nfs4xdr.c| 85 ++++++++++++++++++++++++++++++++++++++++++++--2 files changed, 100 insertions(+), 2 deletions(-)fs/nfsd/nfs4proc.c +17
-0@@ -2523,6 +2523,16 @@ static inline u32 nfsd4_read_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op)return (op_encode_hdr_size +2 +XDR_QUADLEN (rlen)) *sizeof (__be32); } +static inline u32 nfsd4_read_plus_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) +{ +u32 maxcount =svc_max_payload (rqstp); +u32 rlen =min (op->u.read.rd_length, maxcount); +/* enough extra xdr space for encoding either a hole or data segment. */ +u32 segments =1 +2 +2 ; + +return (op_encode_hdr_size +2 + segments +XDR_QUADLEN (rlen)) *sizeof (__be32); +} +static inline u32 nfsd4_readdir_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) {u32 maxcount =0 , rlen =0 ;@@ -3059,6 +3069,13 @@ static const struct nfsd4_operation nfsd4_ops[] = { .op_name = "OP_COPY", .op_rsize_bop = nfsd4_copy_rsize, }, + [OP_READ_PLUS] = { + .op_func = nfsd4_read, + .op_release = nfsd4_read_release, + .op_name = "OP_READ_PLUS", + .op_rsize_bop = nfsd4_read_plus_rsize, + .op_get_currentstateid = nfsd4_get_readstateid, + }, [OP_SEEK] = { .op_func = nfsd4_seek, .op_name = "OP_SEEK",fs/nfsd/nfs4xdr.c +17
-1@@ -1957,7 +1957,7 @@ static const nfsd4_dec nfsd4_dec_ops[] = { [OP_LAYOUTSTATS] = (nfsd4_dec)nfsd4_decode_notsupp, [OP_OFFLOAD_CANCEL] = (nfsd4_dec)nfsd4_decode_offload_status, [OP_OFFLOAD_STATUS] = (nfsd4_dec)nfsd4_decode_offload_status, - [OP_READ_PLUS] = (nfsd4_dec)nfsd4_decode_notsupp, + [OP_READ_PLUS] = (nfsd4_dec)nfsd4_decode_read, [OP_SEEK] = (nfsd4_dec)nfsd4_decode_seek, [OP_WRITE_SAME] = (nfsd4_dec)nfsd4_decode_notsupp, [OP_CLONE] = (nfsd4_dec)nfsd4_decode_clone,@@ -4367,6 +4367,87 @@ nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,return nfserr_resource; p =xdr_encode_hyper (p, os->count); *p++ =cpu_to_be32 (0 ); +return nfserr; +} + +static __be32 +nfsd4_encode_read_plus_data (struct nfsd4_compoundres *resp, +struct nfsd4_read *read, +unsigned long maxcount,u32 *eof) +{ +struct xdr_stream *xdr = &resp->xdr; +struct file *file = read->rd_nf->nf_file; +int starting_len = xdr->buf->len; +__be32 nfserr; +__be32 *p, tmp; +__be64 tmp64; + +/* Content type, offset, byte count */ fs/nfsd/nfs4xdr.c nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr, · line 4395
Quoted 10 lines
fs/nfsd/nfs4xdr.c +10
-0+ p =xdr_reserve_space (xdr,4 +8 +4 ); +if (!p) +return nfserr_resource; + + read->rd_vlen =xdr_reserve_space_vec (xdr, resp->rqstp->rq_vec, maxcount); +if (read->rd_vlen <0 ) +return nfserr_resource; + + nfserr =nfsd_readv (resp->rqstp, read->rd_fhp, file, read->rd_offset, + resp->rqstp->rq_vec, read->rd_vlen, &maxcount, eof);
nfsd4_encode_read()has this:3904 if (
file->f_op->splice_read&&
3905test_bit(RQ_SPLICE_OK, &resp->rqstp->rq_flags))
3906 nfserr =nfsd4_encode_splice_read(resp, read, file, maxcount);
3907 else
3908 nfserr =nfsd4_encode_readv(resp, read, file, maxcount);
Quoted 5 lines
So
READ_PLUSnever uses ->splice_read.readv is known to be less efficient than splice. Is there a measurable server performance impact (either latency or CPU utilization) when reading a file with no holes?
Good question, and I haven't measured that. For a while I was doing the first read data segment with a splice, but that was having issues. I should circle back and see if that's still a problem now that I've fixed a bunch of other bugs.
Quoted 11 lines
fs/nfsd/nfs4xdr.c nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr, · line 4399
Quoted 6 lines
fs/nfsd/nfs4xdr.c +4
-0+if (nfserr) +return nfserr; +if (svc_encode_read_payload (resp->rqstp, starting_len +16 , maxcount)) +return nfserr_io;Not sure you want a
read_payloadcall here. This is to notify the transport that there is a section of the message that can be sent via RDMA, butREAD_PLUShas no DDP-eligible data items; especially not holes!
Sure thing. I'll remove that for the next version.
Anna
Quoted 77 lines
Although, the call is not likely to be much more than a no-op, since clients won't be providing any Write chunks for
READ_PLUS.Quoted 64 lines
fs/nfsd/nfs4xdr.c +52
-2+ + tmp =htonl (NFS4_CONTENT_DATA ); +write_bytes_to_xdr_buf (xdr->buf, starting_len, &tmp,4 ); + tmp64 =cpu_to_be64 (read->rd_offset); +write_bytes_to_xdr_buf (xdr->buf, starting_len +4 , &tmp64,8 ); + tmp =htonl (maxcount); +write_bytes_to_xdr_buf (xdr->buf, starting_len +12 , &tmp,4 ); +return nfs_ok; +} + +static __be32 +nfsd4_encode_read_plus (struct nfsd4_compoundres *resp,__be32 nfserr, +struct nfsd4_read *read) +{ +unsigned long maxcount; +struct xdr_stream *xdr = &resp->xdr; +struct file *file; +int starting_len = xdr->buf->len; +int segments =0 ; +__be32 *p, tmp; +u32 eof; + +if (nfserr) +return nfserr; + file = read->rd_nf->nf_file; + +/* eof flag, segment count */ + p =xdr_reserve_space (xdr,4 +4 ); +if (!p) +return nfserr_resource; +xdr_commit_encode (xdr); + + maxcount =svc_max_payload (resp->rqstp); + maxcount = min_t(unsigned long, maxcount, + (xdr->buf->buflen - xdr->buf->len)); + maxcount = min_t(unsigned long, maxcount, read->rd_length); + + eof = read->rd_offset >=i_size_read (file_inode (file)); +if (!eof) { + nfserr =nfsd4_encode_read_plus_data (resp, read, maxcount, &eof); + segments++; + } + +if (nfserr) +xdr_truncate_encode (xdr, starting_len); +else { + tmp =htonl (eof); +write_bytes_to_xdr_buf (xdr->buf, starting_len, &tmp,4 ); + tmp =htonl (segments); +write_bytes_to_xdr_buf (xdr->buf, starting_len +4 , &tmp,4 ); + }return nfserr; }@@ -4509,7 +4590,7 @@ static const nfsd4_enc nfsd4_enc_ops[] = { [OP_LAYOUTSTATS] = (nfsd4_enc)nfsd4_encode_noop, [OP_OFFLOAD_CANCEL] = (nfsd4_enc)nfsd4_encode_noop, [OP_OFFLOAD_STATUS] = (nfsd4_enc)nfsd4_encode_offload_status, - [OP_READ_PLUS] = (nfsd4_enc)nfsd4_encode_noop, + [OP_READ_PLUS] = (nfsd4_enc)nfsd4_encode_read_plus, [OP_SEEK] = (nfsd4_enc)nfsd4_encode_seek, [OP_WRITE_SAME] = (nfsd4_enc)nfsd4_encode_noop, [OP_CLONE] = (nfsd4_enc)nfsd4_encode_noop, --Quoted 2 lines
2.27.0
--
Chuck Lever
PATCH v3 4/6 NFSD: Add READ_PLUS hole segment encoding
In reply to [email protected]From: Anna Schumaker <[email protected]>
However, we still only reply to the READ_PLUS call with a single segment at this time.
--- fs/nfsd/nfs4proc.c | 2 +- fs/nfsd/nfs4xdr.c | 40 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 40 insertions(+), 2 deletions(-)
fs/nfsd/nfs4proc.c +1 -1
@@ -2528,7 +2528,7 @@ static inline u32 nfsd4_read_plus_rsize(struct svc_rqst *rqstp, struct nfsd4_opu32 maxcount =svc_max_payload (rqstp);u32 rlen =min (op->u.read.rd_length, maxcount);/* enough extra xdr space for encoding either a hole or data segment. */ -u32 segments =1 +2 +2 ; +u32 segments =2 * (1 +2 +2 );return (op_encode_hdr_size +2 + segments +XDR_QUADLEN (rlen)) *sizeof (__be32); }
fs/nfsd/nfs4xdr.c +39 -1
@@ -4378,10 +4378,14 @@ nfsd4_encode_read_plus_data(struct nfsd4_compoundres *resp,struct xdr_stream *xdr = &resp->xdr;struct file *file = read->rd_nf->nf_file;int starting_len = xdr->buf->len; +loff_t hole_pos =vfs_llseek (file, read->rd_offset,SEEK_HOLE );__be32 nfserr;__be32 *p, tmp;__be64 tmp64; +if (hole_pos > read->rd_offset) + maxcount = min_t(unsigned long, maxcount, hole_pos - read->rd_offset); +/* Content type, offset, byte count */ p =xdr_reserve_space (xdr,4 +8 +4 );if (!p)
@@ -4407,6 +4411,27 @@ nfsd4_encode_read_plus_data(struct nfsd4_compoundres *resp,return nfs_ok; } +static __be32 +nfsd4_encode_read_plus_hole (struct nfsd4_compoundres *resp, +struct nfsd4_read *read, +unsigned long maxcount,u32 *eof) +{ +struct file *file = read->rd_nf->nf_file; +__be32 *p; + +/* Content type, offset, byte count */ + p =xdr_reserve_space (&resp->xdr,4 +8 +8 ); +if (!p) +return nfserr_resource; + + *p++ =htonl (NFS4_CONTENT_HOLE ); + p =xdr_encode_hyper (p, read->rd_offset); + p =xdr_encode_hyper (p, maxcount); + + *eof = (read->rd_offset + maxcount) >=i_size_read (file_inode (file)); +return nfs_ok; +} +static __be32 nfsd4_encode_read_plus (struct nfsd4_compoundres *resp,__be32 nfserr,struct nfsd4_read *read)
@@ -4417,6 +4442,7 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr,int starting_len = xdr->buf->len;int segments =0 ;__be32 *p, tmp; +loff_t pos;u32 eof;if (nfserr)
@@ -4435,11 +4461,23 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr, maxcount = min_t(unsigned long, maxcount, read->rd_length); eof = read->rd_offset >=i_size_read (file_inode (file)); -if (!eof) { +if (eof) + goto out; + + pos =vfs_llseek (file, read->rd_offset,SEEK_DATA ); +if (pos == -ENXIO ) + pos =i_size_read (file_inode (file)); + +if (pos > read->rd_offset) { + maxcount = pos - read->rd_offset; + nfserr =nfsd4_encode_read_plus_hole (resp, read, maxcount, &eof); + segments++; + }else { nfserr =nfsd4_encode_read_plus_data (resp, read, maxcount, &eof); segments++; } +out:if (nfserr)xdr_truncate_encode (xdr, starting_len);else {
Signature · 1 line
2.27.0
PATCH v3 5/6 NFSD: Return both a hole and a data segment
In reply to [email protected]From: Anna Schumaker <[email protected]>
But only one of each right now. We'll expand on this in the next patch.
--- fs/nfsd/nfs4xdr.c | 53 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 38 insertions(+), 15 deletions(-)
fs/nfsd/nfs4xdr.c +38 -15
@@ -4373,7 +4373,7 @@ nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,static __be32 nfsd4_encode_read_plus_data (struct nfsd4_compoundres *resp,struct nfsd4_read *read, -unsigned long maxcount,u32 *eof) +unsigned long *maxcount,u32 *eof) {struct xdr_stream *xdr = &resp->xdr;struct file *file = read->rd_nf->nf_file;
@@ -4384,29 +4384,29 @@ nfsd4_encode_read_plus_data(struct nfsd4_compoundres *resp,__be64 tmp64;if (hole_pos > read->rd_offset) - maxcount = min_t(unsigned long, maxcount, hole_pos - read->rd_offset); + *maxcount = min_t(unsigned long, *maxcount, hole_pos - read->rd_offset);/* Content type, offset, byte count */ p =xdr_reserve_space (xdr,4 +8 +4 );if (!p)return nfserr_resource; - read->rd_vlen =xdr_reserve_space_vec (xdr, resp->rqstp->rq_vec, maxcount); + read->rd_vlen =xdr_reserve_space_vec (xdr, resp->rqstp->rq_vec, *maxcount);if (read->rd_vlen <0 )return nfserr_resource; nfserr =nfsd_readv (resp->rqstp, read->rd_fhp, file, read->rd_offset, - resp->rqstp->rq_vec, read->rd_vlen, &maxcount, eof); + resp->rqstp->rq_vec, read->rd_vlen, maxcount, eof);if (nfserr)return nfserr; -if (svc_encode_read_payload (resp->rqstp, starting_len +16 , maxcount)) +if (svc_encode_read_payload (resp->rqstp, starting_len +16 , *maxcount))return nfserr_io; tmp =htonl (NFS4_CONTENT_DATA );write_bytes_to_xdr_buf (xdr->buf, starting_len, &tmp,4 ); tmp64 =cpu_to_be64 (read->rd_offset);write_bytes_to_xdr_buf (xdr->buf, starting_len +4 , &tmp64,8 ); - tmp =htonl (maxcount); + tmp =htonl (*maxcount);write_bytes_to_xdr_buf (xdr->buf, starting_len +12 , &tmp,4 );return nfs_ok; }
@@ -4414,11 +4414,19 @@ nfsd4_encode_read_plus_data(struct nfsd4_compoundres *resp,static __be32 nfsd4_encode_read_plus_hole (struct nfsd4_compoundres *resp,struct nfsd4_read *read, -unsigned long maxcount,u32 *eof) +unsigned long *maxcount,u32 *eof) {struct file *file = read->rd_nf->nf_file; +loff_t data_pos =vfs_llseek (file, read->rd_offset,SEEK_DATA ); +unsigned long count;__be32 *p; +if (data_pos == -ENXIO ) + data_pos =i_size_read (file_inode (file)); +else if (data_pos <= read->rd_offset) +return nfserr_resource; + count = data_pos - read->rd_offset; +/* Content type, offset, byte count */ p =xdr_reserve_space (&resp->xdr,4 +8 +8 );if (!p)
@@ -4426,9 +4434,10 @@ nfsd4_encode_read_plus_hole(struct nfsd4_compoundres *resp, *p++ =htonl (NFS4_CONTENT_HOLE ); p =xdr_encode_hyper (p, read->rd_offset); - p =xdr_encode_hyper (p, maxcount); + p =xdr_encode_hyper (p, count); - *eof = (read->rd_offset + maxcount) >=i_size_read (file_inode (file)); + *eof = (read->rd_offset + count) >=i_size_read (file_inode (file)); + *maxcount = min_t(unsigned long, count, *maxcount);return nfs_ok; }
@@ -4436,7 +4445,7 @@ static __be32 nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_read *read) { - unsigned long maxcount; + unsigned long maxcount, count;struct xdr_stream *xdr = &resp->xdr;struct file *file;int starting_len = xdr->buf->len;
@@ -4459,6 +4468,7 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr, maxcount = min_t(unsigned long, maxcount, (xdr->buf->buflen - xdr->buf->len)); maxcount = min_t(unsigned long, maxcount, read->rd_length); + count = maxcount; eof = read->rd_offset >=i_size_read (file_inode (file));if (eof)
@@ -4467,13 +4477,26 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr, pos =vfs_llseek (file, read->rd_offset,SEEK_DATA );if (pos == -ENXIO ) pos =i_size_read (file_inode (file)); +else if (pos <0 ) + pos = read->rd_offset; -if (pos > read->rd_offset) { - maxcount = pos - read->rd_offset; - nfserr =nfsd4_encode_read_plus_hole (resp, read, maxcount, &eof); +if (pos == read->rd_offset) { + maxcount = count; + nfserr =nfsd4_encode_read_plus_data (resp, read, &maxcount, &eof); +if (nfserr) + goto out; + count -= maxcount; + read->rd_offset += maxcount; segments++; - }else { - nfserr =nfsd4_encode_read_plus_data (resp, read, maxcount, &eof); + } + +if (count >0 && !eof) { + maxcount = count; + nfserr =nfsd4_encode_read_plus_hole (resp, read, &maxcount, &eof); +if (nfserr) + goto out; + count -= maxcount; + read->rd_offset += maxcount; segments++; }
Signature · 1 line
2.27.0
PATCH v3 6/6 NFSD: Encode a full READ_PLUS reply
In reply to [email protected]From: Anna Schumaker <[email protected]>
Reply to the client with multiple hole and data segments. I use the result of the first vfs_llseek() call for encoding as an optimization so we don't have to immediately repeat the call.
--- fs/nfsd/nfs4xdr.c | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-)
fs/nfsd/nfs4xdr.c +14 -19
@@ -4373,16 +4373,18 @@ nfsd4_encode_offload_status(struct nfsd4_compoundres *resp, __be32 nfserr,static __be32 nfsd4_encode_read_plus_data (struct nfsd4_compoundres *resp,struct nfsd4_read *read, -unsigned long *maxcount,u32 *eof) +unsigned long *maxcount,u32 *eof, +loff_t *pos) {struct xdr_stream *xdr = &resp->xdr;struct file *file = read->rd_nf->nf_file;int starting_len = xdr->buf->len; -loff_t hole_pos =vfs_llseek (file, read->rd_offset,SEEK_HOLE ); +loff_t hole_pos;__be32 nfserr;__be32 *p, tmp;__be64 tmp64; + hole_pos = pos ? *pos :vfs_llseek (file, read->rd_offset,SEEK_HOLE );if (hole_pos > read->rd_offset) *maxcount = min_t(unsigned long, *maxcount, hole_pos - read->rd_offset);
@@ -4451,6 +4453,7 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr,int starting_len = xdr->buf->len;int segments =0 ;__be32 *p, tmp; +bool is_data;loff_t pos;u32 eof;
@@ -4474,29 +4477,21 @@ nfsd4_encode_read_plus(struct nfsd4_compoundres *resp, __be32 nfserr,if (eof) goto out; - pos =vfs_llseek (file, read->rd_offset,SEEK_DATA ); -if (pos == -ENXIO ) - pos =i_size_read (file_inode (file)); -else if (pos <0 ) - pos = read->rd_offset; + pos =vfs_llseek (file, read->rd_offset,SEEK_HOLE ); + is_data = pos > read->rd_offset; -if (pos == read->rd_offset) { +while (count >0 && !eof) { maxcount = count; - nfserr =nfsd4_encode_read_plus_data (resp, read, &maxcount, &eof); -if (nfserr) - goto out; - count -= maxcount; - read->rd_offset += maxcount; - segments++; - } - -if (count >0 && !eof) { - maxcount = count; - nfserr =nfsd4_encode_read_plus_hole (resp, read, &maxcount, &eof); +if (is_data) + nfserr =nfsd4_encode_read_plus_data (resp, read, &maxcount, &eof, + segments ==0 ? &pos :NULL ); +else + nfserr =nfsd4_encode_read_plus_hole (resp, read, &maxcount, &eof);if (nfserr) goto out; count -= maxcount; read->rd_offset += maxcount; + is_data = !is_data; segments++; }
Signature · 1 line
2.27.0
Hi Anna-
Quoted 7 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
These patches add server support for the
READ_PLUSoperation, which breaks read requests into several "data" and "hole" segments when replying to the client.
At this point, I've wrapped up the content of the "feature" pull request for v5.9, so Bruce and I are assuming this series is not for v5.9.
Quoted 6 lines
Here are the results of some performance tests I ran on my own virtual machines, which does still have the slow SEEK_HOLE/SEEK_DATA behavior. A recent minimum-compiler-version patch makes installing newer kernels difficult on the lab machines I have access to, but I'll still try to collect that data since it seemed promising during my last posting
(https://www.spinics.net/lists/linux-nfs/msg76487.html).
Nit: I've seen other maintainers prefer the use of lore.kernel.org over spinics or mail.info for patch descriptions and cover letters.
Quoted 10 lines
I tested by reading various 2G files from a few different underlying filesystems and across several NFS versions. I used the
vmtouchutility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h has 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.
In a VM environment, counting bytes transferred might be a more useful comparison than seconds. I'm not sure if the table reports wall clock time or something else. At what layer was the timing measured?
CPU utilization might be interesting to report on the client because clients would have to more work assembling a file with holes in it.
Also the benefit of READ_PLUS might be more plain if you chose a larger file size (by perhaps a factor of 100).
Not clear what the purpose of the v3, v4.0, and v4.1 columns is.
Quoted 50 lines
| v3 | v4.0 | v4.1 | v4.2 | ext4 | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.697 : 1.365 | 2.539 : 1.482 | 2.246 : 1.595 | 2.516 : 1.716 | hole | 2.389 : 1.480 | 1.692 : 1.444 | 1.904 : 1.539 | 1.042 : 1.005 | mixed-4d | 2.426 : 1.448 | 2.468 : 1.480 | 2.258 : 1.731 | 2.408 : 1.674 | mixed-8d | 2.446 : 1.592 | 2.497 : 1.523 | 2.240 : 1.736 | 2.179 : 1.547 | mixed-16d | 2.608 : 1.575 | 2.446 : 1.582 | 2.330 : 1.670 | 2.192 : 1.488 | mixed-32d | 2.544 : 1.587 | 2.164 : 1.499 | 2.076 : 1.452 | 1.982 : 1.346 | mixed-4h | 2.591 : 1.486 | 2.233 : 1.503 | 2.190 : 1.520 | 3.679 : 1.815 | mixed-8h | 2.498 : 1.547 | 2.168 : 1.510 | 2.098 : 1.511 | 2.815 : 1.484 | mixed-16h | 2.480 : 1.664 | 2.152 : 1.597 | 2.096 : 1.537 | 2.288 : 1.580 | mixed-32h | 2.520 : 1.467 | 2.171 : 1.496 | 2.246 : 1.593 | 2.066 : 1.389 || v3 | v4.0 | v4.1 | v4.2 | xfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.074 : 1.353 | 2.129 : 1.489 | 2.198 : 1.564 | 2.579 : 1.719 | hole | 1.714 : 1.430 | 1.647 : 1.440 | 1.748 : 1.464 | 1.019 : 1.028 | mixed-4d | 2.699 : 1.561 | 2.782 : 1.657 | 2.800 : 1.619 | 2.848 : 2.166 | mixed-8d | 2.204 : 1.540 | 2.346 : 1.595 | 2.356 : 1.589 | 2.335 : 1.809 | mixed-16d | 2.034 : 1.445 | 2.212 : 1.561 | 2.172 : 1.546 | 2.127 : 1.658 | mixed-32d | 1.982 : 1.480 | 2.135 : 1.544 | 2.136 : 1.565 | 2.024 : 1.555 | mixed-4h | 2.600 : 1.549 | 2.790 : 1.660 | 2.745 : 1.634 | 8.949 : 2.529 | mixed-8h | 2.283 : 1.555 | 2.414 : 1.605 | 2.373 : 1.607 | 5.626 : 2.015 | mixed-16h | 2.115 : 1.512 | 2.247 : 1.593 | 2.217 : 1.608 | 3.740 : 1.803 | mixed-32h | 2.069 : 1.499 | 2.212 : 1.582 | 2.235 : 1.631 | 2.819 : 1.542 || v3 | v4.0 | v4.1 | v4.2 | btrfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.417 : 1.317 | 2.095 : 1.445 | 2.145 : 1.523 | 2.615 : 1.713 | hole | 2.107 : 1.483 | 2.121 : 1.496 | 2.106 : 1.461 | 1.011 : 1.061 | mixed-4d | 2.348 : 1.471 | 2.370 : 1.523 | 2.379 : 1.499 | 3.028 : 225.812 | mixed-8d | 2.227 : 1.476 | 2.231 : 1.467 | 2.272 : 1.529 | 2.723 : 36.179 | mixed-16d | 2.175 : 1.460 | 2.208 : 1.457 | 2.200 : 1.464 | 2.526 : 10.371 | mixed-32d | 2.148 : 1.501 | 2.191 : 1.468 | 2.167 : 1.471 | 2.455 : 3.367 | mixed-4h | 2.362 : 1.561 | 2.387 : 1.513 | 2.352 : 1.536 | 5.935 : 41.494 | mixed-8h | 2.241 : 1.477 | 2.251 : 1.503 | 2.256 : 1.496 | 3.672 : 10.261 | mixed-16h | 2.238 : 1.477 | 2.188 : 1.496 | 2.183 : 1.503 | 2.955 : 3.809 | mixed-32h | 2.146 : 1.490 | 2.135 : 1.523 | 2.157 : 1.557 | 2.327 : 2.088 |Anna Schumaker (6):
SUNRPC: Implementxdr_reserve_space_vec()
NFSD:nfsd4_encode_readv()can usexdr_reserve_space_vec()
NFSD: AddREAD_PLUSdata support
NFSD: AddREAD_PLUShole segment encoding
NFSD: Return both a hole and a data segment
NFSD: Encode a fullREAD_PLUSreply
Quoted 9 lines
fs/nfsd/nfs4proc.c | 17 ++++ fs/nfsd/nfs4xdr.c | 169 +++++++++++++++++++++++++++++++------ include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 45 ++++++++++ 4 files changed, 206 insertions(+), 27 deletions(-)Signature · 1 line
2.27.0
--
Chuck Lever
Hi Chuck,
Quoted 7 lines · Chuck Lever
Hi Anna-
Quoted 4 lines · On Aug 3, 2020, at 12:59 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
Quoted 7 lines
Quoted 3 lines
These patches add server support for the
READ_PLUSoperation, which breaks read requests into several "data" and "hole" segments when replying to the client.At this point, I've wrapped up the content of the "feature" pull request for v5.9, so Bruce and I are assuming this series is not for v5.9.
It's definitely too close to the merge for 5.9.
Quoted 11 lines
Quoted 8 lines
Here are the results of some performance tests I ran on my own virtual machines, which does still have the slow SEEK_HOLE/SEEK_DATA behavior. A recent minimum-compiler-version patch makes installing newer kernels difficult on the lab machines I have access to, but I'll still try to collect that data since it seemed promising during my last posting
(https://www.spinics.net/lists/linux-nfs/msg76487.html).Nit: I've seen other maintainers prefer the use of lore.kernel.org over spinics or mail.info for patch descriptions and cover letters.
I'll have to remember lore.kernel.org exists. I usually go to whatever is listed for "Archives" for a mailing list on vger.kernel.org
Quoted 13 lines
Quoted 12 lines
I tested by reading various 2G files from a few different underlying filesystems and across several NFS versions. I used the
vmtouchutility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h has 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.
Quoted 4 lines
In a VM environment, counting bytes transferred might be a more useful comparison than seconds. I'm not sure if the table reports wall clock time or something else. At what layer was the timing measured?
I used the time reported by dd. I can see what other data I can pull out of it.
Quoted 7 lines
CPU utilization might be interesting to report on the client because clients would have to more work assembling a file with holes in it.
Also the benefit of
READ_PLUSmight be more plain if you chose a larger file size (by perhaps a factor of 100).
I'll try, but the way my test is set up now the files exist on the disk all at the same time (so I don't need to spend forever constantly recreating them) so there is an upper limit to how big I can make things.
Quoted 2 lines
Not clear what the purpose of the v3, v4.0, and v4.1 columns is.
It's to compare v4.2 performance against earlier versions. v4.1 gets us a baseline, since it uses the same codepaths as v4.2 without read plus.
Anna
Quoted 67 lines
Quoted 52 lines
| v3 | v4.0 | v4.1 | v4.2 | ext4 | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.697 : 1.365 | 2.539 : 1.482 | 2.246 : 1.595 | 2.516 : 1.716 | hole | 2.389 : 1.480 | 1.692 : 1.444 | 1.904 : 1.539 | 1.042 : 1.005 | mixed-4d | 2.426 : 1.448 | 2.468 : 1.480 | 2.258 : 1.731 | 2.408 : 1.674 | mixed-8d | 2.446 : 1.592 | 2.497 : 1.523 | 2.240 : 1.736 | 2.179 : 1.547 | mixed-16d | 2.608 : 1.575 | 2.446 : 1.582 | 2.330 : 1.670 | 2.192 : 1.488 | mixed-32d | 2.544 : 1.587 | 2.164 : 1.499 | 2.076 : 1.452 | 1.982 : 1.346 | mixed-4h | 2.591 : 1.486 | 2.233 : 1.503 | 2.190 : 1.520 | 3.679 : 1.815 | mixed-8h | 2.498 : 1.547 | 2.168 : 1.510 | 2.098 : 1.511 | 2.815 : 1.484 | mixed-16h | 2.480 : 1.664 | 2.152 : 1.597 | 2.096 : 1.537 | 2.288 : 1.580 | mixed-32h | 2.520 : 1.467 | 2.171 : 1.496 | 2.246 : 1.593 | 2.066 : 1.389 || v3 | v4.0 | v4.1 | v4.2 | xfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.074 : 1.353 | 2.129 : 1.489 | 2.198 : 1.564 | 2.579 : 1.719 | hole | 1.714 : 1.430 | 1.647 : 1.440 | 1.748 : 1.464 | 1.019 : 1.028 | mixed-4d | 2.699 : 1.561 | 2.782 : 1.657 | 2.800 : 1.619 | 2.848 : 2.166 | mixed-8d | 2.204 : 1.540 | 2.346 : 1.595 | 2.356 : 1.589 | 2.335 : 1.809 | mixed-16d | 2.034 : 1.445 | 2.212 : 1.561 | 2.172 : 1.546 | 2.127 : 1.658 | mixed-32d | 1.982 : 1.480 | 2.135 : 1.544 | 2.136 : 1.565 | 2.024 : 1.555 | mixed-4h | 2.600 : 1.549 | 2.790 : 1.660 | 2.745 : 1.634 | 8.949 : 2.529 | mixed-8h | 2.283 : 1.555 | 2.414 : 1.605 | 2.373 : 1.607 | 5.626 : 2.015 | mixed-16h | 2.115 : 1.512 | 2.247 : 1.593 | 2.217 : 1.608 | 3.740 : 1.803 | mixed-32h | 2.069 : 1.499 | 2.212 : 1.582 | 2.235 : 1.631 | 2.819 : 1.542 || v3 | v4.0 | v4.1 | v4.2 | btrfs | uncached: cached | uncached: cached | uncached: cached | uncached: cached | ----------|-------------------|-------------------|-------------------|-------------------| data | 2.417 : 1.317 | 2.095 : 1.445 | 2.145 : 1.523 | 2.615 : 1.713 | hole | 2.107 : 1.483 | 2.121 : 1.496 | 2.106 : 1.461 | 1.011 : 1.061 | mixed-4d | 2.348 : 1.471 | 2.370 : 1.523 | 2.379 : 1.499 | 3.028 : 225.812 | mixed-8d | 2.227 : 1.476 | 2.231 : 1.467 | 2.272 : 1.529 | 2.723 : 36.179 | mixed-16d | 2.175 : 1.460 | 2.208 : 1.457 | 2.200 : 1.464 | 2.526 : 10.371 | mixed-32d | 2.148 : 1.501 | 2.191 : 1.468 | 2.167 : 1.471 | 2.455 : 3.367 | mixed-4h | 2.362 : 1.561 | 2.387 : 1.513 | 2.352 : 1.536 | 5.935 : 41.494 | mixed-8h | 2.241 : 1.477 | 2.251 : 1.503 | 2.256 : 1.496 | 3.672 : 10.261 | mixed-16h | 2.238 : 1.477 | 2.188 : 1.496 | 2.183 : 1.503 | 2.955 : 3.809 | mixed-32h | 2.146 : 1.490 | 2.135 : 1.523 | 2.157 : 1.557 | 2.327 : 2.088 |Anna Schumaker (6):
SUNRPC: Implementxdr_reserve_space_vec()
NFSD:nfsd4_encode_readv()can usexdr_reserve_space_vec()
NFSD: AddREAD_PLUSdata support
NFSD: AddREAD_PLUShole segment encoding
NFSD: Return both a hole and a data segment
NFSD: Encode a fullREAD_PLUSreplyQuoted 9 lines
fs/nfsd/nfs4proc.c | 17 ++++ fs/nfsd/nfs4xdr.c | 169 +++++++++++++++++++++++++++++++------ include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 45 ++++++++++ 4 files changed, 206 insertions(+), 27 deletions(-)--
2.27.0--
Chuck Lever