NFS: Add support for the v4.2 READ_PLUS operation
Subsystems (3)
v2 cover NFS: Add support for the v4.2 READ_PLUS operation
From: Anna Schumaker <[email protected]>
These patches add client support for the READ_PLUS operation, which breaks read requests into several "data" and "hole" segments when replying to the client. I also add a "noreadplus" mount option to allow users to disable the new operation if it becomes a problem, similar to the "nordirplus" mount option that we already have.
Here are the results of some performance tests I ran on Netapp lab machines. I tested by reading various 2G files from a few different undelying filesystems and across several NFS versions. I used the vmtouch utility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h hase 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.
ext4 | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 22.909 : 18.253 | 22.934 : 18.252 | 22.902 : 18.253 | 23.485 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.708 : 0.709 | mixed-4d | 28.261 : 18.253 | 29.616 : 18.252 | 28.341 : 18.252 | 24.508 : 9.150 | mixed-8d | 27.956 : 18.253 | 28.404 : 18.252 | 28.320 : 18.252 | 23.967 : 9.140 | mixed-16d | 28.172 : 18.253 | 27.946 : 18.252 | 27.627 : 18.252 | 23.043 : 9.134 | mixed-32d | 25.350 : 18.253 | 24.406 : 18.252 | 24.384 : 18.253 | 20.698 : 9.132 | mixed-4h | 28.913 : 18.253 | 28.564 : 18.252 | 27.996 : 18.252 | 21.837 : 9.150 | mixed-8h | 28.625 : 18.253 | 27.833 : 18.252 | 27.798 : 18.253 | 21.710 : 9.140 | mixed-16h | 27.975 : 18.253 | 27.662 : 18.252 | 27.795 : 18.253 | 20.585 : 9.134 | mixed-32h | 25.958 : 18.253 | 25.491 : 18.252 | 24.856 : 18.252 | 21.018 : 9.132 |
xfs | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 22.041 : 18.253 | 22.618 : 18.252 | 23.067 : 18.253 | 23.496 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.723 : 0.708 | mixed-4d | 29.417 : 18.253 | 28.503 : 18.252 | 28.671 : 18.253 | 24.957 : 9.150 | mixed-8d | 29.080 : 18.253 | 29.401 : 18.252 | 29.251 : 18.252 | 24.625 : 9.140 | mixed-16d | 27.638 : 18.253 | 28.606 : 18.252 | 27.871 : 18.253 | 25.511 : 9.135 | mixed-32d | 24.967 : 18.253 | 25.239 : 18.252 | 25.434 : 18.252 | 21.728 : 9.132 | mixed-4h | 34.816 : 18.253 | 36.243 : 18.252 | 35.837 : 18.252 | 32.332 : 9.150 | mixed-8h | 43.469 : 18.253 | 44.009 : 18.252 | 43.810 : 18.253 | 37.962 : 9.140 | mixed-16h | 29.280 : 18.253 | 28.563 : 18.252 | 28.241 : 18.252 | 22.116 : 9.134 | mixed-32h | 29.428 : 18.253 | 29.378 : 18.252 | 28.808 : 18.253 | 27.378 : 9.134 |
btrfs | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 25.547 : 18.253 | 25.053 : 18.252 | 24.209 : 18.253 | 32.121 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.252 | 0.702 : 0.724 | mixed-4d | 19.016 : 18.253 | 18.822 : 18.252 | 18.955 : 18.253 | 18.697 : 9.150 | mixed-8d | 19.186 : 18.253 | 19.444 : 18.252 | 18.841 : 18.253 | 18.452 : 9.140 | mixed-16d | 18.480 : 18.253 | 19.010 : 18.252 | 19.167 : 18.252 | 16.000 : 9.134 | mixed-32d | 18.635 : 18.253 | 18.565 : 18.252 | 18.550 : 18.252 | 15.930 : 9.132 | mixed-4h | 19.079 : 18.253 | 18.990 : 18.252 | 19.157 : 18.253 | 27.834 : 9.150 | mixed-8h | 18.613 : 18.253 | 19.234 : 18.252 | 18.616 : 18.253 | 20.177 : 9.140 | mixed-16h | 18.590 : 18.253 | 19.221 : 18.252 | 19.654 : 18.253 | 17.273 : 9.135 | mixed-32h | 18.768 : 18.253 | 19.122 : 18.252 | 18.535 : 18.252 | 15.791 : 9.132 |
ext3 | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 34.292 : 18.253 | 33.810 : 18.252 | 33.450 : 18.253 | 33.390 : 18.254 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.718 : 0.728 | mixed-4d | 46.818 : 18.253 | 47.140 : 18.252 | 48.385 : 18.253 | 42.887 : 9.150 | mixed-8d | 58.554 : 18.253 | 59.277 : 18.252 | 59.673 : 18.253 | 56.760 : 9.140 | mixed-16d | 44.631 : 18.253 | 44.291 : 18.252 | 44.729 : 18.253 | 40.237 : 9.135 | mixed-32d | 39.110 : 18.253 | 38.735 : 18.252 | 38.902 : 18.252 | 35.270 : 9.132 | mixed-4h | 56.396 : 18.253 | 56.387 : 18.252 | 56.573 : 18.253 | 67.661 : 9.150 | mixed-8h | 58.483 : 18.253 | 58.484 : 18.252 | 59.099 : 18.253 | 77.958 : 9.140 | mixed-16h | 42.511 : 18.253 | 42.338 : 18.252 | 42.356 : 18.252 | 51.805 : 9.135 | mixed-32h | 38.419 : 18.253 | 38.504 : 18.252 | 38.643 : 18.252 | 40.411 : 9.132 |
Changes since v1:
- Rebase to 5.6-rc1
- Drop the mount option patch for now
- Fix fallback to READ when the server doesn't support READ_PLUS
Any questions?
Anna
Anna Schumaker (6):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
SUNRPC: Add the ability to shift data to a specific offset
NFS: Add READ_PLUS data segment support
NFS: Add READ_PLUS hole segment decoding
NFS: Decode multiple READ_PLUS segments
fs/nfs/nfs42xdr.c | 169 +++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 ++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 244 ++++++++++++++++++++++++++++++++++++- 8 files changed, 457 insertions(+), 7 deletions(-)
Signature · 1 line
2.25.0
PATCH v2 1/6 SUNRPC: Split out a function for setting current page
From: Anna Schumaker <[email protected]>
I'm going to need this bit of code in a few places for READ_PLUS decoding, so let's make it a helper function.
--- net/sunrpc/xdr.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
net/sunrpc/xdr.c +7 -2
@@ -825,6 +825,12 @@ static int xdr_set_page_base(struct xdr_stream *xdr,return 0 ; } +static void xdr_set_page (struct xdr_stream *xdr,unsigned int base) +{ +if (xdr_set_page_base (xdr, base,PAGE_SIZE ) <0 ) +xdr_set_iov (xdr, xdr->buf->tail, xdr->nwords <<2 ); +} +static void xdr_set_next_page (struct xdr_stream *xdr) {unsigned int newbase;
@@ -832,8 +838,7 @@ static void xdr_set_next_page(struct xdr_stream *xdr) newbase = (1 + xdr->page_ptr - xdr->buf->pages) <<PAGE_SHIFT ; newbase -= xdr->buf->page_base; -if (xdr_set_page_base (xdr, newbase,PAGE_SIZE ) <0 ) -xdr_set_iov (xdr, xdr->buf->tail, xdr->nwords <<2 ); +xdr_set_page (xdr, newbase); }static bool xdr_set_next_buffer (struct xdr_stream *xdr)
Signature · 1 line
2.25.0
PATCH v2 2/6 SUNRPC: Add the ability to expand holes in data pages
In reply to [email protected]From: Anna Schumaker <[email protected]>
This patch adds the ability to "read a hole" into a set of XDR data pages by taking the following steps:
1) Shift all data after the current xdr->p to the right, possibly into the tail,
2) Zero the specified range, and
3) Update xdr->p to point beyond the hole.
--- include/linux/sunrpc/xdr.h | 1 + net/sunrpc/xdr.c | 100 +++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+)
include/linux/sunrpc/xdr.h +1 -0
@@ -263,6 +263,7 @@ extern __be32 *xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes);extern unsigned int xdr_read_pages (struct xdr_stream *xdr,unsigned int len);extern void xdr_enter_page (struct xdr_stream *xdr,unsigned int len);extern int xdr_process_buf (struct xdr_buf *buf,unsigned int offset,unsigned int len,int (*actor)(struct scatterlist *,void *),void *data); +extern size_t xdr_expand_hole (struct xdr_stream *,size_t ,uint64_t );/** * xdr_stream_remaining - Return the number of bytes remaining in the stream
net/sunrpc/xdr.c +100 -0
124 lines not on this page · raw message
Signature · 1 line
2.25.0
PATCH v2 3/6 SUNRPC: Add the ability to shift data to a specific offset
In reply to [email protected]From: Anna Schumaker <[email protected]>
Expanding holes tends to put the data content a few bytes to the right of where we want it. This patch implements a left-shift operation to line everything up properly.
--- include/linux/sunrpc/xdr.h | 1 + net/sunrpc/xdr.c | 135 +++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+)
include/linux/sunrpc/xdr.h +1 -0
@@ -264,6 +264,7 @@ extern unsigned int xdr_read_pages(struct xdr_stream *xdr, unsigned int len);extern void xdr_enter_page (struct xdr_stream *xdr,unsigned int len);extern int xdr_process_buf (struct xdr_buf *buf,unsigned int offset,unsigned int len,int (*actor)(struct scatterlist *,void *),void *data);extern size_t xdr_expand_hole (struct xdr_stream *,size_t ,uint64_t ); +extern uint64_t xdr_align_data (struct xdr_stream *,uint64_t ,uint64_t );/** * xdr_stream_remaining - Return the number of bytes remaining in the stream
net/sunrpc/xdr.c +135 -0
153 lines not on this page · raw message
Signature · 1 line
2.25.0
PATCH v2 4/6 NFS: Add READ_PLUS data segment support
In reply to [email protected]From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single NFS4_CONTENT_DATA segment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.
--- fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)
fs/nfs/nfs42xdr.c +138 -0
@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1 /* rpr_eof */ + \ + 1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \
@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \
@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,const struct nfs42_seek_args *args,struct compound_hdr *hdr)
@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr); +encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ; +encode_nops (&hdr); +} +/* * Encode SEEK request */
@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *rereturn decode_op_hdr (xdr,OP_DEALLOCATE ); } +static uint32_t decode_read_plus_data (struct xdr_stream *xdr,struct nfs_pgio_res *res, +uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int decode_seek (struct xdr_stream *xdr,struct nfs42_seek_res *res) {int status;
@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */
fs/nfs/nfs4proc.c +40 -3
81 lines not on this page · raw message
fs/nfs/nfs4xdr.c +1 -0
@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify),PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror), + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE (nfs4_procedures)];
include/linux/nfs4.h +1 -1
@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */
include/linux/nfs_fs_sb.h +1 -0
@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif
include/linux/nfs_xdr.h +1 -1
@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {
Signature · 1 line
2.25.0
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 86 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATAsegment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +37
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1 /* rpr_eof */ + \ + 1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,const struct nfs42_seek_args *args,struct compound_hdr *hdr)@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args),
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 10 lines
fs/nfs/nfs42xdr.c +10
-0+ }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr); +encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;
IMO this line is incorrect.
RFC 8267 Section 6.1 does not list any part of the result of READ_PLUS as DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.
Just curious, have you tried READ_PLUS with proto=rdma ?
Quoted 249 lines
fs/nfs/nfs42xdr.c +91
-0+encode_nops (&hdr); +} +/* * Encode SEEK request */ @@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *rereturn decode_op_hdr (xdr,OP_DEALLOCATE ); } +static uint32_t decode_read_plus_data (struct xdr_stream *xdr,struct nfs_pgio_res *res, +uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int decode_seek (struct xdr_stream *xdr,struct nfs42_seek_res *res) {int status;@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp,return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */ fs/nfs/nfs4proc.c +40
-3@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(struct rpc_task *task,return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + +if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ] && + server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP ) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int nfs4_read_done (struct rpc_task *task,struct nfs_pgio_header *hdr) { -dprintk ("--> %s\n" , __func__);if (!nfs4_sequence_done (task, &hdr->res.seq_res))return -EAGAIN ;if (nfs4_read_stateid_changed (task, &hdr->args))return -EAGAIN ; +if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status >0 )nfs_invalidate_atime (hdr->inode);return hdr->pgio_done_cb ? hdr->pgio_done_cb (task, hdr) :nfs4_read_done_cb (task, hdr); } +#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ]; +else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#else +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup (struct nfs_pgio_header *hdr,struct rpc_message *msg) { hdr->timestamp = jiffies;if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb; - msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence (&hdr->args.seq_args, &hdr->res.seq_res,0 ,0 ); }@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { | NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client, .match_stateid = nfs41_match_stateid,fs/nfs/nfs4xdr.c +1
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify),PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror), + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE (nfs4_procedures)];include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0
--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 89 lines · Chuck Lever
Quoted 89 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATAsegment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +41
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1 /* rpr_eof */ + \ + 1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,const struct nfs42_seek_args *args,struct compound_hdr *hdr)@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr);
Quoted 8 lines
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 6 lines
fs/nfs/nfs42xdr.c +6
-0+encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;IMO this line is incorrect.
You're right, this line causes problems for RDMA with READ_PLUS. I added it to match how the other xdr read encoders were set up
Quoted 7 lines
RFC 8267 Section 6.1 does not list any part of the result of
READ_PLUSas DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.Just curious, have you tried
READ_PLUSwith proto=rdma ?
I haven't done in-depth performance testing, but I have been able to run it.
Anna
Quoted 266 lines
Quoted 260 lines
fs/nfs/nfs42xdr.c +3
-0+encode_nops (&hdr); +} +/* * Encode SEEK request */
fs/nfs/nfs42xdr.c +0
-0@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr,struct
nfs42_falloc_res*rereturn decode_op_hdr(xdr, OP_DEALLOCATE); }+static
uint32_tdecode_read_plus_data(structxdr_stream*xdr, structnfs_pgio_res*res,fs/nfs/nfs42xdr.c +27
-0+ uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res*res)
fs/nfs/nfs42xdr.c +33
-0+{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int
decode_seek(struct xdr_stream *xdr, struct nfs42_seek_res *res){ int status;fs/nfs/nfs42xdr.c +27
-0@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp, return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */
fs/nfs/nfs4proc.c +20
-0@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(structrpc_task *task, return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + +if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ] && + server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP ) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int
nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
{
-dprintk("--> %s\n", __func__); if (!nfs4_sequence_done(task, &hdr->res.seq_res)) return -EAGAIN; if (nfs4_read_stateid_changed(task, &hdr->args)) return -EAGAIN;fs/nfs/nfs4proc.c +2
-0+if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status > 0) nfs_invalidate_atime(hdr->inode); return hdr->pgio_done_cb ? hdr->pgio_done_cb(task, hdr) : nfs4_read_done_cb(task, hdr); }fs/nfs/nfs4proc.c +2
-0+#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +8
-0+{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ]; +else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#else +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +5
-0+{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup(struct
nfs_pgio_header*hdr,
structrpc_message*msg){ hdr->timestamp = jiffies; if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb;fs/nfs/nfs4proc.c +1
-1- msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0, 0); }fs/nfs/nfs4proc.c +2
-1@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { |NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,.match_stateid =
nfs41_match_stateid,fs/nfs/nfs4xdr.c +1
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify),PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror), + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE (nfs4_procedures)];include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 103 lines · Anna Schumaker
Quoted 89 lines · Chuck Lever
Quoted 89 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATAsegment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +41
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1 /* rpr_eof */ + \ + 1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,const struct nfs42_seek_args *args,struct compound_hdr *hdr)@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr);Quoted 8 lines
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 6 lines
fs/nfs/nfs42xdr.c +6
-0+encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;IMO this line is incorrect.
Quoted 2 lines
You're right, this line causes problems for RDMA with
READ_PLUS. I added it to match how the other xdr read encoders were set up
Ja, I think just removing that line should be sufficient. Better would be replacing it with a comment explaining why this encoder does not set XDRBUF_READ. :-)
Quoted 8 lines
Quoted 6 lines
RFC 8267 Section 6.1 does not list any part of the result of
READ_PLUSas DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.Just curious, have you tried
READ_PLUSwith proto=rdma ?I haven't done in-depth performance testing, but I have been able to run it.
We should figure out whether that will have a regressive impact on NFS/RDMA workloads. I expect that it will, but the client can always set up the Reply chunk so that the READ payload fits precisely in an RDMA segment that lines up with page cache pages. That mitigates some impact.
If your patch set already changes NFSv4.2 mounts to always use READ_PLUS in place of READ, it might be prudent for the "proto=rdma" mount option to also set "noreadplus", at least for the time being.
The down-side here is that would make NFSv4.2 on RDMA unable to recognize holes in files the same way as it does on TCP, and that's a pretty significant variation in behavior. Does "noreadplus" even deal with that?
Quoted 265 lines
Anna
Quoted 263 lines
Quoted 260 lines
fs/nfs/nfs42xdr.c +3
-0+encode_nops (&hdr); +} +/* * Encode SEEK request */
fs/nfs/nfs42xdr.c +0
-0@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr,struct
nfs42_falloc_res*rereturn decode_op_hdr(xdr, OP_DEALLOCATE); }+static
uint32_tdecode_read_plus_data(structxdr_stream*xdr, structnfs_pgio_res*res,fs/nfs/nfs42xdr.c +27
-0+ uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res*res)
fs/nfs/nfs42xdr.c +33
-0+{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int
decode_seek(struct xdr_stream *xdr, struct nfs42_seek_res *res){ int status;fs/nfs/nfs42xdr.c +27
-0@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp, return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */
fs/nfs/nfs4proc.c +20
-0@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(structrpc_task *task, return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + +if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ] && + server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP ) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int
nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
{
-dprintk("--> %s\n", __func__); if (!nfs4_sequence_done(task, &hdr->res.seq_res)) return -EAGAIN; if (nfs4_read_stateid_changed(task, &hdr->args)) return -EAGAIN;fs/nfs/nfs4proc.c +2
-0+if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status > 0) nfs_invalidate_atime(hdr->inode); return hdr->pgio_done_cb ? hdr->pgio_done_cb(task, hdr) : nfs4_read_done_cb(task, hdr); }fs/nfs/nfs4proc.c +2
-0+#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +8
-0+{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ]; +else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#else +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +5
-0+{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup(struct
nfs_pgio_header*hdr,
structrpc_message*msg){ hdr->timestamp = jiffies; if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb;fs/nfs/nfs4proc.c +1
-1- msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0, 0); }fs/nfs/nfs4proc.c +2
-1@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { |NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,.match_stateid =
nfs41_match_stateid,fs/nfs/nfs4xdr.c +1
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify),PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror), + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE (nfs4_procedures)];include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0--
Chuck Lever
--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 128 lines · Chuck Lever
Quoted 104 lines
On Feb 20, 2020, at 9:42 AM, Anna Schumaker <[email protected]> wrote:
Quoted 91 lines · Chuck Lever
Quoted 91 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATAsegment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +9
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ +1 /* rpr_eof */ + \ +1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE)#define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \2 /* offset */ + \fs/nfs/nfs42xdr.c +18
-0@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} + static void encode_seek(struct xdr_stream *xdr, const struct nfs42_seek_args *args,struct
compound_hdr*hdr)fs/nfs/nfs42xdr.c +14
-0@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr);Quoted 8 lines
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 6 lines
fs/nfs/nfs42xdr.c +6
-0+encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;IMO this line is incorrect.
Quoted 3 lines
You're right, this line causes problems for RDMA with
READ_PLUS. I added it to
match how the other xdr read encoders were set upJa, I think just removing that line should be sufficient. Better would be replacing it with a comment explaining why this encoder does not set
XDRBUF_READ. :-)Quoted 8 lines
Quoted 6 lines
RFC 8267 Section 6.1 does not list any part of the result of
READ_PLUSas DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.Just curious, have you tried
READ_PLUSwith proto=rdma ?I haven't done in-depth performance testing, but I have been able to run it.
We should figure out whether that will have a regressive impact on NFS/RDMA workloads. I expect that it will, but the client can always set up the Reply chunk so that the READ payload fits precisely in an RDMA segment that lines up with page cache pages. That mitigates some impact.
Quoted 4 lines
If your patch set already changes NFSv4.2 mounts to always use
READ_PLUSin place of READ, it might be prudent for the "proto=rdma" mount option to also set "noreadplus", at least for the time being.
I can make this change.
Quoted 5 lines
The down-side here is that would make NFSv4.2 on RDMA unable to recognize holes in files the same way as it does on TCP, and that's a pretty significant variation in behavior. Does "noreadplus" even deal with that?
Setting "noreadplus" just causes the client to use the READ operation instead, so there should be no difference between v4.1 and v4.2 if the option is set.
Anna
Quoted 280 lines
Quoted 274 lines
Anna
Quoted 270 lines
Quoted 267 lines
fs/nfs/nfs42xdr.c +3
-0+encode_nops (&hdr); +} +/* * Encode SEEK request */
fs/nfs/nfs42xdr.c +0
-0@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr,struct
nfs42_falloc_res*rereturn decode_op_hdr(xdr, OP_DEALLOCATE); }+static
uint32_tdecode_read_plus_data(structxdr_stream*xdr, structnfs_pgio_res*res,fs/nfs/nfs42xdr.c +27
-0+ uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res*res)
fs/nfs/nfs42xdr.c +33
-0+{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int
decode_seek(struct xdr_stream *xdr, struct nfs42_seek_res *res){ int status;fs/nfs/nfs42xdr.c +27
-0@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp, return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */
fs/nfs/nfs4proc.c +11
-0@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(structrpc_task *task, return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + + if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS]&&
fs/nfs/nfs4proc.c +9
-1+ server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int
nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
{
-dprintk("--> %s\n", __func__); if (!nfs4_sequence_done(task, &hdr->res.seq_res)) return -EAGAIN; if (nfs4_read_stateid_changed(task, &hdr->args)) return -EAGAIN;fs/nfs/nfs4proc.c +2
-0+if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status > 0) nfs_invalidate_atime(hdr->inode); return hdr->pgio_done_cb ? hdr->pgio_done_cb(task, hdr) : nfs4_read_done_cb(task, hdr); }fs/nfs/nfs4proc.c +2
-0+#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +3
-0+{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc =&nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
fs/nfs/nfs4proc.c +5
-0+ else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ]; +} +#else +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +5
-0+{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup(struct
nfs_pgio_header*hdr,
structrpc_message*msg){ hdr->timestamp = jiffies; if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb;fs/nfs/nfs4proc.c +1
-1- msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0,0);
}fs/nfs/nfs4proc.c +2
-1@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { |NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,.match_stateid =
nfs41_match_stateid,fs/nfs/nfs4xdr.c +1
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify) ,PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror) , + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE(nfs4_procedures)];
include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0--
Chuck Lever--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 141 lines · Anna Schumaker
Quoted 128 lines · Chuck Lever
Quoted 104 lines
On Feb 20, 2020, at 9:42 AM, Anna Schumaker <[email protected]> wrote:
Quoted 91 lines · Chuck Lever
Quoted 91 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATAsegment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c | 138 ++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +9
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ +1 /* rpr_eof */ + \ +1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE)#define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \2 /* offset */ + \fs/nfs/nfs42xdr.c +18
-0@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} + static void encode_seek(struct xdr_stream *xdr, const struct nfs42_seek_args *args,struct
compound_hdr*hdr)fs/nfs/nfs42xdr.c +14
-0@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req,encode_nops (&hdr); } +/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr);Quoted 8 lines
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 6 lines
fs/nfs/nfs42xdr.c +6
-0+encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;IMO this line is incorrect.
Quoted 3 lines
You're right, this line causes problems for RDMA with
READ_PLUS. I added it to
match how the other xdr read encoders were set upJa, I think just removing that line should be sufficient. Better would be replacing it with a comment explaining why this encoder does not set
XDRBUF_READ. :-)Quoted 8 lines
Quoted 6 lines
RFC 8267 Section 6.1 does not list any part of the result of
READ_PLUSas DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.Just curious, have you tried
READ_PLUSwith proto=rdma ?I haven't done in-depth performance testing, but I have been able to run it.
We should figure out whether that will have a regressive impact on NFS/RDMA workloads. I expect that it will, but the client can always set up the Reply chunk so that the READ payload fits precisely in an RDMA segment that lines up with page cache pages. That mitigates some impact.
Quoted 4 lines
If your patch set already changes NFSv4.2 mounts to always use
READ_PLUSin place of READ, it might be prudent for the "proto=rdma" mount option to also set "noreadplus", at least for the time being.I can make this change.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
The down-side here is that would make NFSv4.2 on RDMA unable to recognize holes in files the same way as it does on TCP, and that's a pretty significant variation in behavior. Does "noreadplus" even deal with that?
Setting "noreadplus" just causes the client to use the READ operation instead, so there should be no difference between v4.1 and v4.2 if the option is set.
My concern is the difference between NFSv4.2 with noreadplus and NFSv4.2 with readplus. The former is not able to detect holes in files on the server, but the latter is.
Is that worth mentioning in the man page, or in release notes when NFSv4.2 becomes the default?
Quoted 279 lines
Anna
Quoted 277 lines
Quoted 274 lines
Anna
Quoted 270 lines
Quoted 267 lines
fs/nfs/nfs42xdr.c +3
-0+encode_nops (&hdr); +} +/* * Encode SEEK request */
fs/nfs/nfs42xdr.c +0
-0@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr,struct
nfs42_falloc_res*rereturn decode_op_hdr(xdr, OP_DEALLOCATE); }+static
uint32_tdecode_read_plus_data(structxdr_stream*xdr, structnfs_pgio_res*res,fs/nfs/nfs42xdr.c +27
-0+ uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res*res)
fs/nfs/nfs42xdr.c +33
-0+{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int
decode_seek(struct xdr_stream *xdr, struct nfs42_seek_res *res){ int status;fs/nfs/nfs42xdr.c +27
-0@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct rpc_rqst *rqstp, return status; } +/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/* * Decode SEEK request */
fs/nfs/nfs4proc.c +11
-0@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(structrpc_task *task, return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + + if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS]&&
fs/nfs/nfs4proc.c +9
-1+ server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int
nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
{
-dprintk("--> %s\n", __func__); if (!nfs4_sequence_done(task, &hdr->res.seq_res)) return -EAGAIN; if (nfs4_read_stateid_changed(task, &hdr->args)) return -EAGAIN;fs/nfs/nfs4proc.c +2
-0+if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status > 0) nfs_invalidate_atime(hdr->inode); return hdr->pgio_done_cb ? hdr->pgio_done_cb(task, hdr) : nfs4_read_done_cb(task, hdr); }fs/nfs/nfs4proc.c +2
-0+#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +3
-0+{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc =&nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
fs/nfs/nfs4proc.c +5
-0+ else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ]; +} +#else +static void nfs42_read_plus_support(struct nfs_server *server, struct
rpc_message*msg)fs/nfs/nfs4proc.c +5
-0+{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup(struct
nfs_pgio_header*hdr,
structrpc_message*msg){ hdr->timestamp = jiffies; if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb;fs/nfs/nfs4proc.c +1
-1- msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0,0);
}fs/nfs/nfs4proc.c +2
-1@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { |NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,.match_stateid =
nfs41_match_stateid,fs/nfs/nfs4xdr.c +1
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify) ,PROC (LOOKUPP , enc_lookupp, dec_lookupp),PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror) , + PROC42(READ_PLUS, enc_read_plus, dec_read_plus), };static unsigned int nfs_version4_counts[ARRAY_SIZE(nfs4_procedures)];
include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0--
Chuck Lever--
Chuck Lever
--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 153 lines · Chuck Lever
Quoted 153 lines
On Feb 20, 2020, at 1:28 PM, Anna Schumaker <[email protected]> wrote:
Quoted 137 lines · Chuck Lever
Quoted 111 lines
On Feb 20, 2020, at 9:42 AM, Anna Schumaker <[email protected]> wrote:
Quoted 98 lines · Chuck Lever
Quoted 98 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
This patch adds client support for decoding a single
NFS4_CONTENT_DATA
segment returned by the server. This is the simplest implementation possible, since it does not account for any hole segments in the reply.Signed-off-by Anna Schumaker <[email protected]>---
fs/nfs/nfs42xdr.c| 138
++++++++++++++++++++++++++++++++++++++fs/nfs/nfs4proc.c | 43 +++++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 182 insertions(+), 5 deletions(-)fs/nfs/nfs42xdr.c +9
-0@@ -45,6 +45,15 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ + 2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ +1 /* rpr_eof */ + \ +1 /* rpr_contents count */ + \ + NFS42_READ_PLUS_SEGMENT_SIZE)#define encode_seek_maxsz (op_encode_hdr_maxsz + \encode_stateid_maxsz + \ 2 /* offset */ + \fs/nfs/nfs42xdr.c +8
-0@@ -128,6 +137,14 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_max sz + \ + encode_sequence_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_max sz + \ + decode_sequence_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \encode_sequence_maxsz + \ encode_putfh_maxsz + \fs/nfs/nfs42xdr.c +10
-0@@ -252,6 +269,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +const struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} + static void encode_seek(struct xdr_stream *xdr, const struct nfs42_seek_args *args,struct
compound_hdr*hdr)fs/nfs/nfs42xdr.c +0
-0@@ -446,6 +473,29 @@ static void nfs4_xdr_enc_deallocate(struct
rpc_rqst
*req,encode_nops(&hdr); }fs/nfs/nfs42xdr.c +14
-0+/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +const void *data) +{ +const struct nfs_pgio_args *args = data; +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr);Quoted 8 lines
fs/nfs/nfs42xdr.c static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, · line 495
Quoted 6 lines
fs/nfs/nfs42xdr.c +6
-0+encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +rpc_prepare_reply_pages (req, args->pages, args->pgbase, + args->count, hdr.replen); + req->rq_rcv_buf.flags |=XDRBUF_READ ;IMO this line is incorrect.
Quoted 4 lines
You're right, this line causes problems for RDMA with
READ_PLUS. I added it
to
match how the other xdr read encoders were set upJa, I think just removing that line should be sufficient. Better would be replacing it with a comment explaining why this encoder does not set
XDRBUF_READ. :-)Quoted 9 lines
Quoted 6 lines
RFC 8267 Section 6.1 does not list any part of the result of
READ_PLUSas DDP-eligible. There's no way for a client to know how to set up Write chunks, unless it knows exactly where the file's holes are in advance. Even then... racy.Just curious, have you tried
READ_PLUSwith proto=rdma ?I haven't done in-depth performance testing, but I have been able to run it.
We should figure out whether that will have a regressive impact on NFS/RDMA workloads. I expect that it will, but the client can always set up the Reply chunk so that the READ payload fits precisely in an RDMA segment that lines up with page cache pages. That mitigates some impact.
Quoted 4 lines
If your patch set already changes NFSv4.2 mounts to always use
READ_PLUSin place of READ, it might be prudent for the "proto=rdma" mount option to also set "noreadplus", at least for the time being.I can make this change.
Quoted 4 lines
The down-side here is that would make NFSv4.2 on RDMA unable to recognize holes in files the same way as it does on TCP, and that's a pretty significant variation in behavior. Does "noreadplus" even deal with that?
Quoted 7 lines
Quoted 3 lines
Setting "noreadplus" just causes the client to use the READ operation instead,
so there should be no difference between v4.1 and v4.2 if the option is set.My concern is the difference between NFSv4.2 with noreadplus and NFSv4.2 with readplus. The former is not able to detect holes in files on the server, but the latter is.
The client could still use lseek to detect holes. The client throws away the hole information after xdr decoding, and zeroes out the corresponding pages for the page cache.
Quoted 297 lines
Is that worth mentioning in the man page, or in release notes when NFSv4.2 becomes the default?
Quoted 286 lines
Anna
Quoted 284 lines
Quoted 281 lines
Anna
Quoted 278 lines
Quoted 275 lines
fs/nfs/nfs42xdr.c +3
-0+encode_nops (&hdr); +} +/*
* Encode SEEK request
*/fs/nfs/nfs42xdr.c +0
-0@@ -694,6 +744,67 @@ static int decode_deallocate(struct xdr_stream *xdr,struct
nfs42_falloc_res*rereturn decode_op_hdr(xdr, OP_DEALLOCATE); }+static
uint32_tdecode_read_plus_data(structxdr_stream*xdr, structnfs_pgio_res*res,fs/nfs/nfs42xdr.c +27
-0+ uint32_t *eof) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); +if (count ==0 ) +return 0 ; + + recvd =xdr_read_pages (xdr, count); +if (count > recvd) { +dprintk ("NFS: server cheating in read reply: " +"count %u > recvd %u\n" , count, recvd); + count = recvd; + *eof =0 ; + } + +return count; +} + +static int decode_read_plus(struct xdr_stream *xdr, struct
nfs_pgio_res
*res)fs/nfs/nfs42xdr.c +33
-0+{ +__be32 *p; +uint32_t count, eof, segments, type; +int status; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) +return -EIO ; + + eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof); +else +return -EINVAL ; + + res->eof = eof; + res->count = count; +return 0 ; +} +static int
decode_seek(struct xdr_stream *xdr, struct nfs42_seek_res *res){ int status;fs/nfs/nfs42xdr.c +0
-0@@ -870,6 +981,33 @@ static int nfs4_xdr_dec_deallocate(struct
rpc_rqst
*rqstp,return status; }fs/nfs/nfs42xdr.c +27
-0+/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +void *data) +{ +struct nfs_pgio_res *res = data; +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} +/*
* Decode SEEK request
*/fs/nfs/nfs4proc.c +11
-0@@ -69,6 +69,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_BITMASK_SZ 3@@ -5199,28 +5203,60 @@ static bool nfs4_read_stateid_changed(structrpc_task *task, return true; } +static bool nfs4_read_plus_not_supported (struct rpc_task *task, +struct nfs_pgio_header *hdr) +{ +struct nfs_server *server =NFS_SERVER (hdr->inode); +struct rpc_message *msg = &task->tk_msg; + + if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS]&&
fs/nfs/nfs4proc.c +9
-1+ server->caps &NFS_CAP_READ_PLUS && task->tk_status == -ENOTSUPP) { + server->caps &= ~NFS_CAP_READ_PLUS ; + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +rpc_restart_call_prepare (task); +return true; + } +return false; +} +static int nfs4_read_done(struct
rpc_task*task, structnfs_pgio_header
*hdr)
{
-dprintk("--> %s\n", __func__); if (!nfs4_sequence_done(task, &hdr->res.seq_res)) return -EAGAIN; if (nfs4_read_stateid_changed(task, &hdr->args)) return -EAGAIN;fs/nfs/nfs4proc.c +2
-0+if (nfs4_read_plus_not_supported (task, hdr)) +return -EAGAIN ;if (task->tk_status > 0) nfs_invalidate_atime(hdr->inode); return hdr->pgio_done_cb ? hdr->pgio_done_cb(task, hdr) : nfs4_read_done_cb(task, hdr); }fs/nfs/nfs4proc.c +2
-0+#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support(struct nfs_server *server,struct
rpc_message*msg)fs/nfs/nfs4proc.c +3
-0+{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc =&nfs4_procedures[NFSPROC4_CLNT_READ_PLUS];
fs/nfs/nfs4proc.c +5
-0+ else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ]; +} +#else +static void nfs42_read_plus_support(struct nfs_server *server,struct
rpc_message*msg)fs/nfs/nfs4proc.c +5
-0+{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup(struct
nfs_pgio_header*hdr,
structrpc_message*msg){ hdr->timestamp = jiffies; if (!hdr->pgio_done_cb) hdr->pgio_done_cb = nfs4_read_done_cb;fs/nfs/nfs4proc.c +1
-1- msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence(&hdr->args.seq_args, &hdr->res.seq_res, 0,0);
}fs/nfs/nfs4proc.c +2
-1@@ -9970,7 +10006,8 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { |NFS_CAP_SEEK |NFS_CAP_LAYOUTSTATS |NFS_CAP_CLONE - | NFS_CAP_LAYOUTERROR, + |NFS_CAP_LAYOUTERROR + | NFS_CAP_READ_PLUS, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,.match_stateid =
nfs41_match_stateid,fs/nfs/nfs4xdr.c +0
-0@@ -7584,6 +7584,7 @@ const struct rpc_procinfo nfs4_procedures[] = {PROC42 (COPY_NOTIFY , enc_copy_notify, dec_copy_notify) ,PROC (LOOKUPP , enc_lookupp, dec_lookupp) ,PROC42 (LAYOUTERROR , enc_layouterror, dec_layouterror),
fs/nfs/nfs4xdr.c +1
-0+ PROC42(READ_PLUS, enc_read_plus, dec_read_plus),};
static unsigned int
nfs_version4_counts[ARRAY_SIZE(nfs4_procedures)];include/linux/nfs4.h +1
-1@@ -540,8 +540,8 @@ enum {NFSPROC4_CLNT_LOOKUPP ,NFSPROC4_CLNT_LAYOUTERROR , -NFSPROC4_CLNT_COPY_NOTIFY , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -281,5 +281,6 @@ struct nfs_server {#define NFS_CAP_OFFLOAD_CANCEL (1U << 25)#define NFS_CAP_LAYOUTERROR (1U << 26)#define NFS_CAP_COPY_NOTIFY (1U << 27) +#define NFS_CAP_READ_PLUS (1U << 28)#endif include/linux/nfs_xdr.h +1
-1@@ -655,7 +655,7 @@ struct nfs_pgio_args { struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - __u32 count; + __u64 count; __u32 op_status; union { struct {Signature · 1 line
2.25.0--
Chuck Lever--
Chuck Lever--
Chuck Lever
Re: [PATCH v2 4/6] NFS: Add READ_PLUS data segment support
Quoted 18 lines · Anna Schumaker
Quoted 14 lines · Chuck Lever
Quoted 13 lines
On Feb 20, 2020, at 1:28 PM, Anna Schumaker <[email protected]> wrote:
Quoted 4 lines · Chuck Lever
The down-side here is that would make NFSv4.2 on RDMA unable to recognize holes in files the same way as it does on TCP, and that's a pretty significant variation in behavior. Does "noreadplus" even deal with that?
Setting "noreadplus" just causes the client to use the READ operation instead,
so there should be no difference between v4.1 and v4.2 if the option is set.
Quoted 7 lines
Quoted 3 lines
My concern is the difference between NFSv4.2 with noreadplus and NFSv4.2 with readplus. The former is not able to detect holes in files on the server, but the latter is.
The client could still use lseek to detect holes. The client throws away the hole information after xdr decoding, and zeroes out the corresponding pages for the page cache.
Then maybe that's an optimization for another day. The READ_PLUS reply can have hole information. The client could save itself some SEEK_HOLE operations if it cached that hole information somehow.
But if "readplus" and "noreadplus" result in exactly the same hole retention behavior on our client, I guess there's no harm in using "noreadplus" instead, except for possible performance regression.
An alternative to making "noreadplus" the default would be to temporarily add "noreadplus" semantics to the "proto=rdma" mount option, as a separate patch; maybe after some performance results show that it is necessary. That would keep the mount interface a little less complicated.
--
Chuck Lever
PATCH v2 5/6 NFS: Add READ_PLUS hole segment decoding
In reply to [email protected]From: Anna Schumaker <[email protected]>
We keep things simple for now by only decoding a single hole or data segment returned by the server, even if they returned more to us.
--- fs/nfs/nfs42xdr.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
fs/nfs/nfs42xdr.c +30 -2
@@ -47,7 +47,7 @@#define decode_deallocate_maxsz (op_decode_hdr_maxsz)#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + 3) -#define NFS42_READ_PLUS_SEGMENT_SIZE (1 /* data_content4 */ + \ +#define NFS42_READ_PLUS_SEGMENT_SIZE (2 /* data_content4 */ + \ 2 /* data_info4.di_offset */ + \ 2 /* data_info4.di_length */)#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \
@@ -771,6 +771,31 @@ static uint32_t decode_read_plus_data(struct xdr_stream *xdr, struct nfs_pgio_rereturn count; } +static uint32_t decode_read_plus_hole (struct xdr_stream *xdr,struct nfs_pgio_res *res, +uint32_t *eof) +{ +__be32 *p; +uint64_t offset, length; +size_t recvd; + + p =xdr_inline_decode (xdr,8 +8 ); +if (unlikely (!p)) +return -EIO ; + + p =xdr_decode_hyper (p, &offset); + p =xdr_decode_hyper (p, &length); +if (length ==0 ) +return 0 ; + + recvd =xdr_expand_hole (xdr,0 , length); +if (recvd < length) { + *eof =0 ; + length = recvd; + } + +return length; +} +static int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) {__be32 *p;
@@ -798,7 +823,10 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)if (type ==NFS4_CONTENT_DATA ) count =decode_read_plus_data (xdr, res, &eof);else -return -EINVAL ; + count =decode_read_plus_hole (xdr, res, &eof); + +if (segments >1 ) + eof =0 ; res->eof = eof; res->count = count;
Signature · 1 line
2.25.0
PATCH v2 6/6 NFS: Decode multiple READ_PLUS segments
In reply to [email protected]From: Anna Schumaker <[email protected]>
We now have everything we need to read holes and shift data to where it's supposed to be. I switch over to using xdr_align_data() to put data segments in the proper place.
--- fs/nfs/nfs42xdr.c | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-)
fs/nfs/nfs42xdr.c +23 -20
@@ -53,7 +53,7 @@#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ 1 /* rpr_eof */ + \ 1 /* rpr_contents count */ + \ - NFS42_READ_PLUS_SEGMENT_SIZE) + (2 * NFS42_READ_PLUS_SEGMENT_SIZE)) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \
@@ -745,7 +745,7 @@ static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *re }static uint32_t decode_read_plus_data (struct xdr_stream *xdr,struct nfs_pgio_res *res, -uint32_t *eof) +uint32_t *eof,uint64_t total) {__be32 *p;uint32_t count, recvd;
@@ -760,7 +760,7 @@ static uint32_t decode_read_plus_data(struct xdr_stream *xdr, struct nfs_pgio_reif (count ==0 )return 0 ; - recvd =xdr_read_pages (xdr, count); + recvd =xdr_align_data (xdr, total, count);if (count > recvd) {dprintk ("NFS: server cheating in read reply: " "count %u > recvd %u\n" , count, recvd);
@@ -772,7 +772,7 @@ static uint32_t decode_read_plus_data(struct xdr_stream *xdr, struct nfs_pgio_re }static uint32_t decode_read_plus_hole (struct xdr_stream *xdr,struct nfs_pgio_res *res, -uint32_t *eof) +uint32_t *eof,uint64_t total) {__be32 *p;uint64_t offset, length;
@@ -787,7 +787,7 @@ static uint32_t decode_read_plus_hole(struct xdr_stream *xdr, struct nfs_pgio_reif (length ==0 )return 0 ; - recvd =xdr_expand_hole (xdr,0 , length); + recvd =xdr_expand_hole (xdr, total, length);if (recvd < length) { *eof =0 ; length = recvd;
@@ -799,8 +799,9 @@ static uint32_t decode_read_plus_hole(struct xdr_stream *xdr, struct nfs_pgio_restatic int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) {__be32 *p; -uint32_t count, eof, segments, type; -int status; +uint32_t eof, segments, type, total; +int32_t count; +int status, i; status =decode_op_hdr (xdr,OP_READ_PLUS );if (status)
@@ -810,26 +811,28 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res)if (unlikely (!p))return -EIO ; + total =0 ; eof =be32_to_cpup (p++); segments =be32_to_cpup (p++); -if (segments ==0 ) -return 0 ; - p =xdr_inline_decode (xdr,4 ); -if (unlikely (!p)) -return -EIO ; +for (i =0 ; i < segments; i++) { + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) +return -EIO ; - type =be32_to_cpup (p++); -if (type ==NFS4_CONTENT_DATA ) - count =decode_read_plus_data (xdr, res, &eof); -else - count =decode_read_plus_hole (xdr, res, &eof); + type =be32_to_cpup (p); +if (type ==NFS4_CONTENT_DATA ) + count =decode_read_plus_data (xdr, res, &eof, total); +else + count =decode_read_plus_hole (xdr, res, &eof, total); -if (segments >1 ) - eof =0 ; +if (count <0 ) +return count; + total += count; + } res->eof = eof; - res->count = count; + res->count = total;return 0 ; }
Signature · 1 line
2.25.0
Hi Anna-
Quoted 9 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
These patches add client support for the
READ_PLUSoperation, which breaks read requests into several "data" and "hole" segments when replying to the client. I also add a "noreadplus" mount option to allow users to disable the new operation if it becomes a problem, similar to the "nordirplus" mount option that we already have.
Hrm, I went looking for the patch that adds "noreadplus", but I don't see it in this series?
Wondering if, to start off, the default should be "noreadplus" until our feet are under us. Just a thought.
Quoted 83 lines
Here are the results of some performance tests I ran on Netapp lab machines. I tested by reading various 2G files from a few different undelying filesystems and across several NFS versions. I used the
vmtouchutility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h hase 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.ext4 | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 22.909 : 18.253 | 22.934 : 18.252 | 22.902 : 18.253 | 23.485 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.708 : 0.709 | mixed-4d | 28.261 : 18.253 | 29.616 : 18.252 | 28.341 : 18.252 | 24.508 : 9.150 | mixed-8d | 27.956 : 18.253 | 28.404 : 18.252 | 28.320 : 18.252 | 23.967 : 9.140 | mixed-16d | 28.172 : 18.253 | 27.946 : 18.252 | 27.627 : 18.252 | 23.043 : 9.134 | mixed-32d | 25.350 : 18.253 | 24.406 : 18.252 | 24.384 : 18.253 | 20.698 : 9.132 | mixed-4h | 28.913 : 18.253 | 28.564 : 18.252 | 27.996 : 18.252 | 21.837 : 9.150 | mixed-8h | 28.625 : 18.253 | 27.833 : 18.252 | 27.798 : 18.253 | 21.710 : 9.140 | mixed-16h | 27.975 : 18.253 | 27.662 : 18.252 | 27.795 : 18.253 | 20.585 : 9.134 | mixed-32h | 25.958 : 18.253 | 25.491 : 18.252 | 24.856 : 18.252 | 21.018 : 9.132 |xfs | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 22.041 : 18.253 | 22.618 : 18.252 | 23.067 : 18.253 | 23.496 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.723 : 0.708 | mixed-4d | 29.417 : 18.253 | 28.503 : 18.252 | 28.671 : 18.253 | 24.957 : 9.150 | mixed-8d | 29.080 : 18.253 | 29.401 : 18.252 | 29.251 : 18.252 | 24.625 : 9.140 | mixed-16d | 27.638 : 18.253 | 28.606 : 18.252 | 27.871 : 18.253 | 25.511 : 9.135 | mixed-32d | 24.967 : 18.253 | 25.239 : 18.252 | 25.434 : 18.252 | 21.728 : 9.132 | mixed-4h | 34.816 : 18.253 | 36.243 : 18.252 | 35.837 : 18.252 | 32.332 : 9.150 | mixed-8h | 43.469 : 18.253 | 44.009 : 18.252 | 43.810 : 18.253 | 37.962 : 9.140 | mixed-16h | 29.280 : 18.253 | 28.563 : 18.252 | 28.241 : 18.252 | 22.116 : 9.134 | mixed-32h | 29.428 : 18.253 | 29.378 : 18.252 | 28.808 : 18.253 | 27.378 : 9.134 |btrfs | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 25.547 : 18.253 | 25.053 : 18.252 | 24.209 : 18.253 | 32.121 : 18.253 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.252 | 0.702 : 0.724 | mixed-4d | 19.016 : 18.253 | 18.822 : 18.252 | 18.955 : 18.253 | 18.697 : 9.150 | mixed-8d | 19.186 : 18.253 | 19.444 : 18.252 | 18.841 : 18.253 | 18.452 : 9.140 | mixed-16d | 18.480 : 18.253 | 19.010 : 18.252 | 19.167 : 18.252 | 16.000 : 9.134 | mixed-32d | 18.635 : 18.253 | 18.565 : 18.252 | 18.550 : 18.252 | 15.930 : 9.132 | mixed-4h | 19.079 : 18.253 | 18.990 : 18.252 | 19.157 : 18.253 | 27.834 : 9.150 | mixed-8h | 18.613 : 18.253 | 19.234 : 18.252 | 18.616 : 18.253 | 20.177 : 9.140 | mixed-16h | 18.590 : 18.253 | 19.221 : 18.252 | 19.654 : 18.253 | 17.273 : 9.135 | mixed-32h | 18.768 : 18.253 | 19.122 : 18.252 | 18.535 : 18.252 | 15.791 : 9.132 |ext3 | v3 | v4.0 | v4.1 | v4.2 | ----------|-----------------|-----------------|-----------------|-----------------| data | 34.292 : 18.253 | 33.810 : 18.252 | 33.450 : 18.253 | 33.390 : 18.254 | hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.718 : 0.728 | mixed-4d | 46.818 : 18.253 | 47.140 : 18.252 | 48.385 : 18.253 | 42.887 : 9.150 | mixed-8d | 58.554 : 18.253 | 59.277 : 18.252 | 59.673 : 18.253 | 56.760 : 9.140 | mixed-16d | 44.631 : 18.253 | 44.291 : 18.252 | 44.729 : 18.253 | 40.237 : 9.135 | mixed-32d | 39.110 : 18.253 | 38.735 : 18.252 | 38.902 : 18.252 | 35.270 : 9.132 | mixed-4h | 56.396 : 18.253 | 56.387 : 18.252 | 56.573 : 18.253 | 67.661 : 9.150 | mixed-8h | 58.483 : 18.253 | 58.484 : 18.252 | 59.099 : 18.253 | 77.958 : 9.140 | mixed-16h | 42.511 : 18.253 | 42.338 : 18.252 | 42.356 : 18.252 | 51.805 : 9.135 | mixed-32h | 38.419 : 18.253 | 38.504 : 18.252 | 38.643 : 18.252 | 40.411 : 9.132 |Changes since v1:
- Rebase to 5.6-rc1
- Drop the mount option patch for now
- Fix fallback to READ when the server doesn't supportREAD_PLUSAny questions?
AnnaAnna Schumaker (6):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
SUNRPC: Add the ability to shift data to a specific offset
NFS: AddREAD_PLUSdata segment support
NFS: AddREAD_PLUShole segment decoding
NFS: Decode multipleREAD_PLUSsegments
Quoted 13 lines
fs/nfs/nfs42xdr.c | 169 +++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 ++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 244 ++++++++++++++++++++++++++++++++++++- 8 files changed, 457 insertions(+), 7 deletions(-)Signature · 1 line
2.25.0
--
Chuck Lever
[email protected]
Quoted 6 lines · Chuck Lever
Hi Anna-
Quoted 4 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
Quoted 8 lines
Quoted 5 lines
These patches add client support for the
READ_PLUSoperation, which breaks read requests into several "data" and "hole" segments when replying to the client. I also add a "noreadplus" mount option to allow users to disable the new operation if it becomes a problem, similar to the "nordirplus" mount option that we already have.Hrm, I went looking for the patch that adds "noreadplus", but I don't see it in this series?
You suggested dropping that patch in the v1 posting and waiting to see if anybody asks for it.
Quoted 3 lines
Wondering if, to start off, the default should be "noreadplus" until our feet are under us. Just a thought.
I could re-add the patch with this as the default if that's the way everybody wants to go.
Anna
Quoted 153 lines
Quoted 133 lines
Here are the results of some performance tests I ran on Netapp lab machines. I tested by reading various 2G files from a few different undelying filesystems and across several NFS versions. I used the
vmtouchutility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h hase 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.ext4 | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 22.909 : 18.253 | 22.934 : 18.252 | 22.902 : 18.253 | 23.485 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.708
: 0.709 |
mixed-4d | 28.261 : 18.253 | 29.616 : 18.252 | 28.341 : 18.252 | 24.508 : 9.150 |
mixed-8d | 27.956 : 18.253 | 28.404 : 18.252 | 28.320 : 18.252 | 23.967 : 9.140 |
mixed-16d | 28.172 : 18.253 | 27.946 : 18.252 | 27.627 : 18.252 | 23.043 : 9.134 |
mixed-32d | 25.350 : 18.253 | 24.406 : 18.252 | 24.384 : 18.253 | 20.698 : 9.132 |
mixed-4h | 28.913 : 18.253 | 28.564 : 18.252 | 27.996 : 18.252 | 21.837 : 9.150 |
mixed-8h | 28.625 : 18.253 | 27.833 : 18.252 | 27.798 : 18.253 | 21.710 : 9.140 |
mixed-16h | 27.975 : 18.253 | 27.662 : 18.252 | 27.795 : 18.253 | 20.585 : 9.134 |
mixed-32h | 25.958 : 18.253 | 25.491 : 18.252 | 24.856 : 18.252 | 21.018 : 9.132 |xfs | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 22.041 : 18.253 | 22.618 : 18.252 | 23.067 : 18.253 | 23.496 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.723
: 0.708 |
mixed-4d | 29.417 : 18.253 | 28.503 : 18.252 | 28.671 : 18.253 | 24.957 : 9.150 |
mixed-8d | 29.080 : 18.253 | 29.401 : 18.252 | 29.251 : 18.252 | 24.625 : 9.140 |
mixed-16d | 27.638 : 18.253 | 28.606 : 18.252 | 27.871 : 18.253 | 25.511 : 9.135 |
mixed-32d | 24.967 : 18.253 | 25.239 : 18.252 | 25.434 : 18.252 | 21.728 : 9.132 |
mixed-4h | 34.816 : 18.253 | 36.243 : 18.252 | 35.837 : 18.252 | 32.332 : 9.150 |
mixed-8h | 43.469 : 18.253 | 44.009 : 18.252 | 43.810 : 18.253 | 37.962 : 9.140 |
mixed-16h | 29.280 : 18.253 | 28.563 : 18.252 | 28.241 : 18.252 | 22.116 : 9.134 |
mixed-32h | 29.428 : 18.253 | 29.378 : 18.252 | 28.808 : 18.253 | 27.378 : 9.134 |btrfs | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 25.547 : 18.253 | 25.053 : 18.252 | 24.209 : 18.253 | 32.121 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.252 | 0.702
: 0.724 |
mixed-4d | 19.016 : 18.253 | 18.822 : 18.252 | 18.955 : 18.253 | 18.697 : 9.150 |
mixed-8d | 19.186 : 18.253 | 19.444 : 18.252 | 18.841 : 18.253 | 18.452 : 9.140 |
mixed-16d | 18.480 : 18.253 | 19.010 : 18.252 | 19.167 : 18.252 | 16.000 : 9.134 |
mixed-32d | 18.635 : 18.253 | 18.565 : 18.252 | 18.550 : 18.252 | 15.930 : 9.132 |
mixed-4h | 19.079 : 18.253 | 18.990 : 18.252 | 19.157 : 18.253 | 27.834 : 9.150 |
mixed-8h | 18.613 : 18.253 | 19.234 : 18.252 | 18.616 : 18.253 | 20.177 : 9.140 |
mixed-16h | 18.590 : 18.253 | 19.221 : 18.252 | 19.654 : 18.253 | 17.273 : 9.135 |
mixed-32h | 18.768 : 18.253 | 19.122 : 18.252 | 18.535 : 18.252 | 15.791 : 9.132 |ext3 | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 34.292 : 18.253 | 33.810 : 18.252 | 33.450 : 18.253 | 33.390 :
18.254 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.718
: 0.728 |
mixed-4d | 46.818 : 18.253 | 47.140 : 18.252 | 48.385 : 18.253 | 42.887 : 9.150 |
mixed-8d | 58.554 : 18.253 | 59.277 : 18.252 | 59.673 : 18.253 | 56.760 : 9.140 |
mixed-16d | 44.631 : 18.253 | 44.291 : 18.252 | 44.729 : 18.253 | 40.237 : 9.135 |
mixed-32d | 39.110 : 18.253 | 38.735 : 18.252 | 38.902 : 18.252 | 35.270 : 9.132 |
mixed-4h | 56.396 : 18.253 | 56.387 : 18.252 | 56.573 : 18.253 | 67.661 : 9.150 |
mixed-8h | 58.483 : 18.253 | 58.484 : 18.252 | 59.099 : 18.253 | 77.958 : 9.140 |
mixed-16h | 42.511 : 18.253 | 42.338 : 18.252 | 42.356 : 18.252 | 51.805 : 9.135 |
mixed-32h | 38.419 : 18.253 | 38.504 : 18.252 | 38.643 : 18.252 | 40.411 : 9.132 |Changes since v1:
- Rebase to 5.6-rc1
- Drop the mount option patch for now
- Fix fallback to READ when the server doesn't supportREAD_PLUSAny questions?
AnnaAnna Schumaker (6):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
SUNRPC: Add the ability to shift data to a specific offset
NFS: AddREAD_PLUSdata segment support
NFS: AddREAD_PLUShole segment decoding
NFS: Decode multipleREAD_PLUSsegmentsQuoted 13 lines
fs/nfs/nfs42xdr.c | 169 +++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 ++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 244 ++++++++++++++++++++++++++++++++++++- 8 files changed, 457 insertions(+), 7 deletions(-)Signature · 1 line
2.25.0--
Chuck Lever
[email protected]
Quoted 17 lines · Anna Schumaker
Quoted 12 lines · Chuck Lever
Hi Anna-
Quoted 9 lines · On Feb 14, 2020, at 4:12 PM, [email protected] wrote:
From: Anna Schumaker <[email protected]>
These patches add client support for the
READ_PLUSoperation, which breaks read requests into several "data" and "hole" segments when replying to the client. I also add a "noreadplus" mount option to allow users to disable the new operation if it becomes a problem, similar to the "nordirplus" mount option that we already have.
Quoted 5 lines
Quoted 2 lines
Hrm, I went looking for the patch that adds "noreadplus", but I don't see it in this series?
You suggested dropping that patch in the v1 posting and waiting to see if anybody asks for it.
Yes, recalling that now.
I requested that because I don't like to add administrative interfaces if there isn't a clear need. I think we might have one now!
Quoted 5 lines
Quoted 2 lines
Wondering if, to start off, the default should be "noreadplus" until our feet are under us. Just a thought.
I could re-add the patch with this as the default if that's the way everybody wants to go.
Yep, I'm interested in other opinions too.
Quoted 152 lines
Anna
Quoted 150 lines
Quoted 133 lines
Here are the results of some performance tests I ran on Netapp lab machines. I tested by reading various 2G files from a few different undelying filesystems and across several NFS versions. I used the
vmtouchutility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h hase 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server.ext4 | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 22.909 : 18.253 | 22.934 : 18.252 | 22.902 : 18.253 | 23.485 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.708
: 0.709 |
mixed-4d | 28.261 : 18.253 | 29.616 : 18.252 | 28.341 : 18.252 | 24.508 : 9.150 |
mixed-8d | 27.956 : 18.253 | 28.404 : 18.252 | 28.320 : 18.252 | 23.967 : 9.140 |
mixed-16d | 28.172 : 18.253 | 27.946 : 18.252 | 27.627 : 18.252 | 23.043 : 9.134 |
mixed-32d | 25.350 : 18.253 | 24.406 : 18.252 | 24.384 : 18.253 | 20.698 : 9.132 |
mixed-4h | 28.913 : 18.253 | 28.564 : 18.252 | 27.996 : 18.252 | 21.837 : 9.150 |
mixed-8h | 28.625 : 18.253 | 27.833 : 18.252 | 27.798 : 18.253 | 21.710 : 9.140 |
mixed-16h | 27.975 : 18.253 | 27.662 : 18.252 | 27.795 : 18.253 | 20.585 : 9.134 |
mixed-32h | 25.958 : 18.253 | 25.491 : 18.252 | 24.856 : 18.252 | 21.018 : 9.132 |xfs | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 22.041 : 18.253 | 22.618 : 18.252 | 23.067 : 18.253 | 23.496 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.723
: 0.708 |
mixed-4d | 29.417 : 18.253 | 28.503 : 18.252 | 28.671 : 18.253 | 24.957 : 9.150 |
mixed-8d | 29.080 : 18.253 | 29.401 : 18.252 | 29.251 : 18.252 | 24.625 : 9.140 |
mixed-16d | 27.638 : 18.253 | 28.606 : 18.252 | 27.871 : 18.253 | 25.511 : 9.135 |
mixed-32d | 24.967 : 18.253 | 25.239 : 18.252 | 25.434 : 18.252 | 21.728 : 9.132 |
mixed-4h | 34.816 : 18.253 | 36.243 : 18.252 | 35.837 : 18.252 | 32.332 : 9.150 |
mixed-8h | 43.469 : 18.253 | 44.009 : 18.252 | 43.810 : 18.253 | 37.962 : 9.140 |
mixed-16h | 29.280 : 18.253 | 28.563 : 18.252 | 28.241 : 18.252 | 22.116 : 9.134 |
mixed-32h | 29.428 : 18.253 | 29.378 : 18.252 | 28.808 : 18.253 | 27.378 : 9.134 |btrfs | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 25.547 : 18.253 | 25.053 : 18.252 | 24.209 : 18.253 | 32.121 :
18.253 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.252 | 0.702
: 0.724 |
mixed-4d | 19.016 : 18.253 | 18.822 : 18.252 | 18.955 : 18.253 | 18.697 : 9.150 |
mixed-8d | 19.186 : 18.253 | 19.444 : 18.252 | 18.841 : 18.253 | 18.452 : 9.140 |
mixed-16d | 18.480 : 18.253 | 19.010 : 18.252 | 19.167 : 18.252 | 16.000 : 9.134 |
mixed-32d | 18.635 : 18.253 | 18.565 : 18.252 | 18.550 : 18.252 | 15.930 : 9.132 |
mixed-4h | 19.079 : 18.253 | 18.990 : 18.252 | 19.157 : 18.253 | 27.834 : 9.150 |
mixed-8h | 18.613 : 18.253 | 19.234 : 18.252 | 18.616 : 18.253 | 20.177 : 9.140 |
mixed-16h | 18.590 : 18.253 | 19.221 : 18.252 | 19.654 : 18.253 | 17.273 : 9.135 |
mixed-32h | 18.768 : 18.253 | 19.122 : 18.252 | 18.535 : 18.252 | 15.791 : 9.132 |ext3 | v3 | v4.0 | v4.1 | v4.2
|----------|-----------------|-----------------|-----------------|----------- ------|
data | 34.292 : 18.253 | 33.810 : 18.252 | 33.450 : 18.253 | 33.390 :
18.254 |
hole | 18.256 : 18.253 | 18.255 : 18.252 | 18.256 : 18.253 | 0.718
: 0.728 |
mixed-4d | 46.818 : 18.253 | 47.140 : 18.252 | 48.385 : 18.253 | 42.887 : 9.150 |
mixed-8d | 58.554 : 18.253 | 59.277 : 18.252 | 59.673 : 18.253 | 56.760 : 9.140 |
mixed-16d | 44.631 : 18.253 | 44.291 : 18.252 | 44.729 : 18.253 | 40.237 : 9.135 |
mixed-32d | 39.110 : 18.253 | 38.735 : 18.252 | 38.902 : 18.252 | 35.270 : 9.132 |
mixed-4h | 56.396 : 18.253 | 56.387 : 18.252 | 56.573 : 18.253 | 67.661 : 9.150 |
mixed-8h | 58.483 : 18.253 | 58.484 : 18.252 | 59.099 : 18.253 | 77.958 : 9.140 |
mixed-16h | 42.511 : 18.253 | 42.338 : 18.252 | 42.356 : 18.252 | 51.805 : 9.135 |
mixed-32h | 38.419 : 18.253 | 38.504 : 18.252 | 38.643 : 18.252 | 40.411 : 9.132 |Changes since v1:
- Rebase to 5.6-rc1
- Drop the mount option patch for now
- Fix fallback to READ when the server doesn't supportREAD_PLUSAny questions?
AnnaAnna Schumaker (6):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
SUNRPC: Add the ability to shift data to a specific offset
NFS: AddREAD_PLUSdata segment support
NFS: AddREAD_PLUShole segment decoding
NFS: Decode multipleREAD_PLUSsegmentsQuoted 13 lines
fs/nfs/nfs42xdr.c | 169 +++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 43 ++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 2 +- include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 244 ++++++++++++++++++++++++++++++++++++- 8 files changed, 457 insertions(+), 7 deletions(-)Signature · 1 line
2.25.0--
Chuck Lever
[email protected]
--
Chuck Lever
[email protected]