NFS: Add READ_PLUS support
Subsystems (3)
v3 cover NFS: Add READ_PLUS support
These patches add client support for the NFS v4.2 operation READ_PLUS. This operation is triggered by doing any kind of read on a NFS v4.2 mounted filesystem. `
I tested these patches using xfstests and compared the runtime between NFS v4.1, NFS v4.2 without READ_PLUS, and NFS v4.2 with READ_PLUS. Here are the results:
Test | v4.1 | no READ_PLUS | READ_PLUS
---------------------------------------------------
generic/013 | 135s | 143s | 123s generic/075 | 4s | 11s | 4s generic/091 | 9s | 16s | 15s generic/112 | 4s | 7s | 6s generic/127 | 62s | 117s | 114s generic/213 | [not run] | 1s | 1s generic/214 | [not run] | 0s | 0s generic/228 | [not run] | 1s | 2s generic/236 | 1s | 1s | 1s generic/263 | 4s | 6s | 8s generic/285 | 0s | 1s | 3s generic/315 | [not run] | 2s | 1s
---------------------------------------------------
Total | 3:47.47 | 5:11.85 | 4:43.77
Using the READ_PLUS operation does have an impact on reading sparse files over the network. There is still a big difference between v4.1 and v4.2 runtimes, but this looks to be a result of fallocate() tests that only run over v4.2.
Changes since v2:
- Merge together patches adding DATA and HOLE segment support.
- Simplify hole zeroing code by using the zero_user_segment() function.
These patches and the corresponding server changes are available in the [read_plus] branch of
git://git.linux-nfs.org/projects/anna/linux-nfs.git
Questions? Comments? Thoughts?
Anna
Anna Schumaker (5):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
NFS: Add basic READ_PLUS support
SUNRPC: Add the ability to shift data to a specific offset
NFS: Add support for decoding multiple segments
fs/nfs/nfs42xdr.c | 162 ++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 30 +++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 1 + include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 240 ++++++++++++++++++++++++++++++++++++++++++++- 8 files changed, 434 insertions(+), 5 deletions(-)
Signature · 1 line
2.3.3
PATCH v3 1/5 SUNRPC: Split out a function for setting current page
I'm going to need this bit of code in a few places for READ_PLUS decoding, so let's make it a helper function.
--- net/sunrpc/xdr.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
net/sunrpc/xdr.c +7 -2
@@ -759,6 +759,12 @@ static int xdr_set_page_base(struct xdr_stream *xdr,return 0 ; } +static void xdr_set_page (struct xdr_stream *xdr,unsigned int base) +{ +if (xdr_set_page_base (xdr, base,PAGE_SIZE ) <0 ) +xdr_set_iov (xdr, xdr->buf->tail, xdr->buf->len); +} +static void xdr_set_next_page (struct xdr_stream *xdr) {unsigned int newbase;
@@ -766,8 +772,7 @@ static void xdr_set_next_page(struct xdr_stream *xdr) newbase = (1 + xdr->page_ptr - xdr->buf->pages) <<PAGE_SHIFT ; newbase -= xdr->buf->page_base; -if (xdr_set_page_base (xdr, newbase,PAGE_SIZE ) <0 ) -xdr_set_iov (xdr, xdr->buf->tail, xdr->buf->len); +xdr_set_page (xdr, newbase); }static bool xdr_set_next_buffer (struct xdr_stream *xdr)
Signature · 1 line
2.3.3
PATCH v3 2/5 SUNRPC: Add the ability to expand holes in data pages
In reply to Anna SchumakerThis patch adds the ability to "read a hole" into a set of XDR data pages by taking the following steps:
1) Shift all data after the current xdr->p to the right, possibly into the tail,
2) Zero the specified range, and
3) Update xdr->p to point beyond the hole.
--- include/linux/sunrpc/xdr.h | 1 + net/sunrpc/xdr.c | 99 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+)
include/linux/sunrpc/xdr.h +1 -0
@@ -229,6 +229,7 @@ extern __be32 *xdr_inline_decode(struct xdr_stream *xdr, size_t nbytes);extern unsigned int xdr_read_pages (struct xdr_stream *xdr,unsigned int len);extern void xdr_enter_page (struct xdr_stream *xdr,unsigned int len);extern int xdr_process_buf (struct xdr_buf *buf,unsigned int offset,unsigned int len,int (*actor)(struct scatterlist *,void *),void *data); +extern size_t xdr_expand_hole (struct xdr_stream *,size_t ,size_t );#endif /* __KERNEL__ */
net/sunrpc/xdr.c +99 -0
123 lines not on this page · raw message
Signature · 1 line
2.3.3
PATCH v3 3/5 NFS: Add basic READ_PLUS support
In reply to Anna SchumakerThis patch adds support for decoding a single NFS4_CONTENT_DATA or NFS4_CONTENT_HOLE segment returned by the server. This gives a simple implementation that does not need to spent a lot of time shifting data arount.
--- fs/nfs/nfs42xdr.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 30 ++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 1 + include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 190 insertions(+), 3 deletions(-)
fs/nfs/nfs42xdr.c +158 -0
@@ -13,6 +13,14 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1/* rpr_eof */ + \ +1 /* rpr_contents count */ + \ +1 /* data_content4 */ + \ +2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \
@@ -39,6 +47,12 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_putfh_maxsz + \ encode_seek_maxsz)
@@ -71,6 +85,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,struct nfs42_seek_args *args,struct compound_hdr *hdr)
@@ -120,6 +144,28 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, }/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +struct nfs_pgio_args *args) +{ +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr); +encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +xdr_inline_pages (&req->rq_rcv_buf, hdr.replen <<2 , + args->pages, args->pgbase, args->count); + req->rq_rcv_buf.flags |=XDRBUF_READ ; +encode_nops (&hdr); +} + +/* * Encode SEEK request */ static void nfs4_xdr_enc_seek(struct rpc_rqst *req,
@@ -147,6 +193,92 @@ static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *rereturn decode_op_hdr (xdr,OP_DEALLOCATE ); } +static int decode_read_plus_data (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) + goto out_overflow; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); + + recvd =xdr_read_pages (xdr, count); +if (recvd < count) + res->eof =0 ; + + res->count = recvd; +return 0 ; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} + +static int decode_read_plus_hole (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint64_t offset, length; +size_t recvd; + + p =xdr_inline_decode (xdr,8 +8 ); +if (unlikely (!p)) + goto out_overflow; + + p =xdr_decode_hyper (p, &offset); + p =xdr_decode_hyper (p, &length); + + recvd =xdr_expand_hole (xdr,0 , length); +if (recvd < length) + res->eof =0 ; + + res->count = recvd; +return 0 ; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} + +static int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +int status, type; +uint32_t segments; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) + goto out_overflow; + + res->count =0 ; + res->eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) + goto out_overflow; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + status =decode_read_plus_data (xdr, res); +else + status =decode_read_plus_hole (xdr, res); + +if (segments >1 ) + res->eof =0 ; +return status; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} +static int decode_seek (struct xdr_stream *xdr,struct nfs42_seek_res *res) {int status;
@@ -224,6 +356,32 @@ out: }/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +struct nfs_pgio_res *res) +{ +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} + +/* * Decode SEEK request */ static int nfs4_xdr_dec_seek(struct rpc_rqst *rqstp,
fs/nfs/nfs4proc.c +28 -2
@@ -68,6 +68,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_POLL_RETRY_MIN (HZ/10)
@@ -4187,9 +4191,15 @@ static bool nfs4_read_stateid_changed(struct rpc_task *task,static int nfs4_read_done (struct rpc_task *task,struct nfs_pgio_header *hdr) { - +struct nfs_server *server =NFS_SERVER (hdr->inode);dprintk ("--> %s\n" , __func__); +if ((server->caps &NFS_CAP_READ_PLUS ) && (task->tk_status == -ENOTSUPP )) { + server->caps &= ~NFS_CAP_READ_PLUS ; +if (rpc_restart_call_prepare (task)) + task->tk_status =0 ; +return -EAGAIN ; + }if (!nfs4_sequence_done (task, &hdr->res.seq_res))return -EAGAIN ;if (nfs4_read_stateid_changed (task, &hdr->args))
@@ -4198,12 +4208,27 @@ static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)nfs4_read_done_cb (task, hdr); } +#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ]; +else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#else +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup (struct nfs_pgio_header *hdr,struct rpc_message *msg) { hdr->timestamp = jiffies; hdr->pgio_done_cb = nfs4_read_done_cb; - msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence (&hdr->args.seq_args, &hdr->res.seq_res,0 ); }
@@ -8556,6 +8581,7 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { | NFS_CAP_ATOMIC_OPEN_V1 |NFS_CAP_ALLOCATE |NFS_CAP_DEALLOCATE + |NFS_CAP_READ_PLUS | NFS_CAP_SEEK, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,
fs/nfs/nfs4xdr.c +1 -0
@@ -7421,6 +7421,7 @@ struct rpc_procinfo nfs4_procedures[] = {PROC (SEEK , enc_seek, dec_seek),PROC (ALLOCATE , enc_allocate, dec_allocate),PROC (DEALLOCATE , enc_deallocate, dec_deallocate), + PROC(READ_PLUS, enc_read_plus, dec_read_plus), #endif /* CONFIG_NFS_V4_2 */ };
include/linux/nfs4.h +1 -0
@@ -493,6 +493,7 @@ enum {NFSPROC4_CLNT_SEEK ,NFSPROC4_CLNT_ALLOCATE ,NFSPROC4_CLNT_DEALLOCATE , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */
include/linux/nfs_fs_sb.h +1 -0
@@ -237,5 +237,6 @@ struct nfs_server {#define NFS_CAP_SEEK (1U << 19)#define NFS_CAP_ALLOCATE (1U << 20)#define NFS_CAP_DEALLOCATE (1U << 21) +#define NFS_CAP_READ_PLUS (1U << 22)#endif
include/linux/nfs_xdr.h +1 -1
@@ -520,7 +520,7 @@ struct nfs_pgio_args {struct nfs_pgio_res {struct nfs4_sequence_res seq_res;struct nfs_fattr * fattr; -__u32 count; +__u64 count;__u32 op_status;int eof;/* used by read */ struct nfs_writeverf * verf;/* used by write */
Signature · 1 line
2.3.3
Re: [PATCH v3 3/5] NFS: Add basic READ_PLUS support
Quoted 4 lines · Anna Schumaker
This patch adds support for decoding a single
NFS4_CONTENT_DATAorNFS4_CONTENT_HOLEsegment returned by the server. This gives a simple implementation that does not need to spent a lot of time shifting data arount.
So to make sure I understand the code correctly: if the server returns (say) 10 segments, the client won't error out--it will just use the information from the first segment, ignore the remaining 9, and send more reads if necessary to get the rest.
So that looks simple and correct. The worst case for performance would probably be reading a file that alternates small data and hole segments.
--b.
Quoted 339 lines
Signed-off-by Anna Schumaker <[email protected]>--- fs/nfs/nfs42xdr.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 30 ++++++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 1 + include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- 6 files changed, 190 insertions(+), 3 deletions(-)fs/nfs/nfs42xdr.c +158
-0@@ -13,6 +13,14 @@#define encode_deallocate_maxsz (op_encode_hdr_maxsz + \ encode_fallocate_maxsz)#define decode_deallocate_maxsz (op_decode_hdr_maxsz) +#define encode_read_plus_maxsz (op_encode_hdr_maxsz + \ + encode_stateid_maxsz + 3) +#define decode_read_plus_maxsz (op_decode_hdr_maxsz + \ + 1/* rpr_eof */ + \ +1 /* rpr_contents count */ + \ +1 /* data_content4 */ + \ +2 /* data_info4.di_offset */ + \ + 2 /* data_info4.di_length */) #define encode_seek_maxsz (op_encode_hdr_maxsz + \ encode_stateid_maxsz + \ 2 /* offset */ + \@@ -39,6 +47,12 @@ decode_putfh_maxsz + \ decode_deallocate_maxsz + \ decode_getattr_maxsz) +#define NFS4_enc_read_plus_sz (compound_encode_hdr_maxsz + \ + encode_putfh_maxsz + \ + encode_read_plus_maxsz) +#define NFS4_dec_read_plus_sz (compound_decode_hdr_maxsz + \ + decode_putfh_maxsz + \ + decode_read_plus_maxsz)#define NFS4_enc_seek_sz (compound_encode_hdr_maxsz + \ encode_putfh_maxsz + \ encode_seek_maxsz)@@ -71,6 +85,16 @@ static void encode_deallocate(struct xdr_stream *xdr,encode_fallocate (xdr, args); } +static void encode_read_plus (struct xdr_stream *xdr, +struct nfs_pgio_args *args, +struct compound_hdr *hdr) +{ +encode_op_hdr (xdr,OP_READ_PLUS , decode_read_plus_maxsz, hdr); +encode_nfs4_stateid (xdr, &args->stateid); +encode_uint64 (xdr, args->offset); +encode_uint32 (xdr, args->count); +} +static void encode_seek (struct xdr_stream *xdr,struct nfs42_seek_args *args,struct compound_hdr *hdr)@@ -120,6 +144,28 @@ static void nfs4_xdr_enc_deallocate(struct rpc_rqst *req, }/* +* Encode READ_PLUS request +*/ +static void nfs4_xdr_enc_read_plus (struct rpc_rqst *req, +struct xdr_stream *xdr, +struct nfs_pgio_args *args) +{ +struct compound_hdr hdr = { + .minorversion =nfs4_xdr_minorversion (&args->seq_args), + }; + +encode_compound_hdr (xdr, req, &hdr); +encode_sequence (xdr, &args->seq_args, &hdr); +encode_putfh (xdr, args->fh, &hdr); +encode_read_plus (xdr, args, &hdr); + +xdr_inline_pages (&req->rq_rcv_buf, hdr.replen <<2 , + args->pages, args->pgbase, args->count); + req->rq_rcv_buf.flags |=XDRBUF_READ ; +encode_nops (&hdr); +} + +/* * Encode SEEK request */ static void nfs4_xdr_enc_seek(struct rpc_rqst *req,@@ -147,6 +193,92 @@ static int decode_deallocate(struct xdr_stream *xdr, struct nfs42_falloc_res *rereturn decode_op_hdr (xdr,OP_DEALLOCATE ); } +static int decode_read_plus_data (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint32_t count, recvd; +uint64_t offset; + + p =xdr_inline_decode (xdr,8 +4 ); +if (unlikely (!p)) + goto out_overflow; + + p =xdr_decode_hyper (p, &offset); + count =be32_to_cpup (p); + + recvd =xdr_read_pages (xdr, count); +if (recvd < count) + res->eof =0 ; + + res->count = recvd; +return 0 ; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} + +static int decode_read_plus_hole (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +uint64_t offset, length; +size_t recvd; + + p =xdr_inline_decode (xdr,8 +8 ); +if (unlikely (!p)) + goto out_overflow; + + p =xdr_decode_hyper (p, &offset); + p =xdr_decode_hyper (p, &length); + + recvd =xdr_expand_hole (xdr,0 , length); +if (recvd < length) + res->eof =0 ; + + res->count = recvd; +return 0 ; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} + +static int decode_read_plus (struct xdr_stream *xdr,struct nfs_pgio_res *res) +{ +__be32 *p; +int status, type; +uint32_t segments; + + status =decode_op_hdr (xdr,OP_READ_PLUS ); +if (status) +return status; + + p =xdr_inline_decode (xdr,4 +4 ); +if (unlikely (!p)) + goto out_overflow; + + res->count =0 ; + res->eof =be32_to_cpup (p++); + segments =be32_to_cpup (p++); +if (segments ==0 ) +return 0 ; + + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) + goto out_overflow; + + type =be32_to_cpup (p++); +if (type ==NFS4_CONTENT_DATA ) + status =decode_read_plus_data (xdr, res); +else + status =decode_read_plus_hole (xdr, res); + +if (segments >1 ) + res->eof =0 ; +return status; +out_overflow: +print_overflow_msg (__func__, xdr); +return -EIO ; +} +static int decode_seek (struct xdr_stream *xdr,struct nfs42_seek_res *res) {int status;@@ -224,6 +356,32 @@ out: }/* +* Decode READ_PLUS request +*/ +static int nfs4_xdr_dec_read_plus (struct rpc_rqst *rqstp, +struct xdr_stream *xdr, +struct nfs_pgio_res *res) +{ +struct compound_hdr hdr; +int status; + + status =decode_compound_hdr (xdr, &hdr); +if (status) + goto out; + status =decode_sequence (xdr, &res->seq_res, rqstp); +if (status) + goto out; + status =decode_putfh (xdr); +if (status) + goto out; + status =decode_read_plus (xdr, res); +if (!status) + status = res->count; +out: +return status; +} + +/* * Decode SEEK request */ static int nfs4_xdr_dec_seek(struct rpc_rqst *rqstp,fs/nfs/nfs4proc.c +28
-2@@ -68,6 +68,10 @@#include "nfs4trace.h" +#ifdef CONFIG_NFS_V4_2 +#include "nfs42.h" +#endif /* CONFIG_NFS_V4_2 */ +#define NFSDBG_FACILITY NFSDBG_PROC#define NFS4_POLL_RETRY_MIN (HZ/10)@@ -4187,9 +4191,15 @@ static bool nfs4_read_stateid_changed(struct rpc_task *task,static int nfs4_read_done (struct rpc_task *task,struct nfs_pgio_header *hdr) { - +struct nfs_server *server =NFS_SERVER (hdr->inode);dprintk ("--> %s\n" , __func__); +if ((server->caps &NFS_CAP_READ_PLUS ) && (task->tk_status == -ENOTSUPP )) { + server->caps &= ~NFS_CAP_READ_PLUS ; +if (rpc_restart_call_prepare (task)) + task->tk_status =0 ; +return -EAGAIN ; + }if (!nfs4_sequence_done (task, &hdr->res.seq_res))return -EAGAIN ;if (nfs4_read_stateid_changed (task, &hdr->args))@@ -4198,12 +4208,27 @@ static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)nfs4_read_done_cb (task, hdr); } +#ifdef CONFIG_NFS_V4_2 +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ +if (server->caps &NFS_CAP_READ_PLUS ) + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ_PLUS ]; +else + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#else +static void nfs42_read_plus_support (struct nfs_server *server,struct rpc_message *msg) +{ + msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +} +#endif /* CONFIG_NFS_V4_2 */ +static void nfs4_proc_read_setup (struct nfs_pgio_header *hdr,struct rpc_message *msg) { hdr->timestamp = jiffies; hdr->pgio_done_cb = nfs4_read_done_cb; - msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ ]; +nfs42_read_plus_support (NFS_SERVER (hdr->inode), msg);nfs4_init_sequence (&hdr->args.seq_args, &hdr->res.seq_res,0 ); }@@ -8556,6 +8581,7 @@ static const struct nfs4_minor_version_ops nfs_v4_2_minor_ops = { | NFS_CAP_ATOMIC_OPEN_V1 |NFS_CAP_ALLOCATE |NFS_CAP_DEALLOCATE + |NFS_CAP_READ_PLUS | NFS_CAP_SEEK, .init_client = nfs41_init_client, .shutdown_client = nfs41_shutdown_client,fs/nfs/nfs4xdr.c +1
-0@@ -7421,6 +7421,7 @@ struct rpc_procinfo nfs4_procedures[] = {PROC (SEEK , enc_seek, dec_seek),PROC (ALLOCATE , enc_allocate, dec_allocate),PROC (DEALLOCATE , enc_deallocate, dec_deallocate), + PROC(READ_PLUS, enc_read_plus, dec_read_plus), #endif /* CONFIG_NFS_V4_2 */ };include/linux/nfs4.h +1
-0@@ -493,6 +493,7 @@ enum {NFSPROC4_CLNT_SEEK ,NFSPROC4_CLNT_ALLOCATE ,NFSPROC4_CLNT_DEALLOCATE , + NFSPROC4_CLNT_READ_PLUS, };/* nfs41 types */ include/linux/nfs_fs_sb.h +1
-0@@ -237,5 +237,6 @@ struct nfs_server {#define NFS_CAP_SEEK (1U << 19)#define NFS_CAP_ALLOCATE (1U << 20)#define NFS_CAP_DEALLOCATE (1U << 21) +#define NFS_CAP_READ_PLUS (1U << 22)#endif include/linux/nfs_xdr.h +1
-1@@ -520,7 +520,7 @@ struct nfs_pgio_args {struct nfs_pgio_res {struct nfs4_sequence_res seq_res;struct nfs_fattr * fattr; -__u32 count; +__u64 count;__u32 op_status;int eof;/* used by read */ struct nfs_writeverf * verf;/* used by write */ Signature · 6 lines
2.3.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v3 4/5 SUNRPC: Add the ability to shift data to a specific offset
In reply to Anna SchumakerExpanding holes tends to put the data content a few bytes to the right of where we want it. This patch implements a left-shift operation to line everything up properly.
--- include/linux/sunrpc/xdr.h | 1 + net/sunrpc/xdr.c | 132 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 133 insertions(+)
include/linux/sunrpc/xdr.h +1 -0
@@ -230,6 +230,7 @@ extern unsigned int xdr_read_pages(struct xdr_stream *xdr, unsigned int len);extern void xdr_enter_page (struct xdr_stream *xdr,unsigned int len);extern int xdr_process_buf (struct xdr_buf *buf,unsigned int offset,unsigned int len,int (*actor)(struct scatterlist *,void *),void *data);extern size_t xdr_expand_hole (struct xdr_stream *,size_t ,size_t ); +extern uint64_t xdr_align_data (struct xdr_stream *,uint64_t ,uint64_t );#endif /* __KERNEL__ */
net/sunrpc/xdr.c +132 -0
150 lines not on this page · raw message
Signature · 1 line
2.3.3
PATCH v3 5/5 NFS: Add support for decoding multiple segments
In reply to Anna SchumakerWe now have everything we need to read holes and then shift data to where it's supposed to be.
--- fs/nfs/nfs42xdr.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)
fs/nfs/nfs42xdr.c +22 -18
@@ -206,11 +206,11 @@ static int decode_read_plus_data(struct xdr_stream *xdr, struct nfs_pgio_res *re p =xdr_decode_hyper (p, &offset); count =be32_to_cpup (p); - recvd =xdr_read_pages (xdr, count); + recvd =xdr_align_data (xdr, res->count, count);if (recvd < count) res->eof =0 ; - res->count = recvd; + res->count += recvd;return 0 ; out_overflow:print_overflow_msg (__func__, xdr);
@@ -230,11 +230,11 @@ static int decode_read_plus_hole(struct xdr_stream *xdr, struct nfs_pgio_res *re p =xdr_decode_hyper (p, &offset); p =xdr_decode_hyper (p, &length); - recvd =xdr_expand_hole (xdr,0 , length); + recvd =xdr_expand_hole (xdr, res->count, length);if (recvd < length) res->eof =0 ; - res->count = recvd; + res->count += recvd;return 0 ; out_overflow:print_overflow_msg (__func__, xdr);
@@ -245,7 +245,7 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res) {__be32 *p;int status, type; -uint32_t segments; +uint32_t i, segments; status =decode_op_hdr (xdr,OP_READ_PLUS );if (status)
@@ -258,20 +258,24 @@ static int decode_read_plus(struct xdr_stream *xdr, struct nfs_pgio_res *res) res->count =0 ; res->eof =be32_to_cpup (p++); segments =be32_to_cpup (p++); -if (segments ==0 ) -return 0 ; - p =xdr_inline_decode (xdr,4 ); -if (unlikely (!p)) - goto out_overflow; - - type =be32_to_cpup (p++); -if (type ==NFS4_CONTENT_DATA ) - status =decode_read_plus_data (xdr, res); -else - status =decode_read_plus_hole (xdr, res); - -if (segments >1 ) +for (i =0 ; i < segments; i++) { + p =xdr_inline_decode (xdr,4 ); +if (unlikely (!p)) + goto out_overflow; + + type =be32_to_cpup (p); +if (type ==NFS4_CONTENT_DATA ) + status =decode_read_plus_data (xdr, res); +else + status =decode_read_plus_hole (xdr, res); +if (status) +break ; +if (res->count == xdr->buf->page_len) +break ; + } + +if (i < segments) res->eof =0 ;return status; out_overflow:
Signature · 1 line
2.3.3
Quoted 14 lines · Anna Schumaker
Test | v4.1 | no
READ_PLUS|READ_PLUS
---------------------------------------------------generic/013 | 135s | 143s | 123s generic/075 | 4s | 11s | 4s generic/091 | 9s | 16s | 15s generic/112 | 4s | 7s | 6s generic/127 | 62s | 117s | 114s generic/213 | [not run] | 1s | 1s generic/214 | [not run] | 0s | 0s generic/228 | [not run] | 1s | 2s generic/236 | 1s | 1s | 1s generic/263 | 4s | 6s | 8s generic/285 | 0s | 1s | 3s generic/315 | [not run] | 2s | 1s
Quoted 2 lines
---------------------------------------------------
Total | 3:47.47 | 5:11.85 | 4:43.77
So why is 4.2 generally slower than 4.1? I guess we really should look into that first.
Quoted 18 lines · Christoph Hellwig
Quoted 14 lines · Anna Schumaker
Test | v4.1 | no
READ_PLUS|READ_PLUS
---------------------------------------------------generic/013 | 135s | 143s | 123s generic/075 | 4s | 11s | 4s generic/091 | 9s | 16s | 15s generic/112 | 4s | 7s | 6s generic/127 | 62s | 117s | 114s generic/213 | [not run] | 1s | 1s generic/214 | [not run] | 0s | 0s generic/228 | [not run] | 1s | 2s generic/236 | 1s | 1s | 1s generic/263 | 4s | 6s | 8s generic/285 | 0s | 1s | 3s generic/315 | [not run] | 2s | 1sQuoted 2 lines
---------------------------------------------------
Total | 3:47.47 | 5:11.85 | 4:43.77
Quoted 3 lines
So why is 4.2 generally slower than 4.1? I guess we really should look into that first.
I did! There are several tests that run on v4.1 but skip over fallocate() calls. v4.2 no longer skips these tests.
Quoted 7 lines · Anna Schumaker
These patches add client support for the NFS v4.2 operation
READ_PLUS. This operation is triggered by doing any kind of read on a NFS v4.2 mounted filesystem. `I tested these patches using xfstests and compared the runtime between NFS v4.1, NFS v4.2 without
READ_PLUS, and NFS v4.2 withREAD_PLUS. Here are the results:
These are interesting but don't tell us much on their own. As far as I know xfstests isn't really intended to be used for performance comparisons, and there are some mysteries here:
Quoted 3 lines
Test | v4.1 | no
READ_PLUS|READ_PLUS
---------------------------------------------------
generic/013 | 135s | 143s | 123s
This is an fsstress run. Looking quickly at ltp/fsstress.c, I see it can do fallocate calls, so presumably that explains the difference.
Quoted 1 line
generic/075 | 4s | 11s | 4s
Ditto for fsx.
Quoted 3 lines
generic/091 | 9s | 16s | 15s generic/112 | 4s | 7s | 6s generic/127 | 62s | 117s | 114s
fsx again; this one must really have a ton of fallocate calls? Might be interesting to look at strace or /proc/self/mountstats to confirm that.
Quoted 9 lines
generic/213 | [not run] | 1s | 1s generic/214 | [not run] | 0s | 0s generic/228 | [not run] | 1s | 2s generic/236 | 1s | 1s | 1s generic/263 | 4s | 6s | 8s generic/285 | 0s | 1s | 3s generic/315 | [not run] | 2s | 1s---------------------------------------------------
Total | 3:47.47 | 5:11.85 | 4:43.77
I'm already inclined to believe that the bandwidth savings are more important than zero-copy in the bad cases. And the xfstests timings count for something. But I'd still rather see more of an argument.
It'd be interesting to see the best possible case (reading a large file that's all one hole), and the worst (reading a file with alternating 4K data and holes?).
--b.
Quoted 46 lines
Using the
READ_PLUSoperation does have an impact on reading sparse files over the network. There is still a big difference between v4.1 and v4.2 runtimes, but this looks to be a result offallocate()tests that only run over v4.2.Changes since v2:
- Merge together patches adding DATA and HOLE segment support.
- Simplify hole zeroing code by using thezero_user_segment()function.These patches and the corresponding server changes are available in the [
read_plus] branch of
git://git.linux-nfs.org/projects/anna/linux-nfs.gitQuestions? Comments? Thoughts?
Anna
Anna Schumaker (5):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
NFS: Add basicREAD_PLUSsupport
SUNRPC: Add the ability to shift data to a specific offset
NFS: Add support for decoding multiple segmentsfs/nfs/nfs42xdr.c | 162 ++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 30 +++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 1 + include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 240 ++++++++++++++++++++++++++++++++++++++++++++- 8 files changed, 434 insertions(+), 5 deletions(-)Signature · 6 lines
2.3.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 41 lines · J. Bruce Fields
Quoted 7 lines · Anna Schumaker
These patches add client support for the NFS v4.2 operation
READ_PLUS. This operation is triggered by doing any kind of read on a NFS v4.2 mounted filesystem. `I tested these patches using xfstests and compared the runtime between NFS v4.1, NFS v4.2 without
READ_PLUS, and NFS v4.2 withREAD_PLUS. Here are the results:These are interesting but don't tell us much on their own. As far as I know xfstests isn't really intended to be used for performance comparisons, and there are some mysteries here:
Quoted 3 lines
Test | v4.1 | no
READ_PLUS|READ_PLUS
---------------------------------------------------
generic/013 | 135s | 143s | 123sThis is an fsstress run. Looking quickly at
ltp/fsstress.c, I see it can do fallocate calls, so presumably that explains the difference.Quoted 1 line
generic/075 | 4s | 11s | 4s
Ditto for fsx.
Quoted 3 lines
generic/091 | 9s | 16s | 15s generic/112 | 4s | 7s | 6s generic/127 | 62s | 117s | 114sfsx again; this one must really have a ton of fallocate calls? Might be interesting to look at strace or
/proc/self/mountstatsto confirm that.Quoted 9 lines
generic/213 | [not run] | 1s | 1s generic/214 | [not run] | 0s | 0s generic/228 | [not run] | 1s | 2s generic/236 | 1s | 1s | 1s generic/263 | 4s | 6s | 8s generic/285 | 0s | 1s | 3s generic/315 | [not run] | 2s | 1s---------------------------------------------------
Total | 3:47.47 | 5:11.85 | 4:43.77
Quoted 7 lines
I'm already inclined to believe that the bandwidth savings are more important than zero-copy in the bad cases. And the xfstests timings count for something. But I'd still rather see more of an argument.
It'd be interesting to see the best possible case (reading a large file that's all one hole), and the worst (reading a file with alternating 4K data and holes?).
I'm writing up this test right now, so I'll hopefully have this data soon!
Anna
Quoted 49 lines
--b.
Quoted 46 lines
Using the
READ_PLUSoperation does have an impact on reading sparse files over the network. There is still a big difference between v4.1 and v4.2 runtimes, but this looks to be a result offallocate()tests that only run over v4.2.Changes since v2:
- Merge together patches adding DATA and HOLE segment support.
- Simplify hole zeroing code by using thezero_user_segment()function.These patches and the corresponding server changes are available in the [
read_plus] branch of
git://git.linux-nfs.org/projects/anna/linux-nfs.gitQuestions? Comments? Thoughts?
Anna
Anna Schumaker (5):
SUNRPC: Split out a function for setting current page
SUNRPC: Add the ability to expand holes in data pages
NFS: Add basicREAD_PLUSsupport
SUNRPC: Add the ability to shift data to a specific offset
NFS: Add support for decoding multiple segmentsfs/nfs/nfs42xdr.c | 162 ++++++++++++++++++++++++++++++ fs/nfs/nfs4proc.c | 30 +++++- fs/nfs/nfs4xdr.c | 1 + include/linux/nfs4.h | 1 + include/linux/nfs_fs_sb.h | 1 + include/linux/nfs_xdr.h | 2 +- include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 240 ++++++++++++++++++++++++++++++++++++++++++++- 8 files changed, 434 insertions(+), 5 deletions(-)Signature · 6 lines
2.3.3 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html