NFS: size the LAYOUTGET reply buffer for wide flexfiles layouts
Subsystems (2)
v2 cover NFS: size the LAYOUTGET reply buffer for wide flexfiles layouts
The flexfiles layout driver caps its LAYOUTGET reply buffer at a single page, which limits a striped layout segment to roughly 28 ff_data_server4 entries -- far below the 4096-stripe decode limit the client otherwise advertises. A server striping wider than that has no way to hand the client a layout: it returns NFS4ERR_TOOSMALL, the client falls back to I/O through the MDS, and pNFS never engages for those files.
This series first makes the client behave sanely when a layout does not fit the size it advertised in loga_maxcount, and then lets the reply buffer grow on demand up to the session's maximum response size.
Patch 1 is a standalone fix (Cc: stable). A server's NFS4ERR_TOOSMALL is mapped to -ETOOSMALL at decode and then handled nowhere, so the client falls back to the MDS for that one request and re-sends a doomed LAYOUTGET on every subsequent pageio attempt. Suspend pNFS via the layout fail bit instead, the way NFS4ERR_LAYOUTUNAVAILABLE already does.
Patches 2-5 make wide layouts work. loga_maxcount is derived from the reply buffer actually allocated rather than a fixed 4096 (block and SCSI layouts were advertising 4KB against a session-sized buffer, so a server whose extent list encodes larger than a page got a needless TOOSMALL). A TOOSMALL LAYOUTGET is then retried once with the buffer raised to the session's maximum response size -- the same bound GETDEVICEINFO already uses -- and a non-conformant server that ignores loga_maxcount and overruns the buffer outright takes that same recovery path instead of today's -EINVAL. Finally the escalated size is remembered per-server, so subsequent opens skip the attempt that is known to fail; this is also what lets the LAYOUTGET attached to OPEN succeed against a wide-striping server, since that path is best-effort and has no retry of its own.
The common path is unchanged throughout: the first LAYOUTGET for a layout still goes out with the layout driver's default reply buffer, and larger buffers are only ever allocated against servers that actually hand out wide layouts.
Patch 6 moves the decoded per-mirror stripe array to kvzalloc_objs(), so that a wide stripe array does not depend on a high-order allocation succeeding.
Wire-validated against reffs at stripe widths 2 and 64, exercising both the conformant NFS4ERR_TOOSMALL path and the buffer-overrun path.
Two known gaps are deliberately left for follow-up work:
- The decoded per-stripe footprint is heavy: nfs4_ff_layout_ds_stripe is roughly 300 bytes and embeds localio and layoutstats state that most stripes never use. Both the structure and its array container want rework before very large stripe counts are comfortable.
- The client decodes only the first logr_layout entry of a LAYOUTGET reply and silently discards the rest, so it can accept less coverage than it asked for in loga_minlength, and cannot amortize a multi-segment reply.
A related series, "NFS: flexfiles device notifications and caching for wide striped layouts", handles CB_NOTIFY_DEVICEID and scales the device caches for the layouts this one makes fetchable. The two are independent and apply cleanly in either order.
Changes on v2:
Rebased on v7.3-rc1
Jeff: trim the comment on the NFS4ERR_TOOSMALL case in patch 1 Collected Jeff's Reviewed-by
Based on v7.3-rc1.
v1: https://lore.kernel.org/linux-nfs/[email protected]/
Benjamin Coddington (6):
NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET NFSv4.1/pnfs: derive loga_maxcount from the LAYOUTGET reply buffer NFSv4.1/pnfs: retry LAYOUTGET with a larger reply buffer on
NFS4ERR_TOOSMALL
NFSv4.1/pnfs: treat an oversized LAYOUTGET reply as -EMSGSIZE NFSv4.1/pnfs: remember when a server needs a larger LAYOUTGET reply
buffer
NFSv4/flexfiles: allocate the per-mirror stripe array with
kvzalloc_objs
fs/nfs/flexfilelayout/flexfilelayout.c | 6 +-- fs/nfs/nfs4proc.c | 8 ++++ fs/nfs/nfs4xdr.c | 2 +- fs/nfs/pnfs.c | 54 +++++++++++++++++++++++--- include/linux/nfs_fs_sb.h | 4 ++ 5 files changed, 64 insertions(+), 10 deletions(-)
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
Signature · 1 line
2.53.0
PATCH v2 1/6 NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET
If the layout for the requested range is larger than the size the client advertised in loga_maxcount, RFC 8881 Section 18.43.3 has the metadata server return NFS4ERR_TOOSMALL. The client caps loga_maxcount at a single page, so a flexfiles server that stripes a layout segment across several dozen data servers produces this error today.
The client has no handling for it: nfs4_stat_to_errno() maps the error to -ETOOSMALL during decode, which nothing in the layoutget path recognizes and nfs_error_is_fatal() does not consider fatal, so pnfs_update_layout() clears the layout fail bit and returns no segment. The I/O falls back to the MDS, but because no fail bit was set, every subsequent pageio attempt sends another LAYOUTGET that is doomed to the same NFS4ERR_TOOSMALL. Files whose layouts do not fit the reply buffer never use pNFS and pay an extra round trip on every pageio.
Map -ETOOSMALL to -EMSGSIZE in the layoutget exception handler and have pnfs_update_layout() treat it like NFS4ERR_LAYOUTUNAVAILABLE:
mark the layout mode as failed and fall back to I/O through the MDS.
--- fs/nfs/nfs4proc.c | 8 ++++++++ fs/nfs/pnfs.c | 2 ++ 2 files changed, 10 insertions(+)
fs/nfs/nfs4proc.c +8 -0
@@ -9680,6 +9680,14 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,case -NFS4ERR_BADLAYOUT : status = -EOVERFLOW ; goto out; +/* +* NFS4ERR_TOOSMALL means the layout for the requested range +* exceeds what the client advertised in loga_maxcount (see +* RFC8881 section 18.43.3). +*/ +case -ETOOSMALL : + status = -EMSGSIZE ; + goto out;/* * NFS4ERR_LAYOUTTRYLATER is a conflict with another client * (or clients) writing to the same RAID stripe except when
fs/nfs/pnfs.c +2 -0
@@ -2331,6 +2331,8 @@ pnfs_update_layout(struct inode *ino,break ;case -ENODATA :/* The server returned NFS4ERR_LAYOUTUNAVAILABLE */ +case -EMSGSIZE : +/* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */ pnfs_layout_set_fail_bit ( lo,pnfs_iomode_to_fail_bit (iomode)); lseg =NULL ;
Signature · 1 line
2.53.0
Anna Schumaker
maintainerRe: [PATCH v2 1/6] NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET
Hi Ben,
Quoted 36 lines · Benjamin Coddington
If the layout for the requested range is larger than the size the client advertised in
loga_maxcount, RFC 8881 Section 18.43.3 has the metadata server returnNFS4ERR_TOOSMALL. The client capsloga_maxcountat a single page, so a flexfiles server that stripes a layout segment across several dozen data servers produces this error today.The client has no handling for it:
nfs4_stat_to_errno()maps the error to-ETOOSMALLduring decode, which nothing in the layoutget path recognizes andnfs_error_is_fatal()does not consider fatal, sopnfs_update_layout()clears the layout fail bit and returns no segment. The I/O falls back to the MDS, but because no fail bit was set, every subsequent pageio attempt sends another LAYOUTGET that is doomed to the sameNFS4ERR_TOOSMALL. Files whose layouts do not fit the reply buffer never use pNFS and pay an extra round trip on every pageio.Map
-ETOOSMALLto-EMSGSIZEin the layoutget exception handler and havepnfs_update_layout()treat it likeNFS4ERR_LAYOUTUNAVAILABLE:
mark the layout mode as failed and fall back to I/O through the MDS.Fixes d600ad1f2bdb ("NFS41: pop some layoutget errors to application")Assisted-by Claude:claude-opus-4-8Signed-off-by Benjamin Coddington <[email protected]>Reviewed-by Jeff Layton <[email protected]>--- fs/nfs/nfs4proc.c | 8 ++++++++ fs/nfs/pnfs.c | 2 ++ 2 files changed, 10 insertions(+)fs/nfs/nfs4proc.c +0
-0
fs/nfs/nfs4proc.c nfs4_layoutget_handle_exception(struct rpc_task *task, · line 9688
Quoted 9 lines
fs/nfs/nfs4proc.c +6
-0@@ -9680,6 +9680,14 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,case -NFS4ERR_BADLAYOUT : status = -EOVERFLOW ; goto out; +/* +* NFS4ERR_TOOSMALL means the layout for the requested range +* exceeds what the client advertised in loga_maxcount (see +* RFC8881 section 18.43.3). +*/ +case -ETOOSMALL :
Should this be NFS4ERR_TOOSMALL instead of ETOOSMALL?
Thanks,
Anna
Quoted 20 lines
fs/nfs/nfs4proc.c +2
-0+ status = -EMSGSIZE ; + goto out;/* * NFS4ERR_LAYOUTTRYLATER is a conflict with another client * (or clients) writing to the same RAID stripe except when fs/nfs/pnfs.c +2
-0@@ -2331,6 +2331,8 @@ pnfs_update_layout(struct inode *ino,break ;case -ENODATA :/* The server returned NFS4ERR_LAYOUTUNAVAILABLE */ +case -EMSGSIZE : +/* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */ pnfs_layout_set_fail_bit ( lo,pnfs_iomode_to_fail_bit (iomode)); lseg =NULL ;Signature · 1 line
2.53.0
Re: [PATCH v2 1/6] NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET
Quoted 34 lines · Anna Schumaker
Hi Ben,
Quoted 31 lines · Benjamin Coddington
If the layout for the requested range is larger than the size the client advertised in
loga_maxcount, RFC 8881 Section 18.43.3 has the metadata server returnNFS4ERR_TOOSMALL. The client capsloga_maxcountat a single page, so a flexfiles server that stripes a layout segment across several dozen data servers produces this error today.The client has no handling for it:
nfs4_stat_to_errno()maps the error to-ETOOSMALLduring decode, which nothing in the layoutget path recognizes andnfs_error_is_fatal()does not consider fatal, sopnfs_update_layout()clears the layout fail bit and returns no segment. The I/O falls back to the MDS, but because no fail bit was set, every subsequent pageio attempt sends another LAYOUTGET that is doomed to the sameNFS4ERR_TOOSMALL. Files whose layouts do not fit the reply buffer never use pNFS and pay an extra round trip on every pageio.Map
-ETOOSMALLto-EMSGSIZEin the layoutget exception handler and havepnfs_update_layout()treat it likeNFS4ERR_LAYOUTUNAVAILABLE:
mark the layout mode as failed and fall back to I/O through the MDS.Fixes d600ad1f2bdb ("NFS41: pop some layoutget errors to application")Assisted-by Claude:claude-opus-4-8Signed-off-by Benjamin Coddington <[email protected]>Reviewed-by Jeff Layton <[email protected]>--- fs/nfs/nfs4proc.c | 8 ++++++++ fs/nfs/pnfs.c | 2 ++ 2 files changed, 10 insertions(+)
Quoted 12 lines
fs/nfs/nfs4proc.c nfs4_layoutget_handle_exception(struct rpc_task *task, · line 9686
Quoted 12 lines
fs/nfs/nfs4proc.c +4
-0@@ -9680,6 +9680,14 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,case -NFS4ERR_BADLAYOUT : status = -EOVERFLOW ; goto out; +/* +* NFS4ERR_TOOSMALL means the layout for the requested range +* exceeds what the client advertised in loga_maxcount (see +* RFC8881 section 18.43.3).
^^ gah - Jeff asked me to drop this comment and I forgot. I agree its better without the comment.
Quoted 4 lines
fs/nfs/nfs4proc.c nfs4_layoutget_handle_exception(struct rpc_task *task, · line 9688
Quoted 2 lines
fs/nfs/nfs4proc.c +2
-0+*/ +case -ETOOSMALL :Should this be
NFS4ERR_TOOSMALLinstead of ETOOSMALL?
At this point, its already gone through nfs4_stat_to_errno():
{ NFS4ERR_TOOSMALL, -ETOOSMALL }
I definitely tested this case thoroughly..
.. but yeah - the mixed error types and the comment in there don't make it clear.
Ben
Anna Schumaker
maintainerRe: [PATCH v2 1/6] NFSv4.1/pnfs: suspend pNFS on NFS4ERR_TOOSMALL from LAYOUTGET
Quoted 52 lines · Benjamin Coddington
Quoted 34 lines · Anna Schumaker
Hi Ben,
Quoted 31 lines · Benjamin Coddington
If the layout for the requested range is larger than the size the client advertised in
loga_maxcount, RFC 8881 Section 18.43.3 has the metadata server returnNFS4ERR_TOOSMALL. The client capsloga_maxcountat a single page, so a flexfiles server that stripes a layout segment across several dozen data servers produces this error today.The client has no handling for it:
nfs4_stat_to_errno()maps the error to-ETOOSMALLduring decode, which nothing in the layoutget path recognizes andnfs_error_is_fatal()does not consider fatal, sopnfs_update_layout()clears the layout fail bit and returns no segment. The I/O falls back to the MDS, but because no fail bit was set, every subsequent pageio attempt sends another LAYOUTGET that is doomed to the sameNFS4ERR_TOOSMALL. Files whose layouts do not fit the reply buffer never use pNFS and pay an extra round trip on every pageio.Map
-ETOOSMALLto-EMSGSIZEin the layoutget exception handler and havepnfs_update_layout()treat it likeNFS4ERR_LAYOUTUNAVAILABLE:
mark the layout mode as failed and fall back to I/O through the MDS.Fixes d600ad1f2bdb ("NFS41: pop some layoutget errors to application")Assisted-by Claude:claude-opus-4-8Signed-off-by Benjamin Coddington <[email protected]>Reviewed-by Jeff Layton <[email protected]>--- fs/nfs/nfs4proc.c | 8 ++++++++ fs/nfs/pnfs.c | 2 ++ 2 files changed, 10 insertions(+)Quoted 12 lines
fs/nfs/nfs4proc.c nfs4_layoutget_handle_exception(struct rpc_task *task, · line 9686
Quoted 12 lines
fs/nfs/nfs4proc.c +4
-0@@ -9680,6 +9680,14 @@ nfs4_layoutget_handle_exception(struct rpc_task *task,case -NFS4ERR_BADLAYOUT : status = -EOVERFLOW ; goto out; +/* +* NFS4ERR_TOOSMALL means the layout for the requested range +* exceeds what the client advertised in loga_maxcount (see +* RFC8881 section 18.43.3). ^^ gah - Jeff asked me to drop this comment and I forgot. I agree its better without the comment.
Quoted 6 lines
Quoted 4 lines
fs/nfs/nfs4proc.c nfs4_layoutget_handle_exception(struct rpc_task *task, · line 9688
Quoted 2 lines
fs/nfs/nfs4proc.c +2
-0+*/ +case -ETOOSMALL :Should this be
NFS4ERR_TOOSMALLinstead of ETOOSMALL?At this point, its already gone through
nfs4_stat_to_errno():
Oh, I see. I was looking for for the call to nfs4_stat_to_errno() in nfs4proc.c, but didn't think to check nfs4xdr.c too. I think that's cleared up my confusion!
Anna
Quoted 9 lines
{
NFS4ERR_TOOSMALL,-ETOOSMALL}I definitely tested this case thoroughly..
.. but yeah - the mixed error types and the comment in there don't make it clear.
Ben
PATCH v2 2/6 NFSv4.1/pnfs: derive loga_maxcount from the LAYOUTGET reply buffer
In reply to Benjamin CoddingtonThe client advertises a fixed PNFS_LAYOUT_MAXSIZE (4096) in loga_maxcount no matter what reply buffer it actually allocated. The two only happen to agree for layout drivers that cap max_layoutget_response at a single page. Block and SCSI layouts allocate a session-sized reply buffer, yet still tell the server it must not send more than 4KB of layout: a server whose extent list encodes larger than a page returns NFS4ERR_TOOSMALL even though the client has ample room to receive it.
Set loga_maxcount from the size of the page array allocated for the reply. The client never advertises more than it can receive, and any future change to the reply buffer sizing automatically carries its wire advertisement with it.
--- fs/nfs/pnfs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/pnfs.c +1 -1
@@ -1210,7 +1210,7 @@ pnfs_alloc_init_layoutget_args(struct inode *ino, lgp->args.minlength = i_size - range->offset; } } - lgp->args.maxcount =PNFS_LAYOUT_MAXSIZE ; + lgp->args.maxcount = lgp->args.layout.pglen;pnfs_copy_range (&lgp->args.range, range); lgp->args.type = server->pnfs_curr_ld->id; lgp->args.inode = ino;
Signature · 1 line
2.53.0
PATCH v2 3/6 NFSv4.1/pnfs: retry LAYOUTGET with a larger reply buffer on NFS4ERR_TOOSMALL
In reply to Benjamin CoddingtonLayout drivers cap the LAYOUTGET reply buffer with max_layoutget_response; the flexfiles driver caps it at a single page, which limits a striped layout segment to roughly 28 stripes. A server striping wider than that returns NFS4ERR_TOOSMALL, and the client falls back to I/O through the MDS -- pNFS never engages for those files.
Instead of failing over to the MDS on the first NFS4ERR_TOOSMALL, retry the LAYOUTGET once with the reply buffer raised to the session's maximum response size, the same bound GETDEVICEINFO already uses. The server offers no size hint in the TOOSMALL error, and the page array is transient (freed when the RPC completes), so a single jump to the ceiling is preferred over incremental growth. If the layout does not fit even the session-sized buffer, fall back to the MDS as before.
The common path is unchanged: the first LAYOUTGET for a layout is still sent with the driver's default reply buffer, and larger buffers are only ever allocated against servers that actually hand out wide layouts.
--- fs/nfs/pnfs.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
fs/nfs/pnfs.c +38 -7
@@ -1167,11 +1167,12 @@ pnfs_alloc_init_layoutget_args(struct inode *ino,struct nfs_open_context *ctx, const nfs4_stateid *stateid, const struct pnfs_layout_range *range, - gfp_t gfp_flags) + size_t min_reply_sz, gfp_t gfp_flags) { struct nfs_server *server = pnfs_find_server(ino, ctx);size_t max_reply_sz = server->pnfs_curr_ld->max_layoutget_response; -size_t max_pages =max_response_pages (server); +size_t session_pages =max_response_pages (server); +size_t max_pages = session_pages;struct nfs4_layoutget *lgp;dprintk ("--> %s\n" , __func__);
@@ -1186,6 +1187,17 @@ pnfs_alloc_init_layoutget_args(struct inode *ino, max_pages = npages; } +/* +* A previous LAYOUTGET for this layout did not fit the reply +* buffer: raise the layout driver's default up to the session's +* maximum response size. +*/ +if (min_reply_sz) { +size_t npages = (min_reply_sz +PAGE_SIZE -1 ) >>PAGE_SHIFT ; +if (npages > max_pages) + max_pages =min (npages, session_pages); + } + lgp->args.layout.pages =nfs4_alloc_pages (max_pages, gfp_flags);if (!lgp->args.layout.pages) {kfree (lgp);
@@ -2145,6 +2157,7 @@ pnfs_update_layout(struct inode *ino, .inode = ino, };unsigned long giveup = jiffies + (clp->cl_lease_time <<1 ); +size_t reply_sz =0 ;bool first;if (!pnfs_enabled_sb (NFS_SERVER (ino))) {
@@ -2304,7 +2317,8 @@ pnfs_update_layout(struct inode *ino,if (arg.length !=NFS4_MAX_UINT64 ) arg.length =PAGE_ALIGN (arg.length); - lgp =pnfs_alloc_init_layoutget_args (ino, ctx, &stateid, &arg, gfp_flags); + lgp =pnfs_alloc_init_layoutget_args (ino, ctx, &stateid, &arg, reply_sz, + gfp_flags);if (!lgp) { lseg =ERR_PTR (-ENOMEM ); trace_pnfs_update_layout(ino, pos, count, iomode, lo, NULL,
@@ -2331,12 +2345,29 @@ pnfs_update_layout(struct inode *ino,break ;case -ENODATA :/* The server returned NFS4ERR_LAYOUTUNAVAILABLE */ -case -EMSGSIZE : -/* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */ pnfs_layout_set_fail_bit ( lo,pnfs_iomode_to_fail_bit (iomode)); lseg =NULL ; goto out_put_layout_hdr; +case -EMSGSIZE : { +/* +* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL): +* retry once with the reply buffer raised to the +* session's maximum response size before falling back +* to I/O through the MDS. +*/ +size_t max =max_response_pages (server) <<PAGE_SHIFT ; + +if (reply_sz < max) { + reply_sz = max; + exception.retry =1 ; +break ; + } +pnfs_layout_set_fail_bit ( + lo,pnfs_iomode_to_fail_bit (iomode)); + lseg =NULL ; + goto out_put_layout_hdr; + }default :if (!nfs_error_is_fatal (PTR_ERR (lseg))) {pnfs_layout_clear_fail_bit (lo,pnfs_iomode_to_fail_bit (iomode));
@@ -2450,7 +2481,7 @@ static void _lgopen_prepare_attached(struct nfs4_opendata *data, lo =_pnfs_grab_empty_layout (ino, ctx);if (!lo)return ; - lgp =pnfs_alloc_init_layoutget_args (ino, ctx, ¤t_stateid, &rng, + lgp =pnfs_alloc_init_layoutget_args (ino, ctx, ¤t_stateid, &rng,0 ,nfs_io_gfp_mask ());if (!lgp) {clear_and_wake_up_bit (NFS_LAYOUT_FIRST_LAYOUTGET , &lo->plh_flags);
@@ -2476,7 +2507,7 @@ static void _lgopen_prepare_floating(struct nfs4_opendata *data, };struct nfs4_layoutget *lgp; - lgp =pnfs_alloc_init_layoutget_args (ino, ctx, ¤t_stateid, &rng, + lgp =pnfs_alloc_init_layoutget_args (ino, ctx, ¤t_stateid, &rng,0 ,nfs_io_gfp_mask ());if (!lgp)return ;
Signature · 1 line
2.53.0
PATCH v2 4/6 NFSv4.1/pnfs: treat an oversized LAYOUTGET reply as -EMSGSIZE
In reply to Benjamin Coddingtondecode_layoutget() already detects a server that sends a layout body larger than the reply buffer we provided ("server cheating in layoutget reply") but maps it to -EINVAL, which pnfs_update_layout() treats as a transient error: the client falls back to the MDS for this I/O only and re-sends a doomed LAYOUTGET on every subsequent pageio attempt.
A server that overruns the reply buffer has ignored loga_maxcount, but the client can recover the same way it recovers from a conformant server's NFS4ERR_TOOSMALL: return -EMSGSIZE from exactly this check so that the layoutget path retries once with a session-sized reply buffer and otherwise marks the layout mode failed and falls back to the MDS.
--- fs/nfs/nfs4xdr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfs/nfs4xdr.c +1 -1
@@ -6187,7 +6187,7 @@ static int decode_layoutget(struct xdr_stream *xdr, struct rpc_rqst *req,dprintk ("NFS: server cheating in layoutget reply: " "layout len %u > recvd %u\n" , res->layoutp->len, recvd); - status = -EINVAL ; + status = -EMSGSIZE ; goto out; }
Signature · 1 line
2.53.0
PATCH v2 5/6 NFSv4.1/pnfs: remember when a server needs a larger LAYOUTGET reply buffer
In reply to Benjamin CoddingtonWhen a LAYOUTGET only succeeds after escalating the reply buffer, every layout fetched from that server is likely to need the larger buffer:
remember the escalated size on the nfs_server and use it as the floor for subsequent LAYOUTGET reply buffers, skipping the doomed attempt at the layout driver's default size.
This also lets the LAYOUTGET attached to OPEN benefit: the lgopen path is best-effort with no retry of its own, so without the learned size it would fail with NFS4ERR_TOOSMALL at every open against a server handing out wide layouts, and layouts would only ever be acquired by the I/O path's retry.
The field is a hint: reads and writes are racy by design, the value only ever grows toward the session's maximum response size, and a stale-low read merely costs one escalation round trip.
--- fs/nfs/pnfs.c | 15 ++++++++++++--- include/linux/nfs_fs_sb.h | 4 ++++ 2 files changed, 16 insertions(+), 3 deletions(-)
fs/nfs/pnfs.c +12 -3
@@ -1188,10 +1188,12 @@ pnfs_alloc_init_layoutget_args(struct inode *ino, }/* -* A previous LAYOUTGET for this layout did not fit the reply -* buffer: raise the layout driver's default up to the session's -* maximum response size. +* A previous LAYOUTGET on this layout or on this server did not +* fit the reply buffer: raise the layout driver's default up to +* the session's maximum response size. */ +if (!min_reply_sz) + min_reply_sz =READ_ONCE (server->lg_reply_sz);if (min_reply_sz) {size_t npages = (min_reply_sz +PAGE_SIZE -1 ) >>PAGE_SHIFT ;if (npages > max_pages)
@@ -2387,6 +2389,13 @@ pnfs_update_layout(struct inode *ino, goto lookup_again; } }else { +/* +* A LAYOUTGET that only succeeded with an escalated reply +* buffer: remember the size so that future LAYOUTGETs to +* this server skip the attempt at the driver's default. +*/ +if (reply_sz) +WRITE_ONCE (server->lg_reply_sz, reply_sz);pnfs_layout_clear_fail_bit (lo,pnfs_iomode_to_fail_bit (iomode)); }
include/linux/nfs_fs_sb.h +4 -0
@@ -248,6 +248,10 @@ struct nfs_server {that are supported on this filesystem */ struct pnfs_layoutdriver_type *pnfs_curr_ld;/* Active layout driver */ +unsigned int lg_reply_sz;/* Learned LAYOUTGET reply +buffer size, when the layout +driver's default has proved +too small */ struct rpc_wait_queue roc_rpcwaitq;/* the following fields are protected by nfs_client->cl_lock */
Signature · 1 line
2.53.0
PATCH v2 6/6 NFSv4/flexfiles: allocate the per-mirror stripe array with kvzalloc_objs
In reply to Benjamin CoddingtonEach mirror's stripe array is a single contiguous allocation of dss_count * sizeof(struct nfs4_ff_layout_ds_stripe) -- roughly 300 bytes per stripe. With the LAYOUTGET reply buffer no longer capped at a single page, a wide striped layout can push this well past the high-order allocation comfort zone (a 2048-stripe mirror is a ~600KB contiguous allocation) where it can fail under memory fragmentation.
Use kvzalloc_objs() so wide stripe arrays fall back to vmalloc. Note the vmalloc fallback is unavailable when the pageio path allocates under memalloc_noio (swap over pNFS); that case simply behaves as before.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +3 -3
@@ -285,8 +285,8 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count, mirror->dss_count = dss_count; mirror->dss = - kzalloc_objs(struct nfs4_ff_layout_ds_stripe, dss_count, - gfp_flags); + kvzalloc_objs(struct nfs4_ff_layout_ds_stripe, dss_count, + gfp_flags);if (mirror->dss ==NULL ) {kfree (mirror);return NULL ;
@@ -315,7 +315,7 @@ static void ff_layout_free_mirror(struct nfs4_ff_layout_mirror *mirror)nfs4_ff_layout_put_deviceid (mirror->dss[dss_id].mirror_ds); } -kfree (mirror->dss); +kvfree (mirror->dss);kfree (mirror); }
Signature · 1 line
2.53.0