NFSv4.0: nfs4_do_fsinfo() should not do implicit lease renewals
Subsystems (2)
PATCH v3 1/1 NFSv4.0: nfs4_do_fsinfo() should not do implicit lease renewals
From: Robert Milkowski <[email protected]>
Currently, each time nfs4_do_fsinfo() is called it will do an implicit NFS4 lease renewal, which is not compliant with the NFS4 specification. This can result in a lease being expired by an NFS server.
Commit 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managing leases") introduced implicit client lease renewal in nfs4_do_fsinfo(), which can result in the NFSv4.0 lease to expire on a server side, and servers returning NFS4ERR_EXPIRED or NFS4ERR_STALE_CLIENTID.
This can easily be reproduced by frequently unmounting a sub-mount, then stat'ing it to get it mounted again, which will delay or even completely prevent client from sending RENEW operations if no other NFS operations are issued. Eventually nfs server will expire client's lease and return an error on file access or next RENEW.
This can also happen when a sub-mount is automatically unmounted due to inactivity (after nfs_mountpoint_expiry_timeout), then it is mounted again via stat(). This can result in a short window during which client's lease will expire on a server but not on a client. This specific case was observed on production systems.
This patch makes an explicit lease renewal instead of an implicit one, by adding RENEW to a compound operation issued by nfs4_do_fsinfo(), similarly to NFSv4.1 which adds SEQUENCE operation.
--- fs/nfs/nfs4proc.c | 4 ++++ fs/nfs/nfs4xdr.c | 13 +++++++++++-- include/linux/nfs_xdr.h | 3 +++ 3 files changed, 18 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +4 -0
@@ -4998,12 +4998,16 @@ static int nfs4_proc_statfs(struct nfs_server *server, struct nfs_fh *fhandle, sstatic int _nfs4_do_fsinfo (struct nfs_server *server,struct nfs_fh *fhandle,struct nfs_fsinfo *fsinfo) { +struct nfs_client *clp = server->nfs_client;struct nfs4_fsinfo_arg args = { .fh = fhandle, .bitmask = server->attr_bitmask, + .clientid = clp->cl_clientid, + .renew =nfs4_has_session (clp) ?0 :1 ,/* append RENEW */ };struct nfs4_fsinfo_res res = { .fsinfo = fsinfo, + .renew =nfs4_has_session (clp) ?0 :1 , };struct rpc_message msg = { .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_FSINFO ],
fs/nfs/nfs4xdr.c +11 -2
@@ -555,11 +555,13 @@ static int decode_layoutget(struct xdr_stream *xdr, struct rpc_rqst *req,#define NFS4_enc_fsinfo_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \ - encode_fsinfo_maxsz) + encode_fsinfo_maxsz + \ + encode_renew_maxsz)#define NFS4_dec_fsinfo_sz (compound_decode_hdr_maxsz + \ decode_sequence_maxsz + \ decode_putfh_maxsz + \ - decode_fsinfo_maxsz) + decode_fsinfo_maxsz + \ + decode_renew_maxsz)#define NFS4_enc_renew_sz (compound_encode_hdr_maxsz + \ encode_renew_maxsz)#define NFS4_dec_renew_sz (compound_decode_hdr_maxsz + \
@@ -2646,6 +2648,8 @@ static void nfs4_xdr_enc_fsinfo(struct rpc_rqst *req, struct xdr_stream *xdr,encode_sequence (xdr, &args->seq_args, &hdr);encode_putfh (xdr, args->fh, &hdr);encode_fsinfo (xdr, args->bitmask, &hdr); +if (args->renew) +encode_renew (xdr, args->clientid, &hdr);encode_nops (&hdr); }
@@ -6778,6 +6782,11 @@ static int nfs4_xdr_dec_fsinfo(struct rpc_rqst *req, struct xdr_stream *xdr, status =decode_putfh (xdr);if (!status) status =decode_fsinfo (xdr, res->fsinfo); +if (status) + goto out; +if (res->renew) + status =decode_renew (xdr); +out:return status; }
include/linux/nfs_xdr.h +3 -0
@@ -1025,11 +1025,14 @@ struct nfs4_fsinfo_arg {struct nfs4_sequence_args seq_args;const struct nfs_fh * fh;const u32 * bitmask; +clientid4 clientid; + unsigned char renew:1; };struct nfs4_fsinfo_res {struct nfs4_sequence_res seq_res;struct nfs_fsinfo *fsinfo; +unsigned char renew:1 ; };struct nfs4_getattr_arg {
Signature · 1 line
1.8.3.1
Quoted 27 lines · Robert Milkowski
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit NFS4 lease renewal, which is not compliant with the NFS4 specification. This can result in a lease being expired by an NFS server.Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases") introduced implicit client lease renewal innfs4_do_fsinfo(), which can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub-mount, then stat'ing it to get it mounted again, which will delay or even completely prevent client from sending RENEW operations if no other NFS operations are issued. Eventually nfs server will expire client's lease and return an error on file access or next RENEW.
This can also happen when a sub-mount is automatically unmounted due to inactivity (after
nfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during which client's lease will expire on a server but not on a client. This specific case was observed on production systems.
Quoted 6 lines
This patch makes an explicit lease renewal instead of an implicit one, by adding RENEW to a compound operation issued by
nfs4_do_fsinfo(), similarly to NFSv4.1 which adds SEQUENCE operation.Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managing leases")Signed-off-by Robert Milkowski <[email protected]>
Quoted 91 lines
---
fs/nfs/nfs4proc.c | 4 ++++ fs/nfs/nfs4xdr.c | 13 +++++++++++-- include/linux/nfs_xdr.h | 3 +++ 3 files changed, 18 insertions(+), 2 deletions(-)fs/nfs/nfs4proc.c +4
-0@@ -4998,12 +4998,16 @@ static int nfs4_proc_statfs(struct nfs_server *server, struct nfs_fh *fhandle, sstatic int _nfs4_do_fsinfo (struct nfs_server *server,struct nfs_fh *fhandle,struct nfs_fsinfo *fsinfo) { +struct nfs_client *clp = server->nfs_client;struct nfs4_fsinfo_arg args = { .fh = fhandle, .bitmask = server->attr_bitmask, + .clientid = clp->cl_clientid, + .renew =nfs4_has_session (clp) ?0 :1 ,/* append RENEW */ };struct nfs4_fsinfo_res res = { .fsinfo = fsinfo, + .renew =nfs4_has_session (clp) ?0 :1 , };struct rpc_message msg = { .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_FSINFO ],fs/nfs/nfs4xdr.c +11
-2@@ -555,11 +555,13 @@ static int decode_layoutget(struct xdr_stream *xdr, struct rpc_rqst *req,#define NFS4_enc_fsinfo_sz (compound_encode_hdr_maxsz + \ encode_sequence_maxsz + \ encode_putfh_maxsz + \ - encode_fsinfo_maxsz) + encode_fsinfo_maxsz + \ + encode_renew_maxsz)#define NFS4_dec_fsinfo_sz (compound_decode_hdr_maxsz + \ decode_sequence_maxsz + \ decode_putfh_maxsz + \ - decode_fsinfo_maxsz) + decode_fsinfo_maxsz + \ + decode_renew_maxsz)#define NFS4_enc_renew_sz (compound_encode_hdr_maxsz + \ encode_renew_maxsz)#define NFS4_dec_renew_sz (compound_decode_hdr_maxsz + \@@ -2646,6 +2648,8 @@ static void nfs4_xdr_enc_fsinfo(struct rpc_rqst *req, struct xdr_stream *xdr,encode_sequence (xdr, &args->seq_args, &hdr);encode_putfh (xdr, args->fh, &hdr);encode_fsinfo (xdr, args->bitmask, &hdr); +if (args->renew) +encode_renew (xdr, args->clientid, &hdr);encode_nops (&hdr); }@@ -6778,6 +6782,11 @@ static int nfs4_xdr_dec_fsinfo(struct rpc_rqst *req, struct xdr_stream *xdr, status =decode_putfh (xdr);if (!status) status =decode_fsinfo (xdr, res->fsinfo); +if (status) + goto out; +if (res->renew) + status =decode_renew (xdr); +out:return status; }include/linux/nfs_xdr.h +3
-0@@ -1025,11 +1025,14 @@ struct nfs4_fsinfo_arg {struct nfs4_sequence_args seq_args;const struct nfs_fh * fh;const u32 * bitmask; +clientid4 clientid; + unsigned char renew:1; };struct nfs4_fsinfo_res {struct nfs4_sequence_res seq_res;struct nfs_fsinfo *fsinfo; +unsigned char renew:1 ; };struct nfs4_getattr_arg {Signature · 1 line
1.8.3.1
--
Chuck Lever
Quoted 41 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust <[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit lease renewalsQuoted 1 line
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <[email protected]>
Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 19 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases") introduced implicit client lease renewal innfs4_do_fsinfo(), which can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub-mount, then stat'ing it to get it mounted again, which will delay or even completely prevent client from sending RENEW operations if no other NFS operations are issued. Eventually nfs server will expire client's lease and return an error on file access or next RENEW.
This can also happen when a sub-mount is automatically unmounted due to inactivity (after
nfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during which client's lease will expire on a server but not on a client. This specific case was observed on production systems.
Quoted 10 lines
Quoted 6 lines
This patch makes an explicit lease renewal instead of an implicit one, by adding RENEW to a compound operation issued by
nfs4_do_fsinfo(), similarly to NFSv4.1 which adds SEQUENCE operation.Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managing leases")Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>
How do we progress it further?
Signature · 1 line
Robert Milkowski
Hi Robert,
Quoted 42 lines · Robert Milkowski
Quoted 42 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust <[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit lease renewalsQuoted 1 line
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <[email protected]>
Quoted 4 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 23 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases") introduced implicit client lease renewal innfs4_do_fsinfo(), which can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub-mount, then stat'ing it to get it mounted again, which will delay or even completely prevent client from sending RENEW operations if no other NFS operations are issued. Eventually nfs server will expire client's lease and return an error on file access or next RENEW.
This can also happen when a sub-mount is automatically unmounted due to inactivity (after
nfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during which client's lease will expire on a server but not on a client. This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one, by adding RENEW to a compound operation issued by
nfs4_do_fsinfo(), similarly to NFSv4.1 which adds SEQUENCE operation.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managing leases")Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>How do we progress it further?
Thanks for following up! I have the patch included in my linux-next branch for the next merge window.
Anna
Quoted 1 line
Quoted 68 lines
-----Original Message-----
From: Schumaker, Anna <[email protected]>
Sent: 22 January 2020 19:11
To: [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit lease renewalsHi Robert,
Quoted 28 lines · Robert Milkowski
Quoted 28 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust <[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski
<[email protected]>Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 7 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases") introduced implicit client lease renewal innfs4_do_fsinfo(), which can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This can easily be reproduced by frequently unmounting a sub-mount, then stat'ing it to get it mounted again, which will delay or even completely prevent client from sending RENEW operations if no other NFS operations are issued. Eventually nfs server will expire client's lease and return an error on file access
or next RENEW.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This can also happen when a sub-mount is automatically unmounted due to inactivity (after
nfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during which client's lease will expire on a server but not on aclient.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This specific case was observed on production systems.
This patch makes an explicit lease renewal instead of an implicit one, by adding RENEW to a compound operation issued by
nfs4_do_fsinfo(), similarly to NFSv4.1 which adds SEQUENCEoperation.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managing leases")Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>
Quoted 6 lines
Quoted 1 line
How do we progress it further?
Thanks for following up! I have the patch included in my linux-next branch for the next merge window.
Anna
Nice. Thanks!
Signature · 1 line
Robert Milkowski
Quoted 68 lines · Anna
Hi Robert,
Quoted 65 lines · Robert Milkowski
Quoted 64 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust
<[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit
lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <
[email protected]>Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 34 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases")
introduced implicit client lease renewal innfs4_do_fsinfo(), which
can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub- mount,
then stat'ing it to get it mounted again, which will delay or even
completely prevent client from sending RENEW operations if no other
NFS operations are issued. Eventually nfs server will expire client's
lease and return an error on file access or next RENEW.This can also happen when a sub-mount is automatically unmounted due
to inactivity (afternfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during
which client's lease will expire on a server but not on a client.
This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one,
by adding RENEW to a compound operation issued bynfs4_do_fsinfo(),
similarly to NFSv4.1 which adds SEQUENCE operation.Quoted 3 lines
Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managingleases")
Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>
Quoted 5 lines
Quoted 1 line
How do we progress it further?
Thanks for following up! I have the patch included in my linux-next branch for
the next merge window.
NACK. This is the wrong way to solve the problem. Lease renewal in NFSv4 does not need to be tied to fsinfo. It creates an unnecessary extra error condition that has absolutely nothing to do with the functionality of retrieving per-filesystem attributes.
All that needs to be done here is to move the setting of clp-
Quoted 1 line
cl_last_renewal_out_ ofnfs4_set_lease_period(), and just have
nfs4_proc_setclientid_confirm() and nfs4_update_session() call do_renew_lease().
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 72 lines · Trond Myklebust
Quoted 70 lines · Anna
Hi Robert,
Quoted 66 lines · Robert Milkowski
Quoted 57 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust
<[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit
lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <
[email protected]>Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 34 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases")
introduced implicit client lease renewal innfs4_do_fsinfo(), which
can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub- mount,
then stat'ing it to get it mounted again, which will delay or even
completely prevent client from sending RENEW operations if no other
NFS operations are issued. Eventually nfs server will expire client's
lease and return an error on file access or next RENEW.This can also happen when a sub-mount is automatically unmounted due
to inactivity (afternfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during
which client's lease will expire on a server but not on a client.
This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one,
by adding RENEW to a compound operation issued bynfs4_do_fsinfo(),
similarly to NFSv4.1 which adds SEQUENCE operation.Quoted 7 lines
Quoted 3 lines
Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managingleases")
Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>How do we progress it further?
Quoted 8 lines
Quoted 3 lines
Thanks for following up! I have the patch included in my linux-next branch for
the next merge window.NACK. This is the wrong way to solve the problem. Lease renewal in NFSv4 does not need to be tied to fsinfo. It creates an unnecessary extra error condition that has absolutely nothing to do with the functionality of retrieving per-filesystem attributes.
Well, we also do it for NFSv4.1+ with the SEQUENCE operation being send by fsinfo, and as Chuck pointed out
it makes sense to do similarly for 4.0, which is what this patch does.
Or as per the v2 version of the patch, not do the implicit RENEW for 4.0, which was a simpler patch,
but then not in-line with 4.1+.
Quoted 6 lines
All that needs to be done here is to move the setting of clp-
Quoted 1 line
cl_last_renewal_out_ ofnfs4_set_lease_period(), and just have
nfs4_proc_setclientid_confirm()andnfs4_update_session()calldo_renew_lease().
This would also require nfs4_setup_state_renewal() to call do_renew_lease() I think - at least it currently calls nfs4_set_lease_period().
Also, iirc fsinfo() not setting cl_last_renewal leads to cl_last_renewal initialization issues under some circumstances.
Then the RFC 7530 in section 16.34.5 states:
"SETCLIENTID_CONFIRM does not establish or renew a lease.", so calling do_renew_lease() from nfs4_setclientid_confirm() doesn't seem to be ok.
I'm not sure if is is valid to do implicit lease renewal in nfs4_update_session() either...
Anyway, the patch would be something like (haven't tested it yet):
fs/nfs/nfs4proc.c +0 -0
@@ -5019,16 +5019,13 @@ static int nfs4_do_fsinfo(struct nfs_server
*server, struct nfs_fh *fhandle, str
struct nfs4_exception exception = {
.interruptible = true,
};
- unsigned long now = jiffies;
int err;
do {
err = _nfs4_do_fsinfo(server, fhandle, fsinfo);
trace_nfs4_fsinfo(server, fhandle, fsinfo->fattr, err);
if (err == 0) {- nfs4_set_lease_period(server->nfs_client,
- fsinfo->lease_time * HZ,
- now);
+ nfs4_set_lease_period(server->nfs_client,fsinfo->lease_time * HZ)
break;
}
err = nfs4_handle_exception(server, err, &exception);
@@ -6146,6 +6143,10 @@ int nfs4_proc_setclientid_confirm(struct nfs_client *clp,
clp->cl_clientid);
status = rpc_call_sync(clp->cl_rpcclient, &msg,
RPC_TASK_TIMEOUT | RPC_TASK_NO_ROUND_ROBIN);+ if(status == 0) {
+ unsigned long now = jiffies;
+ do_renew_lease(clp, now);
+ }
trace_nfs4_setclientid_confirm(clp, status);
dprintk("NFS reply setclientid_confirm: %d\n", status);
return status;
@@ -8590,6 +8591,8 @@ static void nfs4_update_session(struct
nfs4_session *session,
if (res->flags & SESSION4_BACK_CHAN)
memcpy(&session->bc_attrs, &res->bc_attrs,
sizeof(session->bc_attrs));+ unsigned long now = jiffies;
+ do_renew_lease(session->clp, now);
}
static int _nfs4_proc_create_session(struct nfs_client *clp,
fs/nfs/nfs4renewd.c +1 -4
@@ -138,15 +138,12 @@* * @clp: pointer to nfs_client * @lease: new value for lease period -* @lastrenewed: time at which lease was last renewed */ void nfs4_set_lease_period (struct nfs_client *clp, -unsigned long lease, -unsigned long lastrenewed) +unsigned long lease) {spin_lock (&clp->cl_lock); clp->cl_lease_time = lease; - clp->cl_last_renewal = lastrenewed;spin_unlock (&clp->cl_lock);/* Cap maximum reconnect timeout at 1/2 lease period */
fs/nfs/nfs4state.c +2 -1
@@ -102,7 +102,8 @@ static int nfs4_setup_state_renewal(struct nfs_client *clp) now = jiffies; status =nfs4_proc_get_lease_time (clp, &fsinfo);if (status ==0 ) { -nfs4_set_lease_period (clp, fsinfo.lease_time *HZ , now); +nfs4_set_lease_period (clp, fsinfo.lease_time *HZ ); +nfs4_do_renew_lease (clp, now);nfs4_schedule_state_renewal (clp); }
But it potentially has issues, as just pointed out, mainly with not being compliant with rfc again.
Also see the comments above about the SEQUENCE.
Trond - ?
Chuck, Anna - which way do you prefer, the one proposed by Chuck or the one now proposed by Trond?
Or perhaps my v2 patch which is least invasive and just stops doing implicit renewals for 4.0 if cl_last_renewal is already set.
I personally think the way proposed by Chuck is fully compliant with RFC and in-line with what we currently do for 4.1 via SEQUENCE, so perhaps the best option. ?
Signature · 1 line
Robert Milkowski
Quoted 81 lines · Robert Milkowski
Quoted 76 lines · Trond Myklebust
Quoted 73 lines · Anna
Hi Robert,
Quoted 61 lines · Robert Milkowski
Quoted 61 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust
<[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit
lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <
[email protected]>Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 37 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases")
introduced implicit client lease renewal innfs4_do_fsinfo(), which
can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub- mount,
then stat'ing it to get it mounted again, which will delay or even
completely prevent client from sending RENEW operations if no other
NFS operations are issued. Eventually nfs server will expire client's
lease and return an error on file access or next RENEW.This can also happen when a sub-mount is automatically unmounted due
to inactivity (afternfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during
which client's lease will expire on a server but not on a client.
This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one,
by adding RENEW to a compound operation issued bynfs4_do_fsinfo(),
similarly to NFSv4.1 which adds SEQUENCE operation.Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managingleases")
Signed-off-by Robert Milkowski <[email protected]>Quoted 5 lines
Quoted 3 lines
Reviewed-by Chuck Lever <[email protected]>How do we progress it further?
Thanks for following up! I have the patch included in my linux-next branch for
the next merge window.
Quoted 8 lines
Quoted 4 lines
NACK. This is the wrong way to solve the problem. Lease renewal in NFSv4 does not need to be tied to fsinfo. It creates an unnecessary extra error condition that has absolutely nothing to do with the functionality of retrieving per-filesystem attributes.
Well, we also do it for NFSv4.1+ with the SEQUENCE operation being send by fsinfo, and as Chuck pointed out
it makes sense to do similarly for 4.0, which is what this patch does.
I did say that.
However, I can see that for NFSv4.1+, the client code handling the SEQUENCE response will update cl_last_renewal. It does not need to be done in the fsinfo code.
The NFSv4.0 behavior should be correct if cl_last_renewal is not updated. That should force the client to send a separate RENEW operation so that both the client and server agree that the lease is active.
If I understand Trond correctly?
Quoted 119 lines
Or as per the v2 version of the patch, not do the implicit RENEW for 4.0, which was a simpler patch,
but then not in-line with 4.1+.Quoted 6 lines
All that needs to be done here is to move the setting of clp-
Quoted 1 line
cl_last_renewal_out_ ofnfs4_set_lease_period(), and just have
nfs4_proc_setclientid_confirm()andnfs4_update_session()calldo_renew_lease().This would also require
nfs4_setup_state_renewal()to calldo_renew_lease()I think - at least it currently callsnfs4_set_lease_period().
Also, iircfsinfo()not settingcl_last_renewalleads tocl_last_renewalinitialization issues under some circumstances.Then the RFC 7530 in section 16.34.5 states:
"SETCLIENTID_CONFIRMdoes not establish or renew a lease.", so callingdo_renew_lease()fromnfs4_setclientid_confirm()doesn't seem to be ok.I'm not sure if is is valid to do implicit lease renewal in
nfs4_update_session()either...Anyway, the patch would be something like (haven't tested it yet):
fs/nfs/nfs4proc.c +1
-4@@ -5019,16 +5019,13 @@ static int nfs4_do_fsinfo(struct nfs_server *server, struct nfs_fh *fhandle, strstruct nfs4_exception exception = { .interruptible = true, }; -unsigned long now = jiffies;int err; do { err = _nfs4_do_fsinfo(server, fhandle, fsinfo); trace_nfs4_fsinfo(server, fhandle, fsinfo->fattr, err); if (err == 0) { - nfs4_set_lease_period(server->nfs_client, - fsinfo->lease_time * HZ, - now); + nfs4_set_lease_period(server->nfs_client, fsinfo->lease_time * HZ) break;} err = nfs4_handle_exception(server, err, &exception);fs/nfs/nfs4proc.c +6
-0@@ -6146,6 +6143,10 @@ int nfs4_proc_setclientid_confirm(struct nfs_client *clp, clp->cl_clientid); status =rpc_call_sync (clp->cl_rpcclient, &msg,RPC_TASK_TIMEOUT |RPC_TASK_NO_ROUND_ROBIN ); +if (status ==0 ) { +unsigned long now = jiffies; +do_renew_lease (clp, now); + }trace_nfs4_setclientid_confirm (clp, status);dprintk ("NFS reply setclientid_confirm: %d\n" , status);return status;@@ -8590,6 +8591,8 @@ static void nfs4_update_session(struct nfs4_session *session, if (res->flags & SESSION4_BACK_CHAN)memcpy (&session->bc_attrs, &res->bc_attrs,sizeof (session->bc_attrs)); +unsigned long now = jiffies; +do_renew_lease (session->clp, now); }static int _nfs4_proc_create_session(struct
nfs_client*clp,fs/nfs/nfs4renewd.c +1
-4@@ -138,15 +138,12 @@* * @clp: pointer to nfs_client * @lease: new value for lease period -* @lastrenewed: time at which lease was last renewed */ void nfs4_set_lease_period (struct nfs_client *clp, -unsigned long lease, -unsigned long lastrenewed) +unsigned long lease) {spin_lock (&clp->cl_lock); clp->cl_lease_time = lease; - clp->cl_last_renewal = lastrenewed;spin_unlock (&clp->cl_lock);/* Cap maximum reconnect timeout at 1/2 lease period */ fs/nfs/nfs4state.c +2
-1@@ -102,7 +102,8 @@ static int nfs4_setup_state_renewal(struct nfs_client *clp) now = jiffies; status =nfs4_proc_get_lease_time (clp, &fsinfo);if (status ==0 ) { -nfs4_set_lease_period (clp, fsinfo.lease_time *HZ , now); +nfs4_set_lease_period (clp, fsinfo.lease_time *HZ ); +nfs4_do_renew_lease (clp, now);nfs4_schedule_state_renewal (clp); }But it potentially has issues, as just pointed out, mainly with not being compliant with rfc again.
Also see the comments above about the SEQUENCE.
Trond - ?Chuck, Anna - which way do you prefer, the one proposed by Chuck or the one now proposed by Trond?
Quoted 9 lines
Or perhaps my v2 patch which is least invasive and just stops doing implicit renewals for 4.0 if
cl_last_renewalis already set.I personally think the way proposed by Chuck is fully compliant with RFC and in-line with what we currently do for 4.1 via SEQUENCE, so perhaps the best option. ?
Signature · 1 line
Robert Milkowski
--
Chuck Lever
Quoted 91 lines · Chuck Lever
Quoted 82 lines · Robert Milkowski
Quoted 76 lines · Trond Myklebust
Quoted 73 lines · Anna
Hi Robert,
Quoted 61 lines · Robert Milkowski
Quoted 61 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust
<[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do implicit
lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <
[email protected]>Quoted 5 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 37 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases")
introduced implicit client lease renewal innfs4_do_fsinfo(), which
can result in the NFSv4.0 lease to expire on a server side, and servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub- mount,
then stat'ing it to get it mounted again, which will delay or even
completely prevent client from sending RENEW operations if no other
NFS operations are issued. Eventually nfs server will expire client's
lease and return an error on file access or next RENEW.This can also happen when a sub-mount is automatically unmounted due
to inactivity (afternfs_mountpoint_expiry_timeout), then it is mounted again viastat(). This can result in a short window during
which client's lease will expire on a server but not on a client.
This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one,
by adding RENEW to a compound operation issued bynfs4_do_fsinfo(),
similarly to NFSv4.1 which adds SEQUENCE operation.Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managingleases")
Signed-off-by Robert Milkowski <[email protected]>Quoted 5 lines
Quoted 3 lines
Reviewed-by Chuck Lever <[email protected]>How do we progress it further?
Thanks for following up! I have the patch included in my linux-next branch for
the next merge window.Quoted 8 lines
Quoted 4 lines
NACK. This is the wrong way to solve the problem. Lease renewal in NFSv4 does not need to be tied to fsinfo. It creates an unnecessary extra error condition that has absolutely nothing to do with the functionality of retrieving per-filesystem attributes.
Well, we also do it for NFSv4.1+ with the SEQUENCE operation being send by fsinfo, and as Chuck pointed out
it makes sense to do similarly for 4.0, which is what this patch does.
Quoted 5 lines
I did say that.
However, I can see that for NFSv4.1+, the client code handling the SEQUENCE response will update
cl_last_renewal. It does not need to be done in the fsinfo code.
I think it is the case, yes.
Quoted 6 lines
The NFSv4.0 behavior should be correct if
cl_last_renewalis not updated. That should force the client to send a separate RENEW operation so that both the client and server agree that the lease is active.
I was thinking about removing the call to update the last renewal entirely in do_fsinfo(),
however as briefly discussed back in Dec there is an issue with cl_last_renewal initialization on initial mount in 4.0
I observed it to be 0, as nfs4_setup_state_renewal() calls nfs4_proc_get_lease_time() on initial mount and if no error it calls nfs4_set_lease_period().
However with sec=krb the call to nfs4_proc_get_lease_time() returns NFS4ERR_WRONGSEC) during initial mount (which seems to be ok), which results in not setting cl_last_renewal,
which iirc prevented scheduling RENEW operations altogether. As discussed then, this is really a separate issue which should be fixed separately.
Once fixed then fsinfo() shouldn't need to set cl_last_renewal at all, still sending RENEW in fsinfo() seems like a good idea to make it more inline with what we do for 4.1.
Quoted 2 lines
If I understand Trond correctly?
The problem with what Trond proposed is that it seems to go against the rfc, although it should fix the initialization issue.
Signature · 1 line
Robert Milkowski
Quoted 93 lines · Robert Milkowski
Quoted 86 lines · Trond Myklebust
Quoted 78 lines · Anna
Hi Robert,
Quoted 75 lines · Robert Milkowski
Quoted 74 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: 30 December 2019 15:37
To: Robert Milkowski <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; Trond Myklebust
<[email protected]>; Anna Schumaker <[email protected]>; [email protected]
Subject: Re: [PATCH v3] NFSv4.0:nfs4_do_fsinfo()should not do
implicit
lease renewalsQuoted 2 lines
On Dec 30, 2019, at 10:20 AM, Robert Milkowski <
[email protected]>Quoted 6 lines · wrote:
From: Robert Milkowski <[email protected]>
Currently, each time
nfs4_do_fsinfo()is called it will do an
implicit
NFS4 lease renewal, which is not compliant with the NFS4specification.
Quoted 42 lines
This can result in a lease being expired by an NFS server.
Commit
83ca7f5ab31f("NFS: Avoid PUTROOTFH when managing leases")
introduced implicit client lease renewal innfs4_do_fsinfo(),
which
can result in the NFSv4.0 lease to expire on a server side, and
servers returningNFS4ERR_EXPIREDorNFS4ERR_STALE_CLIENTID.This can easily be reproduced by frequently unmounting a sub-
mount,
then stat'ing it to get it mounted again, which will delay or
even
completely prevent client from sending RENEW operations if no
other
NFS operations are issued. Eventually nfs server will expire
client's
lease and return an error on file access or next RENEW.This can also happen when a sub-mount is automatically unmounted due
to inactivity (afternfs_mountpoint_expiry_timeout), then it is
mounted again viastat(). This can result in a short window during
which client's lease will expire on a server but not on a client.
This specific case was observed on production systems.This patch makes an explicit lease renewal instead of an implicit one,
by adding RENEW to a compound operation issued bynfs4_do_fsinfo(),
similarly to NFSv4.1 which adds SEQUENCE operation.Quoted 3 lines
Fixes 83ca7f5ab31f ("NFS: Avoid PUTROOTFH when managingleases")
Signed-off-by Robert Milkowski <[email protected]>Reviewed-by Chuck Lever <[email protected]>Quoted 6 lines
Quoted 1 line
How do we progress it further?
Thanks for following up! I have the patch included in my linux- next
branch for
the next merge window.Quoted 4 lines
NACK. This is the wrong way to solve the problem. Lease renewal in NFSv4 does not need to be tied to fsinfo. It creates an unnecessary extra error condition that has absolutely nothing to do with the functionality of retrieving per-filesystem attributes.
Quoted 4 lines
Well, we also do it for NFSv4.1+ with the SEQUENCE operation being send by fsinfo, and as Chuck pointed out
it makes sense to do similarly for 4.0, which is what this patch does.
That's a bogus argument. The NFSv4.x SEQUENCE is mandated by the protocol in order to manage the only once semantics via the session slot mechanism. While it does implicitly renew the lease, that's not at all the reason why it is present in this particular RPC call.
Quoted 5 lines
Or as per the v2 version of the patch, not do the implicit RENEW for 4.0, which was a simpler patch,
but then not in-line with 4.1+.Quoted 1 line
All that needs to be done here is to move the setting of clp-
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
cl_last_renewal_out_ ofnfs4_set_lease_period(), and just have
nfs4_proc_setclientid_confirm()andnfs4_update_session()calldo_renew_lease().This would also require
nfs4_setup_state_renewal()to calldo_renew_lease()I think - at least it currently callsnfs4_set_lease_period().
This is the mechanism we're replacing. There is no need for a call to do_renew_lease() in nfs4_setup_state_renewal().
Quoted 8 lines
Also, iirc
fsinfo()not settingcl_last_renewalleads tocl_last_renewalinitialization issues under some circumstances.Then the RFC 7530 in section 16.34.5 states:
"SETCLIENTID_CONFIRMdoes not establish or renew a lease.", so callingdo_renew_lease()fromnfs4_setclientid_confirm()doesn't seem to be ok.
So just move the do_renew_lease() to nfs4_proc_setclientid(). The successful combination SETCLIENTID+SETCLIENTID_CONFIRM definitely _does_ establish a lease.
Quoted 2 lines
I'm not sure if is is valid to do implicit lease renewal in
nfs4_update_session()either...
Ditto. Move to the exchange_id call.
Quoted 29 lines
Anyway, the patch would be something like (haven't tested it yet):
fs/nfs/nfs4proc.c +1
-4@@ -5019,16 +5019,13 @@ static int nfs4_do_fsinfo(struct nfs_server *server, struct nfs_fh *fhandle, strstruct nfs4_exception exception = { .interruptible = true, }; -unsigned long now = jiffies;int err; do { err = _nfs4_do_fsinfo(server, fhandle, fsinfo); trace_nfs4_fsinfo(server, fhandle, fsinfo->fattr, err); if (err == 0) { - nfs4_set_lease_period(server->nfs_client, - fsinfo->lease_time * HZ, - now); + nfs4_set_lease_period(server->nfs_client, fsinfo->lease_time * HZ)break; } err = nfs4_handle_exception(server, err, &exception);
fs/nfs/nfs4proc.c int nfs4_proc_setclientid_confirm(struct · line 6149
Quoted 7 lines
fs/nfs/nfs4proc.c +2
-0@@ -6146,6 +6143,10 @@ int nfs4_proc_setclientid_confirm(struct nfs_client *clp, clp->cl_clientid); status =rpc_call_sync (clp->cl_rpcclient, &msg,RPC_TASK_TIMEOUT |RPC_TASK_NO_ROUND_ROBIN ); +if (status ==0 ) { +unsigned long now = jiffies;
The variable 'now' needs to be set before the RPC call in order to avoid overestimating the remaining lease period.
Quoted 41 lines
fs/nfs/nfs4proc.c +2
-0+do_renew_lease (clp, now); + }trace_nfs4_setclientid_confirm (clp, status);dprintk("NFS reply setclientid_confirm: %d\n", status); return status;fs/nfs/nfs4proc.c +2
-0@@ -8590,6 +8591,8 @@ static void nfs4_update_session(struct nfs4_session *session, if (res->flags & SESSION4_BACK_CHAN)memcpy (&session->bc_attrs, &res->bc_attrs,sizeof (session->bc_attrs)); +unsigned long now = jiffies; +do_renew_lease (session->clp, now); }static int _nfs4_proc_create_session(struct
nfs_client*clp,fs/nfs/nfs4renewd.c +1
-4@@ -138,15 +138,12 @@* * @clp: pointer to nfs_client * @lease: new value for lease period -* @lastrenewed: time at which lease was last renewed */ void nfs4_set_lease_period (struct nfs_client *clp, -unsigned long lease, -unsigned long lastrenewed) +unsigned long lease) {spin_lock (&clp->cl_lock); clp->cl_lease_time = lease; - clp->cl_last_renewal = lastrenewed;spin_unlock (&clp->cl_lock);/* Cap maximum reconnect timeout at 1/2 lease period */ fs/nfs/nfs4state.c +0
-0
fs/nfs/nfs4state.c static int nfs4_setup_state_renewal(struct · line 108
Quoted 8 lines
fs/nfs/nfs4state.c +2
-1@@ -102,7 +102,8 @@ static int nfs4_setup_state_renewal(struct nfs_client *clp) now = jiffies; status =nfs4_proc_get_lease_time (clp, &fsinfo);if (status ==0 ) { -nfs4_set_lease_period (clp, fsinfo.lease_time *HZ , now); +nfs4_set_lease_period (clp, fsinfo.lease_time *HZ ); +nfs4_do_renew_lease (clp, now);
As stated above, this call to nfs4_do_renew_lease() is not needed.
Quoted 7 lines
fs/nfs/nfs4state.c +0
-0nfs4_schedule_state_renewal (clp);}But it potentially has issues, as just pointed out, mainly with not being compliant with rfc again.
See above.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
All that needs to be done here is to move the setting of clp-
Quoted 1 line
cl_last_renewal_out_ ofnfs4_set_lease_period(), and just have
nfs4_proc_setclientid_confirm()andnfs4_update_session()calldo_renew_lease().
Quoted 6 lines
Quoted 3 lines
This would also require
nfs4_setup_state_renewal()to calldo_renew_lease()I think - at least it currently callsnfs4_set_lease_period().This is the mechanism we're replacing. There is no need for a call to
do_renew_lease()innfs4_setup_state_renewal().
ok
Quoted 10 lines
Quoted 9 lines
Also, iirc
fsinfo()not settingcl_last_renewalleads tocl_last_renewalinitialization issues under some circumstances.Then the RFC 7530 in section 16.34.5 states:
"SETCLIENTID_CONFIRMdoes not establish or renew a lease.", so callingdo_renew_lease()fromnfs4_setclientid_confirm()doesn't seem to be ok.
Quoted 4 lines
So just move the
do_renew_lease()tonfs4_proc_setclientid(). The successful combination SETCLIENTID+SETCLIENTID_CONFIRM definitely _does_ establish a lease.
ok
Quoted 5 lines
Quoted 2 lines
I'm not sure if is is valid to do implicit lease renewal in
nfs4_update_session()either...Ditto. Move to the
exchange_idcall.
ok
Quoted 3 lines
Quoted 3 lines
Anyway, the patch would be something like (haven't tested it yet):
...
Quoted 12 lines
Review int nfs4_proc_setclientid_confirm(struct · line 6149
Quoted 8 lines
(quoted diff) +2
-0@@ -6146,6 +6143,10 @@ int nfs4_proc_setclientid_confirm(struct nfs_client *clp, clp->cl_clientid); status = rpc_call_sync(clp->cl_rpcclient, &msg, RPC_TASK_TIMEOUT | RPC_TASK_NO_ROUND_ROBIN); + if(status == 0) { + unsigned long now = jiffies;The variable 'now' needs to be set before the RPC call in order to avoid overestimating the remaining lease period.
yes, thx.
I will get a new patch tested and submitted here this week.
Signature · 1 line
Robert Milkowski