Skip to contentSkip to conversation

4.0 client and server restart with decreased lease time

PATCH New Donald Buczek 2 participants 3 messages Feb 7, 2019 – Feb 14, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

The nfsd default lease time has been changed from 90 seconds to 45 seconds between Linux 4.14 and 4.19 by commit d6ebf5088f09 ("nfsd4:
return default lease period").

After we did an upgrade of a nfs server from 4.14 to 4.19, we noticed a failing process and dmesg logs "NFS: nfs4_reclaim_open_state: Lock reclaim failed!" on a client (Linux 4.14.87). The client had the file system mounted with vers=4.0.

Network trace indicated, that the client continued to use the 90 seconds lease period of the previous server incarnation and sent RENEWs every 60 seconds (2/3 of 90 seconds). Sometimes the server answered with NFS4ERR_EXPIRED.

When this happened, the client executed recovery (SETCLIENTID...) but did non query the server for a new lease_time. So the problem was persistent even after the first failure.

As an experiment, I've also restarted a server with the lease time decrement from 90 to 45 seconds, but the grace period fixed to 90 seconds. Now the client got NFS4ERR_STALE_CLIENTID but still did not query the server for a new lease_time and continued to send RENEWs in 60 second intervals.

At least for the later case, the RFC say, a client should refetch the lease_time:

Quoted 8 lines

A server may, upon restart, establish a new value for the lease period. Therefore, clients should, once a new client ID is established, refetch the lease_time attribute and use it as the basis for lease renewal for the lease associated with that server. However, the server must establish, for this restart event, a grace period at least as long as the lease period for the previous server instantiation. This allows the client state obtained during the previous server instance to be reliably re-established.

[ https://tools.ietf.org/html/rfc7530 ]

I understand that a restart with a grace period smaller than the previous lease time is never save.

Aside from that, is a server restart with a decreased lease time supposed to be supported by the Linux client? If not, this is kind of a trap for server upgrades when just relying on the defaults.

Best
Donald

#1 lore raw
Quoted 8 lines · Donald Buczek

The nfsd default lease time has been changed from 90 seconds to 45 seconds between Linux 4.14 and 4.19 by commit d6ebf5088f09 ("nfsd4:
return default lease period").

After we did an upgrade of a nfs server from 4.14 to 4.19, we noticed a failing process and dmesg logs "NFS: nfs4_reclaim_open_state: Lock reclaim failed!" on a client (Linux 4.14.87). The client had the file system mounted with vers=4.0.

Argh. My fault for changing that default without thinking of the impact on a server upgrade on existing client mounts, sorry.

Quoted 8 lines

Network trace indicated, that the client continued to use the 90 seconds lease period of the previous server incarnation and sent RENEWs every 60 seconds (2/3 of 90 seconds). Sometimes the server answered with NFS4ERR_EXPIRED.

When this happened, the client executed recovery (SETCLIENTID...) but did non query the server for a new lease_time. So the problem was persistent even after the first failure.

That certainly sounds like a client bug, though.

Quoted 18 lines

As an experiment, I've also restarted a server with the lease time decrement from 90 to 45 seconds, but the grace period fixed to 90 seconds. Now the client got NFS4ERR_STALE_CLIENTID but still did not query the server for a new lease_time and continued to send RENEWs in 60 second intervals.

At least for the later case, the RFC say, a client should refetch the lease_time:

Quoted 8 lines

A server may, upon restart, establish a new value for the lease period. Therefore, clients should, once a new client ID is established, refetch the lease_time attribute and use it as the basis for lease renewal for the lease associated with that server. However, the server must establish, for this restart event, a grace period at least as long as the lease period for the previous server instantiation. This allows the client state obtained during the previous server instance to be reliably re-established.

Quoted 8 lines

[ https://tools.ietf.org/html/rfc7530 ]

I understand that a restart with a grace period smaller than the previous lease time is never save.

Aside from that, is a server restart with a decreased lease time supposed to be supported by the Linux client? If not, this is kind of a trap for server upgrades when just relying on the defaults.

The client should be following the RFC and requesting the lease period again after restart.

With that fixed, it may still fail to reclaim locks in this case, since the upgraded server isn't providing a 90-second grace period as it should, but that should at least avoid problems caused by it continuing to renew only at 60-second intervals.

The server shouldn't have changed the defaults. The only safe way to decrease the lease period is to do it from userspace (e.g., change NFSD_V4_LEASE value in /etc/sysconfig/nfs, restart, wait at least 90 seconds for every client to have a chance to get the new lease time, then change NFSD_V4_GRACE). The kernel on its own can't do this safely since it doesn't know how to find the old value of the lease time.

--b.

#2 lore raw

PATCH 1/1 Revert "nfsd4: return default lease period"

From: "J. Bruce Fields" <[email protected]>

This reverts commit d6ebf5088f09472c1136cd506bdc27034a6763f8.

I forgot that the kernel's default lease period should never be decreased!

After a kernel upgrade, the kernel has no way of knowing on its own what the previous lease time was. Unless userspace tells it otherwise, it will assume the previous lease period was the same.

So if we decrease this value in a kernel upgrade, we end up enforcing a grace period that's too short, and clients will fail to reclaim state in time. Symptoms may include EIO and log messages like "NFS:
nfs4_reclaim_open_state: Lock reclaim failed!"

There was no real justification for the lease period decrease anyway.

Fixes d6ebf5088f09 "nfsd4: return default lease period"
---
 fs/nfsd/nfsctl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 6 lines · bfields
Quoted 5 lines · Donald Buczek

As an experiment, I've also restarted a server with the lease time decrement from 90 to 45 seconds, but the grace period fixed to 90 seconds. Now the client got NFS4ERR_STALE_CLIENTID but still did not query the server for a new lease_time and continued to send RENEWs in 60 second intervals.

So this doesn't fix that client bug, but it does fix the server.

fs/nfsd/nfsctl.c +2 -2
index b33f9785b756..72a7681f4046 100644
@@ -1239,8 +1239,8 @@ static __net_init int nfsd_init_net(struct net *net)
 	retval = nfsd_idmap_init(net);
 	if (retval)
 		goto out_idmap_error;
-	nn->nfsd4_lease = 45;	/* default lease time */
-	nn->nfsd4_grace = 45;
+	nn->nfsd4_lease = 90;	/* default lease time */
+	nn->nfsd4_grace = 90;
 	nn->somebody_reclaimed = false;
 	nn->clverifier_counter = prandom_u32();
 	nn->clientid_counter = prandom_u32();
Signature · 1 line
2.20.1