NFSv4: open() should try lease recovery on NFS4ERR_EXPIRED
Subsystems (2)
Hi,
Currently if nfsv4 lease expires on a server Linux client will return EIO to open and won't try to recover.
For example:
$ date; strace -t -e trace=open head /mnt/3/var/log/vmware-vmsvc.log
Quoted 1 line
/dev/null
Mon 16 Dec 17:37:16 GMT 2019
...
17:37:16 open("/mnt/3/var/log/vmware-vmsvc.log", O_RDONLY) = -1 EIO (Input/output error)
...
The network/nfs traffic:
2638 17:37:16.286642918 10.50.2.57 -> 10.50.2.59 NFS 246 V4 Call ACCESS
FH: 0x62d40c52, [Check: RD LU MD XT DL]
2639 17:37:16.286929287 10.50.2.59 -> 10.50.2.57 NFS 194 V4 Reply (Call In 2638) ACCESS, [Access Denied: MD XT DL], [Allowed: RD LU]
2640 17:37:16.287295942 10.50.2.57 -> 10.50.2.59 NFS 258 V4 Call ACCESS
FH: 0xeddb7439, [Check: RD LU MD XT DL]
2641 17:37:16.287448815 10.50.2.59 -> 10.50.2.57 NFS 194 V4 Reply (Call In 2640) ACCESS, [Access Denied: MD XT DL], [Allowed: RD LU]
2642 17:37:16.287666576 10.50.2.57 -> 10.50.2.59 NFS 266 V4 Call ACCESS
FH: 0x8503e2cd, [Check: RD LU MD XT DL]
2643 17:37:16.287786851 10.50.2.59 -> 10.50.2.57 NFS 194 V4 Reply (Call In 2642) ACCESS, [Access Denied: MD XT DL], [Allowed: RD LU]
2644 17:37:16.287984808 10.50.2.57 -> 10.50.2.59 NFS 350 V4 Call OPEN DH:
0x8503e2cd/vmware-vmsvc.log
2645 17:37:16.288194276 10.50.2.59 -> 10.50.2.57 NFS 122 V4 Reply (Call In 2644) OPEN Status: NFS4ERR_EXPIRED
Both Linux and Solaris NFS servers return NFS4ERR_EXPIRED if nfsv4 lease expire.
Recent ONTAP versions return NFS4ERR_STALE_CLIENTID which is handled correctly.
The patch changes handling of NFS4ERR_EXPIRED error to also try to recover.
The issue can be triggered by exporting over NFSv4 a filesystem with a sub-filesystem,
for example by exporting / and /var (assuming /var is a separate filesystem).
How to reproduce the issue:
On an NFS server run:
# cat /etc/exports / *(rw,sync) /var *(rw,sync)
On a Linux NFS client:
# mount -o vers=4 10.50.2.59:/ /mnt/3
$ date; strace -t -e trace=open head /mnt/3/var/log/vmware-vmsvc.log
Quoted 1 line
/dev/null
Mon 16 Dec 17:28:45 GMT 2019
...
17:28:45 open("/mnt/3/var/log/vmware-vmsvc.log", O_RDONLY) = 3
...
Now, on the NFS client run:
# while [ 1 ]; do date; umount /mnt/3/var; ls /mnt/3/var >/dev/null; sleep 10; done ...
Because of another bug (see email with subject: [PATCH] NFSv4:nfs4_do_fsinfo() should not do implicit lease renewals)
the above while loop will prevent the NFS client from sending RENEW operations to server as it currently assumes an implicit
lease renewal which I believe is not compliant with the RFC. If you now wait long enough for the NFS server to expire the lease,
and then try to open a file (with no other NFS activity) it should result in server sending NFS4ERR_EXPIRED and the NFS LINUX client
will return EIO to an application as shown above.
This also gets triggered if a sub-mount in automatically unmounted after nfs_mountpoint_expiry_timeout, then shortly after there is some access to it which will trigger for it to be automatically mounted again which will delay the client in sending the RENEW operation. This results in a short window during which open() will return EIO to any files from the file server.
This was tested with 5.5.0-rc2 and the provided patch is applied on top of the 5.5.0-rc2 as well.
Best regards,
Robert Milkowski
Quoted 17 lines · Robert Milkowski
From: Robert Milkowski <[email protected]>
Currently, if an nfs server returns
NFS4ERR_EXPIREDto open, we return EIO to applications without even trying to recover.Signed-off-by Robert Milkowski <[email protected]>--- fs/nfs/nfs4proc.c | 4 ++++ 1 file changed, 4 insertions(+)fs/nfs/nfs4proc.c +0
-0@@ -481,6 +481,10 @@ static int nfs4_do_handle_exception(struct
nfs_server
*server,
fs/nfs/nfs4proc.c
Quoted 10 lines
fs/nfs/nfs4proc.c +4
-0stateid); goto wait_on_recovery; } +if (state ==NULL ) { +nfs4_schedule_lease_recovery (clp); + goto wait_on_recovery; + }/* Fall through */ case -NFS4ERR_OPENMODE :if (inode) {
Ugh. If we need a state==NULL for NFSv4.0, then let's get rid of the switch() fall through in the above case, and rather add a delegation return to the EXPIRED handling.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
PATCH 1/1 NFSv4: open() should try lease recovery on NFS4ERR_EXPIRED
From: Robert Milkowski <[email protected]>
Currently, if an nfs server returns NFS4ERR_EXPIRED to open, we return EIO to applications without even trying to recover.
--- fs/nfs/nfs4proc.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/nfs/nfs4proc.c +0 -0
@@ -481,6 +481,10 @@ static int nfs4_do_handle_exception(struct nfs_server
*server,
stateid);
goto wait_on_recovery;
}
+ if (state == NULL) {
+ nfs4_schedule_lease_recovery(clp);
+ goto wait_on_recovery;
+ }
/* Fall through */
case -NFS4ERR_OPENMODE:
if (inode) {
Signature · 1 line
1.8.3.1