persistent, quasi-random -ESTALE at mount time
Subsystems (2)
For what it's worth, I'm seeing the same problem.
The server was just rebooted with 2.6.38-rc3, and the client reported "STALE NFS file handle". I wish I understood why; I thought the point of a stateless protocol was that it could survive server reboots.
Anyway, I found all the affected processes, killed them, unmounted, tried to remount, and lo and behold:
mount("server:/path/dir", "/client/dir", "nfs", MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC, "addr=ww.xx.yy.zz,vers=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=2050") = -1 ESTALE (Stale NFS file handle)
The server is just logging
send(10, "<29>Feb 1 22:39:49 mountd[4167]: authenticated mount request from $CLIENT:912 for /path/dir (/path/dir)", 125, MSG_NOSIGNAL) = 125
/proc/fs/nfs/exports is reporting:
/path/dir *.dom.ain,client.dom.ain(ro,root_squash,async,wdelay,no_subtree_check,uuid=3210ba59:586b43f2:8780109f:d915f4ab)
Debian packaged nfs utilities 1.2.2-4 on both server and client. 32 bits in both cases. (Server is running a 64-bit kernel, but 32-bit userland.)
It worked immediately before the reboot (when the server was runnign 2.6.26-rcX).
"exportfs -a" several times did NOT fix it, but restarting mountd and nfsd ("/etc/init.d/nfs-kernel-server restart") fixed it.
Anyway, quite annoying. Unfortunately, that's a server I don't like to reboot. (But I can restart the NFS server safely if that would help testing.)
Quoted 23 lines · George Spelvin
For what it's worth, I'm seeing the same problem.
The server was just rebooted with 2.6.38-rc3, and the client reported "STALE NFS file handle". I wish I understood why; I thought the point of a stateless protocol was that it could survive server reboots.
Anyway, I found all the affected processes, killed them, unmounted, tried to remount, and lo and behold:
mount("server:/path/dir", "/client/dir", "nfs", MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC, "addr=ww.xx.yy.zz,vers=3,proto=tcp,mountvers=3,mountproto=tcp,mountport=2050") = -1 ESTALE (Stale NFS file handle)
The server is just logging
send(10, "<29>Feb 1 22:39:49 mountd[4167]: authenticated mount request from $CLIENT:912 for /path/dir (/path/dir)", 125,MSG_NOSIGNAL) = 125
/proc/fs/nfs/exportsis reporting:
/path/dir *.dom.ain,client.dom.ain(ro,root_squash,async,wdelay,no_subtree_check,uuid=3210ba59:586b43f2:8780109f:d915f4ab)Debian packaged nfs utilities 1.2.2-4 on both server and client. 32 bits in both cases. (Server is running a 64-bit kernel, but 32-bit userland.)
It worked immediately before the reboot (when the server was runnign 2.6.26-rcX).
Quoted 6 lines
"exportfs -a" several times did NOT fix it, but restarting mountd and nfsd ("
/etc/init.d/nfs-kernel-serverrestart") fixed it.Anyway, quite annoying. Unfortunately, that's a server I don't like to reboot. (But I can restart the NFS server safely if that would help testing.)
So the reboot was for an upgrade from 2.6.26-rcX to 2.6.38-rc3? I wonder if a reboot (or just a server restart) without changing kernels would see the same problem?
In which case some change in the filehandle format or perhaps in the way the uuid's are calculated might explain the problem.
We work quite hard to ensure that filehandles returned from older nfsd's will still be accepted by newer ones. But that doesn't mean there couldn't failed at that somehow in some case....
If you manage to reproduce the problem, /proc/fs/nfs/exports before and after the reboot would be interesting, and ideally also a network trace showing traffic before and after the reboot (including the operation that returned the STALE error).
--b.
Quoted 3 lines
So the reboot was for an upgrade from 2.6.26-rcX to 2.6.38-rc3? I wonder if a reboot (or just a server restart) without changing kernels would see the same problem?
Whoops, typo. It was from 2.6.36-rcX (I think -rc5, but it's scrolled off the logs), not .26.
Quoted 3 lines
We work quite hard to ensure that filehandles returned from older nfsd's will still be accepted by newer ones. But that doesn't mean there couldn't failed at that somehow in some case....
I understand that sometimes there's an incompatible server change, but I don't ever remember a Linux-linux nfs mount surviving a server reboot.
However, the problem I'm complaining about here is more alarming. With a clean client, attempting to *mount* is failing with -ESTALE.
Quoted 4 lines
If you manage to reproduce the problem,
/proc/fs/nfs/exportsbefore and after the reboot would be interesting, and ideally also a network trace showing traffic before and after the reboot (including the operation that returned the STALE error).
Can do. How much detail do you want in the packet trace? Is -vvv enough, or do you want -X as well?
Thank you very much for the response; I'll try to reproduce and capture the problem.
Signature · 1 line
-Colib
Quoted 11 lines · George Spelvin
Quoted 3 lines
So the reboot was for an upgrade from 2.6.26-rcX to 2.6.38-rc3? I wonder if a reboot (or just a server restart) without changing kernels would see the same problem?
Whoops, typo. It was from 2.6.36-rcX (I think -rc5, but it's scrolled off the logs), not .26.
Quoted 3 lines
We work quite hard to ensure that filehandles returned from older nfsd's will still be accepted by newer ones. But that doesn't mean there couldn't failed at that somehow in some case....
Quoted 6 lines
I understand that sometimes there's an incompatible server change, but I don't ever remember a Linux-linux nfs mount surviving a server reboot.
However, the problem I'm complaining about here is more alarming. With a clean client, attempting to *mount* is failing with
-ESTALE.
Oh, apologies, I missed that.
Quoted 7 lines
Quoted 4 lines
If you manage to reproduce the problem,
/proc/fs/nfs/exportsbefore and after the reboot would be interesting, and ideally also a network trace showing traffic before and after the reboot (including the operation that returned the STALE error).Can do. How much detail do you want in the packet trace? Is -vvv enough, or do you want -X as well?
Actually, the raw packet data would be most useful; so something like:
tcpdump -s0 -wtmp.pcap
then send me tmp.pcap.
And the contents of
/proc/net/rpc/nfsd.fh/content
/proc/net/rpc/nfsd.export/content
after the failure.
--b.
Quoted 4 lines · George Spelvin
Quoted 3 lines
So the reboot was for an upgrade from 2.6.26-rcX to 2.6.38-rc3? I wonder if a reboot (or just a server restart) without changing kernels would see the same problem?
Quoted 8 lines
Whoops, typo. It was from 2.6.36-rcX (I think -rc5, but it's scrolled off the logs), not .26.
Quoted 3 lines
We work quite hard to ensure that filehandles returned from older nfsd's will still be accepted by newer ones. But that doesn't mean there couldn't failed at that somehow in some case....
I understand that sometimes there's an incompatible server change, but
Separate from the current problem, but: no, that shouldn't be true. If you find a case where a kernel upgrade causes clients to fail, that's a bug and we'd like to know about it.
Quoted 2 lines
I don't ever remember a Linux-linux nfs mount surviving a server reboot.
Ouch. I do test that regularly, and haven't seen problems. Again, if you have reproduceable problem, I'd appreciate details.--b.
On 3 Feb 2011, J. Bruce Fields stated:
Quoted 2 lines
Ouch. I do test that regularly, and haven't seen problems. Again, if you have reproduceable problem, I'd appreciate details.--b.
I can give you a packet dump from my (completely reproducible) failure with the same symptoms, if you like. (Again, it is fixed by running exportfs repeatedly, each time fixing another few mounts: currently, I have to run it six times.)
Quoted 9 lines · Nix
On 3 Feb 2011, J. Bruce Fields stated:
Quoted 2 lines
Ouch. I do test that regularly, and haven't seen problems. Again, if you have reproduceable problem, I'd appreciate details.--b.
I can give you a packet dump from my (completely reproducible) failure with the same symptoms, if you like. (Again, it is fixed by running exportfs repeatedly, each time fixing another few mounts: currently, I have to run it six times.)
That would be great, thanks. And could you collect the contents of /proc/net/rpc/{nfsd.fh,nfsd.export}/content at the same time? (I think you already sent some; but I'd be curious to match them up with the packet trace.) Also, the output of "blkid" might help diagnose any problems with uuid's.
(Also, I think it's probably a totally different problem, but I found a problem yesterday with out-of-date uuid information that appears to be due to selinux preventing rpc.mountd from opening block devices; are you using selinux, and if so does turning it off (setenforce 0) help?)
--b.
On 3 Feb 2011, J. Bruce Fields uttered the following:
Quoted 11 lines · Nix
Quoted 4 lines
I can give you a packet dump from my (completely reproducible) failure with the same symptoms, if you like. (Again, it is fixed by running exportfs repeatedly, each time fixing another few mounts: currently, I have to run it six times.)
That would be great, thanks. And could you collect the contents of
/proc/net/rpc/{nfsd.fh,nfsd.export}/contentat the same time? (I think you already sent some; but I'd be curious to match them up with the packet trace.) Also, the output of "blkid" might help diagnose any problems with uuid's.
I jinxed it. I just recompiled and the problem went away. The only differences between the old and new kernel are a so-far-entirely- unconfigured IPv6 setup (so nothing but link-local private addresses, no routing, nothing) and a new compiler (4.5.x rather than 4.4.x).
Which of these caused the change in behaviour I do not yet know, but will try to figure out (by turning IPv6 off again and seeing if the problem recurs).
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 13 lines · Nix
On 3 Feb 2011, J. Bruce Fields uttered the following:
Quoted 11 lines · Nix
Quoted 4 lines
I can give you a packet dump from my (completely reproducible) failure with the same symptoms, if you like. (Again, it is fixed by running exportfs repeatedly, each time fixing another few mounts: currently, I have to run it six times.)
That would be great, thanks. And could you collect the contents of
/proc/net/rpc/{nfsd.fh,nfsd.export}/contentat the same time? (I think you already sent some; but I'd be curious to match them up with the packet trace.) Also, the output of "blkid" might help diagnose any problems with uuid's.
Quoted 8 lines
I jinxed it. I just recompiled and the problem went away. The only differences between the old and new kernel are a so-far-entirely- unconfigured IPv6 setup (so nothing but link-local private addresses, no routing, nothing) and a new compiler (4.5.x rather than 4.4.x).
Which of these caused the change in behaviour I do not yet know, but will try to figure out (by turning IPv6 off again and seeing if the problem recurs).
OK, thanks.
--b.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · David Brodbeck
Quoted 3 lines · George Spelvin
I don't ever remember a Linux-linux nfs mount surviving a server reboot.
Quoted 7 lines
I do it semi-regularly and it *usually* works...but I never reboot the server unless I'm prepared to reboot hung clients, just in case.
Actually, I've seen more problems with Solaris-Linux NFSv4 mounts. For a while there was a pretty good chance that rebooting a server would result in state recovery failure loops on the client. I see less of that lately, but I'm also doing fewer server reboots...
There have been some recovery bugs fixed; if you know of any more, details would be welcomed.
--b.