nfs_statfs: statfs error = 116
martin.knoblauch
Hi,
sorry if OT, but what is above message trying to tell me? Where can I
find a translation of the numbers? We are seeing 116 very frequently, 512 and 5 on occasion.
We have a bunch of Linux clients (Dual P4, RH7.3, 2.4.20-18.7smp
errata kernel) hanging off two Sun NFS Servers (Solaris 8) in a Veritas/VCS HA configuration. All of the clients show the 116 messages, while some of them show the 512 in addition. Those with the 512s seem to "hang" for some periods of time.
The mounts are "vers=3,proto=tcp,hard,intr,bg". Some of them mounted
at boottime, quite a few via "amd".
Any ideas are welcome.
Thanks
Martin
Quoted 6 lines · martin.knoblauch
Hi,
sorry if OT, but what is above message trying to tell me? Where can I
find a translation of the numbers? We are seeing 116 very frequently, 512 and 5 on occasion.
ESTALE is "errno" 116
EIO is "errno" 5
ERESTARTSYS is "errno" 512
You can find these in /usr/include/asm/errno.h (not good to directly include in a program).
The program reporting these errors should have included:
<errno.h>
<string.h>
Then used...
strerror(errno);
or
perror("");etc.
Errno 512 should never be seen by user-mode program, so the header file, /usr/include/linux/errno.h, states...
ESTALE happens when a mounted file-system is on a server that went down or re-booted. The file-handles are then "stale".
EIO is a general catch-all for an I/O error.
ERESTARTSYS is the error returned by a server that has re-booted that is supposed to tell the client-side software to get a new file-handle because of an attempt to access with a stale file-handle. When getting this error, the client should have reopened the file(s) to obtain a new handle.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.
"Richard B. Johnson" <[email protected]> wrote on 11/13/2003 03:39:53 PM:
Quoted 13 lines · martin.knoblauch
Quoted 7 lines
Hi,
sorry if OT, but what is above message trying to tell me? Where can I
find a translation of the numbers? We are seeing 116 very frequently, 512 and 5 on occasion.ESTALE is "errno" 116
EIO is "errno" 5
ERESTARTSYS is "errno" 512
Quoted 8 lines
You can find these in
/usr/include/asm/errno.h(not good to directly include in a program).The program reporting these errors should have included:
<
errno.h>
<string.h>
The messages actually come out of the kernel-nfs code (inode.c). Should
have mentioned "dmesg" :-)
Quoted 10 lines
Then used...
strerror(errno);
or
perror("");
etc.Errno 512 should never be seen by user-mode program, so the header file,
/usr/include/linux/errno.h, states...
This worries me a bit :-)
Quoted 3 lines
ESTALE happens when a mounted file-system is on a server that went down or re-booted. The file-handles are then "stale".
I am "alomost" sure that there were no reboot or failover events at the
time of most of the stale messages. But I'm not going to lay my hand on the book for that.
Quoted 8 lines
EIO is a general catch-all for an I/O error.
ERESTARTSYS is the error returned by a server that has re-booted that is supposed to tell the client-side software to get a new file-handle because of an attempt to access with a stale file-handle. When getting this error, the client should have reopened the file(s) to obtain a new handle.
Definitely no server reboot or HA Failover at the time of the messages.
Thanks
Martin
Quoted 4 lines · On Thursday 13 November 2003 08:52, [email protected] wrote:
"Richard B. Johnson" <[email protected]> wrote on 11/13/2003 03:39:53
PM:
Quoted 1 line · martin.knoblauch
[snip]
Quoted 6 lines
Quoted 2 lines
ESTALE happens when a mounted file-system is on a server that went down or re-booted. The file-handles are then "stale".
I am "alomost" sure that there were no reboot or failover events at the
time of most of the stale messages. But I'm not going to lay my hand on the book for that.
ESTALE should occur whenever the client looses connection to the server, or thinks it has lost connection. It isn't directly related to the server other than the fact that a server reboot will also cause it to happen.
This should be a transient failure that recovers when communication verified from some of the timeouts/retries associated with NFS.
At worst, it can require a remount of the NFS volumn.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jesse Pollard <[email protected]> writes:
> ESTALE should occur whenever the client looses connection to > the server, or thinks it has lost connection.
No it should not.
Cheers,
Trond
Trond Myklebust <[email protected]> wrote on 11/13/2003 09:34:55 PM:
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Jesse Pollard <[email protected]> writes:
> ESTALE should occur whenever the client looses connection to > the server, or thinks it has lost connection.
No it should not.
Cheers,
Trond
Hi Trond,
just by incident I found one reason when an user space application can get
the ESTALE in our setup (Linux client RH-2.4.20-18.7smp, Solaris 2.8 Server). I accidentally run iozone on two clients with the output file being the same and residing on the NFS Server. Pure luser error, but it produced ESTALE pretty much reproducibly.
B^HCheers
Martin
--
Martin Knoblauch
Senior System Architect
MSC.software GmbH
Am Moosfeld 13
D-81829 Muenchen, Germany
e-mail: [email protected]
http://www.mscsoftware.com
Phone/Fax: +49-89-431987-189 / -7189
Mobile: +49-174-3069245
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Martin Knoblauch <[email protected]> writes:
> I accidentally run iozone on two clients with the output file > being the same and residing on the NFS Server. Pure luser > error, but it produced ESTALE pretty much reproducibly.
Sure. This is a prime example of where ESTALE *is* appropriate. One NFS client is deleting a file on the server while the other is still using it.
In the NFSv2/v3 protocols, the assumption is that filehandles are valid for the entire lifetime of the file on the server. IOW only "unlink()" can cause a valid filehandle to become stale. This is mainly because there is no notion of open()/close(), so the server would never be capable of determining when your client has stopped using the filehandle.
If your 2 processes were running on the same machine, you would have seen the kernel temporarily rename your file to .nfsXXXXXX in order to work around the above problem. Delete that file, and you will generate ESTALE reproducibly too....
Cheers,
Trond
Trond Myklebust <[email protected]> wrote on 11/14/2003 02:49:31 PM:
Quoted 17 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Martin Knoblauch <[email protected]> writes:
> I accidentally run iozone on two clients with the output file > being the same and residing on the NFS Server. Pure luser > error, but it produced ESTALE pretty much reproducibly.
Sure. This is a prime example of where ESTALE *is* appropriate. One NFS client is deleting a file on the server while the other is still using it.
In the NFSv2/v3 protocols, the assumption is that filehandles are valid for the entire lifetime of the file on the server. IOW only "
unlink()" can cause a valid filehandle to become stale. This is mainly because there is no notion of open()/close(), so the server would never be capable of determining when your client has stopped using the filehandle.
Quoted 7 lines
If your 2 processes were running on the same machine, you would have seen the kernel temporarily rename your file to .nfsXXXXXX in order to work around the above problem. Delete that file, and you will generate ESTALE reproducibly too....
Cheers,
Trond
Trond,
cool. Great explanation. Always good if you can get those that know into
talking :-)
Cheers
Martin
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> ESTALE happens when a mounted file-system is on a server that > went down or re-booted. The file-handles are then "stale".
Sort of. It means that the server is unable to find the file that corresponds to the filehandle that the client sent it. If the server strictly follows the NFS specs, then this is only supposed to happen if somebody else has deleted the file (and this is why designing a scheme for generating filehandles is such a difficult job).
Some broken servers do, however, "lose" the file in other interesting and unpredictable ways.
> ERESTARTSYS is the error returned by a server that has > re-booted that is supposed to tell the client-side software to > get a new file-handle because of an attempt to access with a > stale file-handle. When getting this error, the client should > have reopened the file(s) to obtain a new handle.
ERESTARTSYS actually just means that a signal was received while inside a system call. If this results in a interruption of that syscall, the kernel is supposed to translate ERESTARTSYS into the user error EINTR.
Userland should therefore never have to handle ERESTARTSYS errors.
Cheers,
Trond
Quoted 20 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
> ESTALE happens when a mounted file-system is on a server that > went down or re-booted. The file-handles are then "stale".
Sort of. It means that the server is unable to find the file that corresponds to the filehandle that the client sent it. If the server strictly follows the NFS specs, then this is only supposed to happen if somebody else has deleted the file (and this is why designing a scheme for generating filehandles is such a difficult job).
Some broken servers do, however, "lose" the file in other interesting and unpredictable ways.
> ERESTARTSYS is the error returned by a server that has > re-booted that is supposed to tell the client-side software to > get a new file-handle because of an attempt to access with a > stale file-handle. When getting this error, the client should > have reopened the file(s) to obtain a new handle.
Quoted 7 lines
ERESTARTSYS actually just means that a signal was received while inside a system call. If this results in a interruption of that syscall, the kernel is supposed to translate ERESTARTSYS into the user error EINTR.
Userland should therefore never have to handle ERESTARTSYS errors.
Hmmm, Maybe I'm getting confused by all the winning-lottery messages, but it's in the syscall specifications for connect() and even fcntl(). http:/www.infran.ru/Techinfo/syscalls/syscalls_43.html
Also, maybe Linux now claims exclusive ownership and keeps it internal, but some networking software, nfsd and pcnfsd, might not know about that. I've seen ERESTARTSYS returned from a DOS (actually FAT) file-handle use after a server has crashed and come back on-line.
Moot point, though, the reported errors were internal via syslog, which was not previously known when I responded.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Richard B Johnson <[email protected]> writes:
ERESTARTSYS actually just means that a signal was received while inside a system call. If this results in a interruption of that syscall, the kernel is supposed to translate ERESTARTSYS into the user error EINTR.
> Hmmm, Maybe I'm getting confused by all the winning-lottery > messages, but it's in the syscall specifications for connect() > and even
> fcntl(). http:/www.infran.ru/Techinfo/syscalls/syscalls_43.html
AFAICS that documentation was written in 1994, and refers to Linux v1.0. We've come a long way since then...
Todays Linux userland is supposed to try to comply with the Single Unix Specification (see http://www.unix-systems.org/version3/)
whenever possible. ERESTARTSYS is missing altogether from the SUSv3 definitions in <errno.h> (and hence does not appear as a valid return value for any SUSv3-compliant functions).
Note: the Linux manpages do list ERESTARTSYS as still being returned by the accept() and syslog() system call. In both those cases, however, they point out that your libc is supposed to intercept it before it gets to the user.
> Also, maybe Linux now claims exclusive ownership and keeps it > internal, but some networking software, nfsd and pcnfsd, might > not know about that. I've seen ERESTARTSYS returned from a DOS > (actually FAT) file-handle use after a server has crashed and > come back on-line.
Linux used to be buggy/non-compliant w.r.t. NFS exporting of FAT filesystems. I'm not sure if that has been fixed yet.
Cheers,
Trond