possible client stale filehandle bug?

DISCUSSION Garrick Staples 4 participants 8 messages Jan 25, 2005 – Jan 28, 2005

Subsystems (3)

VFS NFS client NFS server

lore

Share & tools

lore

Garrick Staples

author

2005-01-25 17:40 UTC #0 lore raw

Hi all,
I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients. Under some conditions, linux apps have been getting stale filehandles during the normal course of their activity. Various file handling syscalls like read() or open() might error. Lots of renames and setattrs calls seem to trigger the problem. 'ci' and 'cvs commit' are particularly good at this.

It seems that the Solaris clients never report any such errors, only the Linux clients. However, watching 'snoop' on the Solaris NFS server, I see that it IS returning stale file handles to both OSes, but Solaris clients seem to retry the request several times; and the Linux clients immediately pass the error up to the application.

Is there some condition that the 2.4 kernel is handling incorrectly?

Sample snippet from the 'snoop' on the Solaris server with a Solaris client waiting...

rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610 Ack=3071279992 Seq=337022612 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049 Ack=337022752 Seq=3071279992 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610 Ack=3071280516 Seq=337023056 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049 Ack=337023196 Seq=3071280516 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610 Ack=3071280796 Seq=337023348 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610 Ack=3071281040 Seq=337023500 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049 Ack=337023640 Seq=3071281040 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE

Signature · 2 lines

Garrick Staples, Linux/HPCC Administrator
University of Southern California

Trond Myklebust

2005-01-26 06:07 UTC #1 lore raw

ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:

Quoted 7 lines

Hi all,
I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients. Under some conditions, linux apps have been getting stale filehandles during the normal course of their activity. Various file handling syscalls like read() or open() might error. Lots of renames and setattrs calls seem to trigger the problem. 'ci' and 'cvs commit' are particularly good at this.

ESTALE is usually a sign that someone is deleting a file on the server that is in use by the client. It is a sign that you are doing something that violates the caching rules of NFS.

Quoted 7 lines

It seems that the Solaris clients never report any such errors, only the Linux clients. However, watching 'snoop' on the Solaris NFS server, I see that it IS returning stale file handles to both OSes, but Solaris clients seem to retry the request several times; and the Linux clients immediately pass the error up to the application.
Is there some condition that the 2.4 kernel is handling incorrectly?

I do not believe that Solaris redrives ESTALE on read, but they may do it on open(). Linux does not redrive either case. See the many discussions in the NFS list archives for why.

Cheers,
Trond

Signature · 12 lines

Trond Myklebust <[email protected]>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Garrick Staples

author

2005-01-26 06:35 UTC #2 lore raw

On Tue, Jan 25, 2005 at 10:06:27PM -0800, Trond Myklebust alleged:

Quoted 9 lines

ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
Quoted 7 lines
Hi all,
I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients. Under some conditions, linux apps have been getting stale filehandles during the normal course of their activity. Various file handling syscalls like read() or open() might error. Lots of renames and setattrs calls seem to trigger the problem. 'ci' and 'cvs commit' are particularly good at this.

Quoted 3 lines

ESTALE is usually a sign that someone is deleting a file on the server that is in use by the client. It is a sign that you are doing something that violates the caching rules of NFS.

Nothing of the kind is happening here. I've tested this a thousand times over the last few days trying to find a solution. In this case, Sun's samfs filesystem is definitely at fault and doing the wrong thing. Backline engineers at Sun confirm this and are working on a fix.

The reason for _this_ email isn't because of the ESTALEs, it's regarding the handling of the ESTALEs. Right now I need the Solaris client behaviour to deal with this particular buggy server.

Incidentally, 2.6.10 never has a problem. It's behaviour never creates ESTALEs in the first place.

Quoted 11 lines

Quoted 7 lines
It seems that the Solaris clients never report any such errors, only the Linux clients. However, watching 'snoop' on the Solaris NFS server, I see that it IS returning stale file handles to both OSes, but Solaris clients seem to retry the request several times; and the Linux clients immediately pass the error up to the application.
Is there some condition that the 2.4 kernel is handling incorrectly?
I do not believe that Solaris redrives ESTALE on read, but they may do it on open(). Linux does not redrive either case. See the many discussions in the NFS list archives for why.

Did you look at the 'snoop' bits in the previous email? During that time, the process on the Solaris client is hanging in a write() call.

I'd be very happy to see any patches lieing around that might do this behaviour. It would get me through the short term until Sun fixes this bug in samfs.

Signature · 2 lines

Garrick Staples, Linux/HPCC Administrator
University of Southern California

Neil Horman

2005-01-26 13:11 UTC #3 lore raw

Quoted 43 lines · Garrick Staples

On Tue, Jan 25, 2005 at 10:06:27PM -0800, Trond Myklebust alleged:
Quoted 9 lines
ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
Quoted 6 lines
Hi all,
I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients. Under some conditions,
linux apps have been getting stale filehandles during the normal course of
their activity. Various file handling syscalls like read() or open() might
error. Lots of renames and setattrs calls seem to trigger the problem. 'ci' and 'cvs commit' are particularly good at this.
Quoted 3 lines
ESTALE is usually a sign that someone is deleting a file on the server that is in use by the client. It is a sign that you are doing something that violates the caching rules of NFS.
Nothing of the kind is happening here. I've tested this a thousand times over the last few days trying to find a solution. In this case, Sun's samfs filesystem is definitely at fault and doing the wrong thing. Backline engineers at Sun confirm this and are working on a fix.
The reason for _this_ email isn't because of the ESTALEs, it's regarding the handling of the ESTALEs. Right now I need the Solaris client behaviour to deal with this particular buggy server.
Incidentally, 2.6.10 never has a problem. It's behaviour never creates ESTALEs in the first place.
Quoted 11 lines
Quoted 7 lines
It seems that the Solaris clients never report any such errors, only the Linux clients. However, watching 'snoop' on the Solaris NFS server, I see that it IS returning stale file handles to both OSes, but Solaris clients seem to retry the request several times; and the Linux clients immediately pass the error up to the application.
Is there some condition that the 2.4 kernel is handling incorrectly?
I do not believe that Solaris redrives ESTALE on read, but they may do it on open(). Linux does not redrive either case. See the many discussions in the NFS list archives for why.

Quoted 7 lines

Did you look at the 'snoop' bits in the previous email? During that time, the process on the Solaris client is hanging in a write() call.
I'd be very happy to see any patches lieing around that might do this behaviour. It would get me through the short term until Sun fixes this bug in samfs.

The easiest, cleanest thing to do is retry the operation in the sys_* function. Red Hat's latest kernel does this to prevent ESTALE. I've attached the patch for your reference. Mind you, due to the reasons you'll find in the list archives that Trond referenced, you'll not see this get into upstream kernels, so caveat emptor.

HTH
Neil

Signature · 8 lines

/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *[email protected]
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

fs/nfs/inode.c +9 -1

diff -urNp linux-6090/fs/nfs/inode.c linux-6091/fs/nfs/inode.c
@@ -1022,7 +1022,15 @@ int
 nfs_revalidate(struct dentry *dentry)
 {
 	struct inode *inode = dentry->d_inode;
-	return nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	int error;
+
+	error = nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	if (error == -ESTALE) {
+		nfs_zap_caches(dentry->d_parent->d_inode);
+		d_drop(dentry);
+	}
+
+	return error;
 }
 
 /*

fs/nfs/xattr.c +6 -1

diff -urNp linux-6090/fs/nfs/xattr.c linux-6091/fs/nfs/xattr.c
@@ -53,8 +53,13 @@ nfs_getxattr(struct dentry *dentry, cons
 	acl = ERR_PTR(-EOPNOTSUPP);
 	if (NFS_PROTO(inode)->version == 3 && NFS_PROTO(inode)->getacl)
 		acl = NFS_PROTO(inode)->getacl(inode, type);
-	if (IS_ERR(acl))
+	if (IS_ERR(acl)) {
+		if (PTR_ERR(acl) == -ESTALE) {
+			nfs_zap_caches(dentry->d_parent->d_inode);
+			d_drop(dentry);
+		}
 		return PTR_ERR(acl);
+	}
 	else if (acl) {
 		if (type == ACL_TYPE_ACCESS && acl->a_count == 0)
 			error = -ENODATA;

fs/open.c +24 -0

diff -urNp linux-6090/fs/open.c linux-6091/fs/open.c
@@ -807,6 +807,30 @@ asmlinkage long sys_open(const char * fi
 		if (fd >= 0) {
 			struct file *f = filp_open(tmp, flags, mode);
 			error = PTR_ERR(f);
+
+			/*
+			 * ESTALE errors can be a pain.  On some
+			 * filesystems (e.g. NFS), ESTALE can often
+			 * be resolved by retry, as the ESTALE resulted
+			 * in a cache invalidation.  We perform this
+			 * retry here, once for every directory element
+			 * in the path to avoid the case where the removal
+			 * of the nth parent directory of the file we're
+			 * trying to open results in n ESTALE errors.
+			 */
+			if (error == -ESTALE) {
+				int nretries = 1;
+				char *cp;
+
+				for (cp = tmp; *cp; cp++) {
+					if (*cp == '/')
+						nretries++;
+				}
+				do {
+					f = filp_open(tmp, flags, mode);
+					error = PTR_ERR(f);
+				} while (error == -ESTALE && --nretries > 0);
+			}
 			if (IS_ERR(f))
 				goto out_error;
 			fd_install(fd, f);

fs/stat.c +42 -6

126 lines not on this page · raw message

[email protected]

2005-01-26 14:42 UTC #4 lore raw

In reply to Garrick Staples

Quoted 4 lines · Garrick Staples

I'd be very happy to see any patches lieing around that might do this behaviour. It would get me through the short term until Sun fixes this bug in samfs.

Don't hold your breath waiting!

Ian

-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Garrick Staples

author

2005-01-26 17:50 UTC #5 lore raw

On Wed, Jan 26, 2005 at 10:31:57PM +0800, [email protected] alleged:

Quoted 9 lines · Garrick Staples

Quoted 6 lines
I'd be very happy to see any patches lieing around that might do this behaviour. It would get me through the short term until Sun fixes this bug in
samfs.
Don't hold your breath waiting!

Now you are just being cruel!

Signature · 2 lines

Garrick Staples, Linux/HPCC Administrator
University of Southern California

Ian Kent

2005-01-28 00:48 UTC #6 lore raw

Quoted 4 lines · Garrick Staples

On Wed, Jan 26, 2005 at 10:31:57PM +0800, [email protected] alleged:
Quoted 3 lines · Garrick Staples
Quoted 2 lines

Quoted 8 lines

Quoted 6 lines
Quoted 4 lines
I'd be very happy to see any patches lieing around that might do this behaviour. It would get me through the short term until Sun fixes this bug in
samfs.
Don't hold your breath waiting!
Now you are just being cruel!

We don't use SAMfs in the standard configuration either!

Oh joy, oh bliss!

Sun purchasing LSCI was a real bummer for us.

Ian

Neil Horman

2005-01-26 13:07 UTC #7 lore raw

In reply to Trond Myklebust

Quoted 21 lines · Trond Myklebust

ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
Quoted 6 lines
Hi all,
I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients. Under some conditions,
linux apps have been getting stale filehandles during the normal course of
their activity. Various file handling syscalls like read() or open() might
error. Lots of renames and setattrs calls seem to trigger the problem. 'ci' and 'cvs commit' are particularly good at this.
ESTALE is usually a sign that someone is deleting a file on the server that is in use by the client. It is a sign that you are doing something that violates the caching rules of NFS.
Quoted 6 lines
It seems that the Solaris clients never report any such errors, only the Linux clients. However, watching 'snoop' on the Solaris NFS server, I see that it IS returning stale file handles to both OSes, but Solaris clients seem to retry the request several times; and the Linux clients immediately pass the error up to the application.

Quoted 7 lines

Quoted 1 line
Is there some condition that the 2.4 kernel is handling incorrectly?
I do not believe that Solaris redrives ESTALE on read, but they may do it on open(). Linux does not redrive either case. See the many discussions in the NFS list archives for why.

Solaris does in fact retry on operations on ESTALE errors, definately on open, and I think on read/readdir/stat/etc. as well. We had some discussion about tht here recently.

Signature · 18 lines

/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *[email protected]
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs