stdio and nfs-utils · kernelthread

J. Bruce Fields

author

2007-07-25 22:37 UTC #0 lore raw

I ran into some problem yesterday where ls on a directory containing several unexported mountpoints was hanging. Watching writes to the /proc/net/rpc/nfsd.export/channel file, what I saw was:

	write("gss/krb5i /path/a expiry\n") -> -ENOENT
	write("gss/krb5i /path/a expiry\ngss/krb5i /path/b\n" -> -ENOENT
	write("gss/krb5i /path/a expiry\ngss/krb5i /path/b\ngss/krb5i/path/b\n) ->ENOENT

etc. You get the idea. The cache code that handles the write throws away all but the first line, so the extra lines are ignored. The kernel keeps doing upcalls to ask about /path/b, but never sees the results because mountd keeps passing down the /path/a downcall again as the first line each time.

I don't see how nfs-utils itself could be producing multiple lines in the upcall handler, so I assume what's happening is that the stdio code is taking the error returns as a sign that it should leave the data in its buffer and just try again next time another fwrite adds more. Does that make sense? I don't really understand how stdio is supposed to handle errors.

The -ENOENTs are spurious in this case, actually; I'll submit a patch to eliminate them. But could we also just ditch the use of stdio in all the nfs-utils cache code? The current problem aside it seems likely to complicate any error handling (which the current nfs-utils code doesn't even attempt).

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 00:07 UTC #1 lore raw

Quoted 4 lines · J. Bruce Fields

I assume what's happening is that the stdio code is taking the error returns as a sign that it should leave the data in its buffer and just try again next time another fwrite adds more. Does that make sense? I don't really understand how stdio is supposed to handle errors.

So I tried a test program that just does

	while (1) {
		fprintf(f, "%d\n", i++)
		fflush(f);
	}

and pointed it at one of those proc files, which will always return an error on such a write. On my fedora rawhide system, each write only writes the new data. On my debian sid system, each write writes all the data accumulated so far. Huh.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Neil Brown

2007-07-26 02:59 UTC #2 lore raw

Quoted 13 lines · On Wednesday July 25, [email protected] wrote:

Quoted 4 lines · J. Bruce Fields
I assume what's happening is that the stdio code is taking the error returns as a sign that it should leave the data in its buffer and just try again next time another fwrite adds more. Does that make sense? I don't really understand how stdio is supposed to handle errors.
So I tried a test program that just does
	while (1) {
		fprintf(f, "%d\n", i++)
		fflush(f);
	}

Quoted 4 lines

and pointed it at one of those proc files, which will always return an error on such a write. On my fedora rawhide system, each write only writes the new data. On my debian sid system, each write writes all the data accumulated so far. Huh.

:-)

While I can see the value it avoiding the stdio code, I can also see some in hanging on to it, so I would first like to see if we can make the current code work with just minor modifications.

Could you check how calling 'clearerr' interacts with this code on the two different systems?

Thanks,
NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 03:17 UTC #3 lore raw

Quoted 6 lines · Neil Brown

While I can see the value it avoiding the stdio code, I can also see some in hanging on to it, so I would first like to see if we can make the current code work with just minor modifications.
Could you check how calling 'clearerr' interacts with this code on the two different systems?

I tried inserting a clearerr into the loop:

	fprintf();
	fflush();
	clearerr();

No change in behavior.

I see the advantage to doing just one step at a time--so if we can figure out a minimal fix for now, fine. But surely we've got to replace the stdio stuff at some point. We need precise control over the writes, so the buffering can only get in the way.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Neil Brown

2007-07-26 04:10 UTC #4 lore raw

Quoted 8 lines · On Wednesday July 25, [email protected] wrote:

Quoted 6 lines · Neil Brown
While I can see the value it avoiding the stdio code, I can also see some in hanging on to it, so I would first like to see if we can make the current code work with just minor modifications.
Could you check how calling 'clearerr' interacts with this code on the two different systems?

Quoted 7 lines

I tried inserting a clearerr into the loop:
	fprintf();
	fflush();
	clearerr();
No change in behavior.

Hmmm.... (reads man page)...

#include <stdio_ext.h>

...

	__fpurge(f)

seems to do the trick.

Quoted 5 lines

I see the advantage to doing just one step at a time--so if we can figure out a minimal fix for now, fine. But surely we've got to replace the stdio stuff at some point. We need precise control over the writes, so the buffering can only get in the way.

But we want to build a line up from multiple words, and so we need buffering for that.
If we toss stdio, we will need to allocate a buffer, keep track of the end point, and append to the buffer in various places. It feels like reinventing the wheel.

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 20:30 UTC #5 lore raw

PATCH 1/1 Use __fpurge to ensure single-line writes to cache files

From: J. Bruce Fields <[email protected]>

On a recent Debian/Sid machine, I saw libc retrying stdio writes that returned write errors. The result is that if an export downcall returns an error (which it can in normal operation, since it currently (incorrectly) returns -ENOENT on any negative downcall), then subsequent downcalls will write multiple lines (including the original line that received the error).

The result is that the server fails to respond to any rpc call that refers to an unexported mount point (such as a readdir of a directory containing such a mountpoint), so client commands hang.

I don't know whether this libc behavior is correct or expected, but it seems safest to add the __fpurge() (suggested by Neil) to ensure data is thrown away.

Signed-off-by "J. Bruce Fields" <[email protected]>

---
 support/nfs/cacheio.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

Quoted 3 lines · Neil Brown

	__fpurge(f)
seems to do the trick.

Yep, thanks!

Quoted 4 lines

But we want to build a line up from multiple words, and so we need buffering for that. If we toss stdio, we will need to allocate a buffer, keep track of the end point, and append to the buffer in various places. It feels like reinventing the wheel.

Reinventing a rather simple part of the wheel, I'd say. Oh well, another day.

--b.

support/nfs/cacheio.c +12 -0

index a76915b..9d271cd 100644
@@ -17,6 +17,7 @@
 
 #include <nfslib.h>
 #include <stdio.h>
+#include <stdio_ext.h>
 #include <ctype.h>
 #include <unistd.h>
 #include <sys/types.h>

@@ -111,7 +112,18 @@ void qword_printint(FILE *f, int num)
 
 int qword_eol(FILE *f)
 {
+	int err;
+
 	fprintf(f,"\n");
+	err = fflush(f);
+	/*
+	 * We must send one line (and one line only) in a single write
+	 * call.  In case of a write error, libc may accumulate the
+	 * unwritten data and try to write it again later, resulting in a
+	 * multi-line write.  So we must explicitly ask it to throw away
+	 * any such cached data:
+	 */
+	__fpurge(f);
 	return fflush(f);
 }

Signature · 11 lines

1.5.3.rc2


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 20:31 UTC #6 lore raw

PATCH 1/1 knfsd: eliminate unnecessary -ENOENT returns on export downcalls

In reply to Neil Brown

From: J. Bruce Fields <[email protected]>

A succesful downcall with a negative result (which indicates that the given filesystem is not exported to the given user) should not return an error.

Currently mountd is depending on stdio to write these downcalls. With some versions of libc this appears to cause subsequent writes to attempt to write all accumulated data (for which writes previously failed) along with any new data. This can prevent the kernel from seeing responses to later downcalls. Symptoms will be that nfsd fails to respond to certain requests.

Signed-off-by "J. Bruce Fields" <[email protected]>

---
 fs/nfsd/export.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

And here's the kernel fix.--b.

fs/nfsd/export.c +3 -2

index 2d295dd..cba899a 100644
@@ -564,9 +564,10 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen)
 
 	/* flags */
 	err = get_int(&mesg, &an_int);
-	if (err == -ENOENT)
+	if (err == -ENOENT) {
+		err = 0;
 		set_bit(CACHE_NEGATIVE, &exp.h.flags);
-	else {
+	} else {
 		if (err || an_int < 0) goto out;	
 		exp.ex_flags= an_int;

Signature · 11 lines

1.5.3.rc2


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Muntz, Daniel

2007-07-26 18:43 UTC #7 lore raw

Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

In reply to J. Bruce Fields

Could all this be related to a problem that we've been seeing while working on v4/4.1 at netapp, where exportfs doesn't show any output even when there are exported file systems? Ricardo recently observed that if you redirect the output of 'exportfs' then you actually do get the correct output (e.g., exportfs > afile, or exportfs | more). This lead to some experiments with strace that show exportfs behaving differently when its output is going to a tty. We've seen this behavior across a variety of kernels (2.6.18-2.6.20) and a variety of nfs-utils (prepackaged with FC6, RHEL4/5, and built from a local copy of 1.0.12).

N.B. no output from the first exportfs, but correct output when piped to 'more':

[root@teal ~]# exportfs -a
[root@teal ~]# exportfs
[root@teal ~]# exportfs | more
/ <world>

Now looking at straces with and without redirect:

[root@teal ~]# strace exportfs 2> exportfs.noredirect [root@teal ~]# strace exportfs | more 2> exportfs.redirect

In the 'no redirection' case, there's an ioctl that fails, and the output never appears in the xterm window (assuming that's related to the ioctl failure the returns ENOTTY). N.B., we see the 'write' system call in the strace, but don't see any output from that call. From exportfs.noredirect:

ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbf8aabc8) = -1 ENOTTY (Inappropriate ioctl for device)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7fba000
write(1, "/ \t<world>\n", 23) = 23
exit_group(0) = ?

In the case of redirection, we don't do the ioctl, and you can see the 'write' of the exportfs output in the strace as well as the result of the 'write'. From exportfs.redirect:

fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f8e000
write(1, "/ \t<world>\n", 23) = 23
exit_group(0) = ?
/ <world>
Process 8408 detached

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, July 25, 2007 8:18 PM
To: Neil Brown
Cc: [email protected]
Subject: Re: [NFS] stdio and nfs-utils

Quoted 7 lines · Neil Brown

While I can see the value it avoiding the stdio code, I can also see some in hanging on to it, so I would first like to see if we can make the current code work with just minor modifications.
Could you check how calling 'clearerr' interacts with this code on the
two different systems?

I tried inserting a clearerr into the loop:

	fprintf();
	fflush();
	clearerr();

No change in behavior.

I see the advantage to doing just one step at a time--so if we can figure out a minimal fix for now, fine. But surely we've got to replace the stdio stuff at some point. We need precise control over the writes, so the buffering can only get in the way.

--b.

------------------------------------------------------------------------
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 19:36 UTC #8 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

Quoted 5 lines · Daniel

Could all this be related to a problem that we've been seeing while working on v4/4.1 at netapp, where exportfs doesn't show any output even when there are exported file systems? Ricardo recently observed that if you redirect the output of 'exportfs' then you actually do get the correct output (e.g., exportfs > afile, or exportfs | more).

I wouldn't expect that to be related to the problem with i/o to these proc files.

But, yes, I've seen the same problem with exportfs output. (Shame on me for not reporting it at at the time.... And thanks for looking into it!) But I don't know what the problem is.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jeff Layton

2007-07-26 19:48 UTC #9 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

In reply to Muntz, Daniel

Quoted 10 lines · Daniel

Could all this be related to a problem that we've been seeing while working on v4/4.1 at netapp, where exportfs doesn't show any output even when there are exported file systems? Ricardo recently observed that if you redirect the output of 'exportfs' then you actually do get the correct output (e.g., exportfs > afile, or exportfs | more). This lead to some experiments with strace that show exportfs behaving differently when its output is going to a tty. We've seen this behavior across a variety of kernels (2.6.18-2.6.20) and a variety of nfs-utils (prepackaged with FC6, RHEL4/5, and built from a local copy of 1.0.12).

I saw a very similar problem with RHEL5 prior to its release. It turned out to be a selinux policy issue. I believe the latest selinux-policy packages in Fedora and RHEL have fixed it if it's the same problem.

Signature · 10 lines

Jeff Layton <[email protected]>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 20:04 UTC #10 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

Quoted 3 lines · Jeff Layton

I saw a very similar problem with RHEL5 prior to its release. It turned out to be a selinux policy issue. I believe the latest selinux-policy packages in Fedora and RHEL have fixed it if it's the same problem.

That makes sense. I don't remember for sure where I saw this, but it may well have been on my Fedora (rawhide) desktop, and I haven't seen it recently.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Muntz, Daniel

2007-07-26 20:42 UTC #11 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

In reply to Jeff Layton

Any idea what one would need to change to verify if it's the same problem? Our base installs are almost all FC6 and RHEL 4/5 (begging the question of whether it was fixed in RHEL 5 prior to release). Someone here can probably test on an FC7 system, but it would be really handy to know how to fix the many existing installations, assuming it's the same problem.

Thanks,
Dan

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Thursday, July 26, 2007 12:48 PM
To: Muntz, Daniel
Cc: J. Bruce Fields; Neil Brown; [email protected]
Subject: Re: [NFS] Exportfs not showing exported file systems (was RE:
stdio and nfs-utils)

Quoted 11 lines · Daniel

Could all this be related to a problem that we've been seeing while working on v4/4.1 at netapp, where exportfs doesn't show any output even when there are exported file systems? Ricardo recently observed that if you redirect the output of 'exportfs' then you actually do get
the correct output (e.g., exportfs > afile, or exportfs | more). This
lead to some experiments with strace that show exportfs behaving differently when its output is going to a tty. We've seen this behavior across a variety of kernels (2.6.18-2.6.20) and a variety of nfs-utils (prepackaged with FC6, RHEL4/5, and built from a local copy

of 1.0.12).

Quoted 1 line

I saw a very similar problem with RHEL5 prior to its release. It turned out to be a selinux policy issue. I believe the latest selinux-policy packages in Fedora and RHEL have fixed it if it's the same problem.

--
Jeff Layton <[email protected]>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jeff Layton

2007-07-26 21:12 UTC #12 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

Quoted 10 lines · Daniel

Any idea what one would need to change to verify if it's the same problem? Our base installs are almost all FC6 and RHEL 4/5 (begging the question of whether it was fixed in RHEL 5 prior to release). Someone here can probably test on an FC7 system, but it would be really handy to know how to fix the many existing installations, assuming it's the same problem.
Thanks,
Dan

While it's discouraged for us to suggest, you could do what I did and temporarily put selinux in permissive mode: ;-)

# setenforce 0

...and then see if exportfs still has the issue. To turn it back to enforcing do:

# setenforce 1

I've not been able to reproduce the problem since it was reportedly fixed. The BZ I opened on it is here:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221181

...if you find that selinux is at fault then you may want to reopen it. It might also be worthwhile to do the same with Bruce's original problem and see if it behaves any differently.

Signature · 10 lines

Jeff Layton <[email protected]>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

J. Bruce Fields

author

2007-07-26 21:16 UTC #13 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

Quoted 3 lines · Jeff Layton

...if you find that selinux is at fault then you may want to reopen it. It might also be worthwhile to do the same with Bruce's original problem and see if it behaves any differently.

The system where I was seeing the problem was a debian system without selinux.

--b.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Muntz, Daniel

2007-07-26 21:40 UTC #14 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

In reply to Jeff Layton

Yep, disabling selinux fixed it on my FC6 machine.

-Dan

Quoted 39 lines

-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Thursday, July 26, 2007 2:12 PM
To: Muntz, Daniel
Cc: J. Bruce Fields; Neil Brown; [email protected]
Subject: Re: [NFS] Exportfs not showing exported file systems (was RE: stdio and nfs-utils)
Quoted 2 lines · Daniel
Any idea what one would need to change to verify if it's the same problem? Our base installs are almost all FC6 and RHEL 4/5
(begging the
Quoted 1 line
question of whether it was fixed in RHEL 5 prior to
release). Someone
Quoted 1 line
here can probably test on an FC7 system, but it would be
really handy to
Quoted 1 line
know how to fix the many existing installations, assuming
it's the same
Quoted 5 lines
problem.
Thanks,
Dan
While it's discouraged for us to suggest, you could do what I did and temporarily put selinux in permissive mode: ;-)
# setenforce 0
...and then see if exportfs still has the issue. To turn it back to enforcing do:
# setenforce 1
I've not been able to reproduce the problem since it was reportedly fixed. The BZ I opened on it is here:

Quoted 10 lines

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221181
...if you find that selinux is at fault then you may want to reopen it.
It might also be worthwhile to do the same with Bruce's original problem and see if it behaves any differently.
Signature · 1 line
Jeff Layton <[email protected]>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Steve Dickson

2007-07-27 14:40 UTC #15 lore raw

Re: Exportfs not showing exported file systems (was RE: stdio and nfs-utils)

Quoted 1 line · Daniel

Yep, disabling selinux fixed it on my FC6 machine.

A better idea might be to simply 'yum update selinux-policy'

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs