Skip to contentSkip to conversation

NFS regression breaks X

bisect Jeff Garzik 5 participants 9 messages May 9, 2007 – May 11, 2007

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Original bug report, with hardware and software info:
http://lkml.org/lkml/2007/5/8/667

I love bisect :) bisect has identified the following commit as the one that causes my GNOME login to die, within 10 seconds of logging in:

	commit 2bea90d43a050bbc4021d44e59beb34f384438db
	Author: Chuck Lever <[email protected]>
	Date:   Thu Mar 29 16:47:53 2007 -0400

	SUNRPC: RPC buffer size estimates are too large

100% reproducible, verified regression. My home directory is an NFSv4 mount, and the problem appears on my client workstation, so this makes some sense:

Quoted 2 lines

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)

As an aside, let me express the hope that the NFS developers develop better patch creation methods. My bisect compile repeatedly died at

Quoted 8 lines
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHK     include/linux/compile.h

fs/nfs/pagelist.c:239: error: conflicting types for ‘nfs_pageio_init’ include/linux/nfs_page.h:80: error: previous declaration of ‘nfs_pageio_init’ was here make[2]: *** [fs/nfs/pagelist.o] Error 1
make[1]: *** [fs/nfs] Error 2
make: *** [fs] Error 2

which indicates that someone on the NFS team did not create wholly-contained patches when submitted to the kernel. Build breakage should not be fixed in a later commit (unless the breakage already went upstream), because -- as we see here -- it breaks bisection.

Jeff, occasionally guilty of same, and trying to reform

#1 lore raw
Quoted 12 lines · Jeff Garzik

Original bug report, with hardware and software info:
http://lkml.org/lkml/2007/5/8/667

I love bisect :) bisect has identified the following commit as the one that causes my GNOME login to die, within 10 seconds of logging in:

	commit 2bea90d43a050bbc4021d44e59beb34f384438db
	Author: Chuck Lever <[email protected]>
	Date:   Thu Mar 29 16:47:53 2007 -0400

	SUNRPC: RPC buffer size estimates are too large
Quoted 5 lines

100% reproducible, verified regression. My home directory is an NFSv4 mount, and the problem appears on my client workstation, so this makes some sense:

Quoted 2 lines

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)

Known issue. Could you try applying the commit at

http://linux-nfs.org/cgi-bin/gitweb.cgi?p=nfs-2.6.git;a=commitdiff;h=6ce7dc940701cf3fde3c6e826a696b333092cbb1;hp=aa3d1faebe6e214cd96be0e587571477ff6fd9fc

and see if that suffices to fix the issue?

Quoted 13 lines

As an aside, let me express the hope that the NFS developers develop better patch creation methods. My bisect compile repeatedly died at

Quoted 8 lines
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHK     include/linux/compile.h

fs/nfs/pagelist.c:239: error: conflicting types for ‘nfs_pageio_init’ include/linux/nfs_page.h:80: error: previous declaration of ‘nfs_pageio_init’ was here make[2]: *** [fs/nfs/pagelist.o] Error 1
make[1]: *** [fs/nfs] Error 2
make: *** [fs] Error 2

Quoted 6 lines

which indicates that someone on the NFS team did not create wholly-contained patches when submitted to the kernel. Build breakage should not be fixed in a later commit (unless the breakage already went upstream), because -- as we see here -- it breaks bisection.

Jeff, occasionally guilty of same, and trying to reform

The problem is that the above only appears to break on 64-bit compiles. gcc gave no errors at all on a 386, and so the problem was detected only after the merge.

Cheers
Trond

#6 lore raw
In reply to Jeff Garzik
Quoted 2 lines · Jeff Garzik

I love bisect :)

Yeah, me too.

Quoted 4 lines

bisect has identified the following commit as the one that causes my GNOME login to die, within 10 seconds of logging in:

	commit 2bea90d43a050bbc4021d44e59beb34f384438db

Ok, that commit looks nice in many ways, so I'd be loathe to revert it entirely.

Can you try it with this patch that re-introduces the "total overkill" slack calculations, and probably makes them even worse (there's a few left-shifts added, and now it will left-shift the extra slacktoo).

But there are also some unexplained changes in that patch, so maybe the size allocation isn't the big problem. For example, now "call_allocate()" will set task->tk_status to zero which is totally strange.

But if it's the allocation that is too small, this patch may help.

Trond, Chuck?

Linus

--

 net/sunrpc/clnt.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
net/sunrpc/clnt.c +6 -1
index d8fbee4..5b78692 100644
@@ -738,6 +738,8 @@ call_reserveresult(struct rpc_task *task)
 	rpc_exit(task, status);
 }
 
+#define RPC_SLACK_SPACE		(1024u)	/* apparently NOT total overkill */
+
 /*
  * 2.	Allocate the buffer. For details, see sched.c:rpc_malloc.
  *	(Note: buffer memory is freed in xprt_release).
@@ -745,7 +747,7 @@ call_reserveresult(struct rpc_task *task)
 static void
 call_allocate(struct rpc_task *task)
 {
-	unsigned int slack = task->tk_auth->au_cslack;
+	unsigned int slack;
 	struct rpc_rqst *req = task->tk_rqstp;
 	struct rpc_xprt *xprt = task->tk_xprt;
 	struct rpc_procinfo *proc = task->tk_msg.rpc_proc;
@@ -764,6 +766,9 @@ call_allocate(struct rpc_task *task)
 			BUG_ON(proc->p_replen == 0);
 	}
 
+	/* Apparently not total overkill */
+	slack =  max(task->tk_auth->au_cslack, RPC_SLACK_SPACE);
+
 	/*
 	 * Calculate the size (in quads) of the RPC call
 	 * and reply headers, and convert both values
#7 lore raw
Quoted 19 lines · Linus Torvalds
Quoted 2 lines · Jeff Garzik

I love bisect :)

Yeah, me too.

Quoted 4 lines

bisect has identified the following commit as the one that causes my GNOME login to die, within 10 seconds of logging in:

	commit 2bea90d43a050bbc4021d44e59beb34f384438db

Ok, that commit looks nice in many ways, so I'd be loathe to revert it entirely.

Can you try it with this patch that re-introduces the "total overkill" slack calculations, and probably makes them even worse (there's a few left-shifts added, and now it will left-shift the extra slacktoo).

Quoted 7 lines

But there are also some unexplained changes in that patch, so maybe the size allocation isn't the big problem. For example, now "call_allocate()" will set task->tk_status to zero which is totally strange.

But if it's the allocation that is too small, this patch may help.

Trond, Chuck?

We'd really like to fix this by getting the pre-allocation tables right in nfs4xdr.c. Chuck has already identified one incorrect value:

http://linux-nfs.org/cgi-bin/gitweb.cgi?p=nfs-2.6.git;a=commitdiff;h=6ce7dc940701cf3fde3c6e826a696b333092cbb1;hp=aa3d1faebe6e214cd96be0e587571477ff6fd9fc

that I just asked you to pull as part of a series of bugfixes. I'm hoping that will suffice to fix Jeff's case too.

Trond

#8 lore raw
In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

Can you try it with this patch that re-introduces the "total overkill" slack calculations, and probably makes them even worse (there's a few left-shifts added, and now it will left-shift the extra slacktoo).

I'll try this patch, and separately, the Trond push that just appeared on LKML.

But after losing several hours discovering this "known issue," I need to attend to my own patches for today first. So, testing won't be immediate.

Jeff

#2 lore raw
In reply to Trond Myklebust
Quoted 13 lines · On Wed, 09 May 2007 18:03:26 -0400 Trond Myklebust…
Quoted 12 lines · Jeff Garzik

Original bug report, with hardware and software info:
http://lkml.org/lkml/2007/5/8/667

I love bisect :) bisect has identified the following commit as the one that causes my GNOME login to die, within 10 seconds of logging in:

	commit 2bea90d43a050bbc4021d44e59beb34f384438db
	Author: Chuck Lever <[email protected]>
	Date:   Thu Mar 29 16:47:53 2007 -0400

	SUNRPC: RPC buffer size estimates are too large
Quoted 7 lines
Quoted 5 lines

100% reproducible, verified regression. My home directory is an NFSv4 mount, and the problem appears on my client workstation, so this makes some sense:

Quoted 2 lines

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) pretzel:/ on /g type nfs4 (rw,noatime,proto=tcp,addr=10.10.10.1)

Known issue.

It's a bit rough that Jeff spent a large amount of time hunting down an already-known bug. That's normally my job :(

This five-week-old diff only ever appeared in 2.6.21-mm1, which was released four days ago. It was then whizzed into mainline. We thus lost five weeks public testing which would probably have saved Jeff his pain.

What went wrong?

#3 lore raw
Quoted 2 lines · Andrew Morton

It's a bit rough that Jeff spent a large amount of time hunting down an already-known bug. That's normally my job :(

The bug was reported by Florin Iucha (on lkml!) on Saturday. It has only just been debugged, and I was in fact in the middle of marshalling the fixes.

Quoted 5 lines

This five-week-old diff only ever appeared in 2.6.21-mm1, which was released four days ago. It was then whizzed into mainline. We thus lost five weeks public testing which would probably have saved Jeff his pain.

What went wrong?

Probably my fault. I've had a couple of weeks of heavy travel due to various circumstances that were beyond my control, and so I had little time in which to test the stuff and push it out.

Another factor that is affecting us is the slow but gradual collapse of the OSDL NFSv4 regression testing effort.

Trond

#4 lore raw
Quoted 12 lines · Trond Myklebust
Quoted 2 lines · Andrew Morton

It's a bit rough that Jeff spent a large amount of time hunting down an already-known bug. That's normally my job :(

The bug was reported by Florin Iucha (on lkml!) on Saturday. It has only just been debugged, and I was in fact in the middle of marshalling the fixes.

Quoted 4 lines

This five-week-old diff only ever appeared in 2.6.21-mm1, which was released four days ago. It was then whizzed into mainline. We thus lost five weeks public testing which would probably have saved Jeff his pain.

Quoted 8 lines
Quoted 1 line

What went wrong?

Probably my fault. I've had a couple of weeks of heavy travel due to various circumstances that were beyond my control, and so I had little time in which to test the stuff and push it out.

Another factor that is affecting us is the slow but gradual collapse of the OSDL NFSv4 regression testing effort.

I had expected that many of these issues would be caught by the OSDL test harness. I learned only yesterday that it was no longer available, so I am making an effort to broaden my personal test regime.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel/
version:2.1
end:vcard