Skip to contentSkip to conversation

nfs4: 30 second delay during umount of remote fs on system shutdown

regression Frans Pop 3 participants 5 messages Feb 28, 2008 – Feb 29, 2008

Subsystems (2)
lore
Share & tools
lore

Frans Pop

author
#0 lore raw

With 2.6.25-rc2 and -rc3 I noticed a pause of 30 seconds when my system (Debian unstable, x86_64) is being shutdown. The cause is the unmounting of an nfs4 remote file system.
The pause does not happen with 2.6.24. I'm unsure about 2.6.25-rc1 as I was concentrating on other issues when I was testing that.

I can reproduce the same pause on an active system by stopping portmap before doing an umount of the filesystem.

The shutdown sequence on my box includes:
K20nfs-common
[...]
S20sendsigs
[...]
S31umountnfs.sh
S32portmap

So it seems that things should be OK (S32portmap is after S31umountnfs.sh), but in fact portmap has already been killed during S20sendsigs!

I have verified that portmap is also not running during S31umountnfs.sh with 2.6.24, so apparently there has been a change in the kernel that has made umount look for portmap and cause the 30 second delay if it is not running.

Is this a kernel regression?

Cheers,
FJP

#1 lore raw
Quoted 3 lines · Frans Pop

I have verified that portmap is also not running during S31umountnfs.sh with 2.6.24, so apparently there has been a change in the kernel that has made umount look for portmap and cause the 30 second delay if it is not running.

You are probably seeing the effect of lockd going down: it always attempts to unregister from the portmapper (and no, this is _not_ new behaviour).

If debian is killing the portmapper while lockd is still up, then that is a definite debian bug...

Trond

author
#2 lore raw
Quoted 9 lines · Trond Myklebust
Quoted 4 lines · Frans Pop

I have verified that portmap is also not running during S31umountnfs.sh with 2.6.24, so apparently there has been a change in the kernel that has made umount look for portmap and cause the 30 second delay if it is not running.

You are probably seeing the effect of lockd going down: it always attempts to unregister from the portmapper (and no, this is _not_ new behaviour).

It is _very much_ new behavior because as soon as I boot (and shut down) with a 2.6.24 kernel the problem completely disappears. Userland is *100% identical* between my tests.

Quoted 2 lines

If debian is killing the portmapper while lockd is still up, then that is a definite debian bug...

I have no idea about that. AFAICT lockd is not even running (at least, there's no process named lockd, even when the system is running normally. Isn't lockd an nfs3 thing? I'm using nfs4 here.

Cheers,
FJP

#3 lore raw
Quoted 15 lines · On Fri, 29 Feb 2008 02:03:05 +0100 Frans Pop…
Quoted 9 lines · Trond Myklebust
Quoted 4 lines · Frans Pop

I have verified that portmap is also not running during S31umountnfs.sh with 2.6.24, so apparently there has been a change in the kernel that has made umount look for portmap and cause the 30 second delay if it is not running.

You are probably seeing the effect of lockd going down: it always attempts to unregister from the portmapper (and no, this is _not_ new behaviour).

It is _very much_ new behavior because as soon as I boot (and shut down) with a 2.6.24 kernel the problem completely disappears. Userland is *100% identical* between my tests.

Quoted 7 lines
Quoted 2 lines

If debian is killing the portmapper while lockd is still up, then that is a definite debian bug...

I have no idea about that. AFAICT lockd is not even running (at least, there's no process named lockd, even when the system is running normally. Isn't lockd an nfs3 thing? I'm using nfs4 here.

It may be the NFSv4 callback thread going down then. It also tries to unregister with the portmapper and recent patches fixed the reference counting so that it actually tears things down instead of just leaking memory and sockets.

I'm not sure what we can do about that other than try to fix it so that it doesn't try to unregister with the portmapper. It really doesn't need to since it no longer registers with it.

Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw
In reply to Frans Pop
Quoted 7 lines · Frans Pop
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Frans Pop

I have verified that portmap is also not running during S31umountnfs.sh with 2.6.24, so apparently there has been a change in the kernel that has made umount look for portmap and cause the 30 second delay if it is not running.

Quoted 7 lines
Quoted 3 lines

You are probably seeing the effect of lockd going down: it always attempts to unregister from the portmapper (and no, this is _not_ new behaviour).

It is _very much_ new behavior because as soon as I boot (and shut down) with a 2.6.24 kernel the problem completely disappears. Userland is *100% identical* between my tests.

The "new behaviour" is that we fixed a bug whereby the lockd code was failing to flush signals before attempting the rpc unregister. The sad fact is that userland appears to have been 100% broken for some time. The difference is that it was able to get away with it earlier...

Quoted 6 lines
Quoted 2 lines

If debian is killing the portmapper while lockd is still up, then that is a definite debian bug...

I have no idea about that. AFAICT lockd is not even running (at least, there's no process named lockd, even when the system is running normally. Isn't lockd an nfs3 thing? I'm using nfs4 here.

NFSv4 has a delegation recall server to kill. Same difference...

Trond