NFS client hang on attempt to do async blocking posix lock enqueue
Quoted 20 lines · Oleg Drokin
Hello!
There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when
fl_grantis called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately afterfl_grant, there isnlmsvc_grant_blockedmessage (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I seenlmsvc_grant_blocked"lockd:
GRANTing blocked lock" message again with no activity from underlying FS.I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.
Quoted 6 lines
I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.
Thanks, I'll take a look. Replying now just to add Marc to the cc:.
--b.
The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.
Marc.
"J. Bruce Fields" <[email protected]> wrote on 11/29/2007 11:15:32 AM:
Quoted 31 lines · Oleg Drokin
Quoted 20 lines
Hello!
There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when
fl_grantis called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately afterfl_grant, there isnlmsvc_grant_blockedmessage (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I seenlmsvc_grant_blocked"lockd:
GRANTing blocked lock" message again with no activity from underlying FS.I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.
Quoted 6 lines
I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.
Thanks, I'll take a look. Replying now just to add Marc to the cc:.
--b.
Quoted 6 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.
That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
--b.
Quoted 36 lines
Marc.
"J. Bruce Fields" <[email protected]> wrote on 11/29/2007 11:15:32 AM:
Quoted 31 lines · Oleg Drokin
Quoted 20 lines
Hello!
There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when
fl_grantis called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately afterfl_grant, there isnlmsvc_grant_blockedmessage (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I seenlmsvc_grant_blocked"lockd:
GRANTing blocked lock" message again with no activity from underlying FS.I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.
Quoted 6 lines
I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.
Thanks, I'll take a look. Replying now just to add Marc to the cc:.
--b.
Hello!
Quoted 9 lines · J. Bruce Fields
Quoted 6 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.
That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
Bye,
Oleg
Quoted 16 lines · Oleg Drokin
Hello!
Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.
Quoted 5 lines
Quoted 2 lines
That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....
--b.
Quoted 20 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 19 lines · Oleg Drokin
Hello!
Quoted 14 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Quoted 8 lines
Quoted 2 lines
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....
--b.
It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to rpc_ping itself then it probably would hang like this, wouldn't it?
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 30 lines · Jeff Layton
Quoted 27 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 16 lines · Oleg Drokin
Hello!
Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.Quoted 5 lines
Quoted 2 lines
That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....
Quoted 6 lines
Quoted 1 line
--b.
It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to
rpc_pingitself then it probably would hang like this, wouldn't it?
Of course! Yes, that fits.
--b.
Quoted 33 lines · J. Bruce Fields
Quoted 32 lines · Jeff Layton
Quoted 28 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 16 lines · Oleg Drokin
Hello!
Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.Quoted 5 lines
Quoted 2 lines
That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....
--b.
Quoted 8 lines
Quoted 4 lines
It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to
rpc_pingitself then it probably would hang like this, wouldn't it?Of course! Yes, that fits.
--b.
right on, jeff, good catch and thanks for directing my attention
to your patches.
i applied them on top of 2.6.23.1 and tested them on a cluster
exporting GFS2 over NFS, using oleg's reproducer code. your patches fix that lockd hang.
in a bit more detail, oleg's reproducer basically gets a
whole-file read lock, tests the lock, upgrades to a whole-file exclusive lock, tests the lock, then unlocks. the problem was that when getting that exclusive lock things would hang. this only happened when the client and server were on the same machine, and i could reproduce it with NFS exporting GFS2 but not NFS exporting EXT3.
thanks, d .
Quoted 40 lines · On Fri, 8 Feb 2008 13:49:01 -0500 (EST) "david m. richter"…
Quoted 38 lines · J. Bruce Fields
Quoted 36 lines · Jeff Layton
Quoted 20 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 19 lines · Oleg Drokin
Hello!
Quoted 14 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel
The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?
Quoted 8 lines
Quoted 2 lines
Yes, I just pulled your latest nfs tree and I still can reproduce the problem.
OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....
--b.
It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to
rpc_pingitself then it probably would hang like this, wouldn't it?
Quoted 7 lines
Quoted 3 lines
Of course! Yes, that fits.
--b.
right on, jeff, good catch and thanks for directing my attention
to your patches.
Excellent! Glad that took care of it...
Quoted 4 lines
i applied them on top of 2.6.23.1 and tested them on a cluster
exporting GFS2 over NFS, using oleg's reproducer code. your patches fix that lockd hang.
Quoted 8 lines
in a bit more detail, oleg's reproducer basically gets a
whole-file read lock, tests the lock, upgrades to a whole-file exclusive lock, tests the lock, then unlocks. the problem was that when getting that exclusive lock things would hang. this only happened when the client and server were on the same machine, and i could reproduce it with NFS exporting GFS2 but not NFS exporting EXT3.
Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like fl_grant really only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 4 lines · Jeff Layton
Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like
fl_grantreally only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.
The asynchronous locking interface does something slightly cheesy for blocking locks--instead of waiting for the filesystem to respond, it just sends back a deny immediately (even if the lock might actually be available), then responds later with a granted message when it discovers it's available.
That works, but we should make it just wait to send the reply to the original lock request until we've got a real answer, as we do for nonblocking lock requests. And in fact someone submitted a patch to do that--I just haven't gotten the time to review it. Urp.
So anyway the effect is that on ext3 this particular lock wouldn't have required a grant reply, whereas on gfs2 it does.
Of course, what this means is that we'd hit the same problem on ext3 too if the lock request did in fact legitimately block. So grant callbacks probably have never worked on ext3 over the loopback interface either. Oops!
I bet nobody's ever noticed because we manage to recover by retrying the lock after it's available (whereas in the gfs2 case the retry hits the same problem). So in practice for ext3 this probably just means blocking lock requests take a lot longer over loopback then they would otherwise. And probably the only people that care about nlm performance don't usually do local mounts like that.
--b.
Quoted 17 lines · On Fri, 8 Feb 2008 16:12:28 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 4 lines · Jeff Layton
Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like
fl_grantreally only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.The asynchronous locking interface does something slightly cheesy for blocking locks--instead of waiting for the filesystem to respond, it just sends back a deny immediately (even if the lock might actually be available), then responds later with a granted message when it discovers it's available.
That works, but we should make it just wait to send the reply to the original lock request until we've got a real answer, as we do for nonblocking lock requests. And in fact someone submitted a patch to do that--I just haven't gotten the time to review it. Urp.
Quoted 8 lines
So anyway the effect is that on ext3 this particular lock wouldn't have required a grant reply, whereas on gfs2 it does.
Of course, what this means is that we'd hit the same problem on ext3 too if the lock request did in fact legitimately block. So grant callbacks probably have never worked on ext3 over the loopback interface either. Oops!
As best I can tell, the whole problem with rpc_pings was introduced when we moved everything to the rpcbind stuff. Before that we generally never did an rpc_ping when binding the client. This probably did work until that was introduced.
Quoted 7 lines
I bet nobody's ever noticed because we manage to recover by retrying the lock after it's available (whereas in the gfs2 case the retry hits the same problem). So in practice for ext3 this probably just means blocking lock requests take a lot longer over loopback then they would otherwise. And probably the only people that care about nlm performance don't usually do local mounts like that.
Signature · 1 line
Jeff Layton <[email protected]>