Skip to contentSkip to conversation

NFS client hang on attempt to do async blocking posix lock enqueue

DISCUSSION J. Bruce Fields 5 participants 11 messages Nov 29, 2007 – Feb 8, 2008 VFS

lore
Share & tools
lore
#0 lore raw
Quoted 20 lines · Oleg Drokin

Hello!

There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when fl_grant is called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately after fl_grant, there is nlmsvc_grant_blocked message (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I see nlmsvc_grant_blocked "lockd:
GRANTing blocked lock" message again with no activity from underlying FS.

I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.

Quoted 6 lines

I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.

Thanks, I'll take a look. Replying now just to add Marc to the cc:.

--b.

#1 lore raw

The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.
Marc.

"J. Bruce Fields" <[email protected]> wrote on 11/29/2007 11:15:32 AM:

Quoted 31 lines · Oleg Drokin
Quoted 20 lines

Hello!

There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when fl_grant is called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately after fl_grant, there is nlmsvc_grant_blocked message (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I see nlmsvc_grant_blocked "lockd:
GRANTing blocked lock" message again with no activity from underlying FS.

I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.

Quoted 6 lines

I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.

Thanks, I'll take a look. Replying now just to add Marc to the cc:.

--b.

#2 lore raw
Quoted 6 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

--b.

Quoted 36 lines

Marc.

"J. Bruce Fields" <[email protected]> wrote on 11/29/2007 11:15:32 AM:

Quoted 31 lines · Oleg Drokin
Quoted 20 lines

Hello!

There is a problem with blocking async posix lock enqueue in 2.6.22 and 2.6.23 kernels. Lock call to underlying FS is done just fine, but when fl_grant is called to inform lockd of succesful granting, nothing happens, and no reply to client is sent. The end result is client reports that the server is not responding. I enabled dprintks in the code and I see that immediately after fl_grant, there is nlmsvc_grant_blocked message (after callback: label) printed. Then server not responding messages start, and after every message about "coulndn't create RPC handle for localhost" I see nlmsvc_grant_blocked "lockd:
GRANTing blocked lock" message again with no activity from underlying FS.

I am attaching a reproducer that I have, it is quite simple actually. Take note, that path to file to lock is hardcoded, so adjust for your environment please. Lcoking should be performed on a file that resides on nfs client mountpoint.

Quoted 6 lines

I reproduced the problem with 2.6.22 and 2.6.23 with Lustre (I am working on adapting lustre to async posix locks API) and GFS2. Setup is totally local, i.e. I have single node on which there is gfs (both server and client) (or lustre - just client, but that does not make any difference), nfs server and nfs client that mounts exported gfs or lustre.

Thanks, I'll take a look. Replying now just to add Marc to the cc:.

--b.

#3 lore raw

Hello!

Quoted 9 lines · J. Bruce Fields
Quoted 6 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on the same machine. This test work fine with or without an underlaying fs that supports locking when the client and the server are on a different machines. Like you said the server is trying to send the grant message to the client but for some reason it fails when the client is on the same machine.

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

Bye,
Oleg

#4 lore raw
Quoted 16 lines · Oleg Drokin

Hello!

Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.

Quoted 5 lines
Quoted 2 lines

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....

--b.

#5 lore raw
Quoted 20 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 19 lines · Oleg Drokin

Hello!

Quoted 14 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Quoted 8 lines
Quoted 2 lines

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....

--b.

It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to rpc_ping itself then it probably would hang like this, wouldn't it?

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw
Quoted 30 lines · Jeff Layton
Quoted 27 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 16 lines · Oleg Drokin

Hello!

Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.

Quoted 5 lines
Quoted 2 lines

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....

Quoted 6 lines
Quoted 1 line

--b.

It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to rpc_ping itself then it probably would hang like this, wouldn't it?

Of course! Yes, that fits.

--b.

#7 lore raw
Quoted 33 lines · J. Bruce Fields
Quoted 32 lines · Jeff Layton
Quoted 28 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 16 lines · Oleg Drokin

Hello!

Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.

Quoted 5 lines
Quoted 2 lines

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....

--b.

Quoted 8 lines
Quoted 4 lines

It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to rpc_ping itself then it probably would hang like this, wouldn't it?

Of course! Yes, that fits.

--b.

right on, jeff, good catch and thanks for directing my attention
to your patches.

i applied them on top of 2.6.23.1 and tested them on a cluster
exporting GFS2 over NFS, using oleg's reproducer code. your patches fix that lockd hang.

in a bit more detail, oleg's reproducer basically gets a
whole-file read lock, tests the lock, upgrades to a whole-file exclusive lock, tests the lock, then unlocks. the problem was that when getting that exclusive lock things would hang. this only happened when the client and server were on the same machine, and i could reproduce it with NFS exporting GFS2 but not NFS exporting EXT3.

	thanks,

	d
	.
#8 lore raw
Quoted 40 lines · On Fri, 8 Feb 2008 13:49:01 -0500 (EST) "david m. richter"…
Quoted 38 lines · J. Bruce Fields
Quoted 36 lines · Jeff Layton
Quoted 20 lines · On Thu, 7 Feb 2008 18:26:18 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 19 lines · Oleg Drokin

Hello!

Quoted 14 lines · J. Bruce Fields
Quoted 11 lines · Marc Eshel

The problem seems to be with the fact that the client and server are on
the same machine. This test work fine with or without an underlaying fs
that supports locking when the client and the server are on a different
machines. Like you said the server is trying to send the grant message to
the client but for some reason it fails when the client is on the same
machine.

That *shouldn't* make a difference, so we need to take another look at this--Oleg, this problem is still unfixed, right?

Quoted 8 lines
Quoted 2 lines

Yes, I just pulled your latest nfs tree and I still can reproduce the problem.

OK, we have finally reproduced this problem here, and David's working on debugging. It does indeed seem to only be reproduceable with client and server on the same machine. Thanks for the report....

--b.

It might be worth testing this both with and without the patchset I posted to linux-nfs recently to take care of the lockd hang. If lockd is stuck trying to rpc_ping itself then it probably would hang like this, wouldn't it?

Quoted 7 lines
Quoted 3 lines

Of course! Yes, that fits.

--b.

right on, jeff, good catch and thanks for directing my attention
to your patches.

Excellent! Glad that took care of it...

Quoted 4 lines

i applied them on top of 2.6.23.1 and tested them on a cluster
exporting GFS2 over NFS, using oleg's reproducer code. your patches fix that lockd hang.

Quoted 8 lines

in a bit more detail, oleg's reproducer basically gets a
whole-file read lock, tests the lock, upgrades to a whole-file exclusive lock, tests the lock, then unlocks. the problem was that when getting that exclusive lock things would hang. this only happened when the client and server were on the same machine, and i could reproduce it with NFS exporting GFS2 but not NFS exporting EXT3.

Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like fl_grant really only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.

Signature · 1 line
Jeff Layton <[email protected]>
#9 lore raw
Quoted 4 lines · Jeff Layton

Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like fl_grant really only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.

The asynchronous locking interface does something slightly cheesy for blocking locks--instead of waiting for the filesystem to respond, it just sends back a deny immediately (even if the lock might actually be available), then responds later with a granted message when it discovers it's available.

That works, but we should make it just wait to send the reply to the original lock request until we've got a real answer, as we do for nonblocking lock requests. And in fact someone submitted a patch to do that--I just haven't gotten the time to review it. Urp.

So anyway the effect is that on ext3 this particular lock wouldn't have required a grant reply, whereas on gfs2 it does.

Of course, what this means is that we'd hit the same problem on ext3 too if the lock request did in fact legitimately block. So grant callbacks probably have never worked on ext3 over the loopback interface either. Oops!

I bet nobody's ever noticed because we manage to recover by retrying the lock after it's available (whereas in the gfs2 case the retry hits the same problem). So in practice for ext3 this probably just means blocking lock requests take a lot longer over loopback then they would otherwise. And probably the only people that care about nlm performance don't usually do local mounts like that.

--b.

#10 lore raw
Quoted 17 lines · On Fri, 8 Feb 2008 16:12:28 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 4 lines · Jeff Layton

Interesting. It's not clear me why the underlying filesystem would make any difference there. Though now that I look, it looks like fl_grant really only gets called from dlm code, and that queues up the block for an immediate grant callback attempt. So perhaps that's the reason.

The asynchronous locking interface does something slightly cheesy for blocking locks--instead of waiting for the filesystem to respond, it just sends back a deny immediately (even if the lock might actually be available), then responds later with a granted message when it discovers it's available.

That works, but we should make it just wait to send the reply to the original lock request until we've got a real answer, as we do for nonblocking lock requests. And in fact someone submitted a patch to do that--I just haven't gotten the time to review it. Urp.

Quoted 8 lines

So anyway the effect is that on ext3 this particular lock wouldn't have required a grant reply, whereas on gfs2 it does.

Of course, what this means is that we'd hit the same problem on ext3 too if the lock request did in fact legitimately block. So grant callbacks probably have never worked on ext3 over the loopback interface either. Oops!

As best I can tell, the whole problem with rpc_pings was introduced when we moved everything to the rpcbind stuff. Before that we generally never did an rpc_ping when binding the client. This probably did work until that was introduced.

Quoted 7 lines

I bet nobody's ever noticed because we manage to recover by retrying the lock after it's available (whereas in the gfs2 case the retry hits the same problem). So in practice for ext3 this probably just means blocking lock requests take a lot longer over loopback then they would otherwise. And probably the only people that care about nlm performance don't usually do local mounts like that.

Signature · 1 line
Jeff Layton <[email protected]>