lockd and lock cancellation
Subsystems (2)
Hi,
I'm trying to find my way around the lockd code and I'm currently a bit stumped by the code relating to lock cancellation. There is only one call to vfs_cancel_lock() in lockd/svclock.c and its return value isn't checked.
It is used in combination with nlmsvc_unlink_block() which unconditionally calls posix_unblock_lock(). There are also other places where the code also calls nlmsvc_unlink_block() without first canceling the lock. The way in which vfs_cancel_lock() is used suggests that canceling a lock is a synchronous operation, and that it must succeed before returning.
I'd have expected to see (pseudo code) something more like the following:
ret = vfs_cancel_lock();
if (ret == -ENOENT) /* never had the lock in the first place */
do_something_appropriate();
else if (ret == -EINVAL) /* we raced with a grant */
unlock_lock();
else /* lock successfully canceled */
nlmsvc_unlink_block();Is there a reason why that is not required? and indeed, is there a reason why its safe to call nlmsvc_unlink_block() in the cases where the lock isn't canceled first? I'm trying to work out how the underlying fs can tell that a lock has gone away in those particular cases,
Steve.
Quoted 25 lines · Steven Whitehouse
Hi,
I'm trying to find my way around the lockd code and I'm currently a bit stumped by the code relating to lock cancellation. There is only one call to
vfs_cancel_lock()inlockd/svclock.cand its return value isn't checked.It is used in combination with
nlmsvc_unlink_block()which unconditionally callsposix_unblock_lock(). There are also other places where the code also callsnlmsvc_unlink_block()without first canceling the lock. The way in whichvfs_cancel_lock()is used suggests that canceling a lock is a synchronous operation, and that it must succeed before returning.I'd have expected to see (pseudo code) something more like the following:
ret =
vfs_cancel_lock();
if (ret ==-ENOENT) /* never had the lock in the first place */do_something_appropriate(); else if (ret == -EINVAL) /* we raced with a grant */ unlock_lock(); else /* lock successfully canceled */ nlmsvc_unlink_block();
Quoted 5 lines
Is there a reason why that is not required? and indeed, is there a reason why its safe to call
nlmsvc_unlink_block()in the cases where the lock isn't canceled first? I'm trying to work out how the underlying fs can tell that a lock has gone away in those particular cases,
Steve,
I noticed the missing cancel scenario some time ago and reported on it here. Bruce agreed that it was a bug, but I regret that I haven't had time to follow up on it. The problem was that vfs_cancel_lock was not being called in all cases where it should be, possibly resulting in an orphaned lock in the filesystem. See attached message for more detail. (Or http://marc.info/?l=linux-nfs&m=125849395630496&w=2)
By the way, if a lock grant wins a race with a cancel, I do not think it is "safe" to simply unlock the lock at that point.
Rob Gardner
*List: linux-nfs <http://marc.info/?l=linux-nfs&r=1&w=2>
Subject: Re: Potential lockd bug can cause orphaned locks <http://marc.info/?t=125832556700002&r=1&w=2> From: "J. Bruce Fields" <bfields () fieldses ! org> <http://marc.info/?a=100577513600008&r=1&w=2> Date: 2009-11-17 21:39:40 <http://marc.info/?l=linux-nfs&r=1&w=2&b=200911> Message-ID: 20091117213940.GD5121 () fieldses ! org <http://marc.info/?i=20091117213940.GD5121%20%28%29%20fieldses%20%21%20org>*
Quoted 18 lines · Rob Gardner
I've discovered some behavior in lockd that I think is questionable. In conjunction with file systems that provide their own locking functions (ie,
file->f_op->lock()), lockd seems to handle cancel requests inconsistently, with the result that a file may be left with a byte range lock on it but with no owner.There are several scenarios in which lockd would like to cancel a lock request:
1. Client sends explicit unlock or cancel request
2. A non-blocking lock request times out
3. A 'cleanup' type request, ie, client reboot and subsequentSM_NOTIFYwhich attempts to clear all locks and requests for that clientIn all scenarios, lockd believes that the locks and lock requests for the file have been cleaned up, and it closes the file and releases references to the file.
Quoted 7 lines
In the first scenario, lockd calls
vfs_cancel_lockto alert the file system that it would like to cancel the lock request; Then,nlmsvc_unlink_block()callsposix_unblock_lock()which cancels any outstanding posix lock request.But in the other two scenarios,
vfs_cancel_lock()is never called, onlyposix_unblock_lock().
Yes, I agree that this is a bug, thanks for the description.
Quoted 8 lines
It seems to me that scenarios 2 and 3 are perfectly good "cancel lock" situations and lockd should call
vfs_cancel_lock()in both cases to reduce the possibility of a future grant for a file no longer opened by lockd. Depending on the vague and ambiguous advice in thelocks.ccomment seems very dangerous; Releasing a lock may not result in the same state as canceling the lock request that caused the grant so it should always be preferable to cancel a lock if possible, rather than waiting for a grant then unlocking it.
That suggests there are other races between a grant and a cancel that we're not addressing here.
...
Quoted 16 lines
Possible code change (not tested):
svclock.c 2009-11-15 13:57:15.000000000 -0700 renamed from svclock.c.linux-2.6.32-rc6 2009-11-15 13:54:03.000000000 -0700 +2
-0@@ -288,8 +288,9 @@ if (list_empty(&block->b_list)) continue; kref_get(&block->b_count); mutex_unlock(&file->f_mutex); + vfs_cancel_lock(block->b_file->f_file, + &block->b_call->a_args.lock.fl); nlmsvc_unlink_block(block); nlmsvc_release_block(block); goto restart; }
svclock.c 2009-11-15 13:57:15.000000000 -0700
Quoted 10 lines
svclock.c 2009-11-15 13:57:15.000000000 -0700 renamed from svclock.c.linux-2.6.32-rc6 2009-11-15 13:54:03.000000000 -0700 +2
-0@@ -399,8 +400,9 @@ ret = nlm_granted; goto out; } if (block->b_flags & B_TIMED_OUT) { + vfs_cancel_lock(block->b_file->f_file, + &block->b_call->a_args.lock.fl); nlmsvc_unlink_block(block); ret = nlm_lck_denied; goto out; }
Seems reasonable, though it is a bit annoying trying to determine which of these should be called where, so...
Quoted 4 lines
Another possibility is to change
nlmsvc_unlink_block()to make the call tovfs_cancel_lock()and then remove the call tovfs_cancel_lockinnlmsvc_cancel_blocked(). But I don't really like this as most other calls tonlmsvc_unlink_block()do not require a call tovfs_cancel_lock().
..yes, I understand why the ideal initially appeals, but don't have a better suggestion.
--b.
Quoted 10 lines
I am interested in hearing opinions on this... is there a better solution? Or is it not really a problem because of something else?
Rob Gardner
Quoted 47 lines · Steven Whitehouse
Hi,
Thanks for the fast response...
Quoted 33 lines · Rob Gardner
Quoted 25 lines · Steven Whitehouse
Hi,
I'm trying to find my way around the lockd code and I'm currently a bit stumped by the code relating to lock cancellation. There is only one call to
vfs_cancel_lock()inlockd/svclock.cand its return value isn't checked.It is used in combination with
nlmsvc_unlink_block()which unconditionally callsposix_unblock_lock(). There are also other places where the code also callsnlmsvc_unlink_block()without first canceling the lock. The way in whichvfs_cancel_lock()is used suggests that canceling a lock is a synchronous operation, and that it must succeed before returning.I'd have expected to see (pseudo code) something more like the following:
ret =
vfs_cancel_lock();
if (ret ==-ENOENT) /* never had the lock in the first place */do_something_appropriate(); else if (ret == -EINVAL) /* we raced with a grant */ unlock_lock(); else /* lock successfully canceled */ nlmsvc_unlink_block();Quoted 6 lines
Is there a reason why that is not required? and indeed, is there a reason why its safe to call
nlmsvc_unlink_block()in the cases where the lock isn't canceled first? I'm trying to work out how the underlying fs can tell that a lock has gone away in those particular cases,Quoted 5 lines
Steve,
I noticed the missing cancel scenario some time ago and reported on it here. Bruce agreed that it was a bug, but I regret that I haven't had time to follow up on it. The problem was that
vfs_cancel_lockwas not being called in all cases where it should be, possibly resulting in an orphaned lock in the filesystem. See attached message for more detail. (Or http://marc.info/?l=linux-nfs&m=125849395630496&w=2)I have one question relating to that message (see below)
Quoted 8 lines
Quoted 3 lines
By the way, if a lock grant wins a race with a cancel, I do not think it is "safe" to simply unlock the lock at that point.
Why not? If the cancel has failed, then we are left holding the lock just as if we'd requested it and no cancel had been issued. Or another way to ask the same question, if that does occur, what would be the correct way to dispose of the unwanted lock?
If the lock were actually granted, then unlocking in lieu of a cancel can potentially leave a range unlocked that should be left locked. This can happen in the case of a lock upgrade or a coalesce operation; For instance, suppose the client holds a lock on bytes 0-100, then issues another lock request for bytes 50-150, but sends a cancel just after the lock is actually granted. If you now simply unlock 50-150, then the client is left holding only 0-50, and has "lost" the lock on bytes 51-100. In other words, the client will *believe* that he has 0-100 locked, but in reality, only 0-50 are locked.
As for what to do in this situation... well it would be nice if the filesystem treated the cancel request as an "undo" if the grant won the race. But seriously, I think this is just one of the (many) flaws in the protocol and we probably have to live with it. My personal feeling is that it's safer to leave bytes locked rather than have a client believe it holds a lock when it doesn't.
Quoted 14 lines
[snip]
Quoted 12 lines
Seems reasonable, though it is a bit annoying trying to determine which of these should be called where, so...
Quoted 5 lines
Another possibility is to change
nlmsvc_unlink_block()to make the call tovfs_cancel_lock()and then remove the call tovfs_cancel_lockinnlmsvc_cancel_blocked(). But I don't really like this as most other calls tonlmsvc_unlink_block()do not require a call tovfs_cancel_lock()...yes, I understand why the ideal initially appeals, but don't have a better suggestion.
Quoted 9 lines
Quoted 3 lines
--b.
Can we not use a flag to figure out when a cancel needs to be sent? We could set the flag when an async request was sent to the underlying fs and clear it when the reply arrives. It would thus only be valid to send a
vfs_cancel_lock()request when the flag was set.
We could do all this, but I don't see the point, since there is still a race window you could sail a boat through. It's the period of time between when the client sends a cancel request and the time that lockd sends the cancel request to the filesystem. If a grant happens during this time, what can be done? The protocol just doesn't have a way to deal with this.
Quoted 3 lines
My other thought is whether or not
posix_unblock_lock()could be merged intovfs_cancel_lock()or whether there are really cases where that needs to be called without a cancellation having taken place,
I think that the filesystem should do the posix_unblock_lock call when it (successfully?) processes a cancel request. After all, the fs is already calling posix_lock_file when it successfully grants a lock. And just as vfs_lock_file falls through to posix_lock_file when the fs doesn't provide a lock function, so should vfs_cancel_lock fall through to posix_unblock_lock in that situation.
Rob Gardner
Quoted 10 lines · Rob Gardner
If the lock were actually granted, then unlocking in lieu of a cancel can potentially leave a range unlocked that should be left locked. This can happen in the case of a lock upgrade or a coalesce operation; For instance, suppose the client holds a lock on bytes 0-100, then issues another lock request for bytes 50-150, but sends a cancel just after the lock is actually granted. If you now simply unlock 50-150, then the client is left holding only 0-50, and has "lost" the lock on bytes 51-100. In other words, the client will *believe* that he has 0-100 locked, but in reality, only 0-50 are locked.
Quoted 6 lines
As for what to do in this situation... well it would be nice if the filesystem treated the cancel request as an "undo" if the grant won the race. But seriously, I think this is just one of the (many) flaws in the protocol and we probably have to live with it. My personal feeling is that it's safer to leave bytes locked rather than have a client believe it holds a lock when it doesn't.
I wrote some code to address this sort of problem a long time ago, and didn't submit it:
http://git.linux-nfs.org/?p=bfields/linux-topics.git;a=shortlog;h=refs/heads/fair-queueing
The idea was to support correct cancelling by introducing a new type of lock. Let's call it a "provisional lock". It behaves in every way like a posix lock, *except* that it doesn't combine with (or downgrade) existing locks. This allows a provisional lock to be cancelled--by just removing it from the lock list--or to be "upgraded" to a normal posix lock. (Note if you're looking at the code above, "provisional locks" are identified with the FL_BLOCK flag.)
We should probably look back at that idea at some point.
The original motivation was to implement "fair queuing" for NFSv4 locks. (Since NFSv4 clients wait on blocking locks by polling the server, a server is supposed to be willing to temporarily hold a lock on a polling client's behalf. But since posix locks aren't really guaranteed to be granted to waiters in any particular order, I don't know how much that matters.)
A side-effect was to eliminate a potential thundering-herd problem by allowing the lock code to immediately grant a single process a provisional lock, wake up just that process, and allow it to upgrade the provisional lock to a normal posix lock, instead of waking up all the waiters. I was never able to figure out how to measure any benefit from that, though!
You could push the same "provisional lock" idea down into GFS2, and I suspect it would make it easier to avoid some races. But I haven't thoguht that through.
--b.
Hi,
Thanks for the fast response...
Quoted 32 lines · Rob Gardner
Quoted 25 lines · Steven Whitehouse
Hi,
I'm trying to find my way around the lockd code and I'm currently a bit stumped by the code relating to lock cancellation. There is only one call to
vfs_cancel_lock()inlockd/svclock.cand its return value isn't checked.It is used in combination with
nlmsvc_unlink_block()which unconditionally callsposix_unblock_lock(). There are also other places where the code also callsnlmsvc_unlink_block()without first canceling the lock. The way in whichvfs_cancel_lock()is used suggests that canceling a lock is a synchronous operation, and that it must succeed before returning.I'd have expected to see (pseudo code) something more like the following:
ret =
vfs_cancel_lock();
if (ret ==-ENOENT) /* never had the lock in the first place */do_something_appropriate(); else if (ret == -EINVAL) /* we raced with a grant */ unlock_lock(); else /* lock successfully canceled */ nlmsvc_unlink_block();Quoted 5 lines
Is there a reason why that is not required? and indeed, is there a reason why its safe to call
nlmsvc_unlink_block()in the cases where the lock isn't canceled first? I'm trying to work out how the underlying fs can tell that a lock has gone away in those particular cases,
Quoted 9 lines
Steve,
I noticed the missing cancel scenario some time ago and reported on it here. Bruce agreed that it was a bug, but I regret that I haven't had time to follow up on it. The problem was that
vfs_cancel_lockwas not being called in all cases where it should be, possibly resulting in an orphaned lock in the filesystem. See attached message for more detail. (Or http://marc.info/?l=linux-nfs&m=125849395630496&w=2)
I have one question relating to that message (see below)
Quoted 3 lines
By the way, if a lock grant wins a race with a cancel, I do not think it is "safe" to simply unlock the lock at that point.
Why not? If the cancel has failed, then we are left holding the lock just as if we'd requested it and no cancel had been issued. Or another way to ask the same question, if that does occur, what would be the correct way to dispose of the unwanted lock?
[snip]
Quoted 9 lines
Seems reasonable, though it is a bit annoying trying to determine which of these should be called where, so...
Quoted 4 lines
Another possibility is to change
nlmsvc_unlink_block()to make the call tovfs_cancel_lock()and then remove the call tovfs_cancel_lockinnlmsvc_cancel_blocked(). But I don't really like this as most other calls tonlmsvc_unlink_block()do not require a call tovfs_cancel_lock().
Quoted 5 lines
..yes, I understand why the ideal initially appeals, but don't have a better suggestion.
--b.
Can we not use a flag to figure out when a cancel needs to be sent? We could set the flag when an async request was sent to the underlying fs and clear it when the reply arrives. It would thus only be valid to send a vfs_cancel_lock() request when the flag was set.
My other thought is whether or not posix_unblock_lock() could be merged into vfs_cancel_lock() or whether there are really cases where that needs to be called without a cancellation having taken place,
Steve.