Finding and breaking client locks
Subsystems (2)
Hello there,
I run a diskless network where every user NFS mounts pretty much
everything including /home and /var/mail. It's often the case that a misbehaved client will leave a locked file stuck on the server -- today it is a file in a user's mail/ directory.
Is there a way to query what files are being held locked by clients? I'm sure the kernel knows, as it is able to enforce the lock, but it isn't obvious how to extract that information -- lsof is documented to and indeed does not return any information pertaining NFS client locks, and I'm not clear whether /proc/locks (on the server side obviously) does or not.
A related question is whether it is possible to break a client lock without rebooting the server (or restarting the NFS services).
Does anyone have any insight to share? Thanks,
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kikoQuoted 7 lines · On Mon, 21 Mar 2016 11:39:14 -0300 Christian Robottom Reis <[email protected]> wrote:
Hello there,
I run a diskless network where every user NFS mounts pretty much
everything including /home and/var/mail. It's often the case that a misbehaved client will leave a locked file stuck on the server -- today it is a file in a user's mail/ directory.
Quoted 7 lines
Is there a way to query what files are being held locked by clients? I'm sure the kernel knows, as it is able to enforce the lock, but it isn't obvious how to extract that information -- lsof is documented to and indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.
/proc/locks will generally show you all of the locks being held (assuming the filesystem's ->lock routine records the locks). It's not really possible to match those up with a particular client though.
Quoted 4 lines
A related question is whether it is possible to break a client lock without rebooting the server (or restarting the NFS services).
Does anyone have any insight to share? Thanks,
I assume you're using NFSv3? What happens when the client "misbehaves"? Are the clients dropping offline or do you have applications that are just sitting on the lock and not releasing it?
If the clients are just going away without unlocking first, then you could consider using NFSv4, which has a lease-based locking. If the client goes away for a while (90s or so), then it'll lose its lock.
Alternately, there is the /proc/fs/nfsd/unlock_ip interface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used that so YMMV there.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 17 lines · Jeff Layton
Quoted 7 lines · On Mon, 21 Mar 2016 11:39:14 -0300 Christian Robottom Reis <[email protected]> wrote:
Hello there,
I run a diskless network where every user NFS mounts pretty much
everything including /home and/var/mail. It's often the case that a misbehaved client will leave a locked file stuck on the server -- today it is a file in a user's mail/ directory.Quoted 6 lines
Is there a way to query what files are being held locked by clients? I'm sure the kernel knows, as it is able to enforce the lock, but it isn't obvious how to extract that information -- lsof is documented to and indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.
Quoted 2 lines
/proc/lockswill generally show you all of the locks being held (assuming the filesystem's ->lock routine records the locks).
Great -- I couldn't find that documented elsewhere; perhaps that's obvious if that's how locking in the filesystem layer always works (i.e. for the locking operation to fail on an inode it must be recorded in /proc/locks).
I wonder, is there a tool which parses /proc/locks and grabs the filenames from the inode and device information? That would already be quite interesting to help preemptively debug locking problems.
Quoted 2 lines
It's not really possible to match those up with a particular client though.
Because the NFS server doesn't track which client requested a lock, and instead just passes the request through to the filesystem, I assume?
Perhaps there is nowhere scalable to record that today?
Quoted 8 lines
Quoted 4 lines
A related question is whether it is possible to break a client lock without rebooting the server (or restarting the NFS services).
Does anyone have any insight to share? Thanks,
I assume you're using NFSv3? What happens when the client "misbehaves"? Are the clients dropping offline or do you have applications that are just sitting on the lock and not releasing it?
In the situation which happened today my guess (because it's a mbox file) is that a client ran something like mutt and the machine died somewhere during shutdown. It's my guess because AIUI the lock doesn't get stuck if the process is simply KILLed or crashes.
Quoted 3 lines
If the clients are just going away without unlocking first, then you could consider using NFSv4, which has a lease-based locking. If the client goes away for a while (90s or so), then it'll lose its lock.
That's quite interesting -- and by client do you actually mean "client application" or client in the sense of the machine itself?
Quoted 3 lines
Alternately, there is the
/proc/fs/nfsd/unlock_ipinterface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used that so YMMV there.
Oh! That's a very interesting, and I now see it documented here:
http://people.redhat.com/rpeterso/Patches/NFS/NLM/004.txt
I've never seen it mentioned elsewhere. It doesn't seem to work:
kiko@chorus:~$ grep 9178074 /proc/locks
2: POSIX ADVISORY READ 2491 09:04:9178074 0 EOF
kiko@chorus:~$ echo "192.168.99.14" | sudo tee /proc/fs/nfsd/unlock_ip
192.168.99.14
kiko@chorus:~$ grep 9178074 /proc/locks
2: POSIX ADVISORY READ 2491 09:04:9178074 0 EOF(Where .14 is holding the lock -- I have tested and releasing it does make the lock entry disappear)
Also, if unlock_ip does work, that seems to not jive with the assertion above that the kernel doesn't track what locks are held by a given client. Surely if unlock_ip works for a given IP, somewhere that is tracked?
Thanks,
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko CEO, Async Open Source | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko
Quoted 7 lines · Christian Robottom Reis
Quoted 3 lines
Alternately, there is the
/proc/fs/nfsd/unlock_ipinterface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used that so YMMV there.Oh! That's a very interesting, and I now see it documented here:
On second look, I don't think that interface is meant to take a client IP, but rather a server IP:
"They are intended to allow admin or user mode script to release NLM
locks based on either a path name or a server in-bound ip address[...]"
That's why echoing the client IP makes no difference.
I'm surprised -- so far I've found no facility for lock management server-side other than restarting the server.
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kikoQuoted 12 lines · On Mon, 21 Mar 2016 17:56:38 -0300 Christian Robottom Reis <[email protected]> wrote:
Quoted 7 lines · Christian Robottom Reis
Quoted 3 lines
Alternately, there is the
/proc/fs/nfsd/unlock_ipinterface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used that so YMMV there.Oh! That's a very interesting, and I now see it documented here:
On second look, I don't think that interface is meant to take a client IP, but rather a server IP:
Quoted 7 lines
"They are intended to allow admin or user mode script to release NLM
locks based on either a path name or a server in-bound ip address[...]"That's why echoing the client IP makes no difference.
I'm surprised -- so far I've found no facility for lock management server-side other than restarting the server.
Ahh that's exactly right -- my bad. I had forgotten that the idea there was to use that for clustering.
And you're also correct that there is currently no facility for administratively revoking locks. That's something that would be a nice to have, if someone wanted to propose a sane interface and mechanism for it. Solaris had such a thing, IIRC, but I don't know how it was implemented.
There is one other option too -- you can send a SIGKILL to the lockd kernel thread and it will drop _all_ of its locks. That sort of sucks for all of the other clients, but it can unwedge things without restarting NFS.
Quoted 4 lines · your earlier email
In the situation which happened today my guess (because it's a mbox file) is that a client ran something like mutt and the machine died somewhere during shutdown. It's my guess because AIUI the lock doesn't get stuck if the process is simply KILLed or crashes.
What should happen there is that the client notify the server when it comes back up, so it can release its locks. That can fail to occur for all sorts of reasons, and that leads exactly to the problem you have now. It's also possible for the client to just drop off the net indefinitely while holding locks in which case you're just out of luck.
It really is better to use NFSv4 if you can at all get away with it. Lease-based locking puts the onus on the client to stay in contact with the server if it wants to maintain its state.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 5 lines · Jeff Layton
And you're also correct that there is currently no facility for administratively revoking locks. That's something that would be a nice to have, if someone wanted to propose a sane interface and mechanism for it. Solaris had such a thing, IIRC, but I don't know how it was implemented.
I might look into that -- I think the right thing to do is (as you had originally alluded to) dropping all locks pertaining to a specific client, as the only failure scenario that can't be worked around that I'm thinking about is the client disappearing.
I would also like to understand whether the data structure behind /proc/locks could be extended to provide additional metadata which the nfs kernel client could annotate to indicate client information. That would allow one to figure out who the actual culprit machine was.
Quoted 4 lines
There is one other option too -- you can send a SIGKILL to the lockd kernel thread and it will drop _all_ of its locks. That sort of sucks for all of the other clients, but it can unwedge things without restarting NFS.
That's quite useful to know, thanks -- I knew that messing with the initscripts responsible for the nfs kernel services "fixed" the problem, but killing lockd is much more convenient.
I wonder, is it normal application behaviour that any locks dropped would be detected and reestablished on the client side?
Quoted 10 lines
Quoted 4 lines
In the situation which happened today my guess (because it's a mbox file) is that a client ran something like mutt and the machine died somewhere during shutdown. It's my guess because AIUI the lock doesn't get stuck if the process is simply KILLed or crashes.
What should happen there is that the client notify the server when it comes back up, so it can release its locks. That can fail to occur for all sorts of reasons, and that leads exactly to the problem you have now. It's also possible for the client to just drop off the net indefinitely while holding locks in which case you're just out of luck.
That's quite interesting. I had initially thought that a misbehaved application could die while holding a lock, but it seems as though the client kernel tracks any remote locks held and releases them regardless of how the process dies. It seems like the actual problem scenarios are:
- Client disappears off the net while holding a lock
- Client kernel fails to clear NFS locks (likely a bug)
- Rogue or misbehaved client holds a lock indefinitely
In any of these cases, the useful thing to know is which client actually holds the lock.
Quoted 3 lines
It really is better to use NFSv4 if you can at all get away with it. Lease-based locking puts the onus on the client to stay in contact with the server if it wants to maintain its state.
I considered moving a few times, but the setup here is a bit fragile and AIUI NFSv4 isn't a straight drop-in for v3. Beyond modifying nfsvers, IIRC at least idmapd needed to be set up and perhaps there was more.
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kikoQuoted 22 lines · Christian Robottom Reis
Quoted 5 lines · Jeff Layton
And you're also correct that there is currently no facility for administratively revoking locks. That's something that would be a nice to have, if someone wanted to propose a sane interface and mechanism for it. Solaris had such a thing, IIRC, but I don't know how it was implemented.
I might look into that -- I think the right thing to do is (as you had originally alluded to) dropping all locks pertaining to a specific client, as the only failure scenario that can't be worked around that I'm thinking about is the client disappearing.
I would also like to understand whether the data structure behind
/proc/lockscould be extended to provide additional metadata which the nfs kernel client could annotate to indicate client information. That would allow one to figure out who the actual culprit machine was.Quoted 4 lines
There is one other option too -- you can send a SIGKILL to the lockd kernel thread and it will drop _all_ of its locks. That sort of sucks for all of the other clients, but it can unwedge things without restarting NFS.
Quoted 6 lines
That's quite useful to know, thanks -- I knew that messing with the initscripts responsible for the nfs kernel services "fixed" the problem, but killing lockd is much more convenient.
I wonder, is it normal application behaviour that any locks dropped would be detected and reestablished on the client side?
No, you generally don't want that--you don't want an application to believe it's held a lock continuously when it reality it's been dropped (and conflicting locks possibly granted and dropped) and then acquired again.
Client behavior varies. I believe recent linux clients should return -EIO on subsequent attempts to use associated file descriptors after a lock is lost. Other OS's apparently signal the process.
--b.
Quoted 23 lines · Jeff Layton
Quoted 12 lines · On Mon, 21 Mar 2016 17:56:38 -0300 Christian Robottom Reis…
Quoted 7 lines · Christian Robottom Reis
Quoted 3 lines
Alternately, there is the
/proc/fs/nfsd/unlock_ipinterface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used that so YMMV there.Oh! That's a very interesting, and I now see it documented here:
On second look, I don't think that interface is meant to take a client IP, but rather a server IP:
Quoted 7 lines
"They are intended to allow admin or user mode script to release NLM
locks based on either a path name or a server in-bound ip address[...]"That's why echoing the client IP makes no difference.
I'm surprised -- so far I've found no facility for lock management server-side other than restarting the server.
Quoted 7 lines
Ahh that's exactly right -- my bad. I had forgotten that the idea there was to use that for clustering.
And you're also correct that there is currently no facility for administratively revoking locks. That's something that would be a nice to have, if someone wanted to propose a sane interface and mechanism for it.
I was all set to give you an answer until I saw that word "sane"... nearly scared me off, but I chose to persist.
You know this, but let me remind you and inform Christian.
When an NFS client ("bob") asks the NFS server ("jane") to lock a file (the first time), the kernel says to statd "Hey, Bob wants a lock. Can you keep and eye on him and let me know when he reboots - when he does I want to discard his locks".
So statd on Jane talks to statd on Bob saying "Hey Bob, tell me if you ever reboot - OK"? Bob takes note of this request by writing "Jane" in /var/lib/nfs/sm.
When Bob reboots, sm-notify sees "Jane" in /var/lib/nfs/sm and sends a message to statd on Jane "Hey Jane, Bob just rebooted. You're welcome".
statd on Jane then tells the kernel "Bob rebooted" and the kernel drops all those locks.
And there, in that last step, we see the key. It is already possible to tell the kernel "drop all the locks held by Bob", you just have to say "Hey, I'm statd - Bob rebooted". Or maybe we could stay to statd "Hi Jane, this is Bob, I just rebooted" - even though we aren't really Bob.
(Or we could just reboot Bob and let it do the talking).
I'd have to hunt through the statd code to figure out what is possible and what is best. It can't be too hard though.
Christian: If the problem client actually comes back up (instead of staying down) do the locks get drops as they should?
NeilBrown
Quoted 10 lines · Jeff Layton
Quoted 5 lines
Quoted 1 line · On Mon, 21 Mar 2016 17:56:38 -0300 Christian Robottom Reis…
On Mon, Mar 21, 2016 at 02:55:00PM -0300, Christian Robottom Reis
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Alternately, there is the
/proc/fs/nfsd/unlock_ipinterface. Supposedly you can echo an address into there and it'll forcibly drop all of the locks that that that client holds. I've not used
that so
Quoted 11 lines
YMMV there.
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Oh! That's a very interesting, and I now see it documented here:
On second look, I don't think that interface is meant to take a client IP, but rather a server IP:
"They are intended to allow admin or user mode script to release NLM
locks based on either a path name or a server in-bound ip
address[...]"
Quoted 14 lines
Quoted 6 lines
Quoted 5 lines
That's why echoing the client IP makes no difference.
I'm surprised -- so far I've found no facility for lock management server-side other than restarting the server.
Quoted 7 lines
Ahh that's exactly right -- my bad. I had forgotten that the idea there was to use that for clustering.
And you're also correct that there is currently no facility for administratively revoking locks. That's something that would be a nice to have, if someone wanted to propose a sane interface and mechanism for it.
Quoted 6 lines
I was all set to give you an answer until I saw that word "sane"... nearly scared me off, but I chose to persist.
You know this, but let me remind you and inform Christian.
When an NFS client ("bob") asks the NFS server ("jane") to lock a file
(the
Quoted 5 lines
first time), the kernel says to statd "Hey, Bob wants a lock. Can you keep and eye on him and let me know when he reboots - when he does I want to discard his locks".
So statd on Jane talks to statd on Bob saying "Hey Bob, tell me if you
ever
Quoted 2 lines
reboot - OK"? Bob takes note of this request by writing "Jane" in
/var/lib/nfs/sm.
Actually, Jane's statd and Bob's statd don't talk until one of them comes back up after reboot...
The actual process is:
1. application on Bob makes a lock request on a mount from jane
2. Bob's lockd asks Bob's statd to record jane as a party of interest after reboot
3. Bob's lockd requests lock from Jane's lockd
4. Jane's lockd asks Jane's statd to record Bob as a party of interest after reboot
5. Jane's lockd completes lock request and responds to Bob's lockd
6. Bob crashes
7. Jane is oblivious to this crash for now...
8. Bob restarts and Bob's statd NOW sends a message to Jane's statd that it rebooted
9 Jane's statd notes that Jane's lockd took and interest in Bob, and passes on the fact that Bob rebooted
10. Jane's lockd cleans up the lock
Quoted 4 lines
When Bob reboots, sm-notify sees "Jane" in
/var/lib/nfs/smand sends a message to statd on Jane "Hey Jane, Bob just rebooted. You're welcome".statd on Jane then tells the kernel "Bob rebooted" and the kernel drops
all
Quoted 3 lines
those locks.
And there, in that last step, we see the key. It is already possible to
tell the
Quoted 1 line
kernel "drop all the locks held by Bob", you just have to say "Hey, I'm
statd -
Quoted 1 line
Bob rebooted". Or maybe we could stay to statd "Hi Jane, this is Bob, I
just
Quoted 1 line
rebooted" - even though we aren't really Bob.
Yes, either of those would work. I'm pretty sure there are tools out there that do this.
What is sometimes more interesting to sysadmins is being able to free up a specific lock rather than ALL locks held by that client.
Quoted 6 lines
(Or we could just reboot Bob and let it do the talking).
I'd have to hunt through the statd code to figure out what is possible and what is best. It can't be too hard though.
Christian: If the problem client actually comes back up (instead of
staying
Quoted 3 lines
down) do the locks get drops as they should?
NeilBrown
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Quoted 3 lines · Christian Robottom Reis
indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.
Somewhat OT, but I find it a PITA that /proc/locks gives inode numbers that then need to be looked up individually. I have often been surprised no tool exists to parse that and give you back a report of filenames, so I just put together a small tool that just offloads the work to debugfs.
I've attached it in case others might find it useful.
(Interestingly, in a network of Ubuntu desktops, the long-term remote lock holders are basically Spotify, Firefox and Zeitgeist, of which the latter two are essentially sqlite.)
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko
| [+55 16] 991 126 430 | http://launchpad.net/~kiko#!/usr/bin/env python3
import os
import sys
import subprocess
import io
locksdb = open("/proc/locks")device_inodes = {}
for lockline in locksdb:
lock_inode_raw = lockline.split()[5]
lock_inode = lock_inode_raw.split(":")
if len(lock_inode) < 3:
#print("Ignoring %s" % lock_inode)
continue
device = tuple(map(int,lock_inode[0:2]))
try:
target_device = os.readlink("/sys/dev/block/%s:%s" % device) target_device = target_device.split("/")[-1]
except OSError:
#print("Device %s:%s not found" % device) continue
inode = lock_inode[2]
if not device_inodes.get(target_device):
device_inodes[target_device] = []
device_inodes[target_device].append(inode)
for device in device_inodes:
print("-> Inodes on %s" % device)
inode_input = []
for inode in device_inodes[device]:
inode_input.append("ncheck %s" % inode)
debugfs = subprocess.Popen(['debugfs', '/dev/%s' % device],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
debugfs_input = "\n".join(inode_input)
out, err = debugfs.communicate(input=debugfs_input.encode()) debugfs_output = out.decode()
for line in debugfs_output.split("\n"):
if line.startswith("debugfs:"):
continue
if line.startswith("Inode"):
continue
print(line)
Quoted 11 lines · Christian Robottom Reis
Quoted 3 lines · Christian Robottom Reis
indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.Somewhat OT, but I find it a PITA that
/proc/locksgives inode numbers that then need to be looked up individually. I have often been surprised no tool exists to parse that and give you back a report of filenames, so I just put together a small tool that just offloads the work to debugfs.I've attached it in case others might find it useful.
Nice idea, though it only works with extXfs - better than nothing though.
It would be really easy to do this in the kernel.
I would be very much in favour of a /proc/locks-extended (or whatever) that provides the same information as /proc/locks, but in a format which includes full path name, process name, and - for nfs locks - client name etc.
NeilBrown
Quoted 15 lines · NeilBrown
Quoted 11 lines · Christian Robottom Reis
Quoted 3 lines · Christian Robottom Reis
indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.Somewhat OT, but I find it a PITA that
/proc/locksgives inode numbers that then need to be looked up individually. I have often been surprised no tool exists to parse that and give you back a report of filenames, so I just put together a small tool that just offloads the work to debugfs.I've attached it in case others might find it useful.
Quoted 8 lines
Nice idea, though it only works with extXfs - better than nothing though.
It would be really easy to do this in the kernel.
I would be very much in favour of a/proc/locks-extended(or whatever) that provides the same information as/proc/locks, but in a format which includes full path name, process name, and - for nfs locks - client name etc.
Why isn't the current interface sufficient? The 'lslocks' utility appears quite capable of extracting the full path info.
Cheers
Trond
Quoted 19 lines · Trond Myklebust
Quoted 18 lines · NeilBrown
Quoted 11 lines · Christian Robottom Reis
Quoted 3 lines · Christian Robottom Reis
indeed does not return any information pertaining NFS client locks, and I'm not clear whether
/proc/locks(on the server side obviously) does or not.Somewhat OT, but I find it a PITA that
/proc/locksgives inode numbers that then need to be looked up individually. I have often been surprised no tool exists to parse that and give you back a report of filenames, so I just put together a small tool that just offloads the work to debugfs.I've attached it in case others might find it useful.
Nice idea, though it only works with extXfs - better than nothing though.
Quoted 9 lines
Quoted 5 lines
It would be really easy to do this in the kernel.
I would be very much in favour of a/proc/locks-extended(or whatever) that provides the same information as/proc/locks, but in a format which includes full path name, process name, and - for nfs locks - client name etc.Why isn't the current interface sufficient? The 'lslocks' utility appears quite capable of extracting the full path info.
I don't think I've come across 'lslocks' before - thanks. It does help a bit but when I run it, most of the "PATH" column is empty.
Which is odd because I can fill in some of the blanks by cross referencing pid/inode from /proc/locks with inodes numbers from
ls -liL /proc/$PID/fd
to get full names.
Any way, you are right that more information is available if you are willing to hunt around, but not everything. In particular, for locks held by lockd (or NFSv4 locks held by nfsd) it would be nice to know which client host it was held on behalf of.
NeilBrown