automatic freeing of space on ENOSPC
Subsystems (3)
Is there anything analogous to a "shrinker", but for disk space? So, some hook that a filesystem could call to say "I'm running out of space, could you please free something?", before giving up and returning ENOSPC?
The NFS server currently revokes a client's state if the client fails to contact it within a lease period (90 seconds by default). That's harsher than necessary--if a network partition lasts longer than a lease period, but if nobody else needs that client's resources, it'd be nice to be able to hang on to them so that the client could resume normal operation after the network comes back. So we'd delay revoking the client's state until there's an actual conflict. But that means we need a way to clean up the client as soon as there is a conflict, to avoid unnecessarily failing operations that conflict with resources held by an expired client.
At first I thought we only needed to worry about file locks, but then I realized clients can also hold references to files, which might be unlinked. I don't want a long-expired client to result in ENOSPC to other filesystem users.
Any ideas?
I searched around and found this discussion of volatile ranges
https://lwn.net/Articles/522135/, which seems close, but I don't know if anything came of that in the end.
--b.
Quoted 22 lines · J. Bruce Fields
Is there anything analogous to a "shrinker", but for disk space? So, some hook that a filesystem could call to say "I'm running out of space,
could you please free something?", before giving up and returning ENOSPC?The NFS server currently revokes a client's state if the client fails to
contact it within a lease period (90 seconds by default). That's harsher than necessary--if a network partition lasts longer than a lease
period, but if nobody else needs that client's resources, it'd be nice
to be able to hang on to them so that the client could resume normal operation after the network comes back. So we'd delay revoking the client's state until there's an actual conflict. But that means we need
a way to clean up the client as soon as there is a conflict, to avoid unnecessarily failing operations that conflict with resources held by an
expired client.
Quoted 8 lines
At first I thought we only needed to worry about file locks, but then I
realized clients can also hold references to files, which might be unlinked. I don't want a long-expired client to result in ENOSPC to other filesystem users.Any ideas?
How about just setting up a notification for unlink on those files, the same way we set up notifications for close with the NFSv3 filecache in nfsd?
Quoted 6 lines
I searched around and found this discussion of volatile ranges
https://lwn.net/Articles/522135/, which seems close, but I don't know if
anything came of that in the end.--b.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 3 lines · Trond Myklebust
How about just setting up a notification for unlink on those files, the same way we set up notifications for close with the NFSv3 filecache in nfsd?
Yes, that'd probably work. It'd be better if we didn't have to throw away unlinked files when the client expires, but it'd still be an incremental improvement over what we do now.
--b.
Quoted 8 lines · On Tue, 29 Jun 2021, [email protected] wrote:
Quoted 3 lines · Trond Myklebust
How about just setting up a notification for unlink on those files, the same way we set up notifications for close with the NFSv3 filecache in nfsd?
Yes, that'd probably work. It'd be better if we didn't have to throw away unlinked files when the client expires, but it'd still be an incremental improvement over what we do now.
I wonder how important this is. If an NFS client unlinks a file that it has open, it will be silly_renamed, and if the client then goes silent, it might never be removed. So we already theoretically have a possibilty of ENOSPC due to silent clients. Have we heard of this becoming a problem?
Is there reason to think that the Courteous server changes will make this problem more likely?
NeilBrown
Quoted 11 lines · NeilBrown
Quoted 8 lines · On Tue, 29 Jun 2021, [email protected] wrote:
Quoted 3 lines · Trond Myklebust
How about just setting up a notification for unlink on those files, the same way we set up notifications for close with the NFSv3 filecache in nfsd?
Yes, that'd probably work. It'd be better if we didn't have to throw away unlinked files when the client expires, but it'd still be an incremental improvement over what we do now.
Quoted 8 lines
I wonder how important this is. If an NFS client unlinks a file that it has open, it will be
silly_renamed, and if the client then goes silent, it might never be removed. So we already theoretically have a possibilty of ENOSPC due to silent clients. Have we heard of this becoming a problem?
Is there reason to think that the Courteous server changes will make this problem more likely?
To me, stale silly renamed files sounds like a problem worth fixing not as an excuse to create another similar problem.
w.r.t pre-ENOSPC notification, I don't know of such notification in filesystems. It exists for some thin-provisioned storage devices (thinp as well I think), but that is not very useful for nfsd.
OTOH, ENOSPC is rarely a surprising event.
I believe you can get away with tunable for nfsd, such as % of available storage space that may consumed for "opportunistic caching".
Polling for available storage space every least time or so in case there are possibly forgotten unlinked files should be sufficient for any practical purpose IMO.
Thanks,
Amir.
Quoted 2 lines · Amir Goldstein
To me, stale silly renamed files sounds like a problem worth fixing not as an excuse to create another similar problem.
Yeah, it's ugly, I'd like to fix it some day, but given that people have lived with it since forever it's not the highest priority.
Quoted 4 lines
w.r.t pre-ENOSPC notification, I don't know of such notification in filesystems. It exists for some thin-provisioned storage devices (thinp as well I think), but that is not very useful for nfsd.
Quoted 8 lines
OTOH, ENOSPC is rarely a surprising event.
I believe you can get away with tunable for nfsd, such as % of available storage space that may consumed for "opportunistic caching".Polling for available storage space every least time or so in case there are possibly forgotten unlinked files should be sufficient for any practical purpose IMO.
Makes sense, thanks for the perspective.
--b.
Quoted 5 lines · NeilBrown
I wonder how important this is. If an NFS client unlinks a file that it has open, it will be
silly_renamed, and if the client then goes silent, it might never be removed. So we already theoretically have a possibilty of ENOSPC due to silent clients. Have we heard of this becoming a problem?
Oh, that's a good point. I've seen complaints about sillyrename files, but I can't recall ever seen a single complaint about their causing ENOSPC.
Quoted 2 lines
Is there reason to think that the Courteous server changes will make this problem more likely?
So I guess the only new cases the courteous server will introduce are even less likely (probably just the case where a file is unlinked by a client other than the one that has it open).
So I think doing nothing for now is an acceptable alternative....
--b.
Quoted 15 lines · On Jun 29, 2021, at 2:32 PM, [email protected] wrote:
Quoted 5 lines · NeilBrown
I wonder how important this is. If an NFS client unlinks a file that it has open, it will be
silly_renamed, and if the client then goes silent, it might never be removed. So we already theoretically have a possibilty of ENOSPC due to silent clients. Have we heard of this becoming a problem?Oh, that's a good point. I've seen complaints about sillyrename files, but I can't recall ever seen a single complaint about their causing ENOSPC.
Quoted 8 lines
Quoted 2 lines
Is there reason to think that the Courteous server changes will make this problem more likely?
So I guess the only new cases the courteous server will introduce are even less likely (probably just the case where a file is unlinked by a client other than the one that has it open).
So I think doing nothing for now is an acceptable alternative....
I'm comfortable with that too. We can keep a careful eye on this as the courteous server moves further along.
--
Chuck Lever
Quoted 4 lines · J. Bruce Fields
Is there anything analogous to a "shrinker", but for disk space? So, some hook that a filesystem could call to say "I'm running out of space, could you please free something?", before giving up and returning ENOSPC?
The only thing the filesystem can do at this point is run internal garbage collection operations to free up space that it has either speculatively allocated or has deferred the freeing and/or cleanupi of. In general, filesystems already do this when they are approaching/at ENOSPC, so it seems to me like there's little scope for an external "free some space" trigger to be able to make much difference.
I mean, we do have an ioctl in XFS to trigger garbage collection of speculative preallocation from userspace (XFS_IOC_FREE_EOFBLOCKS and the prealloc command in xfs_spaceman(8)) but this was written because some people want accurate space accounting for things like quota reports....
Regardless, we run these same GC triggers internally before we declare ENOSPC, so there's no real need for applications to run them when they get an ENOSPC error....
Quoted 10 lines
The NFS server currently revokes a client's state if the client fails to contact it within a lease period (90 seconds by default). That's harsher than necessary--if a network partition lasts longer than a lease period, but if nobody else needs that client's resources, it'd be nice to be able to hang on to them so that the client could resume normal operation after the network comes back. So we'd delay revoking the client's state until there's an actual conflict. But that means we need a way to clean up the client as soon as there is a conflict, to avoid unnecessarily failing operations that conflict with resources held by an expired client.
I'm not sure what you are asking for filesystems to do here. This seems like an application problem - revoking the client's open file state and cleaning up silly rename files is application level garbage collection, not filesystem level stuff. Maybe I've misunderstood what you are trying to do, perhaps you could clarify what you're expecting the filesystems to be able to clean up here?
Quoted 3 lines
I searched around and found this discussion of volatile ranges
https://lwn.net/Articles/522135/, which seems close, but I don't know if anything came of that in the end.
Nothing that I know of.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 12 lines · Dave Chinner
Quoted 10 lines · J. Bruce Fields
The NFS server currently revokes a client's state if the client fails to contact it within a lease period (90 seconds by default). That's harsher than necessary--if a network partition lasts longer than a lease period, but if nobody else needs that client's resources, it'd be nice to be able to hang on to them so that the client could resume normal operation after the network comes back. So we'd delay revoking the client's state until there's an actual conflict. But that means we need a way to clean up the client as soon as there is a conflict, to avoid unnecessarily failing operations that conflict with resources held by an expired client.
Quoted 4 lines
I'm not sure what you are asking for filesystems to do here. This seems like an application problem - revoking the client's open file state and cleaning up silly rename files is application level garbage collection, not filesystem level stuff.
Right, the "application" in this case is knfsd. It may be keeping some unlinked files around that it doesn't really need to. So I'm basically wondering if I could get a notification from the filesystem that now would be a good time to close those files.
I think Neil's convinced me this isn't a priority, though....
--b.
Quoted 4 lines · J. Bruce Fields
Is there anything analogous to a "shrinker", but for disk space? So, some hook that a filesystem could call to say "I'm running out of space, could you please free something?", before giving up and returning ENOSPC?
In addition to the issues raised by Neil, Amir, Dave, and others, the other challenge with the file system calling a "please try to free something before I return ENOSPC" is that this would almost certainly require blocking a system call while some userspace daemon tried to free up some space --- or were you thinking that the nfsd kernel code would be tracking all of the silly-rename files so it could release space really quickly on demand?
Even if this is only a kernel callback, I'd be concerned about potential locking hierarchy problems if we are calling out from block allocation subsystem to nfsd, only to have nfsd call back in to request unlinking a silly-renamed file.
So the suggestion that we not wait until we're down to 0 blocks free, but when we hit some threshold (say, free space dips below N minutes worth of worst or average case block allocations), trigger code which deletes silly-renamed files, is probably the best way to go. In which case, a callback is not what is needed; and if N is large enough, this could done via a pure user-space-only solution.
- Ted
Quoted 6 lines · Theodore Ts'o
Quoted 4 lines · J. Bruce Fields
Is there anything analogous to a "shrinker", but for disk space? So, some hook that a filesystem could call to say "I'm running out of space, could you please free something?", before giving up and returning ENOSPC?
Quoted 7 lines
In addition to the issues raised by Neil, Amir, Dave, and others, the other challenge with the file system calling a "please try to free something before I return ENOSPC" is that this would almost certainly require blocking a system call while some userspace daemon tried to free up some space --- or were you thinking that the nfsd kernel code would be tracking all of the silly-rename files so it could release space really quickly on demand?
Something like that, yep.
Quoted 11 lines
Even if this is only a kernel callback, I'd be concerned about potential locking hierarchy problems if we are calling out from block allocation subsystem to nfsd, only to have nfsd call back in to request unlinking a silly-renamed file.
So the suggestion that we not wait until we're down to 0 blocks free, but when we hit some threshold (say, free space dips below N minutes worth of worst or average case block allocations), trigger code which deletes silly-renamed files, is probably the best way to go. In which case, a callback is not what is needed; and if N is large enough, this could done via a pure user-space-only solution.
Makes sense, thanks!
--b.