Network filesystem cache management system call
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the pioctl() and afs() system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.
Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
A brainstorming session on this would be useful.
David
Quoted 10 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
Quoted 5 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
A brainstorming session on this would be useful.
This is definitely something I'd like to discuss as well. The posix_fadvise() call doesn't interact with the filesystem at all, so it isn't a useful source of cache management hints. We've implemented a Lustre-specific ladvise hint using an ioctl to advise client and server caches, but having a generic hint API would be better.
Cheers, Andreas
Quoted 16 lines · Andreas Dilger
Quoted 15 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
Quoted 5 lines
Quoted 1 line
A brainstorming session on this would be useful.
This is definitely something I'd like to discuss as well. The
posix_fadvise()call doesn't interact with the filesystem at all, so it isn't a useful source of cache management hints.
That could be remedied though. Maybe a new file_operations member that posix_fadvise could call when it's defined by the fs? I think it might be useful to explore that avenue before proposing new interfaces.
Quoted 4 lines
We've implemented a Lustre-specific ladvise hint
using an ioctl to advise client and server caches, but having a generic hint API would be better.
Agreed.
Ceph also has a private ioctl to toggle lazy write caching. Also, ISTR that Trond had some patches at one point to let you micromanage the NFS client caches. All of those are potentially useful, but without a standard way to access them, it's hard to write applications that are fs-agnostic.
For that reason, a hinting mechanism like posix_fadvise would seem to be the best approach, IMO. The kernel could be free to ignore any of those calls on filesystems where it's not implemented or not applicable.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 9 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
Quoted 5 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
A brainstorming session on this would be useful.
I guess this would be very interesting for Lustre as well.
Bye,
Oleg
Quoted 9 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
Quoted 5 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
A brainstorming session on this would be useful.
Another thing that I would like to work out is how to do mountpoint management commands for kAFS. This includes doing the following operations:
(*) Creating a mountpoint.
(*) Deleting a mountpoint
(*) Reading a mountpoint.
(*) Invalidating cached mountpoint information.
The problem here is that mountpoints are automatically mounted when you touch them unless you specify the AT_NO_AUTOMOUNT flag - and even then, that's ignored if the mountpoint is already mounted upon.
I could do this by saying that you have to open the parent directory of the mountpoint and do an ioctl() on it, but would it be better to have one or more syscalls for this purpose?
Further, would this be of use to any other filesystems?
David
Quoted 11 lines · David Howells
Quoted 9 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
Quoted 5 lines
Quoted 5 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
A brainstorming session on this would be useful.
I would be happy to participate.
Quoted 11 lines
Another thing that I would like to work out is how to do mountpoint management commands for kAFS. This includes doing the following operations:
(*) Creating a mountpoint.
(*) Deleting a mountpoint
(*) Reading a mountpoint.
(*) Invalidating cached mountpoint information.
Quoted 7 lines
The problem here is that mountpoints are automatically mounted when you touch them unless you specify the
AT_NO_AUTOMOUNTflag - and even then, that's ignored if the mountpoint is already mounted upon.I could do this by saying that you have to open the parent directory of the mountpoint and do an
ioctl()on it, but would it be better to have one or more syscalls for this purpose?
I am uncomfortable with opening the parent directory object because it introduces object reference confusion when the path to the parent directory is itself a symlink or a mount point.
Granted, the AFS3 protocol does not make this easy.
1. An AFS3 mount point is a special type of symlink
2. AFS3 does not provide a method by which a symlink's target can be altered or its type changed after it is created.
3. Legacy AFS3 clients have always exposed mount points to applications as a directory not a symlink.
If I had my preference, I would implement creating a mount point as:
1. create an empty file or symlink
2. open the resulting object and issue an ioctl against that object to set a mount point target which results in it becoming a mount point.
Deleting, Reading, Modifying, and Invalidating all are implemented as
1. open mount point
2. issue ioctl against object
Unfortunately, AFS3 simply doesn't provide the functionality I desire. The best that could be done would be to implement a createMountPoint ioctl which operates by deleting the open object and creating a new mount point object in its place. While dealing with the fact that the AFS FID for the object object will change as a result.
Quoted 2 lines
Further, would this be of use to any other filesystems?
Could SMB use this as a method of defining referrals?
Jeffrey Altman
begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United States email;internet:[email protected]
title:Founder and CEO
tel;work:+1-212-769-9018
note;quoted-printable:LinkedIn: https://www.linkedin.com/in/jeffreyaltman=0D=0A=
Skype: jeffrey.e.altman=0D=0A=
url:https://www.auristor.com/
version:2.1
end:vcard
Quoted 11 lines · Jeffrey Altman
Quoted 3 lines
I could do this by saying that you have to open the parent directory of the mountpoint and do an
ioctl()on it, but would it be better to have one or more syscalls for this purpose?I am uncomfortable with opening the parent directory object because it introduces object reference confusion when the path to the parent directory is itself a symlink or a mount point.
Granted, the AFS3 protocol does not make this easy.
1. An AFS3 mount point is a special type of symlink
Whilst this is true in the protocol, kAFS maps that into a directory that has a ->d_automount() op.
Quoted 2 lines
2. AFS3 does not provide a method by which a symlink's target can be altered or its type changed after it is created.
Can the mountpoint object be renamed and/or moved between directories?
Quoted 2 lines
3. Legacy AFS3 clients have always exposed mount points to applications as a directory not a symlink.
It should be noted here that legacy AFS3 clients have a single superblock that covers all the volumes on all the cells that they mount, so mountpoints aren't actually visible to the VFS.
In contrast, with kAFS, each volume is kept as a separate superblock and that is mounted directly on the mountpoint.
A further note: I believe that a volume must have a directory at the base and cannot have a mountpoint there - so you shouldn't get stacked mountpoints without interference by the local sysadmin manually issuing mount commands.
Quoted 7 lines
If I had my preference, I would implement creating a mount point as:
1. create an empty file or symlink
2. open the resulting object and issue an ioctl against that object to set a mount point target which results in it becoming a mount point.
Opening the mountpoint object, if it's a dangling symlink, could be tricky. There's an O_PATH that you can open with, but I'm not sure that's of use; further, if the automountpoint is actually mounted over, you can't get to the mountpoint without stepping over it. Even O_NOFOLLOW and AT_NO_AUTOMOUNT don't help.
David
Quoted 10 lines · David Howells
Quoted 8 lines · Jeffrey Altman
Quoted 3 lines
I could do this by saying that you have to open the parent directory of the mountpoint and do an
ioctl()on it, but would it be better to have one or more syscalls for this purpose?I am uncomfortable with opening the parent directory object because it introduces object reference confusion when the path to the parent directory is itself a symlink or a mount point.
Quoted 6 lines
Quoted 3 lines
Granted, the AFS3 protocol does not make this easy.
1. An AFS3 mount point is a special type of symlink
Whilst this is true in the protocol, kAFS maps that into a directory that has a ->d_automount() op.
that is a fine representation.
Quoted 4 lines
Quoted 2 lines
2. AFS3 does not provide a method by which a symlink's target can be altered or its type changed after it is created.
Can the mountpoint object be renamed and/or moved between directories?
yes, and it is theoretically possible for its target to be modified. If that happens, the data version of the object will change.
Quoted 6 lines
Quoted 2 lines
3. Legacy AFS3 clients have always exposed mount points to applications as a directory not a symlink.
It should be noted here that legacy AFS3 clients have a single superblock that covers all the volumes on all the cells that they mount, so mountpoints aren't actually visible to the VFS.
correct, and this does cause a variety of problems. there is no method of reporting the size of the volume or the amount of free space in a volume. this is a side effect of /afs being exposed to the VFS as a single device that includes all AFS3 storage everywhere in the world.
Quoted 7 lines
In contrast, with kAFS, each volume is kept as a separate superblock and that is mounted directly on the mountpoint.
A further note: I believe that a volume must have a directory at the base and cannot have a mountpoint there - so you shouldn't get stacked mountpoints without interference by the local sysadmin manually issuing mount commands.
A volume by definition is a rooted directory tree where the volume root directory vnode id is 1 and the root directory has no parent.
Quoted 8 lines
Quoted 7 lines
If I had my preference, I would implement creating a mount point as:
1. create an empty file or symlink
2. open the resulting object and issue an ioctl against that object to set a mount point target which results in it becoming a mount point.
Quoted 5 lines
Opening the mountpoint object, if it's a dangling symlink, could be tricky. There's an
O_PATHthat you can open with, but I'm not sure that's of use; further, if the automountpoint is actually mounted over, you can't get to the mountpoint without stepping over it. EvenO_NOFOLLOWandAT_NO_AUTOMOUNTdon't help.
David and I spoke offline. Create and Unlink are operations on the directory and should remain so. Read is an operation best performed on the mount point itself.
Jeffrey Altman
begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United States email;internet:[email protected]
title:Founder and CEO
tel;work:+1-212-769-9018
note;quoted-printable:LinkedIn: https://www.linkedin.com/in/jeffreyaltman=0D=0A=
Skype: jeffrey.e.altman=0D=0A=
url:https://www.auristor.com/
version:2.1
end:vcard
Quoted 5 lines · David Howells
AFS filesystem implementations like OpenAFS have a number of management calls usually implemented through the
pioctl()andafs()system calls. However, there is general revulsion to the idea of allowing these in Linux as they are very much abused.
Quoted 7 lines
Now, I need to come up with replacements for these. For some I can use getxattr()/setxattr(), for some keyrings and for others procfs/sysfs, but for some I can't.
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
Do you have a summary of the AFS interface to give an idea what's needed?
--b.
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
Do you have a summary of the AFS interface to give an idea what's needed?
I have the pioctls listed here that I need to emulate:
https://www.infradead.org/~dhowells/kafs/user_interface.html
along with my thoughts on how to do that.
For cache wangling, I was thinking of something like:
fcachectl(int dirfd,
const char *pathname,
unsigned atflags,
const char *cmd,
char *result,
size_t *result_len);
The relevant pioctls are:
(*) VIOCGETCACHEPARMS
Get the size of the cache.
(*) VIOCSETCACHESIZE
Set the cache size.
(*) VIOCFLUSH
Invalidate the cached information for an object, both the inode/dentry structs and anything in the local cache.
(*) VIOCFLUSHCB
Invalidate any callbacks/leases outstanding on an object. This might make more sense to be done via the same mechanism as lease/lock management.
(*) VIOC_FLUSHVOLUME
Flush all cached state for a volume, both from RAM and local disk cache as far as possible. Files that are open aren't necessarily affected.
(*) VIOC_FLUSHALL
FLush all cached state for all volumes.
(*) VIOCPREFETCH
Prefetch a file into the cache.
So, maybe:
fcachectl(AT_FDCWD, "/afs/user/dhowells", 0, "flush volume", NULL, NULL);
to flush an AFS volume containing my home directory.
Note that doing this by fcntl() or ioctl() has potential difficulties as it would have to work on non-file objects such as device files or symlinks.
Other functions that this could be used for are cache pinning and fixup/integration should we ever want disconnected operation.
David
Quoted 15 lines · David Howells
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines
One of these is the call to manage local caching on a file or volume. This, however, doesn't really need to be limited to AFS, but could also be applicable to NFS, CIFS, etc. - and possibly even to local filesystems.
Do you have a summary of the AFS interface to give an idea what's needed?
I have the pioctls listed here that I need to emulate:
https://www.infradead.org/~dhowells/kafs/user_interface.html
along with my thoughts on how to do that.
Quoted 9 lines
For cache wangling, I was thinking of something like:
fcachectl(int dirfd,
const char *pathname,
unsigned atflags,
const char *cmd,
char *result,
size_t*result_len);
I think it might be more useful to wire posix_fadvise into the filesystem drivers somehow. A hinting interface really seems like the right approach here, given the differences between different filesystems.
Quoted 6 lines
The relevant pioctls are:
(*) VIOCGETCACHEPARMS
Get the size of the cache.
Global or per-inode cache?
Quoted 9 lines
(*) VIOCSETCACHESIZE
Set the cache size.
(*) VIOCFLUSH
Invalidate the cached information for an object, both the inode/dentry structs and anything in the local cache.
Maybe POSIX_FADV_DONTNEED ?
Quoted 6 lines
(*) VIOCFLUSHCB
Invalidate any callbacks/leases outstanding on an object. This might make more sense to be done via the same mechanism as lease/lock management.
Well...just because we have a delegation or layout on NFS, that doesn't mean we'll have any sort of client VFS-layer lease.
I guess you could use this on NFS to force the client to drop a delegation or layout? That could be useful.
Quoted 5 lines
(*)
VIOC_FLUSHVOLUMEFlush all cached state for a volume, both from RAM and local disk cache as far as possible. Files that are open aren't necessarily affected.
Maybe POSIX_FADV_DONTNEED on the mountpoint?
Quoted 4 lines
(*)
VIOC_FLUSHALLFLush all cached state for all volumes.
How would you implement that in a generic way? Suppose I have a mix of AFS and NFS mountpoints and issue this via some mechanism. Is everything going to drop their caches?
Might want to punt on this one or do it with a private, AFS-only ioctl.
Quoted 4 lines
(*) VIOCPREFETCH
Prefetch a file into the cache.
POSIX_FADV_WILLNEED ?
Quoted 8 lines
So, maybe:
fcachectl(AT_FDCWD, "/afs/user/dhowells", 0, "flush volume", NULL, NULL);
Quoted 5 lines
to flush an AFS volume containing my home directory.
Note that doing this by
fcntl()orioctl()has potential difficulties as it would have to work on non-file objects such as device files or symlinks.
Does AFS allow remote access to devices a'la CIFS?
Could we allow posix_fadvise on O_PATH opens? For symlinks there is always O_NOFOLLOW.
Quoted 3 lines
Other functions that this could be used for are cache pinning and fixup/integration should we ever want disconnected operation.
Yeah, a lot of possibilities there.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 4 lines · Jeff Layton
I think it might be more useful to wire
posix_fadviseinto the filesystem drivers somehow. A hinting interface really seems like the right approach here, given the differences between different filesystems.
The main reason I'm against using an fd-taking interface is that the object to be affected might not be a regular file and could even be mounted over.
Quoted 6 lines
Quoted 4 lines
(*) VIOCGETCACHEPARMS
Get the size of the cache.
Global or per-inode cache?
I think this would have to be whatever cache the target inode is lurking within. fscache permits multiple caches on a system.
Quoted 7 lines
Quoted 5 lines
(*)
VIOC_FLUSHVOLUMEFlush all cached state for a volume, both from RAM and local disk cache as far as possible. Files that are open aren't necessarily affected.
Maybe
POSIX_FADV_DONTNEEDon the mountpoint?
Ugh. No. How would you differentiate flushing just the mountpoint or the root dir from flushing the volume? Also, you cannot open the mountpoint object if it is mounted over.
Also POSIX_FADV_DONTNEED is a hint that an application no longer needs the data and is not a specifically a command to flush that data.
Quoted 10 lines
Quoted 4 lines
(*)
VIOC_FLUSHALLFLush all cached state for all volumes.
How would you implement that in a generic way? Suppose I have a mix of AFS and NFS mountpoints and issue this via some mechanism. Is everything going to drop their caches?
Might want to punt on this one or do it with a private, AFS-only ioctl.
Might be worth making it AFS-only. Possibly it would make sense to implement it in userspace using VIOC_FLUSHVOLUME and iterating over /proc/mounts, but that then begs the question of whether this should be affected by namespaces.
Quoted 1 line
POSIX_FADV_WILLNEED?
Perhaps.
Quoted 1 line
Does AFS allow remote access to devices a'la CIFS?
No. :-)
Quoted 2 lines
Could we allow
posix_fadviseonO_PATHopens? For symlinks there is alwaysO_NOFOLLOW.
Maybe. Al?
This doesn't work for mounted-over mountpoints, however. I guess we could add AT_NO_FOLLOW_MOUNTS to get the basalmost mountpoint.
David
Quoted 10 lines · David Howells
Quoted 4 lines · Jeff Layton
I think it might be more useful to wire
posix_fadviseinto the filesystem drivers somehow. A hinting interface really seems like the right approach here, given the differences between different filesystems.The main reason I'm against using an fd-taking interface is that the object to be affected might not be a regular file and could even be mounted over.
How would you disambiguate the mounted-over case with a path-based interface?
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
(*) VIOCGETCACHEPARMS
Get the size of the cache.
Global or per-inode cache?
I think this would have to be whatever cache the target inode is lurking within. fscache permits multiple caches on a system.
Ok, but does this tell you "how big is this entire cache?" or "how much cache does this inode currently consume" ? Both could be useful...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
(*)
VIOC_FLUSHVOLUMEFlush all cached state for a volume, both from RAM and local disk cache as far as possible. Files that are open aren't necessarily affected.
Quoted 6 lines
Quoted 1 line
Maybe
POSIX_FADV_DONTNEEDon the mountpoint?Ugh. No. How would you differentiate flushing just the mountpoint or the root dir from flushing the volume? Also, you cannot open the mountpoint object if it is mounted over.
Good point.
I don't know...this kind of thing might be better suited to a sysfs- style interface, honestly. Anything where you're dealing at the level of an entire fs doesn't benefit much from a per-inode syscall interface. That said, that could get messy once you start dealing with namespaces and such.
Quoted 18 lines
Also
POSIX_FADV_DONTNEEDis a hint that an application no longer needs the data and is not a specifically a command to flush that data.Quoted 10 lines
Quoted 4 lines
(*)
VIOC_FLUSHALLFLush all cached state for all volumes.
How would you implement that in a generic way? Suppose I have a mix of AFS and NFS mountpoints and issue this via some mechanism. Is everything going to drop their caches?
Might want to punt on this one or do it with a private, AFS-only ioctl.
Might be worth making it AFS-only. Possibly it would make sense to implement it in userspace using
VIOC_FLUSHVOLUMEand iterating over/proc/mounts, but that then begs the question of whether this should be affected by namespaces.
Quoted 8 lines
Quoted 1 line
POSIX_FADV_WILLNEED?Perhaps.
Quoted 1 line
Does AFS allow remote access to devices a'la CIFS?
No. :-)
I'm not sure I get why it's terribly useful to manipulate the cache on a symlink or device file itself. There's generally not much cached anyway, right (generally nothing more than a page anyway).
Quoted 8 lines
Quoted 2 lines
Could we allow
posix_fadviseonO_PATHopens? For symlinks there is alwaysO_NOFOLLOW.Maybe. Al?
This doesn't work for mounted-over mountpoints, however. I guess we could add
AT_NO_FOLLOW_MOUNTSto get the basalmost mountpoint.
Yeah, perhaps.
Signature · 1 line
Jeff Layton <[email protected]>