which ioctls matter across filesystems
Other than the obvious example of
EXT2_IOC_GETFLAGS
and
EXT2_IOC_SETFLAGS
which are implemented by multiple filesystems (and are necessary to support a few commonly used tools), are there any other ioctls which should be able to be sent remotely (optionally)? For it to be worth extending the network protocol (in my case CIFS to servers such as Samba, but presumably cluster filesystems have similar interests in supporting all key local tools across the network), an ioctl would have to be
- used by more than one local filesystem
- not have an equivalent way to do the same thing without an ioctl
I have added the GET/SETFLAGS client support, but am not aware of any others which would need to be remoted. For fcntl there are more, but it requires more research to figure out how to handle setlease/getlease and a few others with network implications without degrading performance. Although I am not a fan of ioctls and fcntls, there are a few that are necessary to achieve 100% local semantics across the network.
The new inotify mechanism being prototyped in -mm currently is the other one which needs work to determine how to map it across the network. Since it was added for support of Samba, the corresponding client part (for cifs) may turn out to map to the network protocol quite well already, and given NFSv4 having various similarities to CIFS, it would be interesting if the semantics of inotify would map to NFSv4 write protocol.
fr den 29.04.2005 Klokka 14:22 (-0500) skreiv Steve French:
Quoted 7 lines
The new inotify mechanism being prototyped in -mm currently is the other one which needs work to determine how to map it across the network. Since it was added for support of Samba, the corresponding client part (for cifs) may turn out to map to the network protocol quite well already, and given NFSv4 having various similarities to CIFS, it would be interesting if the semantics of inotify would map to NFSv4 write protocol.
We are discussing the equivalent of dnotify as a potential candidate for the first minor version of NFSv4, but not inotify.
The purpose of our dnotify implementation is address the needs of things like file browsers that don't really care about synchronous notification of changes, but that do currently cause a lot of unnecessary traffic on the wire due to constantly polling stat() and doing readdir() updates. The jury is still out as to whether or not the callbacks actually do reduce on-the-wire traffic, though, so we may drop it.
What kind of real-world applications exist out there that need inotify functionality, and what sort of requirements do they have (in particular w.r.t. the notification mechanism)?
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>
Quoted 8 lines · Trond Myklebust
We are discussing the equivalent of dnotify as a potential candidate for the first minor version of NFSv4, but not inotify.
The purpose of our dnotify implementation is address the needs of things like file browsers that don't really care about synchronous notification of changes, but that do currently cause a lot of unnecessary traffic on the wire due to constantly pollingstat()and doingreaddir()updates. The jury is still out as to whether or not the callbacks actually do reduce on-the-wire traffic, though, so we may drop it.
What about inotify makes it insufficient for your needs?
Quoted 3 lines
What kind of real-world applications exist out there that need inotify functionality, and what sort of requirements do they have (in particular w.r.t. the notification mechanism)?
A few worksets:
- Current users, such as FAM and Samba, that need simple file change notification
- Random applications that want to watch a file or two
- The Linux desktop
- Real-time live-updating indexing systems, such as Beagle, that compete with f.e. Apple's Spotlight.
Best,
Robert Love
Quoted 8 lines · Trond Myklebust
What kind of real-world applications exist out there that need inotify functionality, and what sort of requirements do they have (in particular w.r.t. the notification mechanism)?
Cheers,
Trond
The two cases I can think of which matter (other than the case you mention) are:
1) KDE File manager - autoupdate of directory listings which today calls D_NOTIFY (a similar feature was first done IIRC in OS/2 for support of the workplace shell). Obviously this is as or more important to support well over the network as it is in the local fs, but the client implications. I don't know whether their needs (and the equivalent in Gnome) map better to fcntl DNOTIFY or inotify.
2) Support of Network File Servers - The Samba example has already been mentioned, but it is important because it would be quite common for a series of Samba servers to export shares that are NFS mounted to a set of NFS servers (or on other platforms mounted to a cluster filesystem). The CIFS network protocol has long had a notify mechanism, and client implementations on various operating systems use it, so there is pressure for Samba to support it better. The Linux CIFS client can issue these calls too, but it is marked experimental and disabled by default as more work needs to be done to clean it up.
A loosely related issue which will matter a lot in the long run are figuring out a way to pass get/setlease requests as the network caching mechanisms would otherwise not work in three tier environments (e.g. SMB/CIFS client -> Samba server over NFS client mounted to -> NFS server, or the reverse).
fr den 29.04.2005 Klokka 16:03 (-0400) skreiv Robert Love:
Quoted 11 lines · Trond Myklebust
Quoted 9 lines
We are discussing the equivalent of dnotify as a potential candidate for the first minor version of NFSv4, but not inotify.
The purpose of our dnotify implementation is address the needs of things like file browsers that don't really care about synchronous notification of changes, but that do currently cause a lot of unnecessary traffic on the wire due to constantly pollingstat()and doingreaddir()updates. The jury is still out as to whether or not the callbacks actually do reduce on-the-wire traffic, though, so we may drop it.
Quoted 1 line
What about inotify makes it insufficient for your needs?
Nothing, if you are just tracking changes that are made by local processes.
For remote filesystems like CIFS and NFS, however, there may be applications that want to monitor changes that are made directly on the server by other clients. In the above paragraph, I'm therefore referring to a debate about whether or not adding officially sanctioned NFSv4 protocol support for something like dnotify/inotify makes sense.
The problem is that having the server call back a bunch of clients every time a file changes does not really scale too well. The current dnotify-like proposal therefore specifies that notification is not synchronous (i.e. there may be a delay of several seconds), and that the server may want to group several notifications into a single callback. (if interested, you can view the current proposal on
http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-directory-delegation-01.txt)
Quoted 4 lines
Quoted 3 lines
What kind of real-world applications exist out there that need inotify functionality, and what sort of requirements do they have (in particular w.r.t. the notification mechanism)?
Quoted 8 lines
A few worksets:
- Current users, such as FAM and Samba, that need simple file change notification
- Random applications that want to watch a file or two
- The Linux desktop
- Real-time live-updating indexing systems, such as Beagle, that compete with f.e. Apple's Spotlight.
Right, but I assume that most of the use-cases (with possible exception of FAM) do not really involve tracking changes made by other NFS clients. Although I imagine in particular that the real-time indexing system workload trying to run on several clients and tracking the same files at the same time might be a lot of fun...
I'm therefore asking whether or not there is a killer-app out there that we need to support and that does want to track changes made by other clients. File browsers are more or less the only thing that come to mind.
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>
Quoted 4 lines · Robert Love
Quoted 4 lines
What about inotify makes it insufficient for your needs?
I am not sure - but it is obviously required that inotify can pass over CIFS (and probably NFS) since change notification is hardest for the user to figure out when they are on a network. I am not sure how the filesystem can detect that a new watch is added to one of its inodes - it looks like you added an ioctl to a device but I am still reading through your latest patch. I was looking for something like an inode operation that cifs could hook so the fs could be told when a new watch was added or one was changed. In any case I need to construct functions somewhat similar to what is in fs/cifs/fcntl.c and need to finish/modify CIFSSMBNotify in fs/cifs/cifssmb.c to map the inotify flags to the flags available in the CIFS network protocol specifications.
The existing network protocol support for ChangeNotify is more straightforward than you might think and the filter flags & actions that I have at my disposal for implementing notify across the network are in fs/cifs/cifspdu.h already (search for FILE_ACTION_ and FILE_NOTIFY_ if you are curious) but obviously they are similar to what the other Samba team guys have already told you.
Quoted 15 lines
Quoted 5 lines
What kind of real-world applications exist out there that need inotify functionality, and what sort of requirements do they have (in particular w.r.t. the notification mechanism)?
A few worksets:
- Current users, such as FAM and Samba, that need simple file change notification
- Random applications that want to watch a file or two
- The Linux desktop
- Real-time live-updating indexing systems, such as Beagle, that compete with f.e. Apple's Spotlight.
Quoted 4 lines
Best,
Robert Love
strongly agree, very strongly agree.
The one example we left out - distributed backup applications (those guys query me from time to time asking for improvements) - they want to know when a server file changes so they can do backup across CIFS.
Quoted 5 lines · Trond Myklebust
The problem is that having the server call back a bunch of clients every time a file changes does not really scale too well. The current dnotify-like proposal therefore specifies that notification is not synchronous (i.e. there may be a delay of several seconds), and that the server may want to group several notifications into a single callback.
Yah, so what I am asking is why not use inotify for the user-side component of this system?
Wouldn't the deferring and coalescing of events occur on the server side? So the server-side stuff would be whatever you need--your own code using whatever protocol you wanted--but the client-side interface would be over inotify.
Even if not, I'd be willing to make changes to inotify to accommodate NFS's needs.
Robert Love
Quoted 9 lines · Steve French
I am not sure - but it is obviously required that inotify can pass over CIFS (and probably NFS) since change notification is hardest for the
user to figure out when they are on a network. I am not sure how the
filesystem can detect that a new watch is added to one of its inodes - it looks like you added an ioctl to a device but I am still reading
through your latest patch. I was looking for something like an inode
operation that cifs could hook so the fs could be told when a new watch
was added or one was changed. In any case I need to construct
functions somewhat similar to what is infs/cifs/fcntl.cand need to
Quoted 8 lines
finish/modify CIFSSMBNotify in
fs/cifs/cifssmb.cto map the inotify flags to the flags available in the CIFS network protocol specifications. The existing network protocol support for ChangeNotify is more straightforward than you might think and the filter flags & actions that I have at my disposal for implementing notify across the network are infs/cifs/cifspdu.halready (search forFILE_ACTION_andFILE_NOTIFY_if
you are curious) but obviously they are similar to what the other Samba team guys have already told you.
So a client adds a watch and the server needs to then physically add the inotify watch?
If you have a user-space, user-space could just add an inotify watch.
But I guess you live entirely in kernel-space? Couldn't we just export our "add watch" interface to you?
Robert Love
Quoted 6 lines · Steve French
I believe that the spotlight facility of MacOS, and the somewhat similar Longhorn feature (think Google desktop search/indexing on steroids)
qualify as killer-apps. I am concerned about how to do better with our
implementations across a distributed (NFS, CIFS etc.) network. And of
course coalescing async notifications most efficiently is a fascinating and difficult area to do right - for servers at least.
If we had some way to efficiently coalesce events, even non-remote stuff would drool. Beagle (our Spotlight killer) would love it.
First thing is, the events cannot be stored in a linked list. ;-)
Robert Love
Quoted 9 lines · Trond Myklebust
I'm therefore asking whether or not there is a killer-app out there that we need to support and that does want to track changes made by other clients. File browsers are more or less the only thing that come to mind.
Cheers,
Trond
I believe that the spotlight facility of MacOS, and the somewhat similar Longhorn feature (think Google desktop search/indexing on steroids) qualify as killer-apps. I am concerned about how to do better with our implementations across a distributed (NFS, CIFS etc.) network. And of course coalescing async notifications most efficiently is a fascinating and difficult area to do right - for servers at least.
Quoted 5 lines · Robert Love
So a client adds a watch and the server needs to then physically add the inotify watch?
Yes, this creates the interesting situation of two responses (one from the local client code, and one from the server) potentially coming as a client changes a file which he has a watch on.
Quoted 9 lines
If you have a user-space, user-space could just add an inotify watch.
But I guess you live entirely in kernel-space? Couldn't we just export our "add watch" interface to you?
Robert Love
Yes - add watch could be exported, I don't see a way around this since a filesystem has to be able to tell the server what to watch. It does not really matter if that were done in kernel or not, but I would prefer it to be done in kernel since that would avoid having to ioctl down to the kernel
Quoted 8 lines · Robert Love
Quoted 3 lines · Steve French
I believe that the spotlight facility of MacOS, and the somewhat similar Longhorn feature (think Google desktop search/indexing on steroids) qualify as killer-apps. I am concerned about how to do better with our implementations across a distributed (NFS, CIFS etc.) network. And of course coalescing async notifications most efficiently is a fascinating and difficult area to do right - for servers at least.
Quoted 8 lines
If we had some way to efficiently coalesce events, even non-remote stuff would drool. Beagle (our Spotlight killer) would love it.
First thing is, the events cannot be stored in a linked list. ;-)
Robert Love
I should mention one obvious thing - there can be two wire protocols here (at least for the CIFS case) - an optimized case which matches -- exactly -- to the [better] new semantics that Linux allows and the interoperability case (which can be less efficient) ie the existing Transact2 NOTIFY mechanism which is widely supported by current servers and currently defined in fs/cifs/cifspdu.h
I don't mind, nor do others on the Samba team, extending the wire protocol for CIFS where it would help that - "killer app" as long as it is
- optional
- persuasive benefit
- helps the network case to better match the -- exact -- local
semantics applications would expect.
fr den 29.04.2005 Klokka 16:47 (-0400) skreiv Robert Love:
Quoted 8 lines · Trond Myklebust
Quoted 6 lines
The problem is that having the server call back a bunch of clients every time a file changes does not really scale too well. The current dnotify-like proposal therefore specifies that notification is not synchronous (i.e. there may be a delay of several seconds), and that the server may want to group several notifications into a single callback.
Quoted 7 lines
Yah, so what I am asking is why not use inotify for the user-side component of this system?
Wouldn't the deferring and coalescing of events occur on the server side? So the server-side stuff would be whatever you need--your own code using whatever protocol you wanted--but the client-side interface would be over inotify.
Sure. We're not talking about inventing new user interfaces here. Just how best to support the existing ones.
Quoted 2 lines
Even if not, I'd be willing to make changes to inotify to accommodate NFS's needs.
I think what the IETF NFS working group rather needs right now is an advocate that is willing to stand up and demonstrate why protocol support for inotify-style callbacks would be a more scalable solution than a solution based on a combination of GETATTR polling and read delegations (essentially the same thing as CIFS' op-locks) for directories.
The current research (see
http://www3.ietf.org/proceedings/05mar/slides/nfsv4-4/sld1.htm) which has uses real-life on-the-wire traffic actually leans more towards the GETATTR solution. That research was based on a set of anonymous tcpdump traces taken at Harvard University, though, so it reflects the traffic in a typical university environment. It may be that other use-cases exist that favour the inotify callbacks case.
If so, now is the time to step forward and say so...
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>
Quoted 9 lines · Trond Myklebust
I think what the IETF NFS working group rather needs right now is an advocate that is willing to stand up and demonstrate why protocol support for inotify-style callbacks would be a more scalable solution than a solution based on a combination of GETATTR polling and read delegations (essentially the same thing as CIFS' op-locks) for directories.
I agree.
Quoted 9 lines
The current research (see
http://www3.ietf.org/proceedings/05mar/slides/nfsv4-4/sld1.htm) which has uses real-life on-the-wire traffic actually leans more towards the GETATTR solution. That research was based on a set of anonymous tcpdump traces taken at Harvard University, though, so it reflects the traffic in a typical university environment. It may be that other use-cases exist that favour the inotify callbacks case.
Very interesting, I had not seen that. FYI - There are many years of real world experience on the current transact2 notify (it is deployed in some form on most clients) but I don't know whether one of the NAS storage companies or researchers has done a good research paper on this topic - although there is no lack of customer traces in SPEC and SNIA.
My gut reaction is that as
1) directory size increases (number of files per directory) and
2) change rate goes down
(both of which could be client heuristics) the notify mechanism (on the directory, or parent directory) is much better, but with small directories and more frequent changes the getattr (Transact2 QueryPathinfo) approach wins. There is no one-size-fits-all that covers both cases.
fr den 29.04.2005 Klokka 16:22 (-0500) skreiv Steve French:
Quoted 5 lines
Very interesting, I had not seen that. FYI - There are many years of
real world experience on the current transact2 notify (it is deployed in some form on most clients) but I don't know whether one of the NAS storage companies or researchers has done a good research paper on this topic - although there is no lack of customer traces in SPEC and SNIA.
If you are able to release those traces to them, then CITI does have a simulator that can be used to test the various models. It is currently only capable of analysing NFSv3 and v4 traffic, but it might perhaps be possible to add a CIFS interpreter onto it.
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>