Cache flushing

Solaris and BSDs support flushing attribute cache safely using fchown(fd, (uid_t)-1, (gid_t)-1). Could Linux be changed to support this as well? If I'm looking at the sources right, this might work (completely untested):

inode.c 2007-11-16 22:19:44.000000000 +0200 renamed from inode.c.old 2007-11-16 22:18:46.000000000 +0200 +1 -0

@@ -322,6 +322,7 @@
 nfs_setattr(struct dentry *dentry, struct iattr *attr)
 {
        struct inode *inode = dentry->d_inode;
+       struct nfs_inode *nfsi = NFS_I(inode);
        struct nfs_fattr fattr;
        int error;
 

@@ -334,8 +335,10 @@
 

Another problem I have is that it's difficult to get a file's data cache flushed. The only way I found was to successfully fcntl() lock the file. This is pretty bad from performance point of view since often I don't want/need to lock the file.

Solaris and BSDs invalidate also a file's data cache when its attribute cache is invalidated. It would be nicer if there was a separate way, but I'd settle for fchown(fd, (uid_t)-1, (gid_t)-1) invalidating data cache as well.

Actually I did also look at posix_fadvise(fd, 0, 0, POSIX_FADVN_DONTNEED). It appears to work, but I'm a bit worried about race conditions that causes pages to randomly not get dropped because invalidate_mapping_pages() doesn't drop locked pages.

Quoted 25 lines · Timo Sirainen

Solaris and BSDs support flushing attribute cache safely using fchown(fd, (uid_t)-1, (gid_t)-1). Could Linux be changed to support this as well? If I'm looking at the sources right, this might work (completely untested):

inode.c 2007-11-16 22:19:44.000000000 +0200 renamed from inode.c.old 2007-11-16 22:18:46.000000000 +0200 +4 -1

@@ -322,6 +322,7 @@
 nfs_setattr(struct dentry *dentry, struct iattr *attr)
 {
        struct inode *inode = dentry->d_inode;
+       struct nfs_inode *nfsi = NFS_I(inode);
        struct nfs_fattr fattr;
        int error;
 

@@ -334,8 +335,10 @@
 
        /* Optimization: if the end result is no change, don't RPC */
        attr->ia_valid &= NFS_VALID_ATTRS;
-       if (attr->ia_valid == 0)
+       if (attr->ia_valid == 0) {
+               nfsi->cache_validity |= NFS_INO_INVALID_ATTR;
                return 0;
+       }
 

inode.c 2007-11-16 22:19:44.000000000 +0200 line 344

Quoted 2 lines

inode.c 2007-11-16 22:19:44.000000000 +0200 renamed from inode.c.old 2007-11-16 22:18:46.000000000 +0200 +0 -0

        lock_kernel();
        nfs_begin_data_update(inode);

Why is this needed?

Quoted 10 lines

Another problem I have is that it's difficult to get a file's data cache flushed. The only way I found was to successfully fcntl() lock the file. This is pretty bad from performance point of view since often I don't want/need to lock the file.

Solaris and BSDs invalidate also a file's data cache when its attribute cache is invalidated. It would be nicer if there was a separate way, but I'd settle for fchown(fd, (uid_t)-1, (gid_t)-1) invalidating data cache as well.

Quoted 4 lines

Actually I did also look at posix_fadvise(fd, 0, 0, POSIX_FADVN_DONTNEED). It appears to work, but I'm a bit worried about race conditions that causes pages to randomly not get dropped because invalidate_mapping_pages() doesn't drop locked pages.

Again, why do you need this level of data cache invalidation? If you don't want cached i/o, then use O_DIRECT.

Quoted 1 line · Trond Myklebust

Why is this needed?

Do you mean why is flushing attribute cache needed, or why is this particular way to flush it needed?

I need to be able to find out if a file has changed, so I need to get its attribute cache flushed. fchown()ing to -1, -1 would work safely in all situations because it's guaranteed not to change the file in any way.

Also O_DIRECT is a bit too much for my use case. I do want the file to be cached for the most part, but there are times occationally when parts of it can be overwritten, and I need to make sure that in those situations the newest data is read.

If you want a wider description of what I'm trying to do: I'm developing Dovecot IMAP server. A lot of people store mails on NFS and want to have multiple IMAP servers be able to access the mails. Dovecot uses somewhat complex index files to speed up accessing the mailboxes, and it's mainly for these index files that I need this explicit control over caching. If two servers are accessing the same mailbox at the same time, the index files get easily corrupted if I can't control the caching.

Quoted 10 lines · Timo Sirainen

Quoted 1 line · Trond Myklebust

Why is this needed?

Do you mean why is flushing attribute cache needed, or why is this particular way to flush it needed?

I need to be able to find out if a file has changed, so I need to get its attribute cache flushed. fchown()ing to -1, -1 would work safely in all situations because it's guaranteed not to change the file in any way.

Why can't you simply close(), and then re-open() the file? That is _the_ standard way to force an attribute cache revalidation on all NFS versions. The close-to-open caching model, which is implemented on most NFS clients guarantees this.

Quoted 5 lines

Also O_DIRECT is a bit too much for my use case. I do want the file to be cached for the most part, but there are times occationally when parts of it can be overwritten, and I need to make sure that in those situations the newest data is read.

Quoted 8 lines

If you want a wider description of what I'm trying to do: I'm developing Dovecot IMAP server. A lot of people store mails on NFS and want to have multiple IMAP servers be able to access the mails. Dovecot uses somewhat complex index files to speed up accessing the mailboxes, and it's mainly for these index files that I need this explicit control over caching. If two servers are accessing the same mailbox at the same time, the index files get easily corrupted if I can't control the caching.

So how are you ensuring that both servers don't try writing to the same locations? You must have some form of synchronisation scheme for this to work.

Quoted 4 lines · Trond Myklebust

Why can't you simply close(), and then re-open() the file? That is _the_ standard way to force an attribute cache revalidation on all NFS versions. The close-to-open caching model, which is implemented on most NFS clients guarantees this.

Interesting. Too bad this is the first time I have heard of it (where as I've seen fchown()/chown() suggested in several mailing lists before). I understood NFS FAQ's close-to-open caching description to mean only how data caching is handled.

close()+open() would have been difficult to handle because open() can fail, but looks like opening another file descriptor and closing it works just as well. Also looks like it works for flushing directories' attribute cache (which doesn't seem to work with FreeBSD though).

There's one potential problem with closing a file descriptor though. It loses all fcntl locks for that file from all fds. I'm not sure if this a problem for me though.

Quoted 14 lines

Quoted 5 lines

Also O_DIRECT is a bit too much for my use case. I do want the file to be cached for the most part, but there are times occationally when parts of it can be overwritten, and I need to make sure that in those situations the newest data is read.

Quoted 8 lines

If you want a wider description of what I'm trying to do: I'm developing Dovecot IMAP server. A lot of people store mails on NFS and want to have multiple IMAP servers be able to access the mails. Dovecot uses somewhat complex index files to speed up accessing the mailboxes, and it's mainly for these index files that I need this explicit control over caching. If two servers are accessing the same mailbox at the same time, the index files get easily corrupted if I can't control the caching.

Quoted 3 lines

So how are you ensuring that both servers don't try writing to the same locations? You must have some form of synchronisation scheme for this to work.

Well, there's no simple answer for that. :) There are 3 different kinds of index files with completely different locking behavior, because I try to avoid long lasting locks. I do use write locks, but reads are mainly lockless. There's this transaction log file which tells me when something has changed, so I know when data cache need to be flushed.

Anyway, all of this is working already, but I'd just like to get the performance a bit better with Linux by avoiding those unnecessary lock +unlock sequences.

Quoted 5 lines

close()+open() would have been difficult to handle because open() can fail, but looks like opening another file descriptor and closing it works just as well. Also looks like it works for flushing directories' attribute cache (which doesn't seem to work with FreeBSD though).

Quoted 3 lines

There's one potential problem with closing a file descriptor though. It loses all fcntl locks for that file from all fds. I'm not sure if this a problem for me though.

Right. I understood that you were not using fcntl() locks. Those will in any case ensure cache revalidation, so you wouldn't have to use anything else.

Again, you can use the close-to-open trick of simply reopening the file whenever you need to revalidate the data cache.

The problem, however, is that on the most common Linux filesystems (ext2/ext3, reiserfs,...) the time resolution on the mtime is 1 second. If your NFS server is running one of those filesystems, then the data cache revalidation may fail to detect a write that happens within 1 second of the previous write.

Most people are using NetApps, and they also seem to have 1 second resolution (at least the one I just tested had). So this isn't a solution. "Let's hope that there are no writes less than 1 second apart" isn't really a solution either. People like their mailboxes uncorrupted.

Right. I understood that you were not using fcntl() locks. Those will in
any case ensure cache revalidation, so you wouldn't have to use anything
else.

Quoted 5 lines

Most people are using NetApps, and they also seem to have 1 second resolution (at least the one I just tested had). So this isn't a solution. "Let's hope that there are no writes less than 1 second apart" isn't really a solution either. People like their mailboxes uncorrupted.

No. NetApp filers and WAFL have nanosecond resolution. They should be fine.

Quoted 4 lines

Why can't you simply close(), and then re-open() the file? That is _the_ standard way to force an attribute cache revalidation on all NFS versions. The close-to-open caching model, which is implemented on most NFS clients guarantees this.

close()+open() would have been difficult to handle because open() can fail, but looks like opening another file descriptor and closing it works just as well. Also looks like it works for flushing directories' attribute cache (which doesn't seem to work with FreeBSD though).

Quoted 4 lines · Timo Sirainen

Quoted 4 lines

Why can't you simply close(), and then re-open() the file? That is _the_ standard way to force an attribute cache revalidation on all NFS versions. The close-to-open caching model, which is implemented on most NFS clients guarantees this.

..

Quoted 4 lines

close()+open() would have been difficult to handle because open() can fail, but looks like opening another file descriptor and closing it works just as well. Also looks like it works for flushing directories' attribute cache (which doesn't seem to work with FreeBSD though).

Actually it works for flushing a directory's attribute cache in v2.6.17-rc2, but not in v2.6.22. chown() works in v2.6.22 also. Is there a reason for this change? I guess it anyway means that I'm back to using chown() for flushing a directory's attribute cache.

Quoted 11 lines · Timo Sirainen

Quoted 4 lines · Timo Sirainen

Quoted 4 lines

Why can't you simply close(), and then re-open() the file? That is _the_ standard way to force an attribute cache revalidation on all NFS versions. The close-to-open caching model, which is implemented on most NFS clients guarantees this.

..

Quoted 4 lines

close()+open() would have been difficult to handle because open() can fail, but looks like opening another file descriptor and closing it works just as well. Also looks like it works for flushing directories' attribute cache (which doesn't seem to work with FreeBSD though).

Quoted 4 lines

Actually it works for flushing a directory's attribute cache in v2.6.17-rc2, but not in v2.6.22. chown() works in v2.6.22 also. Is there a reason for this change? I guess it anyway means that I'm back to using chown() for flushing a directory's attribute cache.

close-to-open caching works fine for me in v2.6.22. It does indeed send a GETATTR and revalidate the inode.

Now that I think of it, I guess the reason is that my 2.6.22 setup has Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So even though open()+close() flushes the directory's attribute cache, it doesn't flush the file name -> NFS handle cache unless mtime changes?

Quoted 4 lines · Timo Sirainen

Now that I think of it, I guess the reason is that my 2.6.22 setup has Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So even though open()+close() flushes the directory's attribute cache, it doesn't flush the file name -> NFS handle cache unless mtime changes?

Right. You're hitting the principal limitation of ext3 as an NFS backend filesystem.
As I said in an earlier mail, the resolution on the time is 1 second, so client 2 can basically not see any changes that happen within < 1 second on client 1. The reason is that the mtime stays the same.

Quoted 10 lines · Trond Myklebust

Quoted 7 lines · Timo Sirainen

Now that I think of it, I guess the reason is that my 2.6.22 setup has
Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So
even though open()+close() flushes the directory's attribute cache, it
doesn't flush the file name -> NFS handle cache unless mtime changes?

Quoted 8 lines

Right. You're hitting the principal limitation of ext3 as an NFS backend
filesystem.
As I said in an earlier mail, the resolution on the time is 1 second, so
client 2 can basically not see any changes that happen within < 1 second
on client 1. The reason is that the mtime stays the same.

With files it's possible to work around this limitation with fcntl locking, but directories can't be locked. So back to my original
question: Would it be possible to have chmod(dir, (uid_t)-1, (gid_t)-1) flush its cache? I don't see what harm there could be in this, since normally no-one would do it anyway and the change is simple.

Quoted 21 lines · Timo Sirainen

Quoted 18 lines · Trond Myklebust

Quoted 7 lines · Timo Sirainen

Now that I think of it, I guess the reason is that my 2.6.22 setup has
Linux+ext3 as NFS server and 2.6.17-rc2 setup has NetApp as server. So
even though open()+close() flushes the directory's attribute cache, it
doesn't flush the file name -> NFS handle cache unless mtime changes?

Right. You're hitting the principal limitation of ext3 as an NFS backend
filesystem.
As I said in an earlier mail, the resolution on the time is 1 second, so
client 2 can basically not see any changes that happen within < 1 second
on client 1. The reason is that the mtime stays the same.

Quoted 5 lines

With files it's possible to work around this limitation with fcntl locking, but directories can't be locked. So back to my original
question: Would it be possible to have chmod(dir, (uid_t)-1, (gid_t)-1) flush its cache? I don't see what harm there could be in this, since normally no-one would do it anyway and the change is simple-

Quoted 1 line · Trond Myklebust

I thought you said this was for flushing the attribute cache?

That was it originally, but this one's more important. :) Also I originally thought that directory's attribute cache flushing also flushed its file handle cache. I'm not actually sure what file handle cache should even really be called. It looks like Linux handles it with page cache the same way as for files?

So I guess I'm asking for chown(-1,-1) call to invalidate the file's attribute and page cache. Although it's interesting why chown(0,-1) flushes a directory's file handle cache (==page cache?), but the same call for a file doesn't flush its page cache. Maybe I'm still misunderstanding something how all of this works.

Quoted 9 lines · Timo Sirainen

Quoted 1 line · Trond Myklebust

I thought you said this was for flushing the attribute cache?

That was it originally, but this one's more important. :) Also I originally thought that directory's attribute cache flushing also flushed its file handle cache. I'm not actually sure what file handle cache should even really be called. It looks like Linux handles it with page cache the same way as for files?

Quoted 5 lines

So I guess I'm asking for chown(-1,-1) call to invalidate the file's attribute and page cache. Although it's interesting why chown(0,-1) flushes a directory's file handle cache (==page cache?), but the same call for a file doesn't flush its page cache. Maybe I'm still misunderstanding something how all of this works.

No. I'm not at all comfortable with the idea of overloading chown() to invalidate data caches.

It is one thing to have it revalidate the attribute cache (you might be able to argue that this is consistent with the mission of chown(), although I'm not yet convinced of that) but the only effect that chown() has on data caching is that we force a sync to disk in order to avoid trouble with cached writes that are no longer accepted by the server. It has never forced a cache invalidation.

IMO, the right interface for manipulating the page cache is rather posix_fadvise(). I can certainly see an argument for adding a "POSIX_FADV_UNCACHE" option in order to force a cache invalidation for those applications that need stronger cache consistency.

Note, however, that even if such a thing is accepted into the mainstream kernel, you still have to convince all the distros to backport this interface to their older kernels. If not, you will still have a problem with legacy clients...