Skip to contentSkip to conversation

NFS: Adding hooks to enable the use of FS-Cache

RFC PATCH In review Steve Dickson 3 participants 6 messages Apr 21, 2006 VFS

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 NFS: Adding hooks to enable the use of FS-Cache

This patch (based on Trond's nfs-2.6 git tree) adds the hooks need to allow NFS to use the generic filesystem caching facility FS-Cache. Caching is done on a per mount bases. To activate caching, the 'fsc' flag is need as a mount option.
Ex:
mount -o fsc server:/export /mnt

(binary and source rpms as well as the actual patch are available at
http://people.redhat.com/steved/cachefs/util-linux/)

NFS uses the cache on a readonly bases; Meaning only reads are caches
not writes. But this *does not* mean filesystems have to be mounted
read only. Filesystems are mounted read/write but only data from
reads will be cached. When a file is opened with either RW or WR
bits, the caching will be turned off (see nfs_open for the details).
The reason for read only is became quite apparent that keeping the cache coherent while jugging both reads and writes is pretty hard. So it was decided to start with a read only implementation and then move on to a
read/write implementation in the future...

There are still a few rough spots in this patch. One being the sysctl variables that show what, if any, caching is being done need to be rolled into NFS I/O metrics code that recently went in.

The second one being, what David termed as, NFS aliasing. The way NFS manages its super blocks would cause double caching since the cookies returned by the cache would not be unique. We got around this problem by making each nfs mount have unique 'fsctag' which would be used to generate the caching cookie. The default fsctag value is the mount point but it also can be set on the command line (which is undocumented at this point since its not clear we really want to do this).

Unfortunately having the mount command passed the kernel a string turned out to a bit difficult since NFS used a binary interface (something that will hopefully change this summer). So David came up with the idea (not too surprising) of using the keyrings to passed down the string which did work very well... But as I said, we have every intending of change the binary interface into an ascii one and when that happens, this interface will change.

Finally there still needs to be some work on mmapped pages. Its not clear NFS is used the cache as well as it could when pages are mmapped. I'm still investigating...

Now I'm sure Trond will find several more rough spots... ;-) but these are all of the ones I know about....

steved.

#1 lore raw
Quoted 9 lines · Steve Dickson

This patch (based on Trond's nfs-2.6 git tree) adds the hooks need to allow NFS to use the generic filesystem caching facility FS-Cache. Caching is done on a per mount bases. To activate caching, the 'fsc' flag is need as a mount option.
Ex:
mount -o fsc server:/export /mnt

(binary and source rpms as well as the actual patch are available at
http://people.redhat.com/steved/cachefs/util-linux/)

Hmm... Are you sure that is based on the latest trees? AFAICS it shouldn't even apply to stock 2.6.17-rc2 since it doesn't take into account the fact that we already use PagePrivate()+nfs_invalidatepage in order to fix truncate races.
Could you look into that please?

Cheers,
Trond

Quoted 41 lines

NFS uses the cache on a readonly bases; Meaning only reads are caches not writes. But this *does not* mean filesystems have to be mounted read only. Filesystems are mounted read/write but only data from reads will be cached. When a file is opened with either RW or WR bits, the caching will be turned off (see nfs_open for the details). The reason for read only is became quite apparent that keeping the cache coherent while jugging both reads and writes is pretty hard. So it was decided to start with a read only implementation and then move on to a read/write implementation in the future...

There are still a few rough spots in this patch. One being the sysctl variables that show what, if any, caching is being done need to be rolled into NFS I/O metrics code that recently went in.

The second one being, what David termed as, NFS aliasing. The way NFS manages its super blocks would cause double caching since the cookies returned by the cache would not be unique. We got around this problem by making each nfs mount have unique 'fsctag' which would be used to generate the caching cookie. The default fsctag value is the mount point but it also can be set on the command line (which is undocumented at this point since its not clear we really want to do this).

Unfortunately having the mount command passed the kernel a string turned out to a bit difficult since NFS used a binary interface (something that will hopefully change this summer). So David came up with the idea (not too surprising) of using the keyrings to passed down the string which did work very well... But as I said, we have every intending of change the binary interface into an ascii one and when that happens, this interface will change.

Finally there still needs to be some work on mmapped pages. Its not clear NFS is used the cache as well as it could when pages are mmapped. I'm still investigating...

Now I'm sure Trond will find several more rough spots... ;-) but these are all of the ones I know about....

steved.

#2 lore raw
Quoted 9 lines · Trond Myklebust
Quoted 7 lines · Steve Dickson

This patch (based on Trond's nfs-2.6 git tree) adds the hooks need to allow NFS to use the generic filesystem caching facility FS-Cache. Caching is done on a per mount bases. To activate caching, the 'fsc' flag is need as a mount option.
Ex:
mount -o fsc server:/export /mnt

Quoted 5 lines
Quoted 2 lines

(binary and source rpms as well as the actual patch are available at
http://people.redhat.com/steved/cachefs/util-linux/)

Hmm... Are you sure that is based on the latest trees?

Just this morning I did an git fetch on my cloned nfs-2.6 tree... and everything seem to apply just fine...

Quoted 3 lines

AFAICS it shouldn't even apply to stock 2.6.17-rc2 since it doesn't take into account the fact that we already use PagePrivate()+nfs_invalidatepage in order to fix truncate races.

hmm... I did not see that... so there must a disconnect somewhere...

Quoted 1 line

Could you look into that please?

Will do...

steved.

#3 lore raw
Quoted 3 lines · Steve Dickson
Quoted 1 line

Hmm... Are you sure that is based on the latest trees?

Just this morning I did an git fetch on my cloned nfs-2.6 tree... and everything seem to apply just fine...

Well... if Steve's using my NFS patch, then it overrides the usage of PG_private to deal with truncate madness. This also needs fixing.

David

#4 lore raw
In reply to Steve Dickson
Quoted 10 lines · Steve Dickson
Quoted 9 lines · Trond Myklebust
Quoted 7 lines · Steve Dickson

This patch (based on Trond's nfs-2.6 git tree) adds the hooks need to allow NFS to use the generic filesystem caching facility FS-Cache. Caching is done on a per mount bases. To activate caching, the 'fsc' flag is need as a mount option.
Ex:
mount -o fsc server:/export /mnt

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

(binary and source rpms as well as the actual patch are available at
http://people.redhat.com/steved/cachefs/util-linux/)

Hmm... Are you sure that is based on the latest trees?

Just this morning I did an git fetch on my cloned nfs-2.6 tree... and everything seem to apply just fine...

'git fetch' or 'git pull'? You almost always want to do the latter, since the former doesn't attempt to merge the changes into your repository.

Cheers,
Trond

#5 lore raw
Quoted 3 lines · Trond Myklebust

'git fetch' or 'git pull'? You almost always want to do the latter, since the former doesn't attempt to merge the changes into your repository.

ah... I did the former... thanks...

steved.