Skip to contentSkip to conversation

NFS using CacheFS

PATCH In review Steve Dickson 6 participants 12 messages Oct 4, 2004 – Oct 13, 2004

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 NFS using CacheFS

Hey Trond,

Here here is the first stab at having NFS use the CacheFS that is currently in the -mm tree. As you know CacheFS is a caching filesystem that allows network filesystems, such as AFS and now NFS, to cache data pages on
the local disk. This is a big win (especially with read-only filesystems) since it drastically cuts down on the amount of data read across the network. Plus the cached data survives umounts and reboots (since CacheFS journals the data). So when a client comes back up or the filesystem is remounted, all that is needed is a few small getattrs to insure the data is still valid. If it is, NFS reads are turned into local disk reads.... A Huge gain in performance!!!!

I have structured that patch so CacheFS is a mount option. I figured this was the safest way to introduce CacheFS and not completely break NFS.... And if for some reason, NFS is not able to get an initial Cachefs cookie (i.e. it can't use CacheFS) , NFS will dynamically turn off the mount option, disabling the use of the cache..

But as usual there are some issues....
1) NFS aliasing. The fact that:

mount hades:/hades /a
mount hades:/hades/xxx /b

creates separate super blocks causes problems for CacheFS. With the current -mm code, these type of mount actual causes CacheFS to crash (although I do have a proposed patch that David is looking at).

2) NFS4 is not supported. I simply have not had time to get this working. But I will!!

3) There is no user level support. I realize this is extremely cheesy but I noticed that the NFS posix mount option (in the 2.6 kernel) was no longer being used, so I high jacked it. Which means to make NFS to used CacheFS you need to use the posix option:

mount -o posix server:/export/home /mnt/server/home

Comments?

SteveD.

PS. The new CacheFS mailing that have been set up is at
http://www.redhat.com/mailman/listinfo/linux-cachefs
I encourage you and anybody else interested in this type of technology to subscribe....

include/linux/nfs_fs.h renamed from include/linux/nfs_fs.h.orig +5 -0
@@ -30,6 +30,7 @@
 #include <linux/nfs_xdr.h>
 #include <linux/rwsem.h>
 #include <linux/workqueue.h>
+#include <linux/cachefs.h>
 
 /*
  * Enable debugging support for nfs client.
@@ -189,6 +190,10 @@ struct nfs_inode {
 	struct rw_semaphore	rwsem;
 #endif /* CONFIG_NFS_V4*/
 
+#ifdef CONFIG_NFS_CACHEFS
+	struct cachefs_cookie	*cachefs;
+#endif
+
 	struct inode		vfs_inode;
 };
 
include/linux/nfs_fs_sb.h renamed from include/linux/nfs_fs_sb.h.orig +5 -0
@@ -3,6 +3,7 @@
 
 #include <linux/list.h>
 #include <linux/backing-dev.h>
+#include <linux/cachefs.h>
 
 /*
  * NFS client parameters stored in the superblock.
@@ -46,6 +47,10 @@ struct nfs_server {
 						   that are supported on this
 						   filesystem */
 #endif
+
+#ifdef CONFIG_NFS_CACHEFS
+	struct cachefs_cookie	*cachefs;	/* cache cookie */
+#endif
 };
 
 /* Server capabilities */
include/linux/nfs_mount.h renamed from include/linux/nfs_mount.h.orig +1 -0
@@ -60,6 +60,7 @@ struct nfs_mount_data {
 #define NFS_MOUNT_BROKEN_SUID	0x0400	/* 4 */
 #define NFS_MOUNT_STRICTLOCK	0x1000	/* reserved for NFSv4 */
 #define NFS_MOUNT_SECFLAVOUR	0x2000	/* 5 */
+#define NFS_MOUNT_CACHEFS		NFS_MOUNT_POSIX
 #define NFS_MOUNT_FLAGMASK	0xFFFF
 
 #endif
fs/nfs/file.c renamed from fs/nfs/file.c.orig +71 -0

92 lines not on this page · raw message

fs/nfs/inode.c renamed from fs/nfs/inode.c.orig +109 -2

195 lines not on this page · raw message

fs/nfs/Makefile renamed from fs/nfs/Makefile.orig +1 -0
@@ -12,4 +12,5 @@ nfs-$(CONFIG_NFS_V4)	+= nfs4proc.o nfs4x
 			   delegation.o idmap.o \
 			   callback.o callback_xdr.o callback_proc.o
 nfs-$(CONFIG_NFS_DIRECTIO) += direct.o
+nfs-$(CONFIG_NFS_CACHEFS) += nfs-cachefs.o
 nfs-objs		:= $(nfs-y)
fs/nfs/nfs-cachefs.c new file +193 -0

193 lines not on this page · raw message

fs/nfs/nfs-cachefs.h new file +27 -0
@@ -0,0 +1,27 @@
+/* nfs-cachefs.h: NFS CacheFS interface definitions
+ *
+ * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _NFS_CACHEFS_H
+#define _NFS_CACHEFS_H
+
+#include <linux/cachefs.h>
+
+#ifdef CONFIG_NFS_CACHEFS
+#ifndef CONFIG_CACHEFS
+#error "CONFIG_NFS_CACHEFS is defined but not CONFIG_CACHEFS"
+#endif
+
+extern struct cachefs_netfs nfs_cache_netfs;
+extern struct cachefs_index_def nfs_cache_server_index_def;
+extern struct cachefs_index_def nfs_cache_fh_index_def;
+
+#endif
+#endif /* _NFS_CACHEFS_H */
fs/nfs/read.c renamed from fs/nfs/read.c.orig +134 -1

179 lines not on this page · raw message

fs/nfs/write.c renamed from fs/nfs/write.c.orig +37 -0
@@ -273,6 +273,38 @@ static int wb_priority(struct writeback_
 }
 
 /*
+ * store an updated page in cachefs
+ */
+#ifdef CONFIG_NFS_CACHEFS
+static void
+nfs_writepage_to_cachefs_complete(void *cookie_data, struct page *page, void *data, int error)
+{
+	/* really need to synchronise the end of writeback, probably using a page flag */
+}
+static inline void
+nfs_writepage_to_cachefs(struct inode *inode, struct page *page)
+{
+	int ret;
+
+	dprintk("NFS: writepage_to_cachefs (0x%p/0x%p/0x%p)\n", 
+		NFS_I(inode)->cachefs, page, inode);
+
+	ret = cachefs_write_page(NFS_I(inode)->cachefs, page,
+		nfs_writepage_to_cachefs_complete, NULL, GFP_KERNEL);
+	if (ret != 0 ) {
+		dprintk("NFS:    cachefs_write_page error %d\n", ret);
+		cachefs_uncache_page(NFS_I(inode)->cachefs, page);
+	}
+}
+#else
+static inline void
+nfs_writepage_to_cachefs(struct inode *inode, struct page *page)
+{
+	BUG();
+}
+#endif
+
+/*
  * Write an mmapped page to the server.
  */
 int nfs_writepage(struct page *page, struct writeback_control *wbc)
@@ -317,6 +349,10 @@ do_it:
 		err = -EBADF;
 		goto out;
 	}
+
+	if (NFS_SERVER(inode)->flags & NFS_MOUNT_CACHEFS)
+		nfs_writepage_to_cachefs(inode, page);
+
 	lock_kernel();
 	if (!IS_SYNC(inode) && inode_referenced) {
 		err = nfs_writepage_async(ctx, inode, page, 0, offset);
@@ -1322,6 +1358,7 @@ nfs_commit_done(struct rpc_task *task)
 			(long long)NFS_FILEID(req->wb_context->dentry->d_inode),
 			req->wb_bytes,
 			(long long)req_offset(req));
+
 		if (task->tk_status < 0) {
 			req->wb_context->error = task->tk_status;
 			nfs_inode_remove_request(req);
fs/Kconfig renamed from fs/Kconfig.orig +7 -0
@@ -1482,6 +1482,13 @@ config NFS_V4
 
 	  If unsure, say N.
 
+config NFS_CACHEFS
+	bool "Provide NFS client caching support through CacheFS"
+	depends on NFS_FS && CACHEFS && EXPERIMENTAL
+	help
+		Say Y here if you want NFS data to be cached locally on disc through
+		the CacheFS filesystem.
+
 config NFS_DIRECTIO
 	bool "Allow direct I/O on NFS files (EXPERIMENTAL)"
 	depends on NFS_FS && EXPERIMENTAL
#1 lore raw

P=E5 m=E5 , 04/10/2004 klokka 22:45, skreiv Steve Dickson:

Quoted 2 lines

3) There is no user level support. I realize this is extremely cheesy but I noticed that the NFS posix mount option (in the 2.6 kerne=

l)

Quoted 4 lines

was no longer being used, so I high jacked it. Which means to make NFS to used CacheFS you need to use the posix option:
=20
mount -o posix server:/export/home /mnt/server/home

This is my one and only real gripe about it. The posix mount option is clearly documented, so we really cannot play around with it. Why can't you just add a separate cachefs flag?

Otherwise, I'm a bit dubious about the wisdom of putting nfs_invalidatepage() and nfs_releasepage() into fs/nfs/file.c. These ar= e
not file operations, but rather pure page cache operations. I would hav= e
thought that either read.c or possibly nfs-cachefs.c would be more appropriate.

Please note too that Chuck has made generic functions for copying and comparing NFS filehandles. They should be used in nfs_cache_fh_match() = &
co.
I'm a bit worried about the use of the raw IP address in nfs_cache_server_match(). It seems to me that when we add the NFSv4.1 support for trunking over several different transport mechanisms (RDMA, IPv4/v6 etc) on the same mountpoint, then we may end up with a problem. We can probably leave it in for now, but later we may want to consider switching to using server->hostname or something equivalent.

Otherwise, it looks good. Looking forward to try it out...

Cheers,
Trond

#2 lore raw
Quoted 4 lines

This is my one and only real gripe about it. The posix mount option is clearly documented, so we really cannot play around with it. Why can't you just add a separate cachefs flag?

That's just for the prototype. It will be changed later. It's also a stand-in for the superblock un-aliasing patch.

Quoted 6 lines

I'm a bit worried about the use of the raw IP address in nfs_cache_server_match(). It seems to me that when we add the NFSv4.1 support for trunking over several different transport mechanisms (RDMA, IPv4/v6 etc) on the same mountpoint, then we may end up with a problem. We can probably leave it in for now, but later we may want to consider switching to using server->hostname or something equivalent.

There needs to be some way of matching NFS file handle sets in the cache if you want persistant caches. The only way I could see was to use IP addr and port number for NFS.

However, changing this later is trivial. If you bump the version number in the netfs definition, then cachefs will nuke existing data in the cache that isn't of the expected format.

David

#3 lore raw
In reply to Steve Dickson

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/05/2004 05:45 AM, Steve Dickson wrote:

| 3) There is no user level support. I realize this is extremely cheesy
|     but I noticed that the NFS posix mount  option (in the 2.6 kernel)
|     was no longer being used, so I high jacked it.  Which means
|     to make NFS to used CacheFS you need to use the posix option:
|
|     mount -o posix server:/export/home /mnt/server/home

brr :) why is it posix, this is so out of the context for me (as a user). Is it possible to have a cachefs flag. Would make it more logical.

- --
[ Clemens Schwaighofer -----=====:::::~ ]
[ TBWA\ && TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-0061, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jp http://www.tbwajapan.co.jp ]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBZhlPjBz/yQjBxz8RAtKbAKCLWJBqWQUcroSaLzlrH1r2nuqI8gCfUzyL kpANJQ4pYt1026VBRm8UlpU=
=NGFC
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 3 lines

brr :) why is it posix, this is so out of the context for me (as a user). Is it possible to have a cachefs flag. Would make it more logical.

It's a prototype, so Steve's made use of an existing flag for convenience sake.

David

#5 lore raw
In reply to Clemens Schwaighofer
Quoted 2 lines · Clemens Schwaighofer

brr :) why is it posix, this is so out of the context for me (as a user). Is it possible to have a cachefs flag. Would make it more logical.

Because it was the easiest way to get things started (i.e. no userlevel changes needed at all).... The 'fscache' flag will be coming along with the nfs4 support, since nfs4 mounting code does not have an open (unused) mounting flag....

SteveD.

#6 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · Steve Dickson
Quoted 2 lines · Clemens Schwaighofer

brr :) why is it posix, this is so out of the context for me (as a user). Is it possible to have a cachefs flag. Would make it more logical.

Because it was the easiest way to get things started (i.e. no userlevel changes needed at all).... The 'fscache' flag will be coming along with the nfs4 support, since nfs4 mounting code does not have an open (unused) mounting flag....

okay, thank you for the information. Anyway inbetween I will try that out with NFS3 ...

lg, clemens

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBZrTAjBz/yQjBxz8RAoddAJwIVyxuMFy0tD9H8z8D/UKjXmvAtgCg4mcY 2NZyYRCsdR6CzbTKKcJiybw=
=sKPN
-----END PGP SIGNATURE-----

#7 lore raw
In reply to Steve Dickson

hi :)

Quoted 2 lines · Steve Dickson

The 'fscache' flag will be coming along with the nfs4 support, since nfs4 mounting code does not have an open (unused) mounting flag....

is such a flag even neccessary?
The way I see fscache is that its operations will be no-ops anyway if you haven't mounted any backing cache.

Signature · 1 line
Martin Waitz

--
Linux-cachefs mailing list
[email protected]
http://www.redhat.com/mailman/listinfo/linux-cachefs

#8 lore raw
Quoted 7 lines
Quoted 3 lines

The 'fscache' flag will be coming along with the nfs4 support, since nfs4 mounting code does not have an open (unused) mounting flag....

is such a flag even neccessary?
The way I see fscache is that its operations will be no-ops anyway if you haven't mounted any backing cache.

There are two reasons:

(1) The fscache middle bit builds up a cookie tree even if there are no backing caches. This means if the cache is added later (NFS root, for example), the netfs then starts caching immediately without requiring a remount.

(2) NFS is not currently safe with respect to fscache in the following situation:

mount warthog:/warthog/x /x
mount warthog:/warthog /y

Because NFS ends up with two superblocks for what is one set of filehandles. This means it tries to get the same cookies out of fscache twice, which fscache is ill-equipped to handle.

This also introduces unnecessary (IMNSHO) aliasing in each client of inodes and pages both.

I have a patch to fix NFS to cope with this by modifying the get_sb() method to allow the filesystem to override the automatic selection of super->s_root as the root dentry. However, Al Viro is leery of the patch for some reason.

David

--
Linux-cachefs mailing list
[email protected]
http://www.redhat.com/mailman/listinfo/linux-cachefs

#9 lore raw
In reply to Martin Waitz

P=E5 m=E5 , 11/10/2004 klokka 16:23, skreiv Martin Waitz:

Quoted 8 lines

hi :)
=20

Quoted 2 lines · Steve Dickson

The 'fscache' flag will be coming along with the nfs4 support, since nfs4 mounting code does not have an open (unused) mounting flag....

=20
is such a flag even neccessary?
The way I see fscache is that its operations will be no-ops anyway if y=

ou

Quoted 1 line

haven't mounted any backing cache.

You may not want to run cachefs on *all* your NFS partitions. It will slow you down on those partitions that have lots of cache contention.

That said, David & co.: why did you choose not to use something similar to the Solaris syntax for cachefs? The "layered filesystem" syntax has the advantage that it would avoid entirely the need to change the NFS mount syntax, and would make it easier to port cachefs to cifs etc.

Cheers,
Trond

--
Linux-cachefs mailing list
[email protected]
http://www.redhat.com/mailman/listinfo/linux-cachefs

#10 lore raw

accessing/modifiying the nfs share files as different users...

hi...

hopefully this is the nfs list.

i've got an nfs share (linux rh8.0) that i can access/copy files to/from...

however, i can't seem to figure out how to access it as a user other than root. basically, i'd like to set it up, to allow the 'apache' user to be able to automatically be able to add/modify files that reside on the share...

in searching through google, i haven't come up with a way of being able to accomplish this...

thanks...

-bruce

-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#11 lore raw
In reply to Trond Myklebust
Quoted 2 lines · Trond Myklebust

That said, David & co.: why did you choose not to use something similar to the Solaris syntax for cachefs?

Well...

(1) when I wrote cachefs I didn't know that Solaris had such a thing

(2) I don't know what Solaris syntax is - I've seen one example, and that didn't make a lot of sense

(3) my way doesn't involve any changes to userspace programs such as mount

Quoted 2 lines

The "layered filesystem" syntax has the advantage that it would avoid entirely the need to change the NFS mount syntax,

And the disadvantage that we'd have to change the VFS to support it, I think. I don't know how this "layered filesystem" thing operates or is used, so I'm guessing.

Quoted 1 line

and would make it easier to port cachefs to cifs etc.

I doubt it. The netfs still has to interact with fscache internally to decide how match netfs files to cache files, which is what most of the interface is about; and to push/pull pages to/from the cache (this bit could possibly be made transparent, but I'm not sure how you'd do it on Linux with the present VM & VFS interfaces).

David

--
Linux-cachefs mailing list
[email protected]
http://www.redhat.com/mailman/listinfo/linux-cachefs