Skip to contentSkip to conversation

Network filesystem caching on local cache files

RFC PATCH New David Howells 1 participant 1 message Mar 21, 2006

Subsystems (5)
lore
Share & tools
lore

David Howells

author maintainer
#0 lore raw

RFC PATCH 1/1 Network filesystem caching on local cache files

I've now prototyped a module to do local caching of network filesystems using files on an already mounted filesystem (such as using EXT3 files to cache NFS files).

Currently the module is activated like this:

insmod cachefiles.ko root=/var/fscache

The cache is then filed in the directory specified as the root. I don't really like the idea of passing it on the insmod command line, but it's handy for development. cachefiles requires extended attributes and bmap to be available on the underlying filesystem.

The cachefiles module requires fscache.ko to be loaded first, as does NFS.

NFS is then told to use fscache by:

/root/mount warthog:/warthog /warthog -o fsc

I have some performance numbers for this:

(1) Lookup performance (1KB ext3 blocks):

Doing a "find" over a 20442 file kernel tree:

No cache: 10s
Cold cache: 23s
Warm cache: 14s

Note that this is with dir_index and user_xattrs turned on on the EXT3 filesystem. Without dir_index, this is really slow.

(2) Read performance:

     Reading a 100MB file (1KB ext3 blocks):

	No cache:			26s
	Cold cache:			44s
	Warm cache:			19s
	Read cachefile:			15s
	dd /dev/zero to disk file:	14s
	dd disk file to /dev/null:	12s
	dd blockdev to /dev/null:	11s

     Reading a 100MB file (4KB ext3 blocks):

	Cold cache:			35s
	Warm cache:			14s
	Read cachefile:			14s
	dd /dev/zero to disk file:	7s
	dd disk file to /dev/null:	12s

     Reading a 100MB file (freshly prepared CacheFS)

	Cold cache:			27s
	Warm cache:			11s

     Reading a 200MB file (1KB ext3 blocks):

	No cache:			46s
	Cold cache:			79s
	Warm cache:			37s
	Read cachefile:			32s
	dd /dev/zero to disk file:	29s
	dd disk file to /dev/null:	30s
	dd blockdev to /dev/null:	22s

     Reading a 200MB file (4KB ext3 blocks):

	Cold cache:			62s
	Warm cache:			29s
	Read cachefile:			25s
	dd /dev/zero to disk file:	22s
	dd disk file to /dev/null:	23s

     Reading a 200MB file (freshly prepared CacheFS)

	Cold cache:			47s
	Warm cache:			23s

     Or to put it another way:


			       100MB File	       200MB File
			=======================	=======================
			1K Ext3	4K Ext3	CacheFS	1K Ext3	4K Ext3	CacheFS	
			=======	=======	=======	=======	=======	=======	
	No Cache	26	26	26	46	46	46
	Cold Cache	44	35	27	79	62	47
	Warm Cache	19	14	11	37	29	23
	-
	Read Cachefile	15	14	-	32	25	-
	Write Disk File	14	7	-	29	22	-
	Read Disk File	12	12	-	30	23	-
	-
	Read Blockdev	11	11	11	22	22	22


     Note that the 100MB file read could, theoretically, fit entirely within
     the pagecache on my test machine as the machine has 128MB of RAM.
     However, CacheFiles on EXT3 doubles the number of pages required because
     it has separate pages for EXT3 and for NFS and copies between them.

So, having a cold cache on EXT3 is going to affect performance quite a lot. I can't contrast CacheFS find performance with CacheFiles yet until I've fixed using NFS with CacheFS.

A large part of the problem is probably that I'm having to copy all the data from the NFS pages into the EXT3 pages and back again. I've looked at doing page stealing, but that's only possible in a tiny set of cases, and in any case does not help with cold-cache performance. CacheFS almost certainly wins here since it submits BIOs that run directly to and from the NFS pages.

Note that CacheFS has a problem with fragmentation, and its warm cache performance degrades over time, and can wind up taking as much as twice as long to read from disk as to read over the network. Cold cache performance doesn't seem to be much affected.

These numbers are made with the data being copied from the NFS page to the EXT3 page inside of the write-to-cache routine (cachefiles_write_page) before we return to NFS's readpage completion routine. If I defer the copy and have keventd do it instead, this costs about 2s/100MB. I wonder if this is because the NFS page is entirely contained within the cache on the CPU at the time cachefiles_write_page() is called, but that it's been evicted by the time keventd is scheduled to deal with it.

Note that the attached patch only does the I/O side of things. It does not yet do cache limit maintenance and culling (much of which will probably be done in userspace).

The full set of patches can be retrieved from:

http://people.redhat.com/~dhowells/cachefs/

They are based on Trond's nfs-2.6 git tree.

David

---

 fs/Kconfig                   |   12 
 fs/Makefile                  |    1 
 fs/cachefiles/Makefile       |   13 +
 fs/cachefiles/cf-interface.c | 1034 ++++++++++++++++++++++++++++++++++++++++++
 fs/cachefiles/cf-main.c      |  245 ++++++++++
 fs/cachefiles/cf-pathwalk.c  |  235 ++++++++++
 fs/cachefiles/cf-xattr.c     |  109 ++++
 fs/cachefiles/internal.h     |  210 +++++++++
 fs/fscache/page.c            |    4 
 fs/nfs/nfs-fscache.c         |    6 
 include/linux/fs.h           |    1 
 include/linux/pagemap.h      |    6 
 mm/filemap.c                 |  102 ++++
 13 files changed, 1972 insertions(+), 6 deletions(-)
fs/Kconfig +10 -2
index 8e74709..787e934 100644
@@ -526,8 +526,8 @@ config CACHEFS
 	depends on FSCACHE
 	help
 	  This filesystem acts as a cache for other filesystems - primarily
-	  networking filesystems - rather than thus allowing fast local disc to
-	  enhance the speed of slower devices.
+	  networking filesystems - thus allowing fast local disk to enhance the
+	  speed of slower devices.
 
 	  It is a filesystem so that raw block devices can be made use of more
 	  efficiently, without suffering any overhead from intermediary
@@ -540,6 +540,14 @@ config CACHEFS
 
 	  See Documentation/filesystems/caching/cachefs.txt for more information.
 
+config CACHEFILES
+	tristate "Filesystem caching on files"
+	depends on FSCACHE
+	help
+	  This permits use of a mounted filesystem as a cache for other
+	  filesystems - primarily networking filesystems - thus allowing fast
+	  local disk to enhance the speed of slower devices.
+
 endmenu
 
 menu "CD-ROM/DVD Filesystems"
fs/Makefile +1 -0
index ba610a5..499ef07 100644
@@ -102,6 +102,7 @@ obj-$(CONFIG_BEFS_FS)		+= befs/
 obj-$(CONFIG_HOSTFS)		+= hostfs/
 obj-$(CONFIG_HPPFS)		+= hppfs/
 obj-$(CONFIG_CACHEFS)		+= cachefs/
+obj-$(CONFIG_CACHEFILES)	+= cachefiles/
 obj-$(CONFIG_DEBUG_FS)		+= debugfs/
 obj-$(CONFIG_CONFIGFS_FS)	+= configfs/
 obj-$(CONFIG_OCFS2_FS)		+= ocfs2/
fs/cachefiles/Makefile new file +13 -0
new file mode 100644
index 0000000..683166f
@@ -0,0 +1,13 @@
+#
+# Makefile for caching in a mounted filesystem
+#
+
+#CFLAGS += -finstrument-functions
+
+cachefiles-objs := \
+	cf-xattr.o \
+	cf-interface.o \
+	cf-main.o \
+	cf-pathwalk.o
+
+obj-$(CONFIG_CACHEFILES) := cachefiles.o
fs/cachefiles/cf-interface.c new file +1034 -0

1,034 lines not on this page · raw message

fs/cachefiles/cf-main.c new file +245 -0

245 lines not on this page · raw message

fs/cachefiles/cf-pathwalk.c new file +235 -0

235 lines not on this page · raw message

fs/cachefiles/cf-xattr.c new file +109 -0

109 lines not on this page · raw message

fs/cachefiles/internal.h new file +210 -0

210 lines not on this page · raw message

fs/fscache/page.c +2 -2
index 1a5edf2..927008f 100644
@@ -397,7 +397,7 @@ int __fscache_write_pages(struct fscache
 	struct fscache_object *object;
 	int ret;
 
-	_enter("%p,{%d},", cookie, pagevec->nr);
+	_enter("%p,{%ld},", cookie, pagevec->nr);
 
 	/* not supposed to use this for indexes */
 	BUG_ON(cookie->def->type == FSCACHE_COOKIE_TYPE_INDEX);
@@ -484,7 +484,7 @@ void __fscache_uncache_pages(struct fsca
 {
 	struct fscache_object *object;
 
-	_enter(",{%d}", pagevec->nr);
+	_enter(",{%ld}", pagevec->nr);
 
 	BUG_ON(pagevec->nr <= 0);
 	BUG_ON(!pagevec->pages[0]);
fs/nfs/nfs-fscache.c +4 -2
index 97efe89..ab237fe 100644
@@ -64,12 +64,13 @@ static uint16_t nfs_server_get_key(const
 		break;
 
 	default:
-		len =0;
+		len = 0;
 		printk(KERN_WARNING "NFS: Unknown network family '%d'\n",
 			server->addr.sin_family);
 		break;
 	}
 
+//	printk("nfs_server_get_key() = %hu\n", len);
 	return len;
 }
 
@@ -91,7 +92,8 @@ static uint16_t nfs_fh_get_key(const voi
 	/* set the file handle */
 	nsize = nfsi->fh.size;
 	memcpy(buffer, nfsi->fh.data, nsize);
-//printk("nfs_fh_get_key: nfsi 0x%p nsize %d\n", nfsi, nsize);
+
+//	printk("nfs_fh_get_key: nfsi 0x%p nsize %hd\n", nfsi, nsize);
 	return nsize;
 }
 
include/linux/fs.h +1 -0
index 074d776..cc9ecc0 100644
@@ -1572,6 +1572,7 @@ extern ssize_t generic_file_direct_write
 		unsigned long *, loff_t, loff_t *, size_t, size_t);
 extern ssize_t generic_file_buffered_write(struct kiocb *, const struct iovec *,
 		unsigned long, loff_t, loff_t *, size_t, ssize_t);
+extern int generic_file_buffered_write_one_kernel_page(struct file *, pgoff_t, struct page *);
 extern ssize_t do_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos);
 extern ssize_t do_sync_write(struct file *filp, const char __user *buf, size_t len, loff_t *ppos);
 ssize_t generic_file_write_nolock(struct file *file, const struct iovec *iov,
include/linux/pagemap.h +6 -0
index 4873f3e..91fe228 100644
@@ -215,6 +215,12 @@ static inline void wait_on_page_fs_misc(
 extern void fastcall end_page_fs_misc(struct page *page);
 
 /*
+ * permit installation of a state change monitor in the queue for a page
+ */
+extern void install_page_waitqueue_monitor(struct page *page,
+					   wait_queue_t *monitor);
+
+/*
  * Fault a userspace page into pagetables.  Return non-zero on a fault.
  *
  * This assumes that two userspace pages are always sufficient.  That's
mm/filemap.c +102 -0

114 lines not on this page · raw message