Network filesystem caching on local cache files
Subsystems (5)
David Howells
maintainerRFC PATCH 1/1 Network filesystem caching on local cache files
I've now prototyped a module to do local caching of network filesystems using files on an already mounted filesystem (such as using EXT3 files to cache NFS files).
Currently the module is activated like this:
insmod cachefiles.ko root=/var/fscache
The cache is then filed in the directory specified as the root. I don't really like the idea of passing it on the insmod command line, but it's handy for development. cachefiles requires extended attributes and bmap to be available on the underlying filesystem.
The cachefiles module requires fscache.ko to be loaded first, as does NFS.
NFS is then told to use fscache by:
/root/mount warthog:/warthog /warthog -o fsc
I have some performance numbers for this:
(1) Lookup performance (1KB ext3 blocks):
Doing a "find" over a 20442 file kernel tree:
No cache: 10s
Cold cache: 23s
Warm cache: 14s
Note that this is with dir_index and user_xattrs turned on on the EXT3 filesystem. Without dir_index, this is really slow.
(2) Read performance:
Reading a 100MB file (1KB ext3 blocks):
No cache: 26s
Cold cache: 44s
Warm cache: 19s
Read cachefile: 15s
dd /dev/zero to disk file: 14s
dd disk file to /dev/null: 12s
dd blockdev to /dev/null: 11s
Reading a 100MB file (4KB ext3 blocks):
Cold cache: 35s
Warm cache: 14s
Read cachefile: 14s
dd /dev/zero to disk file: 7s
dd disk file to /dev/null: 12s
Reading a 100MB file (freshly prepared CacheFS)
Cold cache: 27s
Warm cache: 11s
Reading a 200MB file (1KB ext3 blocks):
No cache: 46s
Cold cache: 79s
Warm cache: 37s
Read cachefile: 32s
dd /dev/zero to disk file: 29s
dd disk file to /dev/null: 30s
dd blockdev to /dev/null: 22s
Reading a 200MB file (4KB ext3 blocks):
Cold cache: 62s
Warm cache: 29s
Read cachefile: 25s
dd /dev/zero to disk file: 22s
dd disk file to /dev/null: 23s
Reading a 200MB file (freshly prepared CacheFS)
Cold cache: 47s
Warm cache: 23s
Or to put it another way:
100MB File 200MB File
======================= =======================
1K Ext3 4K Ext3 CacheFS 1K Ext3 4K Ext3 CacheFS
======= ======= ======= ======= ======= =======
No Cache 26 26 26 46 46 46
Cold Cache 44 35 27 79 62 47
Warm Cache 19 14 11 37 29 23
-
Read Cachefile 15 14 - 32 25 -
Write Disk File 14 7 - 29 22 -
Read Disk File 12 12 - 30 23 -
-
Read Blockdev 11 11 11 22 22 22
Note that the 100MB file read could, theoretically, fit entirely within
the pagecache on my test machine as the machine has 128MB of RAM.
However, CacheFiles on EXT3 doubles the number of pages required because
it has separate pages for EXT3 and for NFS and copies between them.So, having a cold cache on EXT3 is going to affect performance quite a lot. I can't contrast CacheFS find performance with CacheFiles yet until I've fixed using NFS with CacheFS.
A large part of the problem is probably that I'm having to copy all the data from the NFS pages into the EXT3 pages and back again. I've looked at doing page stealing, but that's only possible in a tiny set of cases, and in any case does not help with cold-cache performance. CacheFS almost certainly wins here since it submits BIOs that run directly to and from the NFS pages.
Note that CacheFS has a problem with fragmentation, and its warm cache performance degrades over time, and can wind up taking as much as twice as long to read from disk as to read over the network. Cold cache performance doesn't seem to be much affected.
These numbers are made with the data being copied from the NFS page to the EXT3 page inside of the write-to-cache routine (cachefiles_write_page) before we return to NFS's readpage completion routine. If I defer the copy and have keventd do it instead, this costs about 2s/100MB. I wonder if this is because the NFS page is entirely contained within the cache on the CPU at the time cachefiles_write_page() is called, but that it's been evicted by the time keventd is scheduled to deal with it.
Note that the attached patch only does the I/O side of things. It does not yet do cache limit maintenance and culling (much of which will probably be done in userspace).
The full set of patches can be retrieved from:
http://people.redhat.com/~dhowells/cachefs/
They are based on Trond's nfs-2.6 git tree.
David
---
fs/Kconfig | 12 fs/Makefile | 1 fs/cachefiles/Makefile | 13 + fs/cachefiles/cf-interface.c | 1034 ++++++++++++++++++++++++++++++++++++++++++ fs/cachefiles/cf-main.c | 245 ++++++++++ fs/cachefiles/cf-pathwalk.c | 235 ++++++++++ fs/cachefiles/cf-xattr.c | 109 ++++ fs/cachefiles/internal.h | 210 +++++++++ fs/fscache/page.c | 4 fs/nfs/nfs-fscache.c | 6 include/linux/fs.h | 1 include/linux/pagemap.h | 6 mm/filemap.c | 102 ++++ 13 files changed, 1972 insertions(+), 6 deletions(-)
fs/Kconfig +10 -2
@@ -526,8 +526,8 @@ config CACHEFS depends on FSCACHE help This filesystem acts as a cache for other filesystems - primarily - networking filesystems - rather than thus allowing fast local disc to - enhance the speed of slower devices. + networking filesystems - thus allowing fast local disk to enhance the + speed of slower devices. It is a filesystem so that raw block devices can be made use of more efficiently, without suffering any overhead from intermediary
@@ -540,6 +540,14 @@ config CACHEFS See Documentation/filesystems/caching/cachefs.txt for more information. +config CACHEFILES + tristate "Filesystem caching on files" + depends on FSCACHE + help + This permits use of a mounted filesystem as a cache for other + filesystems - primarily networking filesystems - thus allowing fast + local disk to enhance the speed of slower devices. + endmenu menu "CD-ROM/DVD Filesystems"
fs/Makefile +1 -0
@@ -102,6 +102,7 @@ obj-$(CONFIG_BEFS_FS) += befs/ obj-$(CONFIG_HOSTFS) += hostfs/ obj-$(CONFIG_HPPFS) += hppfs/ obj-$(CONFIG_CACHEFS) += cachefs/ +obj-$(CONFIG_CACHEFILES) += cachefiles/ obj-$(CONFIG_DEBUG_FS) += debugfs/ obj-$(CONFIG_CONFIGFS_FS) += configfs/ obj-$(CONFIG_OCFS2_FS) += ocfs2/
fs/cachefiles/Makefile new file +13 -0
@@ -0,0 +1,13 @@ +# +# Makefile for caching in a mounted filesystem +# + +#CFLAGS += -finstrument-functions + +cachefiles-objs := \ + cf-xattr.o \ + cf-interface.o \ + cf-main.o \ + cf-pathwalk.o + +obj-$(CONFIG_CACHEFILES) := cachefiles.o
fs/cachefiles/cf-interface.c new file +1034 -0
1,034 lines not on this page · raw message
fs/cachefiles/cf-main.c new file +245 -0
245 lines not on this page · raw message
fs/cachefiles/cf-pathwalk.c new file +235 -0
235 lines not on this page · raw message
fs/cachefiles/cf-xattr.c new file +109 -0
109 lines not on this page · raw message
fs/cachefiles/internal.h new file +210 -0
210 lines not on this page · raw message
fs/fscache/page.c +2 -2
@@ -397,7 +397,7 @@ int __fscache_write_pages(struct fscachestruct fscache_object *object;int ret; -_enter ("%p,{%d}," , cookie, pagevec->nr); +_enter ("%p,{%ld}," , cookie, pagevec->nr);/* not supposed to use this for indexes */ BUG_ON (cookie->def->type ==FSCACHE_COOKIE_TYPE_INDEX );
@@ -484,7 +484,7 @@ void __fscache_uncache_pages(struct fsca {struct fscache_object *object; -_enter (",{%d}" , pagevec->nr); +_enter (",{%ld}" , pagevec->nr);BUG_ON (pagevec->nr <=0 );BUG_ON (!pagevec->pages[0 ]);
fs/nfs/nfs-fscache.c +4 -2
@@ -64,12 +64,13 @@ static uint16_t nfs_server_get_key(constbreak ;default : - len =0 ; + len =0 ;printk (KERN_WARNING "NFS: Unknown network family '%d'\n" , server->addr.sin_family);break ; } +// printk("nfs_server_get_key() = %hu\n", len); return len; }
@@ -91,7 +92,8 @@ static uint16_t nfs_fh_get_key(const voi/* set the file handle */ nsize = nfsi->fh.size;memcpy (buffer, nfsi->fh.data, nsize); -//printk("nfs_fh_get_key: nfsi 0x%p nsize %d\n", nfsi, nsize); + +// printk("nfs_fh_get_key: nfsi 0x%p nsize %hd\n", nfsi, nsize); return nsize; }
include/linux/fs.h +1 -0
@@ -1572,6 +1572,7 @@ extern ssize_t generic_file_direct_write unsigned long *, loff_t, loff_t *, size_t, size_t);extern ssize_t generic_file_buffered_write (struct kiocb *,const struct iovec *,unsigned long ,loff_t ,loff_t *,size_t ,ssize_t ); +extern int generic_file_buffered_write_one_kernel_page (struct file *,pgoff_t ,struct page *);extern ssize_t do_sync_read (struct file *filp,char __user *buf,size_t len,loff_t *ppos);extern ssize_t do_sync_write (struct file *filp,const char __user *buf,size_t len,loff_t *ppos); ssize_t generic_file_write_nolock(struct file *file, const struct iovec *iov,
include/linux/pagemap.h +6 -0
@@ -215,6 +215,12 @@ static inline void wait_on_page_fs_misc( extern void fastcall end_page_fs_misc(struct page *page);/* +* permit installation of a state change monitor in the queue for a page +*/ +extern void install_page_waitqueue_monitor (struct page *page, +wait_queue_t *monitor); + +/* * Fault a userspace page into pagetables. Return non-zero on a fault. * * This assumes that two userspace pages are always sufficient. That's
mm/filemap.c +102 -0
114 lines not on this page · raw message