Skip to contentSkip to conversation

FS-Cache: Provide a filesystem-specific sync'able page bit

PATCH In review David Howells 5 participants 31 messages Apr 20, 2006 – Apr 21, 2006

Subsystems (16)
lore
Share & tools
lore

David Howells

author maintainer
#0 lore raw

PATCH 1/7 FS-Cache: Provide a filesystem-specific sync'able page bit

The attached patch provides a filesystem-specific page bit that a filesystem can synchronise upon. This can be used, for example, by a netfs to synchronise with CacheFS writing its pages to disk.

---

 include/linux/page-flags.h |   10 ++++++++++
 include/linux/pagemap.h    |   11 +++++++++++
 mm/filemap.c               |   17 +++++++++++++++++
 3 files changed, 38 insertions(+), 0 deletions(-)
include/linux/page-flags.h +10 -0
index d276a4e..5486874 100644
@@ -74,6 +74,7 @@
 #define PG_slab			 7	/* slab debug (Suparna wants this) */
 
 #define PG_checked		 8	/* kill me in 2.5.<early>. */
+#define PG_fs_misc		 8
 #define PG_arch_1		 9
 #define PG_reserved		10
 #define PG_private		11	/* Has something at ->private */
@@ -376,4 +377,13 @@ static inline void set_page_writeback(st
 	test_set_page_writeback(page);
 }
 
+/*
+ * Filesystem-specific page bit testing
+ */
+#define PageFsMisc(page)		test_bit(PG_fs_misc, &(page)->flags)
+#define SetPageFsMisc(page)		set_bit(PG_fs_misc, &(page)->flags)
+#define TestSetPageFsMisc(page)		test_and_set_bit(PG_fs_misc, &(page)->flags)
+#define ClearPageFsMisc(page)		clear_bit(PG_fs_misc, &(page)->flags)
+#define TestClearPageFsMisc(page)	test_and_clear_bit(PG_fs_misc, &(page)->flags)
+
 #endif	/* PAGE_FLAGS_H */
include/linux/pagemap.h +11 -0
index 9539efd..02e7d8b 100644
@@ -206,6 +206,17 @@ static inline void wait_on_page_writebac
 extern void end_page_writeback(struct page *page);
 
 /*
+ * Wait for filesystem-specific page synchronisation to complete
+ */
+static inline void wait_on_page_fs_misc(struct page *page)
+{
+	if (PageFsMisc(page))
+		wait_on_page_bit(page, PG_fs_misc);
+}
+
+extern void fastcall end_page_fs_misc(struct page *page);
+
+/*
  * Fault a userspace page into pagetables.  Return non-zero on a fault.
  *
  * This assumes that two userspace pages are always sufficient.  That's
mm/filemap.c +17 -0
index 3ef2073..d4a598f 100644
@@ -545,6 +545,23 @@ void fastcall __lock_page(struct page *p
 EXPORT_SYMBOL(__lock_page);
 
 /*
+ * Note completion of filesystem specific page synchronisation
+ *
+ * This is used to allow a page to be written to a filesystem cache in the
+ * background without holding up the completion of readpage
+ */
+void fastcall end_page_fs_misc(struct page *page)
+{
+	smp_mb__before_clear_bit();
+	if (!TestClearPageFsMisc(page))
+		BUG();
+	smp_mb__after_clear_bit();
+	__wake_up_bit(page_waitqueue(page), &page->flags, PG_fs_misc);
+}
+
+EXPORT_SYMBOL(end_page_fs_misc);
+
+/*
  * a rather lightweight function, finding and getting a reference to a
  * hashed page atomically.
  */
author maintainer
#1 lore raw

PATCH 2/7 FS-Cache: Add notification of page becoming writable to VMA ops

The attached patch adds a new VMA operation to notify a filesystem or other driver about the MMU generating a fault because userspace attempted to write to a page mapped through a read-only PTE.

This facility permits the filesystem or driver to:

(*) Implement storage allocation/reservation on attempted write, and so to
deal with problems such as ENOSPC more gracefully (perhaps by generating SIGBUS).

(*) Delay making the page writable until the contents have been written to a
backing cache. This is useful for NFS/AFS when using FS-Cache/CacheFS. It permits the filesystem to have some guarantee about the state of the cache.

(*) Account and limit number of dirty pages. This is one piece of the puzzle
needed to make shared writable mapping work safely in FUSE.

---

 include/linux/mm.h |    4 ++
 mm/memory.c        |   99 +++++++++++++++++++++++++++++++++++++++-------------
 mm/mmap.c          |   12 +++++-
 mm/mprotect.c      |   11 +++++-
 4 files changed, 98 insertions(+), 28 deletions(-)
include/linux/mm.h +4 -0
index 1154684..cd3c2cf 100644
@@ -200,6 +200,10 @@ struct vm_operations_struct {
 	void (*close)(struct vm_area_struct * area);
 	struct page * (*nopage)(struct vm_area_struct * area, unsigned long address, int *type);
 	int (*populate)(struct vm_area_struct * area, unsigned long address, unsigned long len, pgprot_t prot, unsigned long pgoff, int nonblock);
+
+	/* notification that a previously read-only page is about to become
+	 * writable, if an error is returned it will cause a SIGBUS */
+	int (*page_mkwrite)(struct vm_area_struct *vma, struct page *page);
 #ifdef CONFIG_NUMA
 	int (*set_policy)(struct vm_area_struct *vma, struct mempolicy *new);
 	struct mempolicy *(*get_policy)(struct vm_area_struct *vma,
mm/memory.c +75 -24

124 lines not on this page · raw message

mm/mmap.c +10 -2
index e6ee123..6446c61 100644
@@ -1065,7 +1065,8 @@ munmap_back:
 	vma->vm_start = addr;
 	vma->vm_end = addr + len;
 	vma->vm_flags = vm_flags;
-	vma->vm_page_prot = protection_map[vm_flags & 0x0f];
+	vma->vm_page_prot = protection_map[vm_flags &
+				(VM_READ|VM_WRITE|VM_EXEC|VM_SHARED)];
 	vma->vm_pgoff = pgoff;
 
 	if (file) {
@@ -1089,6 +1090,12 @@ munmap_back:
 			goto free_vma;
 	}
 
+	/* Don't make the VMA automatically writable if it's shared, but the
+	 * backer wishes to know when pages are first written to */
+	if (vma->vm_ops && vma->vm_ops->page_mkwrite)
+		vma->vm_page_prot =
+			protection_map[vm_flags & (VM_READ|VM_WRITE|VM_EXEC)];
+
 	/* We set VM_ACCOUNT in a shared mapping's vm_flags, to inform
 	 * shmem_zero_setup (perhaps called through /dev/zero's ->mmap)
 	 * that memory reservation must be checked; but that reservation
@@ -1921,7 +1928,8 @@ unsigned long do_brk(unsigned long addr,
 	vma->vm_end = addr + len;
 	vma->vm_pgoff = pgoff;
 	vma->vm_flags = flags;
-	vma->vm_page_prot = protection_map[flags & 0x0f];
+	vma->vm_page_prot = protection_map[flags &
+				(VM_READ|VM_WRITE|VM_EXEC|VM_SHARED)];
 	vma_link(mm, vma, prev, rb_link, rb_parent);
 out:
 	mm->total_vm += len >> PAGE_SHIFT;
mm/mprotect.c +9 -2
index 4c14d42..2697abd 100644
@@ -106,6 +106,7 @@ mprotect_fixup(struct vm_area_struct *vm
 	unsigned long oldflags = vma->vm_flags;
 	long nrpages = (end - start) >> PAGE_SHIFT;
 	unsigned long charged = 0;
+	unsigned int mask;
 	pgprot_t newprot;
 	pgoff_t pgoff;
 	int error;
@@ -132,8 +133,6 @@ mprotect_fixup(struct vm_area_struct *vm
 		}
 	}
 
-	newprot = protection_map[newflags & 0xf];
-
 	/*
 	 * First try to merge with previous and/or next vma.
 	 */
@@ -160,6 +159,14 @@ mprotect_fixup(struct vm_area_struct *vm
 	}
 
 success:
+	/* Don't make the VMA automatically writable if it's shared, but the
+	 * backer wishes to know when pages are first written to */
+	mask = VM_READ|VM_WRITE|VM_EXEC|VM_SHARED;
+	if (vma->vm_ops && vma->vm_ops->page_mkwrite)
+		mask &= ~VM_SHARED;
+
+	newprot = protection_map[newflags & mask];
+
 	/*
 	 * vm_flags and vm_page_prot are protected by the mmap_sem
 	 * held in write mode.
#2 lore raw

Re: [PATCH 2/7] FS-Cache: Add notification of page becoming writable to VMA ops

Quoted 3 lines · David Howells

The attached patch adds a new VMA operation to notify a filesystem or other driver about the MMU generating a fault because userspace attempted to write to a page mapped through a read-only PTE.

This will almost certainly help OCFS2 get shared writable mmap() right, too, though it probably won't be the whole story.

- z

#3 lore raw

Re: [PATCH 2/7] FS-Cache: Add notification of page becoming writable to VMA ops

Quoted 7 lines · Zach Brown
Quoted 3 lines · David Howells

The attached patch adds a new VMA operation to notify a filesystem or other driver about the MMU generating a fault because userspace attempted to write to a page mapped through a read-only PTE.

This will almost certainly help OCFS2 get shared writable mmap() right, too, though it probably won't be the whole story.

This is also required by NTFS for a correct implementation of "mmap write into sparse region on ntfs volume with cluster size > PAGE_CACHE_SIZE".

Best regards,

Anton

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/
author maintainer
#4 lore raw

PATCH 3/7 FS-Cache: Avoid ENFILE checking for kernel-specific open files

In reply to David Howells

Make it possible to avoid ENFILE checking for kernel specific open files, such as are used by the CacheFiles module.

After, for example, tarring up a kernel source tree over the network, the CacheFiles module may easily have 20000+ files open in the backing filesystem, thus causing all non-root processes to be given error ENFILE when they try to open a file, socket, pipe, etc..

---

 arch/ia64/kernel/perfmon.c            |    2 +-
 drivers/infiniband/core/uverbs_main.c |    2 +-
 fs/eventpoll.c                        |    2 +-
 fs/file_table.c                       |   36 ++++++++++++++++++++++++---------
 fs/hugetlbfs/inode.c                  |    2 +-
 fs/inotify.c                          |    2 +-
 fs/namei.c                            |    2 +-
 fs/open.c                             |   22 +++++++++++++++++++-
 fs/pipe.c                             |    4 ++--
 include/linux/file.h                  |    1 -
 include/linux/fs.h                    |    8 ++++++-
 kernel/futex.c                        |    2 +-
 kernel/sysctl.c                       |    2 +-
 mm/shmem.c                            |    2 +-
 mm/tiny-shmem.c                       |    2 +-
 net/socket.c                          |    2 +-
 16 files changed, 67 insertions(+), 26 deletions(-)
arch/ia64/kernel/perfmon.c +1 -1
index 077f212..f23ab3a 100644
@@ -2162,7 +2162,7 @@ pfm_alloc_fd(struct file **cfile)
 
 	ret = -ENFILE;
 
-	file = get_empty_filp();
+	file = get_empty_filp(0);
 	if (!file) goto out;
 
 	/*
drivers/infiniband/core/uverbs_main.c +1 -1
index ff092a0..4f7137c 100644
@@ -525,7 +525,7 @@ struct file *ib_uverbs_alloc_event_file(
 		goto err;
 	}
 
-	filp = get_empty_filp();
+	filp = get_empty_filp(0);
 	if (!filp) {
 		ret = -ENFILE;
 		goto err_fd;
fs/eventpoll.c +1 -1
index 1b4491c..f774038 100644
@@ -714,7 +714,7 @@ static int ep_getfd(int *efd, struct ino
 
 	/* Get an ready to use file */
 	error = -ENFILE;
-	file = get_empty_filp();
+	file = get_empty_filp(0);
 	if (!file)
 		goto eexit_1;
 
fs/file_table.c +26 -10
index bcea199..300e7c2 100644
@@ -34,6 +34,7 @@ struct files_stat_struct files_stat = {
 __cacheline_aligned_in_smp DEFINE_SPINLOCK(files_lock);
 
 static struct percpu_counter nr_files __cacheline_aligned_in_smp;
+static atomic_t nr_kernel_files;
 
 static inline void file_free_rcu(struct rcu_head *head)
 {
@@ -43,7 +44,10 @@ static inline void file_free_rcu(struct 
 
 static inline void file_free(struct file *f)
 {
-	percpu_counter_dec(&nr_files);
+	if (!(f->f_kernel_flags & FKFLAGS_KERNEL))
+		percpu_counter_dec(&nr_files);
+	else
+		atomic_dec(&nr_kernel_files);
 	call_rcu(&f->f_u.fu_rcuhead, file_free_rcu);
 }
 
@@ -72,6 +76,7 @@ int proc_nr_files(ctl_table *table, int 
                      void __user *buffer, size_t *lenp, loff_t *ppos)
 {
 	files_stat.nr_files = get_nr_files();
+	files_stat.nr_kernel_files = atomic_read(&nr_kernel_files);
 	return proc_dointvec(table, write, filp, buffer, lenp, ppos);
 }
 #else
@@ -86,7 +91,7 @@ int proc_nr_files(ctl_table *table, int 
  * Returns NULL, if there are no more free file structures or
  * we run out of memory.
  */
-struct file *get_empty_filp(void)
+struct file *get_empty_filp(int kernel)
 {
 	struct task_struct *tsk;
 	static int old_max;
@@ -95,20 +100,29 @@ struct file *get_empty_filp(void)
 	/*
 	 * Privileged users can go above max_files
 	 */
-	if (get_nr_files() >= files_stat.max_files && !capable(CAP_SYS_ADMIN)) {
-		/*
-		 * percpu_counters are inaccurate.  Do an expensive check before
-		 * we go and fail.
-		 */
-		if (percpu_counter_sum(&nr_files) >= files_stat.max_files)
-			goto over;
+	if (!kernel) {
+		if (get_nr_files() >= files_stat.max_files &&
+		    !capable(CAP_SYS_ADMIN)
+		    ) {
+			/*
+			 * percpu_counters are inaccurate.  Do an expensive
+			 * check before we go and fail.
+			 */
+			if (percpu_counter_sum(&nr_files) >=
+			    files_stat.max_files)
+				goto over;
+		}
 	}
 
 	f = kmem_cache_alloc(filp_cachep, GFP_KERNEL);
 	if (f == NULL)
 		goto fail;
 
-	percpu_counter_inc(&nr_files);
+	if (!kernel)
+		percpu_counter_inc(&nr_files);
+	else
+		atomic_inc(&nr_kernel_files);
+
 	memset(f, 0, sizeof(*f));
 	if (security_file_alloc(f))
 		goto fail_sec;
@@ -117,6 +131,7 @@ struct file *get_empty_filp(void)
 	INIT_LIST_HEAD(&f->f_u.fu_list);
 	atomic_set(&f->f_count, 1);
 	rwlock_init(&f->f_owner.lock);
+	f->f_kernel_flags = kernel ? FKFLAGS_KERNEL : 0;
 	f->f_uid = tsk->fsuid;
 	f->f_gid = tsk->fsgid;
 	eventpoll_init_file(f);
@@ -235,6 +250,7 @@ struct file fastcall *fget_light(unsigne
 	return file;
 }
 
+EXPORT_SYMBOL(fget_light);
 
 void put_filp(struct file *file)
 {
fs/hugetlbfs/inode.c +1 -1
index 3a5b4e9..cc27ee8 100644
@@ -770,7 +770,7 @@ struct file *hugetlb_zero_setup(size_t s
 		goto out_shm_unlock;
 
 	error = -ENFILE;
-	file = get_empty_filp();
+	file = get_empty_filp(0);
 	if (!file)
 		goto out_dentry;
 
fs/inotify.c +1 -1
index 1f50302..2e66e05 100644
@@ -939,7 +939,7 @@ asmlinkage long sys_inotify_init(void)
 	if (fd < 0)
 		return fd;
 
-	filp = get_empty_filp();
+	filp = get_empty_filp(0);
 	if (!filp) {
 		ret = -ENFILE;
 		goto out_put_fd;
fs/namei.c +1 -1
index 96723ae..6713213 100644
@@ -1146,7 +1146,7 @@ static int __path_lookup_intent_open(int
 		unsigned int lookup_flags, struct nameidata *nd,
 		int open_flags, int create_mode)
 {
-	struct file *filp = get_empty_filp();
+	struct file *filp = get_empty_filp(0);
 	int err;
 
 	if (filp == NULL)
fs/open.c +21 -1
index 53ec28c..cea1538 100644
@@ -962,7 +962,7 @@ struct file *dentry_open(struct dentry *
 	struct file *f;
 
 	error = -ENFILE;
-	f = get_empty_filp();
+	f = get_empty_filp(0);
 	if (f == NULL) {
 		dput(dentry);
 		mntput(mnt);
@@ -974,6 +974,26 @@ struct file *dentry_open(struct dentry *
 EXPORT_SYMBOL(dentry_open);
 
 /*
+ * open a specifically in-kernel file
+ */
+struct file *dentry_open_kernel(struct dentry *dentry, struct vfsmount *mnt, int flags)
+{
+	int error;
+	struct file *f;
+
+	error = -ENFILE;
+	f = get_empty_filp(1);
+	if (f == NULL) {
+		dput(dentry);
+		mntput(mnt);
+		return ERR_PTR(error);
+	}
+
+	return __dentry_open(dentry, mnt, flags, f, NULL);
+}
+EXPORT_SYMBOL(dentry_open_kernel);
+
+/*
  * Find an empty file descriptor entry, and mark it busy.
  */
 int get_unused_fd(void)
fs/pipe.c +2 -2
index 7fefb10..6081367 100644
@@ -795,11 +795,11 @@ int do_pipe(int *fd)
 	int i, j;
 
 	error = -ENFILE;
-	f1 = get_empty_filp();
+	f1 = get_empty_filp(0);
 	if (!f1)
 		goto no_files;
 
-	f2 = get_empty_filp();
+	f2 = get_empty_filp(0);
 	if (!f2)
 		goto close_f1;
 
include/linux/file.h +0 -1
index 9f7c251..da7be8f 100644
@@ -79,7 +79,6 @@ extern void FASTCALL(set_close_on_exec(u
 extern void put_filp(struct file *);
 extern int get_unused_fd(void);
 extern void FASTCALL(put_unused_fd(unsigned int fd));
-struct kmem_cache;
 
 extern struct file ** alloc_fd_array(int);
 extern void free_fd_array(struct file **, int);
include/linux/fs.h +7 -1
index 3de2bfb..979b1d3 100644
@@ -33,6 +33,7 @@ struct files_stat_struct {
 	int nr_files;		/* read only */
 	int nr_free_files;	/* read only */
 	int max_files;		/* tunable */
+	int nr_kernel_files;	/* read only */
 };
 extern struct files_stat_struct files_stat;
 extern int get_max_files(void);
@@ -70,6 +71,8 @@ extern int dir_notify_enable;
    behavior for cross-node execution/opening_for_writing of files */
 #define FMODE_EXEC	16
 
+#define FKFLAGS_KERNEL	1		/* kernel internal file (not accounted) */
+
 #define RW_MASK		1
 #define RWA_MASK	2
 #define READ 0
@@ -640,6 +643,7 @@ struct file {
 	atomic_t		f_count;
 	unsigned int 		f_flags;
 	mode_t			f_mode;
+	unsigned short		f_kernel_flags;
 	loff_t			f_pos;
 	struct fown_struct	f_owner;
 	unsigned int		f_uid, f_gid;
@@ -1377,6 +1381,7 @@ extern long do_sys_open(int fdf, const c
 			int mode);
 extern struct file *filp_open(const char *, int, int);
 extern struct file * dentry_open(struct dentry *, struct vfsmount *, int);
+extern struct file * dentry_open_kernel(struct dentry *, struct vfsmount *, int);
 extern int filp_close(struct file *, fl_owner_t id);
 extern char * getname(const char __user *);
 
@@ -1577,7 +1582,7 @@ static inline void insert_inode_hash(str
 	__insert_inode_hash(inode, inode->i_ino);
 }
 
-extern struct file * get_empty_filp(void);
+extern struct file * get_empty_filp(int kernel);
 extern void file_move(struct file *f, struct list_head *list);
 extern void file_kill(struct file *f);
 struct bio;
@@ -1603,6 +1608,7 @@ extern ssize_t generic_file_direct_write
 		unsigned long *, loff_t, loff_t *, size_t, size_t);
 extern ssize_t generic_file_buffered_write(struct kiocb *, const struct iovec *,
 		unsigned long, loff_t, loff_t *, size_t, ssize_t);
+extern int generic_file_buffered_write_one_kernel_page(struct file *, pgoff_t, struct page *);
 extern ssize_t do_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos);
 extern ssize_t do_sync_write(struct file *filp, const char __user *buf, size_t len, loff_t *ppos);
 ssize_t generic_file_write_nolock(struct file *file, const struct iovec *iov,
kernel/futex.c +1 -1
index 5699c51..7c334f3 100644
@@ -779,7 +779,7 @@ static int futex_fd(unsigned long uaddr,
 	ret = get_unused_fd();
 	if (ret < 0)
 		goto out;
-	filp = get_empty_filp();
+	filp = get_empty_filp(0);
 	if (!filp) {
 		put_unused_fd(ret);
 		ret = -ENFILE;
kernel/sysctl.c +1 -1
index e82726f..e8f9b5f 100644
@@ -943,7 +943,7 @@ static ctl_table fs_table[] = {
 		.ctl_name	= FS_NRFILE,
 		.procname	= "file-nr",
 		.data		= &files_stat,
-		.maxlen		= 3*sizeof(int),
+		.maxlen		= 4*sizeof(int),
 		.mode		= 0444,
 		.proc_handler	= &proc_nr_files,
 	},
mm/shmem.c +1 -1
index 37eaf42..83bbbe8 100644
@@ -2311,7 +2311,7 @@ struct file *shmem_file_setup(char *name
 		goto put_memory;
 
 	error = -ENFILE;
-	file = get_empty_filp();
+	file = get_empty_filp(0);
 	if (!file)
 		goto put_dentry;
 
mm/tiny-shmem.c +1 -1
index f9d6a9c..b014dd5 100644
@@ -71,7 +71,7 @@ struct file *shmem_file_setup(char *name
 		goto put_memory;
 
 	error = -ENFILE;
-	file = get_empty_filp();
+	file = get_empty_filp(0);
 	if (!file)
 		goto put_dentry;
 
net/socket.c +1 -1
index 23898f4..9743df2 100644
@@ -377,7 +377,7 @@ static int sock_alloc_fd(struct file **f
 
 	fd = get_unused_fd();
 	if (likely(fd >= 0)) {
-		struct file *file = get_empty_filp();
+		struct file *file = get_empty_filp(0);
 
 		*filep = file;
 		if (unlikely(!file)) {
#5 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 7 lines · David Howells

Make it possible to avoid ENFILE checking for kernel specific open files, such as are used by the CacheFiles module.

After, for example, tarring up a kernel source tree over the network, the CacheFiles module may easily have 20000+ files open in the backing filesystem, thus causing all non-root processes to be given error ENFILE when they try to open a file, socket, pipe, etc..

No, just increase the limit. The whole point of the limit is to avoid resource exaustion. A file doesn't use any less ressources just becuase it's opened from kernelspace. In doubt increase the limit or even the default limit.

author maintainer
#6 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 8 lines · Christoph Hellwig
Quoted 7 lines

Make it possible to avoid ENFILE checking for kernel specific open files, such as are used by the CacheFiles module.

After, for example, tarring up a kernel source tree over the network, the CacheFiles module may easily have 20000+ files open in the backing filesystem, thus causing all non-root processes to be given error ENFILE when they try to open a file, socket, pipe, etc..

Quoted 4 lines

No, just increase the limit. The whole point of the limit is to avoid resource exaustion. A file doesn't use any less ressources just becuase it's opened from kernelspace. In doubt increase the limit or even the default limit.

As I saw it, the limit is there to prevent userspace from pinning too many file resources. Yes, each userspace process is limited by its rlimit, but you have to multiply that by the number of processes that can be around:

	1024 * 32767 > 32 million files

Besides, you say "increase the limit", but there are two problems with that:

(1) Each AFS or NFS _dentry_ retained in the system pins a file in the backing cache if it's also cached, whether or not it's open. So on my desktop box, I've got about a million dentries cached. That means I might also have anything up to a million files open... Except that the netfs would be denied caching rights on any file beyond the ENFILE limit
- not that that matters, since you wouldn't be able to exec or open anything if you weren't root - and that might include running su and the like.

(2) And what should the limit actually _be_? You haven't said. If you include the cache in the limit, you can at best open ENFILE limit / 2 files on the netfs before you get moaned at by the system... And then closed files aren't immediately released by the cache, so you can quickly find yourself backed into a corner...

With this patch, CacheFiles's consumption of files is controlled by the dcache reclaimer and not by ENFILE, and allocating more files will cause other cache files to be closed automatically.

The cache can't arbitrarily close backing files for which no userspace file descriptor remains open: there may be dirty pages, or the file may be mmapped.

David

#7 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 3 lines · David Howells

(1) Each AFS or NFS _dentry_ retained in the system pins a file in the backing cache if it's also cached, whether or not it's open.

That would seem to be a great shortcoming in fscache.

I guess as memory reclaim reaps the top-level dentries those file*'s will also be freed up, leading to their dentries becoming reclaimable, leading to their inodes being reclaimable.

But still. Is it not possible to release those files-pinned-by-dcache when the top-level files are closed?

author maintainer
#8 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 1 line · Andrew Morton

That would seem to be a great shortcoming in fscache.

It's not something that's easy to get around. The file has to be "open" to be able to do certain operations on it.

Quoted 3 lines

I guess as memory reclaim reaps the top-level dentries those file*'s will also be freed up, leading to their dentries becoming reclaimable, leading to their inodes being reclaimable.

Exactly.

Quoted 2 lines

But still. Is it not possible to release those files-pinned-by-dcache when the top-level files are closed?

No, for three (or maybe four) reasons:

(1) You assume there's a "top-level" file open. AFS lookup(), for example, will read the cache to get the directory contents, but there will _not_ be an open top-level file.

We could open and close the cache file in each lookup(), but that could be very bad for lookup performance.

(2) mmap() may still have the struct file open, even though the last close() has happened.

(3) There may be pages not yet written to the cache outstanding. These belong to the netfs *inode* not the netfs *file*. Whilst the flush or release file operation could be made to wait for these, that's not necessarily within their spec, and could take a long time. How far is the flush op supposed to go anyway?

(4) It prevents the data file going away whilst we have a cookie for it (someone might go into the cache and delete something they shouldn't).

Pinning the dentry might work just as well, I suppose, but it makes little difference to the resource consumption.

We keep at least the dentry available so that we can honour the netfs's requests to update the auxiliary data as quickly as possible.

David

#9 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

In reply to David Howells
Quoted 9 lines · David Howells

Make it possible to avoid ENFILE checking for kernel specific open files, such as are used by the CacheFiles module.

After, for example, tarring up a kernel source tree over the network, the CacheFiles module may easily have 20000+ files open in the backing filesystem, thus causing all non-root processes to be given error ENFILE when they try to open a file, socket, pipe, etc..

Quoted 5 lines

...

static struct percpu_counter nr_files __cacheline_aligned_in_smp;
+static atomic_t nr_kernel_files;

So it's not performance-critical.

fs/file_table.c int proc_nr_files(ctl_table *table, int · line 94

Quoted 2 lines
fs/file_table.c +1 -1
-struct file *get_empty_filp(void)
+struct file *get_empty_filp(int kernel)

I'd suggest a new get_empty_kernel_filp(void) rather than providing a magic argument. (we can still have the magic argument in the new __get_empty_filp(int), but it shouldn't be part of the caller-visible API).

fs/file_table.c struct file *get_empty_filp(void) · line 106

Quoted 4 lines
fs/file_table.c +4 -0
+	if (!kernel) {
+		if (get_nr_files() >= files_stat.max_files &&
+		    !capable(CAP_SYS_ADMIN)
+		    ) {

ugly.

fs/file_table.c struct file *get_empty_filp(void) · line 134

Quoted 1 line
fs/file_table.c +1 -0
+	f->f_kernel_flags = kernel ? FKFLAGS_KERNEL : 0;

It would be more flexible to make the caller pass in the flags directly.

fs/file_table.c struct file fastcall *fget_light(unsigne · line 253

Quoted 8 lines
fs/file_table.c +1 -0
 	f->f_uid = tsk->fsuid;
 	f->f_gid = tsk->fsgid;
 	eventpoll_init_file(f);
@@ -235,6 +250,7 @@ struct file fastcall *fget_light(unsigne
 	return file;
 }
 
+EXPORT_SYMBOL(fget_light);

fget_light is not otherwise referenced in this patch.

this change is not changelogged.

why non-GPL?

fs/open.c struct file *dentry_open(struct dentry * · line 994

Quoted 1 line
fs/open.c +1 -0
+EXPORT_SYMBOL(dentry_open_kernel);

_GPL?

include/linux/fs.h struct file { · line 649

Quoted 8 lines
include/linux/fs.h +1 -0
@@ -640,6 +643,7 @@ struct file {
 	atomic_t		f_count;
 	unsigned int 		f_flags;
 	mode_t			f_mode;
+	unsigned short		f_kernel_flags;
 	loff_t			f_pos;
 	struct fown_struct	f_owner;
 	unsigned int		f_uid, f_gid;

That's unfortunate. There's still room in f_flags. Was it hard to use that?

kernel/sysctl.c static ctl_table fs_table[] = { · line 948

Quoted 8 lines
kernel/sysctl.c +1 -1
@@ -943,7 +943,7 @@ static ctl_table fs_table[] = {
 		.ctl_name	= FS_NRFILE,
 		.procname	= "file-nr",
 		.data		= &files_stat,
-		.maxlen		= 3*sizeof(int),
+		.maxlen		= 4*sizeof(int),
 		.mode		= 0444,
 		.proc_handler	= &proc_nr_files,

This changes the format of /proc/sys/fs/file-nr. What will break?

author maintainer
#10 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 4 lines · Andrew Morton
Quoted 2 lines

static struct percpu_counter nr_files __cacheline_aligned_in_smp;
+static atomic_t nr_kernel_files;

So it's not performance-critical.

Hmmm... nowhere near as critical as the ENFILE accounting, plus the only place we actually read it is for the sysctl file.

It could actually be dispensed with entirely, I suppose.

Quoted 8 lines

fs/file_table.c int proc_nr_files(ctl_table *table, int · line 94

Quoted 2 lines
fs/file_table.c +1 -1
-struct file *get_empty_filp(void)
+struct file *get_empty_filp(int kernel)

I'd suggest a new get_empty_kernel_filp(void) rather than providing a magic argument. (we can still have the magic argument in the new __get_empty_filp(int), but it shouldn't be part of the caller-visible API).
...
It would be more flexible to make the caller pass in the flags directly.

So:

	struct file *get_empty_kernel_filp(unsigned short flags);

which devolves to get_empty_filp() if flags == 0?

Quoted 3 lines
Quoted 1 line

+EXPORT_SYMBOL(fget_light);

fget_light is not otherwise referenced in this patch.

Good point. I'll move it into the cachefiles patch.

Quoted 3 lines
Quoted 1 line

+EXPORT_SYMBOL(dentry_open_kernel);

_GPL?

If you wish.

Quoted 1 line

That's unfortunate. There's still room in f_flags. Was it hard to use that?

Yeah... but the usage of f_flags is constrained by O_xxxx flags that are part of the userspace interface. Using those up for purely kernel things is a bad idea.

Note that I've not actually increased the size of the struct file - f_mode is a 16-bit value, hence why I chose an unsigned short.

Quoted 1 line

This changes the format of /proc/sys/fs/file-nr. What will break?

As far as I can tell, not a lot. I've grepped through various etc, lib and bin directories on my FC5 system, and the only match I've found is:

	/usr/lib64/sa/sadc

I'll present the count through a separate file to make sure.

David

#11 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 9 lines · David Howells
Quoted 8 lines
Quoted 2 lines

+struct file *get_empty_filp(int kernel)

I'd suggest a new get_empty_kernel_filp(void) rather than providing a magic argument. (we can still have the magic argument in the new __get_empty_filp(int), but it shouldn't be part of the caller-visible API).
...
It would be more flexible to make the caller pass in the flags directly.

Quoted 6 lines

So:

	struct file *get_empty_kernel_filp(unsigned short flags);

which devolves to get_empty_filp() if flags == 0?

argh, I forgot about the flag. Oh well. I'd suggest:

static inline struct file *get_empty_filp(void)

{
	return __get_empty_filp(0);
}

static inline struct file *get_empty_kernel_filp(void)

{
	return __get_empty_filp(0);
}
author maintainer
#12 lore raw

Re: [PATCH 3/7] FS-Cache: Avoid ENFILE checking for kernel-specific open files

Quoted 1 line · Andrew Morton

argh, I forgot about the flag. Oh well. I'd suggest:

I presume you mean for the second to pass an argument of 1 to __get_empty_filp().

David

author maintainer
#13 lore raw

PATCH 4/7 FS-Cache: Export find_get_pages()

In reply to David Howells

The attached patch exports find_get_pages() for use by the kAFS filesystem in conjunction with it caching patch.

---

 mm/filemap.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
mm/filemap.c +2 -0
index d4a598f..d6f7ab4 100644
@@ -714,6 +714,8 @@ unsigned find_get_pages(struct address_s
 	return ret;
 }
 
+EXPORT_SYMBOL(find_get_pages);
+
 /*
  * Like find_get_pages, except we only return pages which are tagged with
  * `tag'.   We update *index to index the next page for the traversal.
#14 lore raw

Re: [PATCH 4/7] FS-Cache: Export find_get_pages()

Quoted 2 lines · David Howells

The attached patch exports find_get_pages() for use by the kAFS filesystem in conjunction with it caching patch.

Why don't you use pagevec ?

author maintainer
#15 lore raw

Re: [PATCH 4/7] FS-Cache: Export find_get_pages()

Quoted 4 lines · Christoph Hellwig
Quoted 2 lines

The attached patch exports find_get_pages() for use by the kAFS filesystem in conjunction with it caching patch.

Why don't you use pagevec ?

You mean pagevec_lookup() I suppose... That's probably reasonable, though slower.

David

#16 lore raw

Re: [PATCH 4/7] FS-Cache: Export find_get_pages()

Quoted 11 lines · David Howells
Quoted 4 lines · Christoph Hellwig
Quoted 2 lines

The attached patch exports find_get_pages() for use by the kAFS filesystem in conjunction with it caching patch.

Why don't you use pagevec ?

You mean pagevec_lookup() I suppose... That's probably reasonable, though slower.

But the code's using pagevecs now. In a strange manner.

+ nr_pages = find_get_pages(vnode->vfs_inode.i_mapping, first,
+ PAGEVEC_SIZE, pvec.pages);

that's an open-coded pagevec_lookup().

author maintainer
#17 lore raw

Re: [PATCH 4/7] FS-Cache: Export find_get_pages()

Quoted 1 line · Andrew Morton

that's an open-coded pagevec_lookup().

Whilst that's true, it is still slower to use pagevec_lookup(). But since you insist, I'll do that anyway.

David

author maintainer
#18 lore raw

PATCH 5/7 FS-Cache: Generic filesystem caching facility

In reply to David Howells

The attached patch adds a generic intermediary (FS-Cache) by which filesystems may call on local caching capabilities, and by which local caching backends may make caches available:

	+---------+
	|         |                        +--------------+
	|   NFS   |--+                     |              |
	|         |  |                 +-->|   CacheFS    |
	+---------+  |   +----------+  |   |  /dev/hda5   |
	             |   |          |  |   +--------------+
	+---------+  +-->|          |  |
	|         |      |          |--+
	|   AFS   |----->| FS-Cache |
	|         |      |          |--+
	+---------+  +-->|          |  |
	             |   |          |  |   +--------------+
	+---------+  |   +----------+  |   |              |
	|         |  |                 +-->|  CacheFiles  |
	|  ISOFS  |--+                     |  /var/cache  |
	|         |                        +--------------+
	+---------+

The patch also documents the netfs interface and the cache backend interface provided by the facility.

There are a number of reasons why I'm not using i_mapping to do this. These have been discussed a lot on the LKML and CacheFS mailing lists, but to summarise the basics:

(1) Most filesystems don't do hole reportage. Holes in files are treated as blocks of zeros and can't be distinguished otherwise, making it difficult to distinguish blocks that have been read from the network and cached from those that haven't.

(2) The backing inode must be fully populated before being exposed to userspace through the main inode because the VM/VFS goes directly to the backing inode and does not interrogate the front inode on VM ops.

Therefore:

(a) The backing inode must fit entirely within the cache.

(b) All backed files currently open must fit entirely within the cache at the same time.

(c) A working set of files in total larger than the cache may not be cached.

(d) A file may not grow larger than the available space in the cache.

(e) A file that's open and cached, and remotely grows larger than the cache is potentially stuffed.

(3) Writes go to the backing filesystem, and can only be transferred to the network when the file is closed.

(4) There's no record of what changes have been made, so the whole file must be written back.

(5) The pages belong to the backing filesystem, and all metadata associated with that page are relevant only to the backing filesystem, and not anything stacked atop it.

The attached patch adds a generic core to which both networking filesystems and caches may bind. It transfers requests from networking filesystems to appropriate caches if possible, or else gracefully denies them.

If this facility is disabled in the kernel configuration, then all its operations will be trivially reducible to nothing by the compiler.

FS-Cache provides the following facilities:

(1) Caches can be added / removed at any time, even whilst in use.

(2) Adds a facility by which tags can be used to refer to caches, even if they're not mounted yet.

(3) More than one cache can be used at once. Caches can be selected explicitly by use of tags.

(4) The netfs is provided with an interface that allows either party to withdraw caching facilities from a file (required for (1)).

(5) A netfs may annotate cache objects that belongs to it.

(6) Cache objects can be pinned and reservations made.

(7) The interface to the netfs returns as few errors as possible, preferring rather to let the netfs remain oblivious.

(8) Cookies are used to represent indices, files and other objects to the netfs. The simplest cookie is just a NULL pointer - indicating nothing cached there.

(9) The netfs is allowed to propose - dynamically - any index hierarchy it desires, though it must be aware that the index search function is recursive, stack space is limited, and indices can only be children of indices.

(10) Indices can be used to group files together to reduce key size and to make group invalidation easier. The use of indices may make lookup quicker, but that's cache dependent.

(11) Data I/O is effectively done directly to and from the netfs's pages. The netfs indicates that page A is at index B of the data-file represented by cookie C, and that it should be read or written. The cache backend may or may not start I/O on that page, but if it does, a netfs callback will be invoked to indicate completion. The I/O may be either synchronous or asynchronous.

(12) Cookies can be "retired" upon release. At this point FS-Cache will mark them as obsolete and the index hierarchy rooted at that point will get recycled.

(13) The netfs provides a "match" function for index searches. In addition to saying whether a match was made or not, this can also specify that an entry should be updated or deleted.

FS-Cache maintains a virtual indexing tree in which all indices, files, objects and pages are kept. Bits of this tree may actually reside in one or more caches.

FSDEF
|
+------------------------------------+
| |
NFS AFS
| |
+--------------------------+ +-----------+
| | | |
homedir mirror afs.org redhat.com
| | |
+------------+ +---------------+ +----------+
| | | | | |
00001 00002 00007 00125 vol00001 vol00002

     |            |           |               |                         |
 +---+---+     +-----+      +---+      +------+------+            +-----+----+
 |   |   |     |     |      |   |      |      |      |            |     |    |
PG0 PG1 PG2   PG0  XATTR   PG0 PG1   DIRENT DIRENT DIRENT        R/W   R/O  Bak
                     |                                            |
                    PG0                                       +-------+
                                                              |       |
                                                            00001   00003
                                                              |
                                                          +---+---+
                                                          |   |   |
                                                         PG0 PG1 PG2

In the example above, you can see two netfs's being backed: NFS and AFS. These have different index hierarchies:

(*) The NFS primary index will probably contain per-server indices. Each
server index is indexed by NFS file handles to get data file objects. Each data file objects can have an array of pages, but may also have further child objects, such as extended attributes and directory entries. Extended attribute objects themselves have page-array contents.

(*) The AFS primary index contains per-cell indices. Each cell index contains
per-logical-volume indices. Each of volume index contains up to three indices for the read-write, read-only and backup mirrors of those volumes. Each of these contains vnode data file objects, each of which contains an array of pages.

The very top index is the FS-Cache master index in which individual netfs's have entries.

Any index object may reside in more than one cache, provided it only has index children. Any index with non-index object children will be assumed to only reside in one cache.

The FS-Cache overview can be found in:

Documentation/filesystems/caching/fscache.txt

The netfs API to FS-Cache can be found in:

Documentation/filesystems/caching/netfs-api.txt

The cache backend API to FS-Cache can be found in:

Documentation/filesystems/caching/backend-api.txt

---

 Documentation/filesystems/caching/backend-api.txt |  345 +++++++
 Documentation/filesystems/caching/fscache.txt     |  151 +++
 Documentation/filesystems/caching/netfs-api.txt   |  726 ++++++++++++++
 fs/Kconfig                                        |   15 
 fs/Makefile                                       |    1 
 fs/fscache/Makefile                               |   13 
 fs/fscache/cookie.c                               | 1065 +++++++++++++++++++++
 fs/fscache/fscache-int.h                          |   71 +
 fs/fscache/fsdef.c                                |  113 ++
 fs/fscache/main.c                                 |  150 +++
 fs/fscache/page.c                                 |  548 +++++++++++
 include/linux/fscache-cache.h                     |  220 ++++
 include/linux/fscache.h                           |  484 ++++++++++
 13 files changed, 3902 insertions(+), 0 deletions(-)
Documentation/filesystems/caching/backend-api.txt new file +345 -0
new file mode 100644
index 0000000..896c778
@@ -0,0 +1,345 @@
+			  ==========================
+			  FS-CACHE CACHE BACKEND API
+			  ==========================
+
+The FS-Cache system provides an API by which actual caches can be supplied to
+FS-Cache for it to then serve out to network filesystems and other interested
+parties.
+
+This API is declared in <linux/fscache-cache.h>.
+
+
+====================================
+INITIALISING AND REGISTERING A CACHE
+====================================
+
+To start off, a cache definition must be initialised and registered for each
+cache the backend wants to make available.  For instance, CacheFS does this in
+the fill_super() operation on mounting.
+
+The cache definition (struct fscache_cache) should be initialised by calling:
+
+	void fscache_init_cache(struct fscache_cache *cache,
+				struct fscache_cache_ops *ops,
+				const char *idfmt,
+				...)
+
+Where:
+
+ (*) "cache" is a pointer to the cache definition;
+
+ (*) "ops" is a pointer to the table of operations that the backend supports on
+     this cache;
+
+ (*) and a format and printf-style arguments for constructing a label for the
+     cache.
+
+
+The cache should then be registered with FS-Cache by passing a pointer to the
+previously initialised cache definition to:
+
+	int fscache_add_cache(struct fscache_cache *cache,
+			      struct fscache_object *fsdef,
+			      const char *tagname);
+
+Two extra arguments should also be supplied:
+
+ (*) "fsdef" which should point to the object representation for the FS-Cache
+     master index in this cache.  Netfs primary index entries will be created
+     here.
+
+ (*) "tagname" which, if given, should be a text string naming this cache.  If
+     this is NULL, the identifier will be used instead.  For CacheFS, the
+     identifier is set to name the underlying block device and the tag can be
+     supplied by mount.
+
+This function may return -ENOMEM if it ran out of memory or -EEXIST if the tag
+is already in use.  0 will be returned on success.
+
+
+=====================
+UNREGISTERING A CACHE
+=====================
+
+A cache can be withdrawn from the system by calling this function with a
+pointer to the cache definition:
+
+	void fscache_withdraw_cache(struct fscache_cache *cache)
+
+In CacheFS's case, this is called by put_super().
+
+
+==================
+FS-CACHE UTILITIES
+==================
+
+FS-Cache provides some utilities that a cache backend may make use of:
+
+ (*) Find the parent of an object:
+
+	struct fscache_object *
+	fscache_find_parent_object(struct fscache_object *object)
+
+     This allows a backend to find the logical parent of an index or data file
+     in the cache hierarchy.
+
+ (*) Note occurrence of an I/O error in a cache:
+
+	void fscache_io_error(struct fscache_cache *cache)
+
+     This tells FS-Cache that an I/O error occurred in the cache.  After this
+     has been called, only resource dissociation operations (object and page
+     release) will be passed from the netfs to the cache backend for the
+     specified cache.
+
+     This does not actually withdraw the cache.  That must be done separately.
+
+
+========================
+RELEVANT DATA STRUCTURES
+========================
+
+ (*) Index/Data file FS-Cache representation cookie:
+
+	struct fscache_cookie {
+		struct fscache_object_def	*def;
+		struct fscache_netfs		*netfs;
+		void				*netfs_data;
+		...
+	};
+
+     The fields that might be of use to the backend describe the object
+     definition, the netfs definition and the netfs's data for this cookie.
+     The object definition contain functions supplied by the netfs for loading
+     and matching index entries; these are required to provide some of the
+     cache operations.
+
+ (*) In-cache object representation:
+
+	struct fscache_object {
+		struct fscache_cache		*cache;
+		struct fscache_cookie		*cookie;
+		unsigned long			flags;
+	#define FSCACHE_OBJECT_RECYCLING	1
+		...
+	};
+
+     Structures of this type should be allocated by the cache backend and
+     passed to FS-Cache when requested by the appropriate cache operation.  In
+     the case of CacheFS, they're embedded in CacheFS's internal object
+     structures.
+
+     Each object contains a pointer to the cookie that represents the object it
+     is backing.  It also contains a flag that indicates whether the object is
+     being retired when put_object() is called.  This should be initialised by
+     calling fscache_object_init(object).
+
+
+================
+CACHE OPERATIONS
+================
+
+The cache backend provides FS-Cache with a table of operations that can be
+performed on the denizens of the cache.  These are held in a structure of type:
+
+	struct fscache_cache_ops
+
+ (*) Name of cache provider [mandatory]:
+
+	const char *name
+
+     This isn't strictly an operation, but should be pointed at a string naming
+     the backend.
+
+ (*) Object lookup [mandatory]:
+
+	struct fscache_object *(*lookup_object)(struct fscache_cache *cache,
+						struct fscache_object *parent,
+						struct fscache_cookie *cookie)
+
+     This method is used to look up an object in the specified cache, given a
+     pointer to the parent object and the cookie to which the object will be
+     attached.  This should instantiate that object in the cache if it can, or
+     return -ENOBUFS or -ENOMEM if it can't.
+
+ (*) Increment object refcount [mandatory]:
+
+	struct fscache_object *(*grab_object)(struct fscache_object *object)
+
+     This method is called to increment the reference count on an object.  It
+     may fail (for instance if the cache is being withdrawn) by returning NULL.
+     It should return the object pointer if successful.
+
+ (*) Lock/Unlock object [mandatory]:
+
+	void (*lock_object)(struct fscache_object *object)
+	void (*unlock_object)(struct fscache_object *object)
+
+     These methods are used to exclusively lock an object.  It must be possible
+     to schedule with the lock held, so a spinlock isn't sufficient.
+
+ (*) Pin/Unpin object [optional]:
+
+	int (*pin_object)(struct fscache_object *object)
+	void (*unpin_object)(struct fscache_object *object)
+
+     These methods are used to pin an object into the cache.  Once pinned an
+     object cannot be reclaimed to make space.  Return -ENOSPC if there's not
+     enough space in the cache to permit this.
+
+ (*) Update object [mandatory]:
+
+	int (*update_object)(struct fscache_object *object)
+
+     This is called to update the index entry for the specified object.  The
+     new information should be in object->cookie->netfs_data.  This can be
+     obtained by calling object->cookie->def->get_aux()/get_attr().
+
+ (*) Release object reference [mandatory]:
+
+	void (*put_object)(struct fscache_object *object)
+
+     This method is used to discard a reference to an object.  The object may
+     be destroyed when all the references held by FS-Cache are released.
+
+ (*) Synchronise a cache [mandatory]:
+
+	void (*sync)(struct fscache_cache *cache)
+
+     This is called to ask the backend to synchronise a cache with its backing
+     device.
+
+ (*) Dissociate a cache [mandatory]:
+
+	void (*dissociate_pages)(struct fscache_cache *cache)
+
+     This is called to ask a cache to perform any page dissociations as part of
+     cache withdrawal.
+
+ (*) Set the data size on a cache file [mandatory]:
+
+	int (*set_i_size)(struct fscache_object *object, loff_t i_size);
+
+     This is called to indicate to the cache the maximum size a file may reach.
+     The cache may use this to reserve space on the cache.  It may also return
+     -ENOBUFS to indicate that insufficient space is available to expand the
+     metadata used to track the data.  It should return 0 if successful or
+     -ENOMEM or -EIO on error.
+
+ (*) Reserve cache space for an object's data [optional]:
+
+	int (*reserve_space)(struct fscache_object *object, loff_t size);
+
+     This is called to request that cache space be reserved to hold the data
+     for an object and the metadata used to track it.  Zero size should be
+     taken as request to cancel a reservation.
+
+     This should return 0 if successful, -ENOSPC if there isn't enough space
+     available, or -ENOMEM or -EIO on other errors.
+
+     The reservation may exceed the size of the object, thus permitting future
+     expansion.  If the amount of space consumed by an object would exceed the
+     reservation, it's permitted to refuse requests to allocate pages, but not
+     required.  An object may be pruned down to its reservation size if larger
+     than that already.
+
+ (*) Request page be read from cache [mandatory]:
+
+	int (*read_or_alloc_page)(struct fscache_object *object,
+				  struct page *page,
+				  fscache_rw_complete_t end_io_func,
+				  void *end_io_data,
+				  gfp_t gfp)
+
+     This is called to attempt to read a netfs page from the cache, or to
+     reserve a backing block if not.  FS-Cache will have done as much checking
+     as it can before calling, but most of the work belongs to the backend.
+
+     If there's no page in the cache, then -ENODATA should be returned if the
+     backend managed to reserve a backing block; -ENOBUFS, -ENOMEM or -EIO if
+     it didn't.
+
+     If there is a page in the cache, then a read operation should be queued
+     and 0 returned.  When the read finishes, end_io_func() should be called
+     with the following arguments:
+
+	(*end_io_func)(object->cookie->netfs_data,
+		       page,
+		       end_io_data,
+		       error);
+
+     The mark_pages_cached() cookie operation should be called for the page if
+     any cache metadata is retained.  This will indicate to the netfs that the
+     page needs explicit uncaching.  This operation takes a pagevec, thus
+     allowing several pages to be marked at once.
+
+ (*) Request pages be read from cache [mandatory]:
+
+	int (*read_or_alloc_pages)(struct fscache_object *object,
+				   struct address_space *mapping,
+				   struct list_head *pages,
+				   unsigned *nr_pages,
+				   fscache_rw_complete_t end_io_func,
+				   void *end_io_data,
+				   gfp_t gfp)
+
+     This is like the previous operation, except it will be handed a list of
+     pages instead of one page.  Any pages on which a read operation is started
+     must be added to the page cache for the specified mapping and also to the
+     LRU.  Such pages must also be removed from the pages list and nr_pages
+     decremented per page.
+
+     If there was an error such as -ENOMEM, then that should be returned; else
+     if one or more pages couldn't be read or allocated, then -ENOBUFS should
+     be returned; else if one or more pages couldn't be read, then -ENODATA
+     should be returned.  If all the pages are dispatched then 0 should be
+     returned.
+
+ (*) Request page be allocated in the cache [mandatory]:
+
+	int (*allocate_page)(struct fscache_object *object,
+			     struct page *page,
+			     gfp_t gfp)
+
+     This is like read_or_alloc_page(), except that it shouldn't read from the
+     cache, even if there's data there that could be retrieved.  It should,
+     however, set up any internal metadata required such that write_page() can
+     write to the cache.
+
+     If there's no backing block available, then -ENOBUFS should be returned
+     (or -ENOMEM or -EIO if there were other problems).  If a block is
+     successfully allocated, then the netfs page should be marked and 0
+     returned.
+
+ (*) Request page be written to cache [mandatory]:
+
+	int (*write_page)(struct fscache_object *object,
+			  struct page *page,
+			  fscache_rw_complete_t end_io_func,
+			  void *end_io_data,
+			  gfp_t gfp)
+
+     This is called to write from a page on which there was a previously
+     successful read_or_alloc_page() call.  FS-Cache filters out pages that
+     don't have mappings.
+
+     If there's no backing block available, then -ENOBUFS should be returned
+     (or -ENOMEM or -EIO if there were other problems).
+
+     If the write operation could be queued, then 0 should be returned.  When
+     the write completes, end_io_func() should be called with the following
+     arguments:
+
+	(*end_io_func)(object->cookie->netfs_data,
+		       page,
+		       end_io_data,
+		       error);
+
+ (*) Discard retained per-page metadata [mandatory]:
+
+	void (*uncache_pages)(struct fscache_object *object,
+			      struct pagevec *pagevec)
+
+     This is called when one or more netfs pages are being evicted from the
+     pagecache.  The cache backend should tear down any internal representation
+     or tracking it maintains.
Documentation/filesystems/caching/fscache.txt new file +151 -0

151 lines not on this page · raw message

Documentation/filesystems/caching/netfs-api.txt new file +726 -0

726 lines not on this page · raw message

fs/Kconfig +15 -0
index f9b5842..66acf29 100644
@@ -508,6 +508,21 @@ config FUSE_FS
 	  If you want to develop a userspace FS, or if you want to use
 	  a filesystem based on FUSE, answer Y or M.
 
+menu "Caches"
+
+config FSCACHE
+	tristate "General filesystem cache manager"
+	depends on EXPERIMENTAL
+	help
+	  This option enables a generic filesystem caching manager that can be
+	  used by various network and other filesystems to cache data
+	  locally. Different sorts of caches can be plugged in, depending on the
+	  resources available.
+
+	  See Documentation/filesystems/caching/fscache.txt for more information.
+
+endmenu
+
 menu "CD-ROM/DVD Filesystems"
 
 config ISO9660_FS
fs/Makefile +1 -0
index 83bf478..36ee03b 100644
@@ -50,6 +50,7 @@ obj-y				+= devpts/
 obj-$(CONFIG_PROFILING)		+= dcookies.o
  
 # Do not add any filesystems before this line
+obj-$(CONFIG_FSCACHE)		+= fscache/
 obj-$(CONFIG_REISERFS_FS)	+= reiserfs/
 obj-$(CONFIG_EXT3_FS)		+= ext3/ # Before ext2 so root fs can be ext3
 obj-$(CONFIG_JBD)		+= jbd/
fs/fscache/Makefile new file +13 -0
new file mode 100644
index 0000000..4c91e22
@@ -0,0 +1,13 @@
+#
+# Makefile for general filesystem caching code
+#
+
+#CFLAGS += -finstrument-functions
+
+fscache-objs := \
+	cookie.o \
+	fsdef.o \
+	main.o \
+	page.o
+
+obj-$(CONFIG_FSCACHE) := fscache.o
fs/fscache/cookie.c new file +1065 -0

1,065 lines not on this page · raw message

fs/fscache/fscache-int.h new file +71 -0

71 lines not on this page · raw message

fs/fscache/fsdef.c new file +113 -0

113 lines not on this page · raw message

fs/fscache/main.c new file +150 -0

150 lines not on this page · raw message

fs/fscache/page.c new file +548 -0

338 lines not on this page · raw message

include/linux/fscache-cache.h new file +220 -0

220 lines not on this page · raw message

include/linux/fscache.h new file +484 -0

484 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#19 lore raw

Re: [PATCH 5/7] FS-Cache: Generic filesystem caching facility

Quoted 28 lines · David Howells

...

fs/fscache/cookie.c new file +22 -0
@@ -0,0 +1,1065 @@
+/* cookie.c: general filesystem cache cookie management
+ *
+ * Copyright (C) 2004-5 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include "fscache-int.h"
+
+static LIST_HEAD(fscache_cache_tag_list);
+static LIST_HEAD(fscache_cache_list);
+static LIST_HEAD(fscache_netfs_list);
+static DECLARE_RWSEM(fscache_addremove_sem);
+static struct fscache_cache_tag fscache_nomem_tag;
+
+kmem_cache_t *fscache_cookie_jar;
+

fs/fscache/cookie.c line 32

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+static void fscache_withdraw_object(struct fscache_cache *cache,
+				    struct fscache_object *object);
+
+static void __fscache_cookie_put(struct fscache_cookie *cookie);
+
+static inline void fscache_cookie_put(struct fscache_cookie *cookie)
+{
+#ifdef CONFIG_DEBUG_SLAB
+	BUG_ON((atomic_read(&cookie->usage) & 0xffff0000) == 0x6b6b0000);
+#endif

Interesting. Perhaps a comment in there?

Quoted 16 lines
fs/fscache/cookie.c new file +16 -0
+/*****************************************************************************/
+/*
+ * look up a cache tag
+ */
+struct fscache_cache_tag *__fscache_lookup_cache_tag(const char *name)
+{
+	struct fscache_cache_tag *tag, *xtag;
+
+	/* firstly check for the existence of the tag under read lock */
+	down_read(&fscache_addremove_sem);
+
+	list_for_each_entry(tag, &fscache_cache_tag_list, link) {
+		if (strcmp(tag->name, name) == 0) {
+			atomic_inc(&tag->usage);
+			up_read(&fscache_addremove_sem);
+			return tag;

fs/fscache/cookie.c line 66

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+		}
+	}
+
+	up_read(&fscache_addremove_sem);
+
+	/* the tag does not exist - create a candidate */
+	xtag = kmalloc(sizeof(*tag) + strlen(name) + 1, GFP_KERNEL);
+	if (!xtag) {
+		/* return a dummy tag if out of memory */
+		up_read(&fscache_addremove_sem);

We already did an up_read().

Quoted 14 lines
fs/fscache/cookie.c new file +14 -0
+/*****************************************************************************/
+/*
+ * register a network filesystem for caching
+ */
+int __fscache_register_netfs(struct fscache_netfs *netfs)
+{
+	struct fscache_netfs *ptr;
+	int ret;
+
+	_enter("{%s}", netfs->name);
+
+	INIT_LIST_HEAD(&netfs->link);
+
+	/* allocate a cookie for the primary index */

fs/fscache/cookie.c line 135

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+	netfs->primary_index =
+		kmem_cache_alloc(fscache_cookie_jar, SLAB_KERNEL);
+
+	if (!netfs->primary_index) {
+		_leave(" = -ENOMEM");
+		return -ENOMEM;
+	}
+
+	/* initialise the primary index cookie */
+	memset(netfs->primary_index, 0, sizeof(*netfs->primary_index));

We have kmem_cache_zalloc() now.

fs/fscache/cookie.c line 179

Quoted 1 line
fs/fscache/cookie.c new file +1 -0
+EXPORT_SYMBOL(__fscache_register_netfs);

This code exports a huge number of symbols. Should they be non-GPL?

Quoted 41 lines
fs/fscache/cookie.c new file +41 -0
+void fscache_withdraw_cache(struct fscache_cache *cache)
+{
+	struct fscache_object *object;
+
+	_enter("");
+
+	printk(KERN_NOTICE
+	       "FS-Cache: Withdrawing cache \"%s\"\n",
+	       cache->tag->name);
+
+	/* make the cache unavailable for cookie acquisition */
+	down_write(&cache->withdrawal_sem);
+
+	down_write(&fscache_addremove_sem);
+	list_del_init(&cache->link);
+	cache->tag->cache = NULL;
+	up_write(&fscache_addremove_sem);
+
+	/* mark all objects as being withdrawn */
+	spin_lock(&cache->object_list_lock);
+	list_for_each_entry(object, &cache->object_list, cache_link) {
+		set_bit(FSCACHE_OBJECT_WITHDRAWN, &object->flags);
+	}
+	spin_unlock(&cache->object_list_lock);
+
+	/* make sure all pages pinned by operations on behalf of the netfs are
+	 * written to disc */
+	cache->ops->sync_cache(cache);
+
+	/* dissociate all the netfs pages backed by this cache from the block
+	 * mappings in the cache */
+	cache->ops->dissociate_pages(cache);
+
+	/* we now have to destroy all the active objects pertaining to this
+	 * cache */
+	spin_lock(&cache->object_list_lock);
+
+	while (!list_empty(&cache->object_list)) {
+		object = list_entry(cache->object_list.next,
+				    struct fscache_object, cache_link);
+		list_del_init(&object->cache_link);

fs/fscache/cookie.c line 397

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+		spin_unlock(&cache->object_list_lock);
+
+		_debug("withdraw %p", object->cookie);
+
+		/* we've extracted an active object from the tree - now dispose
+		 * of it */
+		fscache_withdraw_object(cache, object);
+
+		spin_lock(&cache->object_list_lock);
+	}

This looks livelockable.

Quoted 23 lines
fs/fscache/cookie.c new file +20 -0
+	spin_unlock(&cache->object_list_lock);
+
+	fscache_release_cache_tag(cache->tag);
+	cache->tag = NULL;
+
+	_leave("");
+
+} /* end fscache_withdraw_cache() */
+
 
...
 
+static struct fscache_object *fscache_lookup_object(struct fscache_cookie *cookie,
+						    struct fscache_cache *cache)
+{
+	struct fscache_cookie *parent = cookie->parent;
+	struct fscache_object *pobject, *object;
+	struct hlist_node *_p;
+
+	_enter("{%s/%s},",
+	       parent && parent->def ? parent->def->name : "",
+	       cookie->def ? (char *) cookie->def->name : "<file>");
+

fs/fscache/cookie.c line 582

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+	if (test_bit(FSCACHE_IOERROR, &cache->flags))
+		return NULL;
+
+	/* see if we have the backing object for this cookie + cache immediately
+	 * to hand
+	 */
+	object = NULL;
+	hlist_for_each_entry(object, _p,
+			     &cookie->backing_objects, cookie_link
+			     ) {

That's really weird-looking. How about

	hlist_for_each_entry(object, _p, &cookie->backing_objects, cookie_link){

fs/fscache/cookie.c line 607

Quoted 6 lines
fs/fscache/cookie.c new file +6 -0
+	hlist_for_each_entry(pobject, _p,
+			     &parent->backing_objects, cookie_link
+			     ) {
+		if (pobject->cache == cache)
+			break;
+	}

Ditto.

Quoted 39 lines
fs/fscache/cookie.c new file +39 -0
+	if (!pobject) {
+		/* we don't know about the parent object */
+		up_read(&parent->sem);
+		down_write(&parent->sem);
+
+		pobject = fscache_lookup_object(parent, cache);
+		if (IS_ERR(pobject)) {
+			up_write(&parent->sem);
+			_leave(" = %ld [no ipobj]", PTR_ERR(pobject));
+			return pobject;
+		}
+
+		_debug("pobject=%p", pobject);
+
+		BUG_ON(pobject->cookie != parent);
+
+		downgrade_write(&parent->sem);
+	}
+
+	/* now we can attempt to look up this object in the parent, possibly
+	 * creating a representation on disc when we do so
+	 */
+	object = cache->ops->lookup_object(cache, pobject, cookie);
+	up_read(&parent->sem);
+
+	if (IS_ERR(object)) {
+		_leave(" = %ld [no obj]", PTR_ERR(object));
+		return object;
+	}
+
+	/* keep track of it */
+	cache->ops->lock_object(object);
+
+	BUG_ON(!hlist_unhashed(&object->cookie_link));
+
+	/* attach to the cache's object list */
+	if (list_empty(&object->cache_link)) {
+		spin_lock(&cache->object_list_lock);
+		list_add(&object->cache_link, &cache->object_list);

fs/fscache/cookie.c line 657

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+		spin_unlock(&cache->object_list_lock);
+	}
+
+	/* attach to the cookie */
+	object->cookie = cookie;
+	atomic_inc(&cookie->usage);
+	hlist_add_head(&object->cookie_link, &cookie->backing_objects);
+
+	/* done */
+	cache->ops->unlock_object(object);

I assume ->lock_object() provides the locking for the hlist_add_head()?

Quoted 16 lines
fs/fscache/cookie.c new file +13 -0
+	_leave(" = %p [new]", object);
+	return object;
+
+} /* end fscache_lookup_object() */
 
...
 
+/*
+ * update the index entries backing a cookie
+ */
+void __fscache_update_cookie(struct fscache_cookie *cookie)
+{
+	struct fscache_object *object;
+	struct hlist_node *_p;
+
+	if (cookie == FSCACHE_NEGATIVE_COOKIE) {

fs/fscache/cookie.c line 886

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+		_leave(" [no cookie]");
+		return;
+	}
+
+	_enter("{%s}", cookie->def->name);
+
+	BUG_ON(!cookie->def->get_aux);
+
+	down_write(&cookie->sem);
+	down_read(&cookie->parent->sem);

That's a surprising locking order. Normally things are parent-first.

Quoted 43 lines
fs/fscache/cookie.c new file +40 -0
+	/* update the index entry on disc in each cache backing this cookie */
+	hlist_for_each_entry(object, _p,
+			     &cookie->backing_objects, cookie_link
+			     ) {
+		if (!test_bit(FSCACHE_IOERROR, &object->cache->flags))
+			object->cache->ops->update_object(object);
+	}
+
+	up_read(&cookie->parent->sem);
+	up_write(&cookie->sem);
+	_leave("");
+
+} /* end __fscache_update_cookie() */
+
 
...
 
+int __fscache_pin_cookie(struct fscache_cookie *cookie)
+{
+	struct fscache_object *object;
+	int ret;
+
+	_enter("%p", cookie);
+
+	if (hlist_empty(&cookie->backing_objects)) {
+		_leave(" = -ENOBUFS");
+		return -ENOBUFS;
+	}
+
+	/* not supposed to use this for indexes */
+	BUG_ON(cookie->def->type == FSCACHE_COOKIE_TYPE_INDEX);
+
+	/* prevent the file from being uncached whilst we access it and exclude
+	 * read and write attempts on pages
+	 */
+	down_write(&cookie->sem);
+
+	ret = -ENOBUFS;
+	if (!hlist_empty(&cookie->backing_objects)) {
+		/* get and pin the backing object */
+		object = hlist_entry(cookie->backing_objects.first,
+				     struct fscache_object, cookie_link);
+

fs/fscache/cookie.c line 989

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+		if (test_bit(FSCACHE_IOERROR, &object->cache->flags))
+			goto out;
+
+		if (!object->cache->ops->pin_object) {
+			ret = -EOPNOTSUPP;
+			goto out;
+		}
+
+		/* prevent the cache from being withdrawn */
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {

trylock is often a sign that something is mucked up. A comment describing why it is needed is always appropriate.

Quoted 48 lines
fs/fscache/cookie.c new file +48 -0
+			if (object->cache->ops->grab_object(object)) {
+				/* ask the cache to honour the operation */
+				ret = object->cache->ops->pin_object(object);
+
+				object->cache->ops->put_object(object);
+			}
+
+			up_read(&object->cache->withdrawal_sem);
+		}
+	}
+
+out:
+	up_write(&cookie->sem);
+	_leave(" = %d", ret);
+	return ret;
+
+} /* end __fscache_pin_cookie() */
+
+EXPORT_SYMBOL(__fscache_pin_cookie);
+
+/*****************************************************************************/
+/*
+ * unpin an object into the cache
+ */
+void __fscache_unpin_cookie(struct fscache_cookie *cookie)
+{
+	struct fscache_object *object;
+	int ret;
+
+	_enter("%p", cookie);
+
+	if (hlist_empty(&cookie->backing_objects)) {
+		_leave(" [no obj]");
+		return;
+	}
+
+	/* not supposed to use this for indexes */
+	BUG_ON(cookie->def->type == FSCACHE_COOKIE_TYPE_INDEX);
+
+	/* prevent the file from being uncached whilst we access it and exclude
+	 * read and write attempts on pages
+	 */
+	down_write(&cookie->sem);
+
+	ret = -ENOBUFS;
+	if (!hlist_empty(&cookie->backing_objects)) {
+		/* get and unpin the backing object */
+		object = hlist_entry(cookie->backing_objects.first,

fs/fscache/cookie.c line 1047

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+				     struct fscache_object, cookie_link);
+
+		if (test_bit(FSCACHE_IOERROR, &object->cache->flags))
+			goto out;
+
+		if (!object->cache->ops->unpin_object)
+			goto out;
+
+		/* prevent the cache from being withdrawn */
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {

Ditto.

Quoted 24 lines
fs/fscache/cookie.c new file +21 -0
+			if (object->cache->ops->grab_object(object)) {
+				/* ask the cache to honour the operation */
+				object->cache->ops->unpin_object(object);
+
+				object->cache->ops->put_object(object);
+			}
+
+			up_read(&object->cache->withdrawal_sem);
+		}
+	}
+
+out:
+	up_write(&cookie->sem);
+	_leave("");
+
+} /* end __fscache_unpin_cookie() */
+
 
...
 
+/*
+ * debug tracing
+ */
+#define dbgprintk(FMT,...) \

fs/fscache/cookie.c

Quoted 10 lines
fs/fscache/cookie.c new file +10 -0
+	printk("[%-6.6s] "FMT"\n",current->comm ,##__VA_ARGS__)
+#define _dbprintk(FMT,...) do { } while(0)
+
+#define kenter(FMT,...)	dbgprintk("==> %s("FMT")",__FUNCTION__ ,##__VA_ARGS__)
+#define kleave(FMT,...)	dbgprintk("<== %s()"FMT"",__FUNCTION__ ,##__VA_ARGS__)
+#define kdebug(FMT,...)	dbgprintk(FMT ,##__VA_ARGS__)
+
+#define kjournal(FMT,...) _dbprintk(FMT ,##__VA_ARGS__)
+
+#define dbgfree(ADDR)  _dbprintk("%p:%d: FREEING %p",__FILE__,__LINE__,ADDR)

That's your fourth implementation of kenter(). Maybe we need <linux/dhowells.h>?

Quoted 12 lines
fs/fscache/fsdef.c new file +9 -0
@@ -0,0 +1,113 @@
+/* fsdef.c: filesystem index definition
+ *
+ * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.

fs/fscache/fsdef.c line 19

Quoted 10 lines
fs/fscache/fsdef.c new file +10 -0
+ */
+
+#include <linux/module.h>
+#include "fscache-int.h"
+
+static uint16_t fscache_fsdef_netfs_get_key(const void *cookie_netfs_data,
+					    void *buffer, uint16_t bufmax);
+
+static uint16_t fscache_fsdef_netfs_get_aux(const void *cookie_netfs_data,
+					    void *buffer, uint16_t bufmax);

u16 would be preferred.

Quoted 14 lines

new file mode 100644
index 0000000..613c3b1

fs/fscache/main.c new file +9 -0
@@ -0,0 +1,150 @@
+/* main.c: general filesystem caching manager
+ *
+ * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.

fs/fscache/main.c line 19

Quoted 10 lines
fs/fscache/main.c new file +10 -0
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/sched.h>
+#include <linux/completion.h>
+#include <linux/slab.h>
+#include "fscache-int.h"
+
+int fscache_debug = 0;

unneeded initialisation

fs/fscache/main.c line 56

Quoted 5 lines
fs/fscache/main.c new file +5 -0
+/*****************************************************************************/
+/*
+ * initialise the fs caching module
+ */
+static int fscache_init(void)

__init?

fs/fscache/main.c line 105

Quoted 7 lines
fs/fscache/main.c new file +7 -0
+/*****************************************************************************/
+/*
+ * release the ktype
+ */
+static void fscache_ktype_release(struct kobject *kobject)
+{
+} /* end fscache_ktype_release() */

Don't empty kobject release() functions upset Greg?

Quoted 29 lines
fs/fscache/main.c new file +29 -0
+#if 0
+void __cyg_profile_func_enter (void *this_fn, void *call_site)
+__attribute__((no_instrument_function));
+
+void __cyg_profile_func_enter (void *this_fn, void *call_site)
+{
+       asm volatile("  movl    %%esp,%%edi     \n"
+                    "  andl    %0,%%edi        \n"
+                    "  addl    %1,%%edi        \n"
+                    "  movl    %%esp,%%ecx     \n"
+                    "  subl    %%edi,%%ecx     \n"
+                    "  shrl    $2,%%ecx        \n"
+                    "  movl    $0xedededed,%%eax     \n"
+                    "  rep stosl               \n"
+                    :
+                    : "i"(~(THREAD_SIZE-1)), "i"(sizeof(struct thread_info))
+                    : "eax", "ecx", "edi", "memory", "cc"
+                    );
+}
+
+void __cyg_profile_func_exit(void *this_fn, void *call_site)
+__attribute__((no_instrument_function));
+
+void __cyg_profile_func_exit(void *this_fn, void *call_site)
+{
+       asm volatile("  movl    %%esp,%%edi     \n"
+                    "  andl    %0,%%edi        \n"
+                    "  addl    %1,%%edi        \n"
+                    "  movl    %%esp,%%ecx     \n"

fs/fscache/main.c line 150

Quoted 10 lines
fs/fscache/main.c new file +10 -0
+                    "  subl    %%edi,%%ecx     \n"
+                    "  shrl    $2,%%ecx        \n"
+                    "  movl    $0xdadadada,%%eax     \n"
+                    "  rep stosl               \n"
+                    :
+                    : "i"(~(THREAD_SIZE-1)), "i"(sizeof(struct thread_info))
+                    : "eax", "ecx", "edi", "memory", "cc"
+                    );
+}
+#endif

Removeable?

Quoted 49 lines
fs/fscache/page.c new file +43 -0
new file mode 100644
index 0000000..b197be7
@@ -0,0 +1,548 @@
+/* page.c: general filesystem cache cookie management
+ *
+ * Copyright (C) 2004-5 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/fscache-cache.h>
+#include <linux/buffer_head.h>
+#include <linux/pagevec.h>
+#include "fscache-int.h"
+
+/*****************************************************************************/
+/*
+ * set the data file size on an object in the cache
+ */
+int __fscache_set_i_size(struct fscache_cookie *cookie, loff_t i_size)
+{
+	struct fscache_object *object;
+	int ret;
+
+	_enter("%p,%llu,", cookie, i_size);
+
+	if (hlist_empty(&cookie->backing_objects)) {
+		_leave(" = -ENOBUFS");
+		return -ENOBUFS;
+	}
+
+	/* not supposed to use this for indexes */
+	BUG_ON(cookie->def->type == FSCACHE_COOKIE_TYPE_INDEX);
+
+	/* prevent the file from being uncached whilst we access it and exclude
+	 * read and write attempts on pages
+	 */
+	down_write(&cookie->sem);
+
+	ret = -ENOBUFS;
+	if (!hlist_empty(&cookie->backing_objects)) {

fs/fscache/page.c line 53

Quoted 10 lines
fs/fscache/page.c new file +10 -0
+		/* get and pin the backing object */
+		object = hlist_entry(cookie->backing_objects.first,
+				     struct fscache_object, cookie_link);
+
+		if (test_bit(FSCACHE_IOERROR, &object->cache->flags))
+			goto out;
+
+		/* prevent the cache from being withdrawn */
+		if (object->cache->ops->set_i_size &&
+		    down_read_trylock(&object->cache->withdrawal_sem)

another trylock.

Quoted 52 lines
fs/fscache/page.c new file +52 -0
+		    ) {
+			if (object->cache->ops->grab_object(object)) {
+				/* ask the cache to honour the operation */
+				ret = object->cache->ops->set_i_size(object,
+								     i_size);
+
+				object->cache->ops->put_object(object);
+			}
+
+			up_read(&object->cache->withdrawal_sem);
+		}
+	}
+
+out:
+	up_write(&cookie->sem);
+	_leave(" = %d", ret);
+	return ret;
+
+} /* end __fscache_set_i_size() */
+
+EXPORT_SYMBOL(__fscache_set_i_size);
+
+/*****************************************************************************/
+/*
+ * reserve space for an object
+ */
+int __fscache_reserve_space(struct fscache_cookie *cookie, loff_t size)
+{
+	struct fscache_object *object;
+	int ret;
+
+	_enter("%p,%llu,", cookie, size);
+
+	if (hlist_empty(&cookie->backing_objects)) {
+		_leave(" = -ENOBUFS");
+		return -ENOBUFS;
+	}
+
+	/* not supposed to use this for indexes */
+	BUG_ON(cookie->def->type == FSCACHE_COOKIE_TYPE_INDEX);
+
+	/* prevent the file from being uncached whilst we access it and exclude
+	 * read and write attempts on pages
+	 */
+	down_write(&cookie->sem);
+
+	ret = -ENOBUFS;
+	if (!hlist_empty(&cookie->backing_objects)) {
+		/* get and pin the backing object */
+		object = hlist_entry(cookie->backing_objects.first,
+				     struct fscache_object, cookie_link);
+

fs/fscache/page.c line 115

Quoted 10 lines
fs/fscache/page.c new file +10 -0
+		if (test_bit(FSCACHE_IOERROR, &object->cache->flags))
+			goto out;
+
+		if (!object->cache->ops->reserve_space) {
+			ret = -EOPNOTSUPP;
+			goto out;
+		}
+
+		/* prevent the cache from being withdrawn */
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {

and another.

Quoted 19 lines
fs/fscache/page.c new file +19 -0
+			if (object->cache->ops->grab_object(object)) {
+				/* ask the cache to honour the operation */
+				ret = object->cache->ops->reserve_space(object,
+									size);
+
+				object->cache->ops->put_object(object);
+			}
+
+			up_read(&object->cache->withdrawal_sem);
+		}
+	}
+
+out:
+	up_write(&cookie->sem);
+	_leave(" = %d", ret);
+	return ret;
+
+} /* end __fscache_reserve_space() */
+

fs/fscache/page.c

Quoted 10 lines
fs/fscache/page.c new file +7 -0
 
...
 
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {

more.

Quoted 10 lines
fs/fscache/page.c new file +10 -0
+
+/* find the parent index object for a object */
+static inline
+struct fscache_object *fscache_find_parent_object(struct fscache_object *object)
+{
+	struct fscache_object *parent;
+	struct fscache_cookie *cookie = object->cookie;
+	struct fscache_cache *cache = object->cache;
+	struct hlist_node *_p;
+

fs/fscache/page.c

Quoted 10 lines
fs/fscache/page.c new file +10 -0
+	hlist_for_each_entry(parent, _p,
+			     &cookie->parent->backing_objects,
+			     cookie_link
+			     ) {
+		if (parent->cache == cache)
+			return parent;
+	}
+
+	return NULL;
+}

Large?

Quoted 20 lines
fs/fscache/page.c new file +1 -0
+#endif /* _LINUX_FSCACHE_CACHE_H */
include/linux/fscache.h new file +13 -0
new file mode 100644
index 0000000..8aa464b
@@ -0,0 +1,484 @@
+/* fscache.h: general filesystem caching interface
+ *
+ * Copyright (C) 2004-5 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _LINUX_FSCACHE_H
+#define _LINUX_FSCACHE_H

include/linux/fscache.h line 23

Quoted 10 lines
include/linux/fscache.h new file +10 -0
+
+#include <linux/config.h>
+#include <linux/fs.h>
+#include <linux/list.h>
+#include <linux/pagemap.h>
+#include <linux/pagevec.h>
+
+#ifdef CONFIG_FSCACHE_MODULE
+#define CONFIG_FSCACHE
+#endif

Defining symbols which are owned by the Kconfig system isn't very nice.

Quoted 7 lines
include/linux/fscache.h new file +7 -0
+
+/*
+ * request a page be stored in the cache
+ * - this request may be ignored if no cache block is currently allocated, in
+ *   which case it:
+ *   - returns -ENOBUFS
+ * - if a cache block was already allocated:

include/linux/fscache.h line 414

Quoted 10 lines
include/linux/fscache.h new file +10 -0
+ *   - a BIO will be dispatched to write the page (end_io_func will be called
+ *     from the completion function)
+ *   - returns 0
+ */
+#ifdef CONFIG_FSCACHE
+extern int __fscache_write_page(struct fscache_cookie *cookie,
+				struct page *page,
+				fscache_rw_complete_t end_io_func,
+				void *end_io_data,
+				gfp_t gfp);

hm. I tend to find it better for functions to be documented at the definition site, not at the declaration site. I don't expect people even think to go looking in a header file to learn about the function which they're presently looking at.

author maintainer
#20 lore raw

Re: [PATCH 5/7] FS-Cache: Generic filesystem caching facility

Quoted 1 line · Andrew Morton

Interesting. Perhaps a comment in there?

Yep. It might even be worth making the actual check in linux/slab.h.

Quoted 1 line

We already did an up_read().

Yep. Also the initialisation was done on the wrong variable (tag not xtag)... which only worked because list_for_each_entry() always leaves a valid pointer in tag.

Quoted 1 line

We have kmem_cache_zalloc() now.

It's not in the NFS tree I've just pulled. Of course, git may be being a complete git again. I blame the original author:-)

Quoted 1 line

This looks livelockable.

How so? The bit at the top of the function makes sure that we can't get any new objects.

Quoted 3 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+	cache->ops->unlock_object(object);

I assume ->lock_object() provides the locking for the hlist_add_head()?

Half of it... Cookie lookup and withdrawal has to deal with the interaction between two locks:

(1) The cookie lock. This is taken first when approached from the netfs side.

(2) The object lock. This is taken first when approached from the cache side.

The cookie lock is superior to the netfs lock, and fscache_withdraw_object() has to do lock reordering when withdrawing a cache from the system.

fscache_lookup_object() - to which you refer - must be called with the cookie already write-locked.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	down_write(&cookie->sem);
+	down_read(&cookie->parent->sem);

That's a surprising locking order. Normally things are parent-first.

But the operation starts from the child cookie.

The order I've defined is that if both a cookie and its parent already exist, then you must get the lock on the child before the lock on the parent if you want both.

Lookup is okay, since the new cookie doesn't exist to the rest of the system until we've finished, and should we need to walk up the index tree instantiating indices, the locking is in the right order to just do that.

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		/* prevent the cache from being withdrawn */
+		if (down_read_trylock(&object->cache->withdrawal_sem)) {

trylock is often a sign that something is mucked up. A comment describing why it is needed is always appropriate.

In this case, it's not something mucked up. The only thing that write-locks cache->withdrawal_sem is the fscache_withdraw_cache(), and that then prevents any further operations on that cache succeeding.

Obviously, before we withdraw the cache, down_read_trylock() will always succeed, and after we withdraw the cache it will always fail; but not only that, _whilst_ there's an operation in progress, the withdrawal will be made to wait because that does down_write() *not* down_write_trylock().

I wonder if I should write some docs on the locking procedures used by FS-Cache.

Quoted 2 lines

That's your fourth implementation of kenter(). Maybe we need <linux/dhowells.h>?

:-)

Maybe I should move my debugging macros into include/linux, but then everyone else would complain if their own versions weren't put in there, or would complain if they were forced to use mine.

Quoted 1 line

u16 would be preferred.

As I recall, Linus said something like that uint16_t and co are fine in one's own interfaces. I did actually ask him about that.

u16 should die really, IMO.

Quoted 3 lines
Quoted 1 line

+int fscache_debug = 0;

unneeded initialisation

Yep.

Quoted 3 lines
Quoted 1 line

+static int fscache_init(void)

__init?

Yep.

Quoted 1 line

Don't empty kobject release() functions upset Greg?

I hope so... then he might improve his docs:-)

The function would seem to be mandatory, but there really isn't anything for it to do, since the object is static:-/

Quoted 1 line

Removeable?

Yeah. It's handy to have around for debugging, but it's very arch specific, and can be added back later easily enough.

Quoted 1 line

Large?

Depends how you define "large".

It doesn't actually produce very much code.

Quoted 1 line

Defining symbols which are owned by the Kconfig system isn't very nice.

Kconfig is still broken:

	warthog>grep -r CONFIG_FSCACHE include/linux/autoconf.h 
	#define CONFIG_FSCACHE_MODULE 1
	warthog>

Modules that might depend on fscache need to know that it's there, and having to double up every #if to detect both is stupid.

Would you suggest then:

	#if defined(CONFIG_FSCACHE) || defined(CONFIG_FSCACHE_MODULE)
	#define FSCACHE_AVAILABLE 1
	#endif
Quoted 4 lines

hm. I tend to find it better for functions to be documented at the definition site, not at the declaration site. I don't expect people even think to go looking in a header file to learn about the function which they're presently looking at.

Two points:

(1) The function they'll be looking for isn't in a .c file.

(2) They *should* be looking in Documentation/ where all the kernel's interface functions are of course documented. They will find the documentation for the FS-Cache interfaces there. I'll add obvious banners to the tops of the header files telling them where to look. I can even add them to the .c files, I suppose.

Now I admit that I haven't added docs for get_empty_filp(), dentry_open_kernel() and suchlike because there's nowhere to actually add them yet.

David

#21 lore raw

Re: [PATCH 5/7] FS-Cache: Generic filesystem caching facility

Quoted 9 lines · David Howells
Quoted 3 lines

That's your fourth implementation of kenter(). Maybe we need <linux/dhowells.h>?

:-)

Maybe I should move my debugging macros into include/linux, but then everyone else would complain if their own versions weren't put in there, or would complain if they were forced to use mine.

The number of home-made debugging macro implementations we have is quite demented. Developing (and maintaining) a common set would be a good idea, IMO.

Quoted 11 lines

It doesn't actually produce very much code.

Quoted 1 line

Defining symbols which are owned by the Kconfig system isn't very nice.

Kconfig is still broken:

	warthog>grep -r CONFIG_FSCACHE include/linux/autoconf.h 
	#define CONFIG_FSCACHE_MODULE 1
	warthog>

Modules that might depend on fscache need to know that it's there,

In theory, module A isn't supposed to care whether module B was configured, because module B might be compiled separately, or dowloaded from elsewhere or whatever.

Quoted 8 lines

and having
to double up every #if to detect both is stupid.

Would you suggest then:

	#if defined(CONFIG_FSCACHE) || defined(CONFIG_FSCACHE_MODULE)
	#define FSCACHE_AVAILABLE 1
	#endif

yup.

author maintainer
#22 lore raw

Re: [PATCH 5/7] FS-Cache: Generic filesystem caching facility

Quoted 5 lines · Andrew Morton
Quoted 1 line

Modules that might depend on fscache need to know that it's there,

In theory, module A isn't supposed to care whether module B was configured, because module B might be compiled separately, or dowloaded from elsewhere or whatever.

In this case it's sort of necessary - unless you're suggesting I make FS-Cache mandatory...

The problem is that I don't want NFS or whatever to be carrying around the cookie pointers if FS-Cache isn't compiled as that saves memory. But that involves conditionally changing the composition of structures, something that's most clearly done with cpp-conditionals.

David

author maintainer
#23 lore raw

PATCH 6/7 FS-Cache: Make kAFS use FS-Cache

In reply to David Howells

The attached patch makes the kAFS filesystem in fs/afs/ use FS-Cache, and through it any attached caches. The kAFS filesystem will use caching automatically if it's available.

---

 fs/Kconfig         |    7 +
 fs/afs/cache.h     |   27 ------
 fs/afs/cell.c      |  109 ++++++++++++++---------
 fs/afs/cell.h      |   16 +--
 fs/afs/cmservice.c |    2 
 fs/afs/dir.c       |   15 +--
 fs/afs/file.c      |  240 +++++++++++++++++++++++++++++++++-----------------
 fs/afs/fsclient.c  |    4 +
 fs/afs/inode.c     |   43 ++++++---
 fs/afs/internal.h  |   24 ++---
 fs/afs/main.c      |   24 ++---
 fs/afs/mntpt.c     |   12 +--
 fs/afs/proc.c      |    1 
 fs/afs/server.c    |    3 -
 fs/afs/vlocation.c |  185 ++++++++++++++++++++++++---------------
 fs/afs/vnode.c     |  249 +++++++++++++++++++++++++++++++++++++++++++---------
 fs/afs/vnode.h     |   10 +-
 fs/afs/volume.c    |   78 ++++++----------
 fs/afs/volume.h    |   28 +-----
 19 files changed, 660 insertions(+), 417 deletions(-)
fs/Kconfig +7 -0
index 66acf29..6c95e58 100644
@@ -1816,6 +1816,13 @@ config AFS_FS
 
 	  If unsure, say N.
 
+config AFS_FSCACHE
+	bool "Provide AFS client caching support"
+	depends on AFS_FS && FSCACHE && EXPERIMENTAL
+	help
+	  Say Y here if you want AFS data to be cached locally on through the
+	  generic filesystem cache manager
+
 config RXRPC
 	tristate
 
fs/afs/cache.h deleted +0 -27
deleted file mode 100644
index 9eb7722..0000000
@@ -1,27 +0,0 @@
-/* cache.h: AFS local cache management interface
- *
- * Copyright (C) 2002 Red Hat, Inc. All Rights Reserved.
- * Written by David Howells ([email protected])
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- */
-
-#ifndef _LINUX_AFS_CACHE_H
-#define _LINUX_AFS_CACHE_H
-
-#undef AFS_CACHING_SUPPORT
-
-#include <linux/mm.h>
-#ifdef AFS_CACHING_SUPPORT
-#include <linux/cachefs.h>
-#endif
-#include "types.h"
-
-#ifdef __KERNEL__
-
-#endif /* __KERNEL__ */
-
-#endif /* _LINUX_AFS_CACHE_H */
fs/afs/cell.c +65 -44

146 lines not on this page · raw message

fs/afs/cell.h +3 -13
index 4834910..d670502 100644
@@ -13,7 +13,7 @@
 #define _LINUX_AFS_CELL_H
 
 #include "types.h"
-#include "cache.h"
+#include <linux/fscache.h>
 
 #define AFS_CELL_MAX_ADDRS 15
 
@@ -21,16 +21,6 @@ extern volatile int afs_cells_being_purg
 
 /*****************************************************************************/
 /*
- * entry in the cached cell catalogue
- */
-struct afs_cache_cell
-{
-	char			name[64];	/* cell name (padded with NULs) */
-	struct in_addr		vl_servers[15];	/* cached cell VL servers */
-};
-
-/*****************************************************************************/
-/*
  * AFS cell record
  */
 struct afs_cell
@@ -39,8 +29,8 @@ struct afs_cell
 	struct list_head	link;		/* main cell list link */
 	struct list_head	proc_link;	/* /proc cell list link */
 	struct proc_dir_entry	*proc_dir;	/* /proc dir for this cell */
-#ifdef AFS_CACHING_SUPPORT
-	struct cachefs_cookie	*cache;		/* caching cookie */
+#ifdef CONFIG_AFS_FSCACHE
+	struct fscache_cookie	*cache;		/* caching cookie */
 #endif
 
 	/* server record management */
fs/afs/cmservice.c +1 -1
index 3d097fd..f87d5a7 100644
@@ -24,7 +24,7 @@
 #include "internal.h"
 
 static unsigned afscm_usage;		/* AFS cache manager usage count */
-static struct rw_semaphore afscm_sem;	/* AFS cache manager start/stop semaphore */
+static DECLARE_RWSEM(afscm_sem);	/* AFS cache manager start/stop semaphore */
 
 static int afscm_new_call(struct rxrpc_call *call);
 static void afscm_attention(struct rxrpc_call *call);
fs/afs/dir.c +7 -8
index a6dff6a..b8e7c32 100644
@@ -145,7 +145,7 @@ static inline void afs_dir_check_page(st
 	qty /= sizeof(union afs_dir_block);
 
 	/* check them */
-	dbuf = page_address(page);
+	dbuf = kmap_atomic(page, KM_USER0);
 	for (tmp = 0; tmp < qty; tmp++) {
 		if (dbuf->blocks[tmp].pagehdr.magic != AFS_DIR_MAGIC) {
 			printk("kAFS: %s(%lu): bad magic %d/%d is %04hx\n",
@@ -154,12 +154,12 @@ static inline void afs_dir_check_page(st
 			goto error;
 		}
 	}
+	kunmap_atomic(dbuf, KM_USER0);
 
-	SetPageChecked(page);
 	return;
 
  error:
-	SetPageChecked(page);
+	kunmap_atomic(dbuf, KM_USER0);
 	SetPageError(page);
 
 } /* end afs_dir_check_page() */
@@ -170,7 +170,6 @@ static inline void afs_dir_check_page(st
  */
 static inline void afs_dir_put_page(struct page *page)
 {
-	kunmap(page);
 	page_cache_release(page);
 
 } /* end afs_dir_put_page() */
@@ -190,11 +189,9 @@ static struct page *afs_dir_get_page(str
 			       NULL);
 	if (!IS_ERR(page)) {
 		wait_on_page_locked(page);
-		kmap(page);
 		if (!PageUptodate(page))
 			goto fail;
-		if (!PageChecked(page))
-			afs_dir_check_page(dir, page);
+		afs_dir_check_page(dir, page);
 		if (PageError(page))
 			goto fail;
 	}
@@ -359,7 +356,7 @@ static int afs_dir_iterate(struct inode 
 
 		limit = blkoff & ~(PAGE_SIZE - 1);
 
-		dbuf = page_address(page);
+		dbuf = kmap_atomic(page, KM_USER0);
 
 		/* deal with the individual blocks stashed on this page */
 		do {
@@ -368,6 +365,7 @@ static int afs_dir_iterate(struct inode 
 			ret = afs_dir_iterate_block(fpos, dblock, blkoff,
 						    cookie, filldir);
 			if (ret != 1) {
+				kunmap_atomic(dbuf, KM_USER0);
 				afs_dir_put_page(page);
 				goto out;
 			}
@@ -376,6 +374,7 @@ static int afs_dir_iterate(struct inode 
 
 		} while (*fpos < dir->i_size && blkoff < limit);
 
+		kunmap_atomic(dbuf, KM_USER0);
 		afs_dir_put_page(page);
 		ret = 0;
 	}
fs/afs/file.c +160 -80

376 lines not on this page · raw message

fs/afs/fsclient.c +3 -1
index 61bc371..c88c41a 100644
@@ -398,6 +398,8 @@ int afs_rxfs_fetch_file_status(struct af
 		bp++; /* spare6 */
 	}
 
+	_debug("Data Version %llx\n", vnode->status.version);
+
 	/* success */
 	ret = 0;
 
@@ -408,7 +410,7 @@ int afs_rxfs_fetch_file_status(struct af
  out_put_conn:
 	afs_server_release_callslot(server, &callslot);
  out:
-	_leave("");
+	_leave(" = %d", ret);
 	return ret;
 
  abort:
fs/afs/inode.c +30 -13

83 lines not on this page · raw message

fs/afs/internal.h +9 -15
index 72febdf..0bddcdf 100644
@@ -16,15 +16,17 @@
 #include <linux/kernel.h>
 #include <linux/fs.h>
 #include <linux/pagemap.h>
+#include <linux/fscache.h>
 
 /*
  * debug tracing
  */
-#define kenter(FMT, a...)	printk("==> %s("FMT")\n",__FUNCTION__ , ## a)
-#define kleave(FMT, a...)	printk("<== %s()"FMT"\n",__FUNCTION__ , ## a)
-#define kdebug(FMT, a...)	printk(FMT"\n" , ## a)
-#define kproto(FMT, a...)	printk("### "FMT"\n" , ## a)
-#define knet(FMT, a...)		printk(FMT"\n" , ## a)
+#define __kdbg(FMT, a...)	printk("[%05d] "FMT"\n", current->pid , ## a)
+#define kenter(FMT, a...)	__kdbg("==> %s("FMT")", __FUNCTION__ , ## a)
+#define kleave(FMT, a...)	__kdbg("<== %s()"FMT, __FUNCTION__ , ## a)
+#define kdebug(FMT, a...)	__kdbg(FMT , ## a)
+#define kproto(FMT, a...)	__kdbg("### "FMT , ## a)
+#define knet(FMT, a...)		__kdbg(FMT , ## a)
 
 #ifdef __KDEBUG
 #define _enter(FMT, a...)	kenter(FMT , ## a)
@@ -56,9 +58,6 @@ static inline void afs_discard_my_signal
  */
 extern struct rw_semaphore afs_proc_cells_sem;
 extern struct list_head afs_proc_cells;
-#ifdef AFS_CACHING_SUPPORT
-extern struct cachefs_index_def afs_cache_cell_index_def;
-#endif
 
 /*
  * dir.c
@@ -72,11 +71,6 @@ extern const struct file_operations afs_
 extern struct address_space_operations afs_fs_aops;
 extern struct inode_operations afs_file_inode_operations;
 
-#ifdef AFS_CACHING_SUPPORT
-extern int afs_cache_get_page_cookie(struct page *page,
-				     struct cachefs_page **_page_cookie);
-#endif
-
 /*
  * inode.c
  */
@@ -97,8 +91,8 @@ extern void afs_key_unregister(void);
 /*
  * main.c
  */
-#ifdef AFS_CACHING_SUPPORT
-extern struct cachefs_netfs afs_cache_netfs;
+#ifdef CONFIG_AFS_FSCACHE
+extern struct fscache_netfs afs_cache_netfs;
 #endif
 
 /*
fs/afs/main.c +11 -13
index 913c689..5840bb2 100644
@@ -1,6 +1,6 @@
 /* main.c: AFS client file system
  *
- * Copyright (C) 2002 Red Hat, Inc. All Rights Reserved.
+ * Copyright (C) 2002,5 Red Hat, Inc. All Rights Reserved.
  * Written by David Howells ([email protected])
  *
  * This program is free software; you can redistribute it and/or
@@ -14,11 +14,11 @@
 #include <linux/init.h>
 #include <linux/sched.h>
 #include <linux/completion.h>
+#include <linux/fscache.h>
 #include <rxrpc/rxrpc.h>
 #include <rxrpc/transport.h>
 #include <rxrpc/call.h>
 #include <rxrpc/peer.h>
-#include "cache.h"
 #include "cell.h"
 #include "server.h"
 #include "fsclient.h"
@@ -51,12 +51,11 @@ static struct rxrpc_peer_ops afs_peer_op
 struct list_head afs_cb_hash_tbl[AFS_CB_HASH_COUNT];
 DEFINE_SPINLOCK(afs_cb_hash_lock);
 
-#ifdef AFS_CACHING_SUPPORT
-static struct cachefs_netfs_operations afs_cache_ops = {
-	.get_page_cookie	= afs_cache_get_page_cookie,
+#ifdef CONFIG_AFS_FSCACHE
+static struct fscache_netfs_operations afs_cache_ops = {
 };
 
-struct cachefs_netfs afs_cache_netfs = {
+struct fscache_netfs afs_cache_netfs = {
 	.name			= "afs",
 	.version		= 0,
 	.ops			= &afs_cache_ops,
@@ -83,10 +82,9 @@ static int __init afs_init(void)
 	if (ret < 0)
 		return ret;
 
-#ifdef AFS_CACHING_SUPPORT
+#ifdef CONFIG_AFS_FSCACHE
 	/* we want to be able to cache */
-	ret = cachefs_register_netfs(&afs_cache_netfs,
-				     &afs_cache_cell_index_def);
+	ret = fscache_register_netfs(&afs_cache_netfs);
 	if (ret < 0)
 		goto error;
 #endif
@@ -137,8 +135,8 @@ static int __init afs_init(void)
 	afs_key_unregister();
  error_cache:
 #endif
-#ifdef AFS_CACHING_SUPPORT
-	cachefs_unregister_netfs(&afs_cache_netfs);
+#ifdef CONFIG_AFS_FSCACHE
+	fscache_unregister_netfs(&afs_cache_netfs);
  error:
 #endif
 	afs_cell_purge();
@@ -167,8 +165,8 @@ static void __exit afs_exit(void)
 #ifdef CONFIG_KEYS_TURNED_OFF
 	afs_key_unregister();
 #endif
-#ifdef AFS_CACHING_SUPPORT
-	cachefs_unregister_netfs(&afs_cache_netfs);
+#ifdef CONFIG_AFS_FSCACHE
+	fscache_unregister_netfs(&afs_cache_netfs);
 #endif
 	afs_proc_cleanup();
 
fs/afs/mntpt.c +6 -6
index 4e6eeb5..91f57cf 100644
@@ -82,7 +82,7 @@ int afs_mntpt_check_symlink(struct afs_v
 
 	ret = -EIO;
 	wait_on_page_locked(page);
-	buf = kmap(page);
+	buf = kmap_atomic(page, KM_USER0);
 	if (!PageUptodate(page))
 		goto out_free;
 	if (PageError(page))
@@ -105,7 +105,7 @@ int afs_mntpt_check_symlink(struct afs_v
 	ret = 0;
 
  out_free:
-	kunmap(page);
+	kunmap_atomic(buf, KM_USER0);
 	page_cache_release(page);
  out:
 	_leave(" = %d", ret);
@@ -195,9 +195,9 @@ static struct vfsmount *afs_mntpt_do_aut
 	if (!PageUptodate(page) || PageError(page))
 		goto error;
 
-	buf = kmap(page);
+	buf = kmap_atomic(page, KM_USER0);
 	memcpy(devname, buf, size);
-	kunmap(page);
+	kunmap_atomic(buf, KM_USER0);
 	page_cache_release(page);
 	page = NULL;
 
@@ -276,12 +276,12 @@ static void *afs_mntpt_follow_link(struc
  */
 static void afs_mntpt_expiry_timed_out(struct afs_timer *timer)
 {
-	kenter("");
+//	kenter("");
 
 	mark_mounts_for_expiry(&afs_vfsmounts);
 
 	afs_kafstimod_add_timer(&afs_mntpt_expiry_timer,
 				afs_mntpt_expiry_timeout * HZ);
 
-	kleave("");
+//	kleave("");
 } /* end afs_mntpt_expiry_timed_out() */
fs/afs/proc.c +1 -0
index 101d21b..db58488 100644
@@ -177,6 +177,7 @@ int afs_proc_init(void)
  */
 void afs_proc_cleanup(void)
 {
+	remove_proc_entry("rootcell", proc_afs);
 	remove_proc_entry("cells", proc_afs);
 
 	remove_proc_entry("fs/afs", NULL);
fs/afs/server.c +1 -2
index 62b093a..7103e10 100644
@@ -377,7 +377,6 @@ int afs_server_request_callslot(struct a
 	else if (list_empty(&server->fs_callq)) {
 		/* no one waiting */
 		server->fs_conn_cnt[nconn]++;
-		spin_unlock(&server->fs_lock);
 	}
 	else {
 		/* someone's waiting - dequeue them and wake them up */
@@ -395,9 +394,9 @@ int afs_server_request_callslot(struct a
 		}
 		pcallslot->ready = 1;
 		wake_up_process(pcallslot->task);
-		spin_unlock(&server->fs_lock);
 	}
 
+	spin_unlock(&server->fs_lock);
 	rxrpc_put_connection(callslot->conn);
 	callslot->conn = NULL;
 
fs/afs/vlocation.c +113 -72

240 lines not on this page · raw message

fs/afs/vnode.c +204 -45

287 lines not on this page · raw message

fs/afs/vnode.h +5 -5
index b86a971..3f0602d 100644
@@ -13,9 +13,9 @@
 #define _LINUX_AFS_VNODE_H
 
 #include <linux/fs.h>
+#include <linux/fscache.h>
 #include "server.h"
 #include "kafstimod.h"
-#include "cache.h"
 
 #ifdef __KERNEL__
 
@@ -32,8 +32,8 @@ struct afs_cache_vnode
 	afs_dataversion_t	data_version;	/* data version */
 };
 
-#ifdef AFS_CACHING_SUPPORT
-extern struct cachefs_index_def afs_vnode_cache_index_def;
+#ifdef CONFIG_AFS_FSCACHE
+extern struct fscache_cookie_def afs_vnode_cache_index_def;
 #endif
 
 /*****************************************************************************/
@@ -47,8 +47,8 @@ struct afs_vnode
 	struct afs_volume	*volume;	/* volume on which vnode resides */
 	struct afs_fid		fid;		/* the file identifier for this inode */
 	struct afs_file_status	status;		/* AFS status info for this file */
-#ifdef AFS_CACHING_SUPPORT
-	struct cachefs_cookie	*cache;		/* caching cookie */
+#ifdef CONFIG_AFS_FSCACHE
+	struct fscache_cookie	*cache;		/* caching cookie */
 #endif
 
 	wait_queue_head_t	update_waitq;	/* status fetch waitqueue */
fs/afs/volume.c +29 -49

116 lines not on this page · raw message

fs/afs/volume.h +5 -23
index bfdcf19..fc9895a 100644
@@ -12,11 +12,11 @@
 #ifndef _LINUX_AFS_VOLUME_H
 #define _LINUX_AFS_VOLUME_H
 
+#include <linux/fscache.h>
 #include "types.h"
 #include "fsclient.h"
 #include "kafstimod.h"
 #include "kafsasyncd.h"
-#include "cache.h"
 
 typedef enum {
 	AFS_VLUPD_SLEEP,		/* sleeping waiting for update timer to fire */
@@ -45,24 +45,6 @@ struct afs_cache_vlocation
 	time_t			rtime;		/* last retrieval time */
 };
 
-#ifdef AFS_CACHING_SUPPORT
-extern struct cachefs_index_def afs_vlocation_cache_index_def;
-#endif
-
-/*****************************************************************************/
-/*
- * volume -> vnode hash table entry
- */
-struct afs_cache_vhash
-{
-	afs_voltype_t		vtype;		/* which volume variation */
-	uint8_t			hash_bucket;	/* which hash bucket this represents */
-} __attribute__((packed));
-
-#ifdef AFS_CACHING_SUPPORT
-extern struct cachefs_index_def afs_volume_cache_index_def;
-#endif
-
 /*****************************************************************************/
 /*
  * AFS volume location record
@@ -73,8 +55,8 @@ struct afs_vlocation
 	struct list_head	link;		/* link in cell volume location list */
 	struct afs_timer	timeout;	/* decaching timer */
 	struct afs_cell		*cell;		/* cell to which volume belongs */
-#ifdef AFS_CACHING_SUPPORT
-	struct cachefs_cookie	*cache;		/* caching cookie */
+#ifdef CONFIG_AFS_FSCACHE
+	struct fscache_cookie	*cache;		/* caching cookie */
 #endif
 	struct afs_cache_vlocation vldb;	/* volume information DB record */
 	struct afs_volume	*vols[3];	/* volume access record pointer (index by type) */
@@ -109,8 +91,8 @@ struct afs_volume
 	atomic_t		usage;
 	struct afs_cell		*cell;		/* cell to which belongs (unrefd ptr) */
 	struct afs_vlocation	*vlocation;	/* volume location */
-#ifdef AFS_CACHING_SUPPORT
-	struct cachefs_cookie	*cache;		/* caching cookie */
+#ifdef CONFIG_AFS_FSCACHE
+	struct fscache_cookie	*cache;		/* caching cookie */
 #endif
 	afs_volid_t		vid;		/* volume ID */
 	afs_voltype_t		type;		/* type of volume */
author maintainer
#24 lore raw

PATCH 7/7 FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem

In reply to David Howells

Add a cache backend that permits a mounted filesystem to be used as a backing store for the cache.

CacheFiles uses a userspace daemon to do some of the cache management - such as reaping stale nodes and culling. This is called cachefilesd and lives in /sbin. The source for the daemon can be downloaded from:

http://people.redhat.com/~dhowells/cachefs/cachefilesd.c

And an example configuration from:

http://people.redhat.com/~dhowells/cachefs/cachefilesd.conf

The filesystem and data integrity of the cache are only as good as those of the filesystem providing the backing services. Note that CacheFiles does not attempt to journal anything since the journalling interfaces of the various filesystems are very specific in nature.

CacheFiles creates a proc-file - "/proc/fs/cachefiles" - that is used for communication with the daemon. Only one thing may have this open at once, and whilst it is open, a cache is at least partially in existence. The daemon opens this and sends commands down it to control the cache.

CacheFiles is currently limited to a single cache.

CacheFiles attempts to maintain at least a certain percentage of free space on the filesystem, shrinking the cache by culling the objects it contains to make space if necessary - see the "Cache Culling" section. This means it can be placed on the same medium as a live set of data, and will expand to make use of spare space and automatically contract when the set of data requires more space.

============
REQUIREMENTS
============

The use of CacheFiles and its daemon requires the following features to be available in the system and in the cache filesystem:

- dnotify.

- extended attributes (xattrs).

- openat() and friends.

- bmap() support on files in the filesystem (FIBMAP ioctl).

- The use of bmap() to detect a partial page at the end of the file.

It is strongly recommended that the "dir_index" option is enabled on Ext3 filesystems being used as a cache.

=============
CONFIGURATION
=============

The cache is configured by a script in /etc/cachefilesd.conf. These commands set up cache ready for use. The following script commands are available:

(*) brun <N>%
(*) bcull <N>%
(*) bstop <N>%

Configure the culling limits. Optional. See the section on culling The defaults are 7%, 5% and 1% respectively.

(*) dir <path>

Specify the directory containing the root of the cache. Mandatory.

(*) tag <name>

Specify a tag to FS-Cache to use in distinguishing multiple caches. Optional. The default is "CacheFiles".

(*) debug <mask>

Specify a numeric bitmask to control debugging in the kernel module. Optional. The default is zero (all off).

==================
STARTING THE CACHE
==================

The cache is started by running the daemon. The daemon opens the cache proc file, configures the cache and tells it to begin caching. At that point the cache binds to fscache and the cache becomes live.

The daemon is run as follows:

	/sbin/cachefilesd [-d]* [-s] [-n] [-f <configfile>]

The flags are:

(*) -d

Increase the debugging level. This can be specified multiple times and is cumulative with itself.

(*) -s

Send messages to stderr instead of syslog.

(*) -n

Don't daemonise and go into background.

(*) -f <configfile>

Use an alternative configuration file rather than the default one.

===============
THINGS TO AVOID
===============

Do not mount other things within the cache as this will cause problems. The kernel module contains its own very cut-down path walking facility that ignores mountpoints, but the daemon can't avoid them.

Do not create, rename or unlink files and directories in the cache whilst the cache is active, as this may cause the state to become uncertain.

Renaming files in the cache might make objects appear to be other objects (the filename is part of the lookup key).

Do not change or remove the extended attributes attached to cache files by the cache as this will cause the cache state management to get confused.

Do not create files or directories in the cache, lest the cache get confused or serve incorrect data.

Do not chmod files in the cache. The module creates things with minimal permissions to prevent random users being able to access them directly.

=============
CACHE CULLING
=============

The cache may need culling occasionally to make space. This involves discarding objects from the cache that have been used less recently than anything else. Culling is based on the access time of data objects. Empty directories are culled if not in use.

Cache culling is done on the basis of the percentage of blocks available in the underlying filesystem. There are three "limits":

(*) brun

If the amount of available space in the cache rises above this limit, then culling is turned off.

(*) bcull

If the amount of available space in the cache falls below this limit, then culling is started.

(*) bstop

If the amount of available space in the cache falls below this limit, then no further allocation of disk space is permitted until culling has raised the amount above this limit again.

These must be configured thusly:

	0 <= bstop < bcull < brun < 100

Note that these are percentages of available space, and do _not_ appear as 100 minus the percentage displayed by the "df" program.

The userspace daemon scans the cache to build up a table of cullable objects. These are then culled in least recently used order. A new scan of the cache is started as soon as space is made in the table. Objects will be skipped if their atimes have changed or if the kernel module says it is still using them.

===============
CACHE STRUCTURE
===============

The CacheFiles module will create two directories in the directory it was given:

(*) cache/

(*) graveyard/

The active cache objects all reside in the first directory. The CacheFiles kernel module moves any retired or culled objects that it can't simply unlink to the graveyard from which the daemon will actually delete them.

The daemon uses dnotify to monitor the graveyard directory, and will delete anything that appears therein.

The module represents index objects as directories with the filename "I..." or "J...". Note that the "cache/" directory is itself a special index.

Data objects are represented as files if they have no children, or directories if they do. Their filenames all begin "D..." or "E...". If represented as a directory, data objects will have a file in the directory called "data" that actually holds the data.

Special objects are similar to data objects, except their filenames begin "S..." or "T...".

If an object has children, then it will be represented as a directory. Immediately in the representative directory are a collection of directories named for hash values of the child object keys with an '@' prepended. Into this directory, if possible, will be placed the representations of the child objects:

	INDEX     INDEX      INDEX                             DATA FILES
	========= ========== ================================= ================
	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400
	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...DB1ry
	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...N22ry
	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...FP1ry

If the key is so long that it exceeds NAME_MAX with the decorations added on to it, then it will be cut into pieces, the first few of which will be used to make a nest of directories, and the last one of which will be the objects inside the last directory. The names of the intermediate directories will have '+' prepended:

	J1223/@23/+xy...z/+kl...m/Epqr

Note that keys are raw data, and not only may they exceed NAME_MAX in size, they may also contain things like '/' and NUL characters, and so they may not be suitable for turning directly into a filename.

To handle this, CacheFiles will use a suitably printable filename directly and "base-64" encode ones that aren't directly suitable. The two versions of object filenames indicate the encoding:

	OBJECT TYPE	PRINTABLE	ENCODED
	===============	===============	===============
	Index		"I..."		"J..."
	Data		"D..."		"E..."
	Special		"S..."		"T..."

Intermediate directories are always "@" or "+" as appropriate.

Each object in the cache has an extended attribute label that holds the object type ID (required to distinguish special objects) and the auxiliary data from the netfs. The latter is used to detect stale objects in the cache and update or retire them.

Note that CacheFiles will erase from the cache any file it doesn't recognise or any file of an incorrect type (such as a FIFO file or a device file).

This documentation is added by the patch to:

Documentation/filesystems/caching/cachefiles.txt

---

 Documentation/filesystems/caching/cachefiles.txt |  274 +++++
 fs/Kconfig                                       |    8 
 fs/Makefile                                      |    1 
 fs/buffer.c                                      |    2 
 fs/cachefiles/Makefile                           |   16 
 fs/cachefiles/cf-bind.c                          |  283 +++++
 fs/cachefiles/cf-interface.c                     | 1303 ++++++++++++++++++++++
 fs/cachefiles/cf-key.c                           |  160 +++
 fs/cachefiles/cf-main.c                          |  167 +++
 fs/cachefiles/cf-namei.c                         |  837 ++++++++++++++
 fs/cachefiles/cf-proc.c                          |  510 +++++++++
 fs/cachefiles/cf-xattr.c                         |  299 +++++
 fs/cachefiles/internal.h                         |  292 +++++
 fs/fcntl.c                                       |    2 
 include/linux/pagemap.h                          |    6 
 mm/filemap.c                                     |  102 ++
 16 files changed, 4262 insertions(+), 0 deletions(-)
Documentation/filesystems/caching/cachefiles.txt new file +274 -0
new file mode 100644
index 0000000..c9875f2
@@ -0,0 +1,274 @@
+	       ===============================================
+	       CacheFiles: CACHE ON ALREADY MOUNTED FILESYSTEM
+	       ===============================================
+
+Contents:
+
+ (*) Overview.
+
+ (*) Requirements.
+
+ (*) Configuration.
+
+ (*) Starting the cache.
+
+ (*) Things to avoid.
+
+
+========
+OVERVIEW
+========
+
+CacheFiles is a caching backend that's meant to use as a cache a directory on
+an already mounted filesystem of a local type (such as Ext3).
+
+CacheFiles uses a userspace daemon to do some of the cache management - such as
+reaping stale nodes and culling.  This is called cachefilesd and lives in
+/sbin.
+
+The filesystem and data integrity of the cache are only as good as those of the
+filesystem providing the backing services.  Note that CacheFiles does not
+attempt to journal anything since the journalling interfaces of the various
+filesystems are very specific in nature.
+
+CacheFiles creates a proc-file - "/proc/fs/cachefiles" - that is used for
+communication with the daemon.  Only one thing may have this open at once, and
+whilst it is open, a cache is at least partially in existence.  The daemon
+opens this and sends commands down it to control the cache.
+
+CacheFiles is currently limited to a single cache.
+
+CacheFiles attempts to maintain at least a certain percentage of free space on
+the filesystem, shrinking the cache by culling the objects it contains to make
+space if necessary - see the "Cache Culling" section.  This means it can be
+placed on the same medium as a live set of data, and will expand to make use of
+spare space and automatically contract when the set of data requires more
+space.
+
+
+============
+REQUIREMENTS
+============
+
+The use of CacheFiles and its daemon requires the following features to be
+available in the system and in the cache filesystem:
+
+	- dnotify.
+
+	- extended attributes (xattrs).
+
+	- openat() and friends.
+
+	- bmap() support on files in the filesystem (FIBMAP ioctl).
+
+	- The use of bmap() to detect a partial page at the end of the file.
+
+It is strongly recommended that the "dir_index" option is enabled on Ext3
+filesystems being used as a cache.
+
+
+=============
+CONFIGURATION
+=============
+
+The cache is configured by a script in /etc/cachefilesd.conf.  These commands
+set up cache ready for use.  The following script commands are available:
+
+ (*) brun <N>%
+ (*) bcull <N>%
+ (*) bstop <N>%
+
+	Configure the culling limits.  Optional.  See the section on culling
+	The defaults are 7%, 5% and 1% respectively.
+
+ (*) dir <path>
+
+	Specify the directory containing the root of the cache.  Mandatory.
+
+ (*) tag <name>
+
+	Specify a tag to FS-Cache to use in distinguishing multiple caches.
+	Optional.  The default is "CacheFiles".
+
+ (*) debug <mask>
+
+	Specify a numeric bitmask to control debugging in the kernel module.
+	Optional.  The default is zero (all off).
+
+
+==================
+STARTING THE CACHE
+==================
+
+The cache is started by running the daemon.  The daemon opens the cache proc
+file, configures the cache and tells it to begin caching.  At that point the
+cache binds to fscache and the cache becomes live.
+
+The daemon is run as follows:
+
+	/sbin/cachefilesd [-d]* [-s] [-n] [-f <configfile>]
+
+The flags are:
+
+ (*) -d
+
+	Increase the debugging level.  This can be specified multiple times and
+	is cumulative with itself.
+
+ (*) -s
+
+	Send messages to stderr instead of syslog.
+
+ (*) -n
+
+	Don't daemonise and go into background.
+
+ (*) -f <configfile>
+
+	Use an alternative configuration file rather than the default one.
+
+
+===============
+THINGS TO AVOID
+===============
+
+Do not mount other things within the cache as this will cause problems.  The
+kernel module contains its own very cut-down path walking facility that ignores
+mountpoints, but the daemon can't avoid them.
+
+Do not create, rename or unlink files and directories in the cache whilst the
+cache is active, as this may cause the state to become uncertain.
+
+Renaming files in the cache might make objects appear to be other objects (the
+filename is part of the lookup key).
+
+Do not change or remove the extended attributes attached to cache files by the
+cache as this will cause the cache state management to get confused.
+
+Do not create files or directories in the cache, lest the cache get confused or
+serve incorrect data.
+
+Do not chmod files in the cache.  The module creates things with minimal
+permissions to prevent random users being able to access them directly.
+
+
+=============
+CACHE CULLING
+=============
+
+The cache may need culling occasionally to make space.  This involves
+discarding objects from the cache that have been used less recently than
+anything else.  Culling is based on the access time of data objects.  Empty
+directories are culled if not in use.
+
+Cache culling is done on the basis of the percentage of blocks available in the
+underlying filesystem.  There are three "limits":
+
+ (*) brun
+
+     If the amount of available space in the cache rises above this limit, then
+     culling is turned off.
+
+ (*) bcull
+
+     If the amount of available space in the cache falls below this limit, then
+     culling is started.
+
+ (*) bstop
+
+     If the amount of available space in the cache falls below this limit, then
+     no further allocation of disk space is permitted until culling has raised
+     the amount above this limit again.
+
+These must be configured thusly:
+
+	0 <= bstop < bcull < brun < 100
+
+Note that these are percentages of available space, and do _not_ appear as 100
+minus the percentage displayed by the "df" program.
+
+The userspace daemon scans the cache to build up a table of cullable objects.
+These are then culled in least recently used order.  A new scan of the cache is
+started as soon as space is made in the table.  Objects will be skipped if
+their atimes have changed or if the kernel module says it is still using them.
+
+
+===============
+CACHE STRUCTURE
+===============
+
+The CacheFiles module will create two directories in the directory it was
+given:
+
+ (*) cache/
+
+ (*) graveyard/
+
+The active cache objects all reside in the first directory.  The CacheFiles
+kernel module moves any retired or culled objects that it can't simply unlink
+to the graveyard from which the daemon will actually delete them.
+
+The daemon uses dnotify to monitor the graveyard directory, and will delete
+anything that appears therein.
+
+
+The module represents index objects as directories with the filename "I..." or
+"J...".  Note that the "cache/" directory is itself a special index.
+
+Data objects are represented as files if they have no children, or directories
+if they do.  Their filenames all begin "D..." or "E...".  If represented as a
+directory, data objects will have a file in the directory called "data" that
+actually holds the data.
+
+Special objects are similar to data objects, except their filenames begin
+"S..." or "T...".
+
+
+If an object has children, then it will be represented as a directory.
+Immediately in the representative directory are a collection of directories
+named for hash values of the child object keys with an '@' prepended.  Into
+this directory, if possible, will be placed the representations of the child
+objects:
+
+	INDEX     INDEX      INDEX                             DATA FILES
+	========= ========== ================================= ================
+	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400
+	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...DB1ry
+	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...N22ry
+	cache/@4a/I03nfs/@30/Ji000000000000000--fHg8hi8400/@75/Es0g000w...FP1ry
+
+
+If the key is so long that it exceeds NAME_MAX with the decorations added on to
+it, then it will be cut into pieces, the first few of which will be used to
+make a nest of directories, and the last one of which will be the objects
+inside the last directory.  The names of the intermediate directories will have
+'+' prepended:
+
+	J1223/@23/+xy...z/+kl...m/Epqr
+
+
+Note that keys are raw data, and not only may they exceed NAME_MAX in size,
+they may also contain things like '/' and NUL characters, and so they may not
+be suitable for turning directly into a filename.
+
+To handle this, CacheFiles will use a suitably printable filename directly and
+"base-64" encode ones that aren't directly suitable.  The two versions of
+object filenames indicate the encoding:
+
+	OBJECT TYPE	PRINTABLE	ENCODED
+	===============	===============	===============
+	Index		"I..."		"J..."
+	Data		"D..."		"E..."
+	Special		"S..."		"T..."
+
+Intermediate directories are always "@" or "+" as appropriate.
+
+
+Each object in the cache has an extended attribute label that holds the object
+type ID (required to distinguish special objects) and the auxiliary data from
+the netfs.  The latter is used to detect stale objects in the cache and update
+or retire them.
+
+
+Note that CacheFiles will erase from the cache any file it doesn't recognise or
+any file of an incorrect type (such as a FIFO file or a device file).
fs/Kconfig +8 -0
index 6c95e58..9ef9f14 100644
@@ -521,6 +521,14 @@ config FSCACHE
 
 	  See Documentation/filesystems/caching/fscache.txt for more information.
 
+config CACHEFILES
+	tristate "Filesystem caching on files"
+	depends on FSCACHE
+	help
+	  This permits use of a mounted filesystem as a cache for other
+	  filesystems - primarily networking filesystems - thus allowing fast
+	  local disk to enhance the speed of slower devices.
+
 endmenu
 
 menu "CD-ROM/DVD Filesystems"
fs/Makefile +1 -0
index 36ee03b..94ab3f9 100644
@@ -100,6 +100,7 @@ obj-$(CONFIG_AFS_FS)		+= afs/
 obj-$(CONFIG_BEFS_FS)		+= befs/
 obj-$(CONFIG_HOSTFS)		+= hostfs/
 obj-$(CONFIG_HPPFS)		+= hppfs/
+obj-$(CONFIG_CACHEFILES)	+= cachefiles/
 obj-$(CONFIG_DEBUG_FS)		+= debugfs/
 obj-$(CONFIG_CONFIGFS_FS)	+= configfs/
 obj-$(CONFIG_OCFS2_FS)		+= ocfs2/
fs/buffer.c +2 -0
index 23f1f3a..0602bf8 100644
@@ -185,6 +185,8 @@ int fsync_super(struct super_block *sb)
 	return sync_blockdev(sb->s_bdev);
 }
 
+EXPORT_SYMBOL(fsync_super);
+
 /*
  * Write out and wait upon all dirty data associated with this
  * device.   Filesystem data as well as the underlying block
fs/cachefiles/Makefile new file +16 -0
new file mode 100644
index 0000000..fc63c0c
@@ -0,0 +1,16 @@
+#
+# Makefile for caching in a mounted filesystem
+#
+
+#CFLAGS += -finstrument-functions
+
+cachefiles-objs := \
+	cf-bind.o \
+	cf-interface.o \
+	cf-key.o \
+	cf-main.o \
+	cf-namei.o \
+	cf-proc.o \
+	cf-xattr.o
+
+obj-$(CONFIG_CACHEFILES) := cachefiles.o
fs/cachefiles/cf-bind.c new file +283 -0

283 lines not on this page · raw message

fs/cachefiles/cf-interface.c new file +1303 -0

1,303 lines not on this page · raw message

fs/cachefiles/cf-key.c new file +160 -0

160 lines not on this page · raw message

fs/cachefiles/cf-main.c new file +167 -0

167 lines not on this page · raw message

fs/cachefiles/cf-namei.c new file +837 -0

768 lines not on this page · raw message

fs/cachefiles/cf-proc.c new file +510 -0

510 lines not on this page · raw message

fs/cachefiles/cf-xattr.c new file +299 -0

299 lines not on this page · raw message

fs/cachefiles/internal.h new file +292 -0

292 lines not on this page · raw message

fs/fcntl.c +2 -0
index d35cbc6..b43d821 100644

Lines of this file are not shown. View the raw message.

include/linux/pagemap.h +6 -0
index 02e7d8b..bc5eba8 100644

Lines of this file are not shown. View the raw message.

mm/filemap.c +102 -0

102 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#25 lore raw

Re: [PATCH 7/7] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem

Review

Quoted 9 lines · David Howells
(quoted diff) +1 -0
+		/* let keventd have some air occasionally */
 +		max--;
 +		if (max < 0 || need_resched()) {
 +			if (!list_empty(&object->read_list))
 +				schedule_work(&object->read_work);
 +			_leave(" [maxed out]");
 +			return;
 +		}

That's perhaps not a terribly effective way of multiplexing keventd cycles. If someone has done a schedule_work(), that will stick an entry onto keventd's worklist, but it won't necessarily set need_resched().

We'd need to extend the workqueue API to be able to determine whether there's other work pending.

#26 lore raw

Re: [PATCH 7/7] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem

In reply to David Howells

fs/cachefiles/cf-interface.c line 733

Quoted 6 lines · David Howells

...

fs/cachefiles/cf-interface.c new file +3 -0
+		ret = 0;
+	}
+	else if (cachefiles_has_space(cache, 1) == 0) {
	} else if (cachefiles_has_space(cache, 1) == 0) {

fs/cachefiles/cf-interface.c line 740

Quoted 7 lines
fs/cachefiles/cf-interface.c new file +7 -0
+		/* there's space in the cache we can use */
+		pagevec_add(&pagevec, page);
+		cookie->def->mark_pages_cached(cookie->netfs_data,
+					       page->mapping, &pagevec);
+		ret = -ENODATA;
+	}
+	else {
	} else {

(many instances)

Quoted 1 line

+unsigned long cachefiles_debug = 0;

Unneeded initialisation.

Quoted 1 line

+static int cachefiles_init(void)

__init?

Quoted 29 lines
fs/cachefiles/cf-main.c new file +29 -0
+#if 0
+void __cyg_profile_func_enter (void *this_fn, void *call_site)
+__attribute__((no_instrument_function));
+
+void __cyg_profile_func_enter (void *this_fn, void *call_site)
+{
+       asm volatile("  movl    %%esp,%%edi     \n"
+                    "  andl    %0,%%edi        \n"
+                    "  addl    %1,%%edi        \n"
+                    "  movl    %%esp,%%ecx     \n"
+                    "  subl    %%edi,%%ecx     \n"
+                    "  shrl    $2,%%ecx        \n"
+                    "  movl    $0xedededed,%%eax     \n"
+                    "  rep stosl               \n"
+                    :
+                    : "i"(~(THREAD_SIZE-1)), "i"(sizeof(struct thread_info))
+                    : "eax", "ecx", "edi", "memory", "cc"
+                    );
+}
+
+void __cyg_profile_func_exit(void *this_fn, void *call_site)
+__attribute__((no_instrument_function));
+
+void __cyg_profile_func_exit(void *this_fn, void *call_site)
+{
+       asm volatile("  movl    %%esp,%%edi     \n"
+                    "  andl    %0,%%edi        \n"
+                    "  addl    %1,%%edi        \n"
+                    "  movl    %%esp,%%ecx     \n"

fs/cachefiles/cf-main.c line 167

Quoted 10 lines
fs/cachefiles/cf-main.c new file +10 -0
+                    "  subl    %%edi,%%ecx     \n"
+                    "  shrl    $2,%%ecx        \n"
+                    "  movl    $0xdadadada,%%eax     \n"
+                    "  rep stosl               \n"
+                    :
+                    : "i"(~(THREAD_SIZE-1)), "i"(sizeof(struct thread_info))
+                    : "eax", "ecx", "edi", "memory", "cc"
+                    );
+}
+#endif

removeable?

Quoted 17 lines
fs/cachefiles/cf-namei.c new file +17 -0
+/*
+ * delete an object representation from the cache
+ * - file backed objects are unlinked
+ * - directory backed objects are stuffed into the graveyard for userspace to
+ *   delete
+ * - unlocks the directory mutex
+ */
+static int cachefiles_bury_object(struct cachefiles_cache *cache,
+				  struct dentry *dir,
+				  struct dentry *rep)
+{
+	struct dentry *grave, *alt, *trap;
+	struct qstr name;
+	const char *old_name;
+	char nbuffer[8 + 8 + 1];
+	int ret;
+

fs/cachefiles/cf-namei.c line 85

Quoted 10 lines
fs/cachefiles/cf-namei.c new file +10 -0
+	_enter(",'%*.*s','%*.*s'",
+	       dir->d_name.len, dir->d_name.len, dir->d_name.name,
+	       rep->d_name.len, rep->d_name.len, rep->d_name.name);
+
+	/* non-directories can just be unlinked */
+	if (!S_ISDIR(rep->d_inode->i_mode)) {
+		_debug("unlink stale object");
+		ret = dir->d_inode->i_op->unlink(dir->d_inode, rep);
+
+		mutex_unlock(&dir->d_inode->i_mutex);

hm, what's going on here? It's strange for a callee to undo an i_mutex which some caller took.

mm/filemap.c generic_file_buffered_write(struct kiocb · line 2124

Quoted 6 lines

EXPORT_SYMBOL(generic_file_buffered_write);

mm/filemap.c +4 -0
+int
+generic_file_buffered_write_one_kernel_page(struct file *file,
+					    pgoff_t index,
+					    struct page *src)

Some covering comments would be nice.

Quoted 9 lines
mm/filemap.c +9 -0
+{
+	struct address_space *mapping = file->f_mapping;
+	struct address_space_operations *a_ops = mapping->a_ops;
+	struct pagevec	lru_pvec;
+	struct page *page, *cached_page = NULL;
+	void *from, *to;
+	long status = 0;
+
+	pagevec_init(&lru_pvec, 0);

mm/filemap.c generic_file_buffered_write(struct kiocb · line 2143

Quoted 10 lines
mm/filemap.c +10 -0
+
+	page = __grab_cache_page(mapping, index, &cached_page, &lru_pvec);
+	if (!page) {
+		BUG_ON(cached_page);
+		return -ENOMEM;
+	}
+
+	status = a_ops->prepare_write(file, page, 0, PAGE_CACHE_SIZE);
+	if (unlikely(status)) {
+		loff_t isize = i_size_read(mapping->host);

If the hosts's i_mutex is held (it should be, but there are no comments) then we can read inode->i_size directly. Minor thing.

mm/filemap.c generic_file_buffered_write(struct kiocb · line 2163

Quoted 6 lines
mm/filemap.c +6 -0
+
+	from = kmap_atomic(src, KM_USER0);
+	to = kmap_atomic(page, KM_USER1);
+	copy_page(to, from);
+	kunmap_atomic(from, KM_USER0);
+	kunmap_atomic(to, KM_USER1);

that's copy_highpage().

Sigh. It's all a huge pile of new code. And it's only used by AFS, the number of users of which can be counted on the fingers of one foot. An NFS implementation would make a testing phase much more useful.

author maintainer
#27 lore raw

Re: [PATCH 7/7] FS-Cache: CacheFiles: A cache that backs onto a mounted filesystem

Quoted 3 lines · Andrew Morton
Quoted 1 line

+unsigned long cachefiles_debug = 0;

Unneeded initialisation.

Yep.

Quoted 3 lines
Quoted 1 line

+static int cachefiles_init(void)

__init?

Yep.

Quoted 1 line

removeable?

Yep.

Quoted 2 lines

hm, what's going on here? It's strange for a callee to undo an i_mutex which some caller took.

It happens occasionally. The problem here is that I want to call this from three different places, but if I drop the mutex before calling the burial function, I have to get the mutex again to do the unlink; but as it is, I have to drop it before I can do the rename:-/

It's not nice, but...

You have to note also that the directory's i_mutex is quite important for interacting with the daemon also. The wonders of working through an existing filesystem, and the the wonders of co-operating with userspace.

Quoted 6 lines

mm/filemap.c generic_file_buffered_write(struct kiocb · line 2124

Quoted 4 lines
mm/filemap.c +4 -0
+int
+generic_file_buffered_write_one_kernel_page(struct file *file,
+					    pgoff_t index,
+					    struct page *src)

Some covering comments would be nice.

I copied those of generic_file_buffered_write() and rearranged them a bit:-)

I'll add a comment to my function.

Quoted 2 lines

If the hosts's i_mutex is held (it should be, but there are no comments) then we can read inode->i_size directly. Minor thing.

Ah. Do we though? I just copied generic_file_buffered_write() and cut it down. The same is done there. The comments at the top of that function weren't exactly forthcoming on the preconditions for calling that function.

Quoted 1 line

that's copy_highpage().

Good point.

Quoted 3 lines

Sigh. It's all a huge pile of new code. And it's only used by AFS, the number of users of which can be counted on the fingers of one foot. An NFS implementation would make a testing phase much more useful.

Yes... Whilst I have it working with NFS, the NFS anti-aliasing problems are still there and still need to be sorted. I thought I'd got them nailed, but then Trond changed his mind:-(

But that does not preclude putting what I can release up for review.

David

#28 lore raw
In reply to David Howells
Quoted 3 lines · David Howells

#define PG_checked 8 /* kill me in 2.5.<early>. */
+#define PG_fs_misc 8

It would be better to rename PG_checked to PG_fs_misc kernel-wide.

author maintainer
#29 lore raw
Quoted 1 line · Andrew Morton

It would be better to rename PG_checked to PG_fs_misc kernel-wide.

So would deleting PG_checked and changing the PageChecked() macros to:

	#define PageChecked(page)		PageFsMisc((page))
	#define SetPageChecked(page)		SetPageFsMisc((page))
	#define ClearPageChecked(page)		ClearPageFsMisc((page))

be acceptable? Or would you rather I replaced those too?

David

#30 lore raw
Quoted 5 lines · David Howells
Quoted 1 line · Andrew Morton

It would be better to rename PG_checked to PG_fs_misc kernel-wide.

Quoted 8 lines

So would deleting PG_checked and changing the PageChecked() macros to:

	#define PageChecked(page)		PageFsMisc((page))
	#define SetPageChecked(page)		SetPageFsMisc((page))
	#define ClearPageChecked(page)		ClearPageFsMisc((page))

be acceptable? Or would you rather I replaced those too?

PG_checked is presently a misc bit which only filesystems use. So yes, I'd say it's appropriate to remove PageChecked() and friends altogether.

That might break out-of-tree filesystems, but they'll work it out.