Skip to contentSkip to conversation

Fix bugs in Slow-work, FS-Cache and CacheFiles

GIT PULL In review David Howells 3 participants 4 messages Nov 20, 2009 – Dec 1, 2009 VFS

lore
Share & tools
lore
#0 lore raw

Hi Linus,

Could you pull the attached patches before 2.6.32 please? For the most part they are bug fixes for the slow-work, FS-Cache and CacheFiles facilities, with a few patches to produce better debugging information, and a couple of patches to slow work from Jens that all the rest are built on top of.

These vastly improve the stability of FS-Cache under extremely high load - especially when the system is running out of memory regularly. I wrote a better test case:

http://people.redhat.com/~dhowells/slurp.c

which helped enormously in reproducing these bugs. Testing has involved running repeated invocations of slurp over five NFS4 mounted directory trees totalling ~40GB in N file spread over 100,000 dirs on a machine with 1GB of RAM and 4GB of disk cache space. Note that slurp, being massively parallel, will abandon a part of the tree if it hits any error (such as ENOMEM) - so not all the data set will necessarily be used in any particular run.

The patches can also be downloaded as:

http://people.redhat.com/~dhowells/fscache/patches/fscache-fixes.tar.bz2

The changes don't affect very much outside of FS-Cache and its dependents (NFS, AFS and 9P), but the slow-work changes do affect CIFS and GFS2.

Thanks,
David
---
The following changes since commit 66b00a7c93ec782d118d2c03bd599cfd041e80a1:
Linus Torvalds (1):
Merge branch 'fixes' of git://git.kernel.org/.../davej/cpufreq

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-fscache.git master

David Howells (25):
SLOW_WORK: Wait for outstanding work items belonging to a module to clear
SLOW_WORK: Allow the work items to be viewed through a /proc file
SLOW_WORK: Allow the owner of a work item to determine if it is queued or not
SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed
FS-Cache: Annotate slow-work runqueue proc lines for FS-Cache work items
FS-Cache: Allow the current state of all objects to be dumped
FS-Cache: Add counters for entry/exit to/from cache operation functions
FS-Cache: Clear netfs pointers in cookie after detaching object, not before
FS-Cache: Use radix tree preload correctly in tracking of pages to be stored
FS-Cache: Permit cache retrieval ops to be interrupted in the initial wait phase
FS-Cache: The object-available state can't rely on the cookie to be available
FS-Cache: Fix lock misorder in fscache_write_op()
FS-Cache: Don't delete pending pages from the page-store tracking tree
FS-Cache: Handle read request vs lookup, creation or other cache failure
FS-Cache: Handle pages pending storage that get evicted under OOM conditions
FS-Cache: Add a retirement stat counter
FS-Cache: Make sure FSCACHE_COOKIE_LOOKING_UP cleared on lookup failure
FS-Cache: Start processing an object's operations on that object's death
FS-Cache: Actually requeue an object when requested
CacheFiles: Don't write a full page if there's only a partial page to cache
CacheFiles: Handle truncate unlocking the page we're reading
CacheFiles: Mark parent directory locks as I_MUTEX_PARENT to keep lockdep happy
CacheFiles: Better showing of debugging information in active object problems
CacheFiles: Catch an overly long wait for an old active object
CacheFiles: Don't log lookup/create failing with ENOBUFS

Jens Axboe (3):
SLOW_WORK: Make slow_work_ops ->get_ref/->put_ref optional
SLOW_WORK: Add support for cancellation of slow work
SLOW_WORK: Add delayed_slow_work support

 Documentation/filesystems/caching/fscache.txt   |  110 +++++
 Documentation/filesystems/caching/netfs-api.txt |   21 +-
 Documentation/slow-work.txt                     |  160 +++++++-
 fs/9p/cache.c                                   |   14 +-
 fs/afs/file.c                                   |   15 +-
 fs/cachefiles/interface.c                       |   32 ++-
 fs/cachefiles/namei.c                           |  187 +++++++--
 fs/cachefiles/rdwr.c                            |  128 ++++++-
 fs/fscache/Kconfig                              |    7 +
 fs/fscache/Makefile                             |    1 +
 fs/fscache/cache.c                              |    5 +
 fs/fscache/cookie.c                             |   26 +-
 fs/fscache/internal.h                           |   55 +++
 fs/fscache/main.c                               |    6 +-
 fs/fscache/object-list.c                        |  432 ++++++++++++++++++++
 fs/fscache/object.c                             |  104 +++++-
 fs/fscache/operation.c                          |  120 +++++--
 fs/fscache/page.c                               |  273 ++++++++++---
 fs/fscache/proc.c                               |   13 +
 fs/fscache/stats.c                              |   94 ++++-
 fs/gfs2/main.c                                  |    4 +-
 fs/gfs2/recovery.c                              |    1 +
 fs/nfs/fscache.c                                |   10 +-
 include/linux/fscache-cache.h                   |   40 ++-
 include/linux/fscache.h                         |   27 ++
 include/linux/slow-work.h                       |   72 ++++-
 init/Kconfig                                    |   10 +
 kernel/Makefile                                 |    1 +
 kernel/slow-work-proc.c                         |  227 +++++++++++
 kernel/slow-work.c                              |  494 +++++++++++++++++++++--
 kernel/slow-work.h                              |   72 ++++
 lib/radix-tree.c                                |    5 +-
 32 files changed, 2502 insertions(+), 264 deletions(-)
 create mode 100644 fs/fscache/object-list.c
 create mode 100644 kernel/slow-work-proc.c
 create mode 100644 kernel/slow-work.h
#1 lore raw

Hi Linus,

Could you pull the attached patches before 2.6.32 please? I've added three one-liner patches to fix some silly bugs.

For the most part these patches are bug fixes for the slow-work, FS-Cache and CacheFiles facilities, with a few patches to produce better debugging information, and a couple of patches to slow work from Jens that all the rest are built on top of.

These vastly improve the stability of FS-Cache under extremely high load - especially when the system is running out of memory regularly. I wrote a better test case:

http://people.redhat.com/~dhowells/slurp.c

which helped enormously in reproducing these bugs. Testing has involved running repeated invocations of slurp over five NFS4 mounted directory trees totalling ~40GB in N file spread over 100,000 dirs on a machine with 1GB of RAM and 4GB of disk cache space. Note that slurp, being massively parallel, will abandon a part of the tree if it hits any error (such as ENOMEM) - so not all the data set will necessarily be used in any particular run.

The patches can also be downloaded as:

http://people.redhat.com/~dhowells/fscache/patches/fscache-fixes.tar.bz2

The changes don't affect very much outside of FS-Cache and its dependents (NFS, AFS and 9P), but the slow-work changes do affect CIFS and GFS2.

Thanks,
David
---
The following changes since commit 66b00a7c93ec782d118d2c03bd599cfd041e80a1:
Linus Torvalds (1):
Merge branch 'fixes' of git://git.kernel.org/.../davej/cpufreq

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-fscache.git master

David Howells (28):
SLOW_WORK: Wait for outstanding work items belonging to a module to clear
SLOW_WORK: Allow the work items to be viewed through a /proc file
SLOW_WORK: Allow the owner of a work item to determine if it is queued or not
SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed
FS-Cache: Annotate slow-work runqueue proc lines for FS-Cache work items
FS-Cache: Allow the current state of all objects to be dumped
FS-Cache: Add counters for entry/exit to/from cache operation functions
FS-Cache: Clear netfs pointers in cookie after detaching object, not before
FS-Cache: Use radix tree preload correctly in tracking of pages to be stored
FS-Cache: Permit cache retrieval ops to be interrupted in the initial wait phase
FS-Cache: The object-available state can't rely on the cookie to be available
FS-Cache: Fix lock misorder in fscache_write_op()
FS-Cache: Don't delete pending pages from the page-store tracking tree
FS-Cache: Handle read request vs lookup, creation or other cache failure
FS-Cache: Handle pages pending storage that get evicted under OOM conditions
FS-Cache: Add a retirement stat counter
FS-Cache: Make sure FSCACHE_COOKIE_LOOKING_UP cleared on lookup failure
FS-Cache: Start processing an object's operations on that object's death
FS-Cache: Actually requeue an object when requested
CacheFiles: Don't write a full page if there's only a partial page to cache
CacheFiles: Handle truncate unlocking the page we're reading
CacheFiles: Mark parent directory locks as I_MUTEX_PARENT to keep lockdep happy
CacheFiles: Better showing of debugging information in active object problems
CacheFiles: Catch an overly long wait for an old active object
CacheFiles: Don't log lookup/create failing with ENOBUFS
SLOW_WORK: Fix CIFS to pass THIS_MODULE to slow_work_register_user()
SLOW_WORK: Fix GFS2 to #include <linux/module.h> before using THIS_MODULE
FS-Cache: Provide nop fscache_stat_d() if CONFIG_FSCACHE_STATS=n

Jens Axboe (3):
SLOW_WORK: Make slow_work_ops ->get_ref/->put_ref optional
SLOW_WORK: Add support for cancellation of slow work
SLOW_WORK: Add delayed_slow_work support

 Documentation/filesystems/caching/fscache.txt   |  110 +++++
 Documentation/filesystems/caching/netfs-api.txt |   21 +-
 Documentation/slow-work.txt                     |  160 +++++++-
 fs/9p/cache.c                                   |   14 +-
 fs/afs/file.c                                   |   15 +-
 fs/cachefiles/interface.c                       |   32 ++-
 fs/cachefiles/namei.c                           |  187 +++++++--
 fs/cachefiles/rdwr.c                            |  128 ++++++-
 fs/cifs/cifsfs.c                                |    2 +-
 fs/fscache/Kconfig                              |    7 +
 fs/fscache/Makefile                             |    1 +
 fs/fscache/cache.c                              |    5 +
 fs/fscache/cookie.c                             |   26 +-
 fs/fscache/internal.h                           |   56 +++
 fs/fscache/main.c                               |    6 +-
 fs/fscache/object-list.c                        |  432 ++++++++++++++++++++
 fs/fscache/object.c                             |  104 +++++-
 fs/fscache/operation.c                          |  120 +++++--
 fs/fscache/page.c                               |  273 ++++++++++---
 fs/fscache/proc.c                               |   13 +
 fs/fscache/stats.c                              |   94 ++++-
 fs/gfs2/main.c                                  |    4 +-
 fs/gfs2/recovery.c                              |    2 +
 fs/nfs/fscache.c                                |   10 +-
 include/linux/fscache-cache.h                   |   40 ++-
 include/linux/fscache.h                         |   27 ++
 include/linux/slow-work.h                       |   72 ++++-
 init/Kconfig                                    |   10 +
 kernel/Makefile                                 |    1 +
 kernel/slow-work-proc.c                         |  227 +++++++++++
 kernel/slow-work.c                              |  494 +++++++++++++++++++++--
 kernel/slow-work.h                              |   72 ++++
 lib/radix-tree.c                                |    5 +-
 33 files changed, 2505 insertions(+), 265 deletions(-)
 create mode 100644 fs/fscache/object-list.c
 create mode 100644 kernel/slow-work-proc.c
 create mode 100644 kernel/slow-work.h
#2 lore raw

PATCH 1/1 slow-work: Fix build bug in the !CONFIG_MODULES case

Quoted 9 lines · David Howells

Hi Linus,

Could you pull the attached patches before 2.6.32 please? I've added three one-liner patches to fix some silly bugs.

For the most part these patches are bug fixes for the slow-work, FS-Cache and CacheFiles facilities, with a few patches to produce better debugging information, and a couple of patches to slow work from Jens that all the rest are built on top of.

these slow-work changes have introduced a new !CONFIG_MODULES build bug on x86 (and all other architectures i suspect):

kernel/slow-work.c: In function ‘slow_work_execute’:
kernel/slow-work.c:313: error: ‘slow_work_thread_processing’ undeclared (first use in this function) kernel/slow-work.c:313: error: (Each undeclared identifier is reported only once kernel/slow-work.c:313: error: for each function it appears in.)
kernel/slow-work.c: In function ‘slow_work_wait_for_items’:
kernel/slow-work.c:950: error: ‘slow_work_unreg_sync_lock’ undeclared (first use in this function) kernel/slow-work.c:951: error: ‘slow_work_unreg_wq’ undeclared (first use in this function) kernel/slow-work.c:961: error: ‘slow_work_unreg_work_item’ undeclared (first use in this function) kernel/slow-work.c:974: error: ‘slow_work_unreg_module’ undeclared (first use in this function) kernel/slow-work.c:977: error: ‘slow_work_thread_processing’ undeclared (first use in this function)

Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed") broke the !CONFIG_MODULES build by relying on variables that are only available (and only make sense) with module support enabled. Fix it.

Note, to fix it in a minimal way for .32 i've simply extended the existing #ifdef's. In v2.6.33 a cleaner change would be to move the whole wait-for-modules block into a separate slow-work-wait-modules.c file or so - that will allow the removal of 6 #ifdefs.

Thanks,

Ingo

--------------------------->
From ac73995e9be78ba5dcf3db843d7e04295decabcd Mon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Tue, 1 Dec 2009 07:47:52 +0100
Subject: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case

Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed") broke the !CONFIG_MODULES build by relying on variables that are only available (and only make sense) with module support enabled. Fix it.

Cc David Howells <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Andrew Morton <[email protected]>
---
 kernel/slow-work.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
kernel/slow-work.c +4 -0
index da94f3c..5bae807 100644
@@ -310,7 +310,9 @@ auto_requeue:
 	else
 		list_add_tail(&work->link, &slow_work_queue);
 	spin_unlock_irq(&slow_work_queue_lock);
+#ifdef CONFIG_MODULES
 	slow_work_thread_processing[id] = NULL;
+#endif
 	return true;
 }
 
@@ -943,6 +945,7 @@ EXPORT_SYMBOL(slow_work_register_user);
  */
 static void slow_work_wait_for_items(struct module *module)
 {
+#ifdef CONFIG_MODULES
 	DECLARE_WAITQUEUE(myself, current);
 	struct slow_work *work;
 	int loop;
@@ -989,6 +992,7 @@ static void slow_work_wait_for_items(struct module *module)
 
 	remove_wait_queue(&slow_work_unreg_wq, &myself);
 	mutex_unlock(&slow_work_unreg_sync_lock);
+#endif
 }
 
 /**

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

Re: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case

Hi Ingo,

Quoted 3 lines · On Tue, 1 Dec 2009 07:56:05 +0100 Ingo Molnar <[email protected]> wrote:

these slow-work changes have introduced a new !CONFIG_MODULES build bug on x86 (and all other architectures i suspect):

Thanks for the heads up!

Quoted 9 lines

From: Ingo Molnar <[email protected]>
Date: Tue, 1 Dec 2009 07:47:52 +0100
Subject: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case

Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed") broke the !CONFIG_MODULES build by relying on variables that are only available (and only make sense) with module support enabled. Fix it.

Quoted 4 lines
Cc David Howells <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Andrew Morton <[email protected]>

I have applied this to linux-next for today, just so people don't waste their time finding this breakage again.

Signature · 3 lines
Cheers,
Stephen Rothwell                    [email protected]
http://www.canb.auug.org.au/~sfr/