drop_pagecache: Possible circular locking dependency
Subsystems (6)
Hi Ingo,
Got the following message when doing some benchmarks.
I guess we should not hold inode_lock on calling invalidate_inode_pages(). Any ideas?
Fengguang Wu
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0
but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (inode_lock){--..}:
[<ffffffff810b098f>] add_lock_to_list+0x8f/0xd0
[<ffffffff810b35b9>] __lock_acquire+0xb29/0xd40
[<ffffffff810b3863>] lock_acquire+0x93/0xc0
[<ffffffff81072875>] _spin_lock+0x25/0x40
[<ffffffff81014e41>] __mark_inode_dirty+0x101/0x190
[<ffffffff8100fa00>] __set_page_dirty_nobuffers+0x150/0x170
[<ffffffff810314ae>] mark_buffer_dirty+0x1e/0x30
[<ffffffff81139798>] __journal_temp_unlink_buffer+0x1a8/0x1c0
[<ffffffff81139a31>] __journal_unfile_buffer+0x11/0x20
[<ffffffff81139ac7>] __journal_refile_buffer+0x87/0x120
[<ffffffff8113cfe5>] journal_commit_transaction+0x1005/0x1300
[<ffffffff81141285>] kjournald+0xd5/0x230
[<ffffffff810393ea>] kthread+0xda/0x110
[<ffffffff8106bf38>] child_rip+0xa/0x12
[<ffffffffffffffff>] 0xffffffffffffffff-> #0 (&journal->j_list_lock){--..}:
[<ffffffff810b19e5>] print_circular_bug_tail+0x55/0xb0
[<ffffffff810b349e>] __lock_acquire+0xa0e/0xd40
[<ffffffff810b3863>] lock_acquire+0x93/0xc0
[<ffffffff81072875>] _spin_lock+0x25/0x40
[<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0
[<ffffffff8112af49>] ext3_releasepage+0xa9/0xd0
[<ffffffff8102c1ba>] try_to_release_page+0x5a/0x80
[<ffffffff8104f438>] invalidate_mapping_pages+0xa8/0x150
[<ffffffff81066842>] invalidate_inode_pages+0x12/0x20
[<ffffffff810fe895>] drop_pagecache+0xa5/0x120
[<ffffffff810fe932>] drop_caches_sysctl_handler+0x22/0x80
[<ffffffff8109dca7>] do_rw_proc+0xe7/0x150
[<ffffffff8109dd2a>] proc_writesys+0x1a/0x20
[<ffffffff81018344>] vfs_write+0xf4/0x1b0
[<ffffffff81018ea0>] sys_write+0x50/0xa0
[<ffffffff8106b11e>] system_call+0x7e/0x83
[<00002af654962422>] 0x2af654962422
[<ffffffffffffffff>] 0xffffffffffffffffother info that might help us debug this:
2 locks held by rabench.sh/7467:
#0: (&type->s_umount_key#16){----}, at: [<ffffffff810fe843>] drop_pagecache+0x53/0x120 #1: (inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120
stack backtrace: Call Trace: [<ffffffff8107a3e3>] dump_trace+0xb3/0x460 [<ffffffff8107a7d3>] show_trace+0x43/0x70 [<ffffffff8107a815>] dump_stack+0x15/0x20 [<ffffffff810b1a26>] print_circular_bug_tail+0x96/0xb0 [<ffffffff810b349e>] __lock_acquire+0xa0e/0xd40 [<ffffffff810b3863>] lock_acquire+0x93/0xc0 [<ffffffff81072875>] _spin_lock+0x25/0x40 [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0 [<ffffffff8112af49>] ext3_releasepage+0xa9/0xd0 [<ffffffff8102c1ba>] try_to_release_page+0x5a/0x80 [<ffffffff8104f438>] invalidate_mapping_pages+0xa8/0x150 [<ffffffff81066842>] invalidate_inode_pages+0x12/0x20 [<ffffffff810fe895>] drop_pagecache+0xa5/0x120 [<ffffffff810fe932>] drop_caches_sysctl_handler+0x22/0x80 [<ffffffff8109dca7>] do_rw_proc+0xe7/0x150 [<ffffffff8109dd2a>] proc_writesys+0x1a/0x20 [<ffffffff81018344>] vfs_write+0xf4/0x1b0 [<ffffffff81018ea0>] sys_write+0x50/0xa0 [<ffffffff8106b11e>] system_call+0x7e/0x83 [<00002af654962422>]
2 locks held by rabench.sh/7467:
#0: (&type->s_umount_key#16){----}, at: [<ffffffff810fe843>] drop_pagecache+0x53/0x120 #1: (inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120
Quoted 13 lines · On Mon, 4 Dec 2006 16:09:02 +0800 Fengguang Wu <[email protected]> wrote:
Got the following message when doing some benchmarks.
I guess we should not holdinode_lockon callinginvalidate_inode_pages(). Any ideas?Fengguang Wu
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>]journal_try_to_free_buffers+0xdc/0x1c0
Quoted 5 lines
but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>]drop_pagecache+0x67/0x120which lock already depends on the new lock.
drat, I was afraid someone would notice.
It's Hard To Fix. Removing /proc/sys/vm/drop_pagecache would in fact be my preferred fix.
Quoted 19 lines · Andrew Morton
Quoted 16 lines · On Mon, 4 Dec 2006 16:09:02 +0800 Fengguang Wu <[email protected]> wrote:
Got the following message when doing some benchmarks.
I guess we should not holdinode_lockon callinginvalidate_inode_pages(). Any ideas?Fengguang Wu
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>]journal_try_to_free_buffers+0xdc/0x1c0but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>]drop_pagecache+0x67/0x120
Quoted 7 lines
Quoted 2 lines
which lock already depends on the new lock.
drat, I was afraid someone would notice.
It's Hard To Fix. Removing
/proc/sys/vm/drop_pagecachewould in fact be my preferred fix.
Or fix drop_pagecache_sb():
Repeat until all possible pages freed:
grab and save some inodes to a buffer zip their pages outside of inode_lock
Takes much more code though.
I'd like to move this sysctl interface to the upcoming /proc/filecache. Being a module, it helps reduce the kernel size :)
Fengguang Wu
Quoted 1 line · Fengguang Wu
I'd like to move this sysctl interface to the upcoming
/proc/filecache. Being a module, it helps reduce the kernel size :)
What's /proc/filecache?
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 6 lines · Nick Piggin
Quoted 2 lines · Fengguang Wu
I'd like to move this sysctl interface to the upcoming
/proc/filecache. Being a module, it helps reduce the kernel size :)What's
/proc/filecache?
It provides access to the in-memory file cache.
To list an index of all cached files:
echo -n index > /proc/filecache
cat /proc/filecache
The output looks like:
# filecache 1.0
# ino size cached cached% state refcnt dev file
1026334 91 92 100 -- 66 03:02(hda2) /lib/ld-2.3.6.so
233608 1242 972 78 -- 66 03:02(hda2) /lib/tls/libc-2.3.6.so
65203 651 476 73 -- 1 03:02(hda2) /bin/bash
1026445 261 160 61 -- 10 03:02(hda2) /lib/libncurses.so.5.5
235427 10 12 100 -- 44 03:02(hda2) /lib/tls/libdl-2.3.6.soTo list the cached pages of a perticular file:
echo -n /bin/bash > /proc/filecache
cat /proc/filecache
# file /bin/bash
# flags R:referenced A:active U:uptodate D:dirty W:writeback M:mmap
# idx len state refcnt
0 36 RAU__M 3
36 1 RAU__M 2
37 8 RAU__M 3
45 2 RAU___ 1
47 6 RAU__M 3
53 3 RAU__M 2
56 2 RAU__M 3
To instruct the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free:
# drop clean file data cache (i.e. file backed pagecache)
echo drop data > /proc/filecache # drop clean file metadata cache (i.e. dentries and inodes)
echo drop metadata > /proc/filecacheIt's first introduced here: http://lkml.org/lkml/2006/6/12/29
And the latest working patch for -mm:
include/linux/mm.h +1 -0
@@ -29,6 +29,7 @@ extern unsigned long num_physpages;extern void * high_memory;extern unsigned long vmalloc_earlyreserve;extern int page_cluster; +extern char *zone_names[];#ifdef CONFIG_SYSCTL extern int sysctl_legacy_va_layout;
mm/page_alloc.c +1 -1
@@ -86,7 +86,7 @@ int sysctl_lowmem_reserve_ratio[MAX_NR_ZEXPORT_SYMBOL (totalram_pages); -static char *zone_names[MAX_NR_ZONES ] = { +char *zone_names[MAX_NR_ZONES ] = { #ifdef CONFIG_ZONE_DMA"DMA" , #endif
fs/dcache.c +4 -1
@@ -1776,7 +1776,10 @@ static char * __d_path( struct dentry *dif (dentry == root && vfsmnt == rootmnt)break ; -if (dentry == vfsmnt->mnt_root ||IS_ROOT (dentry)) { +if (unlikely (!vfsmnt)) { +if (IS_ROOT (dentry)) +break ; + }else if (dentry == vfsmnt->mnt_root ||IS_ROOT (dentry)) {/* Global root? */ spin_lock (&vfsmount_lock);if (vfsmnt->mnt_parent == vfsmnt) {
fs/seq_file.c +12 -9
@@ -13,6 +13,8 @@#include <asm/uaccess.h> #include <asm/page.h> +#define SEQFILE_SHOW_FROM_NEXT LONG_MAX +/** * seq_open - initialize sequential file * @file: file we initialize
@@ -93,6 +95,7 @@ ssize_t seq_read(struct file *file, char/* if not empty - flush it first */ if (m->count) { n =min (m->count, size); +BUG_ON (m->from ==SEQFILE_SHOW_FROM_NEXT ); err =copy_to_user (buf, m->buf + m->from, n);if (err) goto Efault;
@@ -102,7 +105,7 @@ ssize_t seq_read(struct file *file, char buf += n; copied += n;if (!m->count) - m->index++; + m->from =SEQFILE_SHOW_FROM_NEXT ;if (!size) goto Done; }
@@ -113,9 +116,11 @@ ssize_t seq_read(struct file *file, char err =PTR_ERR (p);if (!p ||IS_ERR (p))break ; - err = m->op->show (m, p); -if (err) -break ; +if (m->from !=SEQFILE_SHOW_FROM_NEXT ) { + err = m->op->show (m, p); +if (err) +break ; + }if (m->count < m->size) goto Fill; m->op->stop (m, p);
@@ -156,7 +161,7 @@ Fill:if (m->count) m->from = n;else - pos++; + m->from =SEQFILE_SHOW_FROM_NEXT ; m->index = pos; Done:if (!copied)
@@ -208,11 +213,9 @@ static int traverse(struct seq_file *m, } pos += m->count; m->count =0 ; -if (pos == offset) { - m->index++; -break ; - } p = m->op->next (m, p, &m->index); +if (pos == offset) +break ; } m->op->stop (m, p);return error;
Documentation/filesystems/proc.txt +83 -0
95 lines not on this page · raw message
fs/proc/filecache.c new file +614 -0
614 lines not on this page · raw message
fs/Kconfig +23 -0
@@ -1047,6 +1047,29 @@ config CONFIGFS_FS Both sysfs and configfs can and should exist together on the same system. One is not a replacement for the other. +config PROC_FILECACHE + tristate "/proc/filecache support" + default m + depends on PROC_FS + help + This option creates a file /proc/filecache which enables one to + query/drop the cached files in memory. + + A quick start guide: + + # echo -n index > /proc/filecache + # cat /proc/filecache + + # echo -n /bin/bash > /proc/filecache + # cat /proc/filecache + + # echo drop data > /proc/filecache + # echo drop metadata > /proc/filecache + + For more details, please check Documentation/filesystems/proc.txt . + + It can be a handy tool for sysadms and desktop users. + endmenu menu "Miscellaneous filesystems"
fs/proc/Makefile +1 -0
@@ -12,5 +12,6 @@ proc-y += inode.o root.o base.o ge proc-$(CONFIG_PROC_KCORE) += kcore.o proc-$(CONFIG_PROC_VMCORE) += vmcore.o +proc-$(CONFIG_PROC_FILECACHE) += filecache.o proc-$(CONFIG_PROC_DEVICETREE) += proc_devtree.o proc-$(CONFIG_PRINTK) += kmsg.o