reiserfs: kill-the-BKL
PATCH 1/1 reiserfs: kill-the-BKL
For the tip:kill-the-BKL tree.
First of all, I'm sorry for such a monolithic patch. I know it doesn't make it easy to review. But I can't really cut it in several subjects because all bits are needed here for the "Bkl to a mutex" conversion.
Although I first thought that half of it was only there for performance optimizations and could also be decoupled, further tests showed me that they all contribute to correctness and not only performance.
This patch is an attempt to remove the Bkl based locking scheme from reiserfs and is intended.
It is a bit inspired from an old attempt by Peter Zijlstra:
http://lkml.indiana.edu/hypermail/linux/kernel/0704.2/2174.html
The bkl is heavily used in this filesystem to prevent from concurrent write accesses on the filesystem.
Reiserfs makes a deep use of the specific properties of the Bkl:
- It can be acqquired recursively by a same task
- It is released on the schedule() calls and reacquired when schedule() returns
The two properties above are a roadmap for the reiserfs write locking so it's very hard to simply replace it with a common mutex.
- We need a recursive-able locking unless we want to restructure several blocks of the code.
- We need to identify the sites where the bkl was implictly relaxed (schedule, wait, sync, etc...) so that we can in turn release and reacquire our new lock explicitly.
Such implicit releases of the lock are often required to let other resources producer/consumer do their job or we can suffer unexpected starvations or deadlocks.
So the new lock that replaces the bkl here is a per superblock mutex with a specific property: it can be acquired recursively by a same task, like the bkl.
For such purpose, we integrate a lock owner and a lock depth field on the superblock information structure.
The first axis on this patch is to turn reiserfs_write_(un)lock() function into a wrapper to manage this mutex. Also some explicit calls to lock_kernel() have been converted to reiserfs_write_lock() helpers.
The second axis is to find the important blocking sites (schedule...(), wait_on_buffer(), sync_dirty_buffer(), etc...) and then apply an explicit release of the write lock on these locations before blocking. Then we can safely wait for those who can give us resources or those who need some. Typically this is a fight between the current writer, the reiserfs workqueue (aka the async commiter) and the pdflush threads.
The third axis is a consequence of the second. The write lock is usually on top of a lock dependency chain which can include the journal lock, the flush lock or the commit lock. So it's dangerous to release and trying to reacquire the write lock while we still hold other locks.
This is fine with the bkl:
T1 T2
lock_kernel()
mutex_lock(A)
unlock_kernel()
// do something
lock_kernel()
mutex_lock(A) -> already locked by T1
schedule() (and then unlock_kernel())
lock_kernel()
mutex_unlock(A)
....This is not fine with a mutex:
T1 T2
mutex_lock(write)
mutex_lock(A)
mutex_unlock(write)
// do something
mutex_lock(write)
mutex_lock(A) -> already locked by T1
schedule()
mutex_lock(write) -> already locked by T2
deadlockThe solution in this patch is to provide a helper which releases the write lock and sleep a bit if we can't lock a mutex that depend on it. It's another simulation of the bkl behaviour.
The last axis is to locate the fs callbacks that are called with the bkl held, according to Documentation/filesystem/Locking.
Those are:
- reiserfs_remount
- reiserfs_fill_super
- reiserfs_put_super
Reiserfs didn't need to explicitly lock because of the context of these callbacks. But now we must take care of that with the new locking.
After this patch, reiserfs suffers from a slight performance regression (for now). On UP, a high volume write with dd reports an average of 27 MB/s instead of 30 MB/s without the patch applied.
--- fs/reiserfs/Makefile | 2 +- fs/reiserfs/bitmap.c | 2 + fs/reiserfs/dir.c | 8 +++ fs/reiserfs/fix_node.c | 10 +++ fs/reiserfs/inode.c | 23 +++++-- fs/reiserfs/ioctl.c | 6 +- fs/reiserfs/journal.c | 134 +++++++++++++++++++++++++++++++--------- fs/reiserfs/lock.c | 63 +++++++++++++++++++ fs/reiserfs/resize.c | 2 + fs/reiserfs/stree.c | 2 + fs/reiserfs/super.c | 41 ++++++++++++- include/linux/reiserfs_fs.h | 12 ++-- include/linux/reiserfs_fs_sb.h | 9 +++ 13 files changed, 271 insertions(+), 43 deletions(-) create mode 100644 fs/reiserfs/lock.c
fs/reiserfs/Makefile +1 -1
@@ -7,7 +7,7 @@ obj-$(CONFIG_REISERFS_FS) += reiserfs.o reiserfs-objs := bitmap.o do_balan.o namei.o inode.o file.o dir.o fix_node.o \ super.o prints.o objectid.o lbalance.o ibalance.o stree.o \ hashes.o tail_conversion.o journal.o resize.o \ - item_ops.o ioctl.o procfs.o xattr.o + item_ops.o ioctl.o procfs.o xattr.o lock.o ifeq ($(CONFIG_REISERFS_FS_XATTR),y) reiserfs-objs += xattr_user.o xattr_trusted.o
fs/reiserfs/bitmap.c +2 -0
@@ -1256,7 +1256,9 @@ struct buffer_head *reiserfs_read_bitmap_block(struct super_block *sb,else {if (buffer_locked (bh)) {PROC_INFO_INC (sb, scan_bitmap.wait); +reiserfs_write_unlock (sb);__wait_on_buffer (bh); +reiserfs_write_lock (sb); }BUG_ON (!buffer_uptodate (bh));BUG_ON (atomic_read (&bh->b_count) ==0 );
fs/reiserfs/dir.c +8 -0
@@ -174,14 +174,22 @@ int reiserfs_readdir_dentry(struct dentry *dentry, void *dirent,// user space buffer is swapped out. At that time // entry can move to somewhere else memcpy (local_buf, d_name, d_reclen); + +/* +* Since filldir might sleep, we can release +* the write lock here for other waiters +*/ +reiserfs_write_unlock (inode->i_sb);if (filldir (dirent, local_buf, d_reclen, d_off, d_ino,DT_UNKNOWN ) <0 ) { +reiserfs_write_lock (inode->i_sb);if (local_buf != small_buf) {kfree (local_buf); } goto end; } +reiserfs_write_lock (inode->i_sb);if (local_buf != small_buf) {kfree (local_buf); }
fs/reiserfs/fix_node.c +10 -0
@@ -1022,7 +1022,11 @@ static int get_far_parent(struct tree_balance *tb,/* Check whether the common parent is locked. */ if (buffer_locked (*pcom_father)) { + +/* Release the write lock while the buffer is busy */ +reiserfs_write_unlock (tb->tb_sb);__wait_on_buffer (*pcom_father); +reiserfs_write_lock (tb->tb_sb);if (FILESYSTEM_CHANGED_TB (tb)) {brelse (*pcom_father);return REPEAT_SEARCH ;
@@ -1927,7 +1931,9 @@ static int get_direct_parent(struct tree_balance *tb, int h)return REPEAT_SEARCH ;if (buffer_locked (bh)) { +reiserfs_write_unlock (tb->tb_sb);__wait_on_buffer (bh); +reiserfs_write_lock (tb->tb_sb);if (FILESYSTEM_CHANGED_TB (tb))return REPEAT_SEARCH ; }
@@ -2278,7 +2284,9 @@ static int wait_tb_buffers_until_unlocked(struct tree_balance *tb) REPEAT_SEARCH : CARRY_ON; }#endif +reiserfs_write_unlock (tb->tb_sb);__wait_on_buffer (locked); +reiserfs_write_lock (tb->tb_sb);if (FILESYSTEM_CHANGED_TB (tb))return REPEAT_SEARCH ; }
@@ -2349,7 +2357,9 @@ int fix_nodes(int op_mode, struct tree_balance *tb,/* if it possible in indirect_to_direct conversion */ if (buffer_locked (tbS0)) { +reiserfs_write_unlock (tb->tb_sb);__wait_on_buffer (tbS0); +reiserfs_write_lock (tb->tb_sb);if (FILESYSTEM_CHANGED_TB (tb))return REPEAT_SEARCH ; }
fs/reiserfs/inode.c +18 -5
@@ -489,10 +489,14 @@ static int reiserfs_get_blocks_direct_io(struct inode *inode,disappeared */ if (REISERFS_I (inode)->i_flags & i_pack_on_close_mask) {int err; -lock_kernel (); + +reiserfs_write_lock (inode->i_sb); + err =reiserfs_commit_for_inode (inode);REISERFS_I (inode)->i_flags &= ~i_pack_on_close_mask; -unlock_kernel (); + +reiserfs_write_unlock (inode->i_sb); +if (err <0 ) ret = err; }
@@ -616,7 +620,6 @@ int reiserfs_get_block(struct inode *inode, sector_t block,loff_t new_offset = (((loff_t ) block) << inode->i_sb->s_blocksize_bits) +1 ; -/* bad.... */ reiserfs_write_lock (inode->i_sb); version =get_inode_item_key_version (inode);
@@ -997,10 +1000,14 @@ int reiserfs_get_block(struct inode *inode, sector_t block,if (retval) goto failure; } -/* inserting indirect pointers for a hole can take a -** long time. reschedule if needed +/* +* inserting indirect pointers for a hole can take a +* long time. reschedule if needed and also release the write +* lock for others. */ +reiserfs_write_unlock (inode->i_sb);cond_resched (); +reiserfs_write_lock (inode->i_sb); retval =search_for_position_by_key (inode->i_sb, &key, &path);if (retval ==IO_ERROR ) {
@@ -2612,7 +2619,10 @@ int reiserfs_prepare_write(struct file *f, struct page *page,int ret;int old_ref =0 ; +reiserfs_write_unlock (inode->i_sb);reiserfs_wait_on_write_block (inode->i_sb); +reiserfs_write_lock (inode->i_sb); +fix_tail_page_for_writing (page);if (reiserfs_transaction_running (inode->i_sb)) {struct reiserfs_transaction_handle *th;
@@ -2762,7 +2772,10 @@ int reiserfs_commit_write(struct file *f, struct page *page,int update_sd =0 ;struct reiserfs_transaction_handle *th =NULL ; +reiserfs_write_unlock (inode->i_sb);reiserfs_wait_on_write_block (inode->i_sb); +reiserfs_write_lock (inode->i_sb); +if (reiserfs_transaction_running (inode->i_sb)) { th = current->journal_info; }
fs/reiserfs/ioctl.c +4 -2
@@ -141,9 +141,11 @@ long reiserfs_compat_ioctl(struct file *file, unsigned int cmd,default : return -ENOIOCTLCMD; } -lock_kernel (); + +reiserfs_write_lock (inode->i_sb); ret =reiserfs_ioctl (inode, file, cmd, (unsigned long )compat_ptr (arg)); -unlock_kernel (); +reiserfs_write_unlock (inode->i_sb); +return ret; }#endif
fs/reiserfs/journal.c +105 -29
297 lines not on this page · raw message
fs/reiserfs/lock.c new file +63 -0
63 lines not on this page · raw message
fs/reiserfs/resize.c +2 -0
@@ -142,7 +142,9 @@ int reiserfs_resize(struct super_block *s, unsigned long block_count_new)set_buffer_uptodate (bh);mark_buffer_dirty (bh); +reiserfs_write_unlock (s);sync_dirty_buffer (bh); +reiserfs_write_lock (s);// update bitmap_info stuff bitmap[i].free_count =sb_blocksize (sb) *8 -1 ;brelse (bh);
fs/reiserfs/stree.c +2 -0
@@ -629,7 +629,9 @@ int search_by_key(struct super_block *sb, const struct cpu_key *key, /* Key to ssearch_by_key_reada (sb, reada_bh, reada_blocks, reada_count);ll_rw_block (READ ,1 , &bh); +reiserfs_write_unlock (sb);wait_on_buffer (bh); +reiserfs_write_lock (sb);if (!buffer_uptodate (bh)) goto io_error; }else {
fs/reiserfs/super.c +40 -1
99 lines not on this page · raw message
include/linux/reiserfs_fs.h +7 -5
@@ -52,11 +52,13 @@#define REISERFS_IOC32_GETVERSION FS_IOC32_GETVERSION#define REISERFS_IOC32_SETVERSION FS_IOC32_SETVERSION -/* Locking primitives */ -/* Right now we are still falling back to (un)lock_kernel, but eventually that -would evolve into real per-fs locks */ -#define reiserfs_write_lock ( sb ) lock_kernel() -#define reiserfs_write_unlock ( sb ) unlock_kernel() +/* +* Locking primitives. The write lock is a per superblock +* special mutex that has properties close to the Big Kernel Lock +* which was used in the previous locking scheme. +*/ +void reiserfs_write_lock (struct super_block *s); +void reiserfs_write_unlock (struct super_block *s);struct fid ;
include/linux/reiserfs_fs_sb.h +9 -0
@@ -7,6 +7,8 @@#ifdef __KERNEL__#include <linux/workqueue.h> #include <linux/rwsem.h> +#include <linux/mutex.h> +#include <linux/sched.h> #endif typedef enum {
@@ -355,6 +357,13 @@ struct reiserfs_sb_info {struct reiserfs_journal *s_journal;/* pointer to journal information */ unsigned short s_mount_state;/* reiserfs state (valid, invalid) */ +/* Serialize writers access, replace the old bkl */ +struct mutex lock; +/* Owner of the lock (can be recursive) */ +struct task_struct *lock_owner; +/* Depth of the lock, start from -1 like the bkl */ +int lock_depth; +/* Comment? -Hans */ void (*end_io_handler) (struct buffer_head *,int );hashf_t s_hash_function;/* pointer to function which is used
Signature · 1 line
1.6.1
Note: I forgot to put the RFC prefix for this patch.
Frederic Weisbecker.
fs/reiserfs/lock.c line 62
Quoted 11 lines · Frederic Weisbecker
fs/reiserfs/lock.c new file +11
-0+/* +* Utility function to force a BUG if it is called without the superblock +* write lock held. caller is the string printed just before calling BUG() +*/ +void reiserfs_check_lock_depth (struct super_block *sb,char *caller) +{ +struct reiserfs_sb_info *sb_i =REISERFS_SB (sb); + +if (sb_i->lock_depth <0 ) +reiserfs_panic (sb,"%s called without kernel lock held %d" , + caller);
s/kernel lock/sb write lock/
fs/reiserfs/super.c static void reiserfs_put_super(struct super_block *s) · line 512
Quoted 4 lines
fs/reiserfs/super.c +2
-0+reiserfs_write_unlock (s); +mutex_destroy (&REISERFS_SB (s)->lock);kfree (s->s_fs_info); s->s_fs_info =NULL ;
ah, mutex_destroy() - it's a small detail but still nice ;-)
Anyway, it still looks good to me, after the earlier patch i reviewed. Would be nice to have the testing feedback of reiserfs users, and the locking review from anyone who knows this code.
Ingo
Hi Frederic
I've applied your patch.
This is UP system.
This warning seems related to your patch.
[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>] might_fault+0x52/0xa0
but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (sysfs_mutex){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160
[<c014ffe4>] lock_acquire+0x74/0x90
[<c0448d70>] __mutex_lock_common+0x50/0x430
[<c0449203>] mutex_lock_nested+0x33/0x40
[<c01cd17c>] sysfs_addrm_start+0x2c/0xc0
[<c01cdd30>] create_dir+0x40/0x80
[<c01cdd9b>] sysfs_create_dir+0x2b/0x50
[<c02c3121>] kobject_add_internal+0xc1/0x220
[<c02c3351>] kobject_add_varg+0x31/0x50
[<c02c33cc>] kobject_add+0x2c/0x60
[<c0337a52>] device_add+0xe2/0x540
[<c01cac02>] add_partition+0x112/0x210
[<c01cb2ef>] rescan_partitions+0x23f/0x320
[<c01b0f13>] __blkdev_get+0x153/0x330
[<c01b10fa>] blkdev_get+0xa/0x10
[<c01caaca>] register_disk+0x10a/0x130
[<c02bd5c9>] add_disk+0x109/0x170
[<c034e85a>] sd_probe_async+0x19a/0x280
[<c0143900>] async_thread+0xd0/0x230
[<c013d86a>] kthread+0x3a/0x70
[<c0103677>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff-> #2 (&bdev->bd_mutex){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160
[<c014ffe4>] lock_acquire+0x74/0x90
[<c0448d70>] __mutex_lock_common+0x50/0x430
[<c0449203>] mutex_lock_nested+0x33/0x40
[<c01b0df3>] __blkdev_get+0x33/0x330
[<c01b10fa>] blkdev_get+0xa/0x10
[<c01b1261>] open_by_devnum+0x21/0x50
[<c01fed2e>] journal_init+0x22e/0x19a0
[<c01ea881>] reiserfs_fill_super+0x3a1/0x1050
[<c018d053>] get_sb_bdev+0x133/0x160
[<c01e8b21>] get_super_block+0x21/0x30
[<c018bd7a>] vfs_kern_mount+0x3a/0xa0
[<c018be39>] do_kern_mount+0x39/0xd0
[<c01a2063>] do_mount+0x3a3/0x790
[<c01a24d9>] sys_mount+0x89/0xc0
[<c06fac3d>] mount_block_root+0xd0/0x261
[<c06fae27>] mount_root+0x59/0x5f
[<c06faf1c>] prepare_namespace+0xef/0x171
[<c06fa4e3>] kernel_init+0xd5/0xee
[<c0103677>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160
[<c014ffe4>] lock_acquire+0x74/0x90
[<c0448d70>] __mutex_lock_common+0x50/0x430
[<c0449203>] mutex_lock_nested+0x33/0x40
[<c020375e>] reiserfs_write_lock+0x1e/0x30
[<c01df2e6>] reiserfs_get_block+0x66/0x14f0
[<c01b32f0>] do_mpage_readpage+0x340/0x490
[<c01b352e>] mpage_readpages+0x9e/0xe0
[<c01dc729>] reiserfs_readpages+0x19/0x20
[<c01696ff>] __do_page_cache_readahead+0x13f/0x200
[<c016980e>] do_page_cache_readahead+0x4e/0x70
[<c0162caf>] filemap_fault+0x30f/0x430
[<c0175f6d>] __do_fault+0x3d/0x360
[<c0176aa0>] handle_mm_fault+0x100/0x590
[<c011aff1>] do_page_fault+0x121/0x280
[<c044b40a>] error_code+0x6a/0x70
[<c01bf810>] load_elf_binary+0x9d0/0x19f0
[<c018fde3>] search_binary_handler+0x163/0x2f0
[<c019018d>] do_execve+0x21d/0x2b0
[<c01016de>] sys_execve+0x3e/0x70
[<c0102ec9>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff-> #0 (&mm->mmap_sem){++++++}:
[<c014fbde>] __lock_acquire+0xdce/0x1160
[<c014ffe4>] lock_acquire+0x74/0x90
[<c0173f83>] might_fault+0x83/0xa0
[<c02ca436>] copy_to_user+0x36/0x130
[<c0197b24>] filldir64+0xa4/0xf0
[<c01ccdf6>] sysfs_readdir+0x116/0x200
[<c0197d9e>] vfs_readdir+0x7e/0xa0
[<c0197e29>] sys_getdents64+0x69/0xb0
[<c0102e48>] sysenter_do_call+0x12/0x36
[<ffffffff>] 0xffffffffother info that might help us debug this:
2 locks held by udevadm/1239:
#0: (&type->i_mutex_dir_key){+.+.+.}, at: [<c0197d6f>] vfs_readdir+0x4f/0xa0 #1: (sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200
stack backtrace: Pid: 1239, comm: udevadm Not tainted 2.6.29-10247-g8e2c4f2-dirty #2 Call Trace: [<c0447c92>] ? printk+0x18/0x1e [<c014db5c>] print_circular_bug_tail+0x8c/0xe0 [<c014fbde>] __lock_acquire+0xdce/0x1160 [<c014bb1d>] ? put_lock_stats+0xd/0x30 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0173f52>] ? might_fault+0x52/0xa0 [<c0173f83>] might_fault+0x83/0xa0 [<c0173f52>] ? might_fault+0x52/0xa0 [<c02ca436>] copy_to_user+0x36/0x130 [<c0197b24>] filldir64+0xa4/0xf0 [<c01ccdf6>] sysfs_readdir+0x116/0x200 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197e29>] sys_getdents64+0x69/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36
Quoted 7 lines · Alexander Beregalov
Hi Frederic
I've applied your patch.
This is UP system.
This warning seems related to your patch.
Quoted 8 lines
[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>]might_fault+0x52/0xa0but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>]sysfs_readdir+0x56/0x200
Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?
Thanks,
Frederic.
Quoted 114 lines
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (sysfs_mutex){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0448d70>] __mutex_lock_common+0x50/0x430 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c01cd17c>] sysfs_addrm_start+0x2c/0xc0 [<c01cdd30>] create_dir+0x40/0x80 [<c01cdd9b>] sysfs_create_dir+0x2b/0x50 [<c02c3121>] kobject_add_internal+0xc1/0x220 [<c02c3351>] kobject_add_varg+0x31/0x50 [<c02c33cc>] kobject_add+0x2c/0x60 [<c0337a52>] device_add+0xe2/0x540 [<c01cac02>] add_partition+0x112/0x210 [<c01cb2ef>] rescan_partitions+0x23f/0x320 [<c01b0f13>] __blkdev_get+0x153/0x330 [<c01b10fa>] blkdev_get+0xa/0x10 [<c01caaca>] register_disk+0x10a/0x130 [<c02bd5c9>] add_disk+0x109/0x170 [<c034e85a>] sd_probe_async+0x19a/0x280 [<c0143900>] async_thread+0xd0/0x230 [<c013d86a>] kthread+0x3a/0x70 [<c0103677>] kernel_thread_helper+0x7/0x10 [<ffffffff>] 0xffffffff-> #2 (&bdev->bd_mutex){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0448d70>] __mutex_lock_common+0x50/0x430 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c01b0df3>] __blkdev_get+0x33/0x330 [<c01b10fa>] blkdev_get+0xa/0x10 [<c01b1261>] open_by_devnum+0x21/0x50 [<c01fed2e>] journal_init+0x22e/0x19a0 [<c01ea881>] reiserfs_fill_super+0x3a1/0x1050 [<c018d053>] get_sb_bdev+0x133/0x160 [<c01e8b21>] get_super_block+0x21/0x30 [<c018bd7a>] vfs_kern_mount+0x3a/0xa0 [<c018be39>] do_kern_mount+0x39/0xd0 [<c01a2063>] do_mount+0x3a3/0x790 [<c01a24d9>] sys_mount+0x89/0xc0 [<c06fac3d>] mount_block_root+0xd0/0x261 [<c06fae27>] mount_root+0x59/0x5f [<c06faf1c>] prepare_namespace+0xef/0x171 [<c06fa4e3>] kernel_init+0xd5/0xee [<c0103677>] kernel_thread_helper+0x7/0x10 [<ffffffff>] 0xffffffff-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[<c014fb2a>] __lock_acquire+0xd1a/0x1160 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0448d70>] __mutex_lock_common+0x50/0x430 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0 [<c01b32f0>] do_mpage_readpage+0x340/0x490 [<c01b352e>] mpage_readpages+0x9e/0xe0 [<c01dc729>] reiserfs_readpages+0x19/0x20 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200 [<c016980e>] do_page_cache_readahead+0x4e/0x70 [<c0162caf>] filemap_fault+0x30f/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c011aff1>] do_page_fault+0x121/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c01bf810>] load_elf_binary+0x9d0/0x19f0 [<c018fde3>] search_binary_handler+0x163/0x2f0 [<c019018d>] do_execve+0x21d/0x2b0 [<c01016de>] sys_execve+0x3e/0x70 [<c0102ec9>] syscall_call+0x7/0xb [<ffffffff>] 0xffffffff-> #0 (&mm->mmap_sem){++++++}:
[<c014fbde>] __lock_acquire+0xdce/0x1160 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0173f83>] might_fault+0x83/0xa0 [<c02ca436>] copy_to_user+0x36/0x130 [<c0197b24>] filldir64+0xa4/0xf0 [<c01ccdf6>] sysfs_readdir+0x116/0x200 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197e29>] sys_getdents64+0x69/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36 [<ffffffff>] 0xffffffffother info that might help us debug this:
2 locks held by udevadm/1239:
#0: (&type->i_mutex_dir_key){+.+.+.}, at: [<c0197d6f>]vfs_readdir+0x4f/0xa0#1: (sysfs_mutex){+.+.+.}, at: [<c01ccd36>]sysfs_readdir+0x56/0x200stack backtrace: Pid: 1239, comm: udevadm Not tainted 2.6.29-10247-g8e2c4f2-dirty #2 Call Trace: [<c0447c92>] ? printk+0x18/0x1e [<c014db5c>] print_circular_bug_tail+0x8c/0xe0 [<c014fbde>] __lock_acquire+0xdce/0x1160 [<c014bb1d>] ? put_lock_stats+0xd/0x30 [<c014ffe4>] lock_acquire+0x74/0x90 [<c0173f52>] ? might_fault+0x52/0xa0 [<c0173f83>] might_fault+0x83/0xa0 [<c0173f52>] ? might_fault+0x52/0xa0 [<c02ca436>] copy_to_user+0x36/0x130 [<c0197b24>] filldir64+0xa4/0xf0 [<c01ccdf6>] sysfs_readdir+0x116/0x200 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197e29>] sys_getdents64+0x69/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36
Quoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Alexander Beregalov
Hi Frederic
I've applied your patch.
This is UP system.
This warning seems related to your patch.[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>]might_fault+0x52/0xa0
Quoted 7 lines
Quoted 2 lines
but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>]sysfs_readdir+0x56/0x200Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?
[Sent in private mail]
This should be related.
System is hung after dbench testing for some time.
SysRq : Emergency Sync
SysRq : Show Blocked State
task PC stack pid father
events/0 D 00000000 5672 6 2
f7067ee8 00000046 00000046 00000000 00000000 f7056720 f70569ac b6acece3 00000b44 00000000 f70569ac c018c1df 00000202 00000202 f6982a60 f7056720 f7067f30 c0448e3a 00000000 00000002 00000000 c018c1df 00000000 0018c3b2
Call Trace: [<c018c1df>] ? lock_super+0x1f/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c018c1df>] ? lock_super+0x1f/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c018c1df>] ? lock_super+0x1f/0x30 [<c018c1df>] lock_super+0x1f/0x30 [<c018c3b9>] sync_supers+0x69/0xc0 [<c01a922c>] do_sync+0x1c/0x60 [<c013a0e6>] ? worker_thread+0xe6/0x250 [<c01a927d>] do_sync_work+0xd/0x20 [<c013a148>] worker_thread+0x148/0x250 [<c013a0e6>] ? worker_thread+0xe6/0x250 [<c01a9270>] ? do_sync_work+0x0/0x20 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c013a000>] ? worker_thread+0x0/0x250 [<c013d86a>] kthread+0x3a/0x70 [<c013d830>] ? kthread+0x0/0x70 [<c0103677>] kernel_thread_helper+0x7/0x10
metalog D 00000000 4936 2183 1
f7161cec 00000046 00000046 00000000 00000000 f6b8d280 f6b8d50c 1eba9be4 00000b44 00000000 f6b8d50c c020375e 00000202 00000202 f670a014 f6b8d280 f7161d34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0018440c
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aaad>] sys_write+0x3d/0x70 [<c0102e48>] sysenter_do_call+0x12/0x36
svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197b70>] ? filldir+0x0/0xd0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197b70>] ? filldir+0x0/0xd0 [<c0197ed8>] sys_getdents+0x68/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36
multilog D 00000000 4800 2236 2228
f731bcec 00000046 00000046 00000000 00000000 f6ba8000 f6ba828c 50105c29 00000b26 00000000 f6ba828c c020375e 00000202 00000202 f670a014 f6ba8000 f731bd34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0014ce9d
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c0173f52>] ? might_fault+0x52/0xa0 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aaad>] sys_write+0x3d/0x70 [<c0102e48>] sysenter_do_call+0x12/0x36
agetty D 00000000 5432 2441 1
f73cbba8 00000046 00000046 00000000 00000000 f73dd280 f73dd50c 6a069cd5 00000b33 00000000 f73dd50c c020375e 00000202 00000202 f670a014 f73dd280 f73cbbf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000246
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c044b07d>] ? _spin_unlock_irq+0x2d/0x50 [<c0124b02>] ? finish_task_switch+0x62/0xb0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c0124aa0>] ? finish_task_switch+0x0/0xb0 [<c044b2df>] ? __reacquire_kernel_lock+0x2f/0x40 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c0332d78>] ? serial8250_tx_empty+0x18/0x50 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0 [<c01b32f0>] do_mpage_readpage+0x340/0x490 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0 [<c016a415>] ? __lru_cache_add+0x65/0xa0 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01b352e>] mpage_readpages+0x9e/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c04480b8>] ? __schedule+0x2f8/0x4f0 [<c01dc710>] ? reiserfs_readpages+0x0/0x20 [<c01dc729>] reiserfs_readpages+0x19/0x20 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200 [<c016980e>] do_page_cache_readahead+0x4e/0x70 [<c0162caf>] filemap_fault+0x30f/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280
screen D 00000000 5244 2529 2528
f612fd90 00000046 00000046 00000000 00000000 f6babde0 f6bac06c 33c73fab 00000b1c 00000000 f6bac06c c0193bf3 00000202 00000202 f6f9615c f6babde0 f612fdd8 c0448e3a 00000000 00000002 00000000 c0193bf3 00000000 0012fed0
Call Trace: [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0193bf3>] do_lookup+0xd3/0x1c0 [<c0193e49>] __link_path_walk+0x169/0xcc0 [<c0194b53>] path_walk+0x53/0xb0 [<c0194c8b>] do_path_lookup+0x8b/0x170 [<c01958ea>] user_path_at+0x4a/0x80 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c018e471>] vfs_stat_fd+0x21/0x50 [<c018e571>] vfs_stat+0x11/0x20 [<c018e594>] sys_stat64+0x14/0x30 [<c018abdf>] ? vfs_read+0xff/0x140 [<c0317d70>] ? tty_read+0x0/0xb0 [<c018acdd>] ? sys_read+0x3d/0x70 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
rtorrent D 00000000 4084 2543 2542
f6325d88 00000046 00000046 00000000 00000000 f6b894a0 f6b8972c d79a38dd 00000af9 00000000 f6b8972c c020375e 00000202 00000202 f670a014 f6b894a0 f6325dd0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197a80>] ? filldir64+0x0/0xf0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197e29>] sys_getdents64+0x69/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36
bash D 00000000 5788 2546 2529
f585dcd8 00000046 00000046 00000000 00000000 f6b8bde0 f6b8c06c 188b381a 00000b1b 00000000 f6b8c06c c020375e 00000202 00000202 f670a014 f6b8bde0 f585dd20 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01db389>] reiserfs_lookup+0x59/0x170 [<c019b923>] ? d_alloc+0x123/0x1b0 [<c0193cb1>] do_lookup+0x191/0x1c0 [<c0194305>] __link_path_walk+0x625/0xcc0 [<c0194b53>] path_walk+0x53/0xb0 [<c0194c8b>] do_path_lookup+0x8b/0x170 [<c01958ea>] user_path_at+0x4a/0x80 [<c018dda4>] ? cp_new_stat64+0xe4/0x100 [<c018e471>] vfs_stat_fd+0x21/0x50 [<c018e571>] vfs_stat+0x11/0x20 [<c018e594>] sys_stat64+0x14/0x30 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4816 20778 1
f44bbd1c 00000046 00000046 00000000 00000000 ce105280 ce10550c d12c8ac7 00000af8 00000000 ce10550c c020375e 00000202 00000202 f670a014 ce105280 f44bbd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766a00
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c018570d>] ? check_object+0x13d/0x200 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4660 20779 1
f44b3de4 00000046 00000046 00000000 00000000 ce106720 ce1069ac d12c836c 00000af8 00000000 ce1069ac c020375e 00000202 00000202 f670a014 ce106720 f44b3e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767400
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c04480b8>] ? __schedule+0x2f8/0x4f0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0133420>] ? process_timeout+0x0/0x10 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4724 20780 1
f6389de4 00000046 00000046 00000000 00000000 ce102940 ce102bcc d12c7533 00000af8 00000000 ce102bcc c020375e 00000202 00000202 f670a014 ce102940 f6389e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c0169c29>] ? pagevec_lookup_tag+0x29/0x40 [<c0168241>] ? write_cache_pages+0xd1/0x350 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4944 20781 1
f589dc84 00000046 00000046 00000000 00000000 ce1014a0 ce10172c d0f2ddbd 00000af8 00000000 ce10172c c020375e 00000202 00000202 f670a014 ce1014a0 f589dccc c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fe400>] journal_begin+0x80/0x110 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e9686>] reiserfs_dirty_inode+0x36/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c02ca436>] ? copy_to_user+0x36/0x130 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aa6b>] sys_pwrite64+0x7b/0x80 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4832 20782 1
f4617a64 00000046 00000046 00000000 00000000 ce103de0 ce10406c d12ca393 00000af8 00000000 ce10406c c020375e 00000202 00000202 f670a014 ce103de0 f4617aac c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767900
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c01fe400>] journal_begin+0x80/0x110 [<c01fe4c2>] ? reiserfs_persistent_transaction+0x32/0x90 [<c01fe4d1>] reiserfs_persistent_transaction+0x41/0x90 [<c01df460>] reiserfs_get_block+0x1e0/0x14f0 [<c018dbbe>] ? inode_get_bytes+0x1e/0x50 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c018570d>] ? check_object+0x13d/0x200 [<c01ab2c3>] ? create_empty_buffers+0x33/0xa0 [<c01ad424>] __block_prepare_write+0x1a4/0x380 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01ad7a8>] block_write_begin+0x48/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01ddfbc>] reiserfs_write_begin+0xec/0x200 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01618f5>] generic_file_buffered_write+0x155/0x2c0 [<c0163520>] __generic_file_aio_write_nolock+0x1c0/0x520 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c0173f52>] ? might_fault+0x52/0xa0 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c02ca436>] ? copy_to_user+0x36/0x130 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aa6b>] sys_pwrite64+0x7b/0x80 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4764 20783 1
f5833e48 00000046 00000046 00000000 00000000 f73de720 f73de9ac d12c96e7 00000af8 00000000 f73de9ac c020375e 00000202 00000202 f670a014 f73de720 f5833e90 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4600 20785 1
f460bd1c 00000046 00000046 00000000 00000000 f6b88000 f6b8828c d12c9d18 00000af8 00000000 f6b8828c c020375e 00000202 00000202 f670a014 f6b88000 f460bd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4548 20786 1
f5903d1c 00000046 00000046 00000000 00000000 f6960000 f696028c d12fcd2a 00000af8 00000000 f696028c c020375e 00000202 00000202 f670a014 f6960000 f5903d64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c018570d>] ? check_object+0x13d/0x200 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
dbench D 00000000 4704 20787 1
f4557d60 00000046 00000046 00000000 00000000 f69614a0 f696172c d12c7cad 00000af8 00000000 f696172c c020375e 00000202 00000202 f670a014 f69614a0 f4557da8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767b80
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c01862b1>] ? __slab_alloc+0xc1/0x5f0 [<c02c4f65>] ? __prop_inc_single+0x65/0x80 [<c01fbfc3>] get_list_bitmap+0x63/0xb0 [<c01fdd5f>] do_journal_end+0xd0f/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36
pdflush D 00000000 6080 20790 2
f09bfe3c 00000046 00000046 00000000 00000000 f7088000 f708828c d12caa96 00000af8 00000000 f708828c c020375e 00000202 00000202 f670a014 f7088000 f09bfe84 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766c80
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fe400>] journal_begin+0x80/0x110 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e95fe>] reiserfs_sync_fs+0x3e/0x80 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c018c1df>] ? lock_super+0x1f/0x30 [<c01e964d>] reiserfs_write_super+0xd/0x10 [<c018c3d4>] sync_supers+0x84/0xc0 [<c0169330>] ? pdflush+0x0/0x1c0 [<c016860b>] wb_kupdate+0x2b/0x100 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c0169330>] ? pdflush+0x0/0x1c0 [<c01693f8>] pdflush+0xc8/0x1c0 [<c01685e0>] ? wb_kupdate+0x0/0x100 [<c013d86a>] kthread+0x3a/0x70 [<c013d830>] ? kthread+0x0/0x70 [<c0103677>] kernel_thread_helper+0x7/0x10
git-daemon D 00000000 6696 20791 2273
f3705ba8 00000046 00000046 00000000 00000000 f66fe720 f66fe9ac 4a31216d 00000afb 00000000 f66fe9ac c020375e 00000202 00000202 f670a014 f66fe720 f3705bf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c018440c>] ? slab_pad_check+0x3c/0x120 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c01666ee>] ? get_page_from_freelist+0x41e/0x530 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0 [<c01b32f0>] do_mpage_readpage+0x340/0x490 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0 [<c016a415>] ? __lru_cache_add+0x65/0xa0 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01b352e>] mpage_readpages+0x9e/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01dc710>] ? reiserfs_readpages+0x0/0x20 [<c01dc729>] reiserfs_readpages+0x19/0x20 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200 [<c016980e>] do_page_cache_readahead+0x4e/0x70 [<c0162caf>] filemap_fault+0x30f/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280
git-daemon D f63ede14 6888 20792 2273
f63ede20 00000046 00000000 f63ede14 00000046 f66fa940 f66fabcc 977b8ab0 00000afe 00000000 f66fabcc f63ede14 c014e52b c057e5a0 000003ae f63ede70 f63ede28 c04482c2 f63ede34 c04482f4 f63ede68 f63ede3c c0160d95 f63ede5c
Call Trace: [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c04482c2>] schedule+0x12/0x30 [<c04482f4>] io_schedule+0x14/0x20 [<c0160d95>] sync_page+0x55/0x60 [<c0448881>] __wait_on_bit_lock+0x41/0x80 [<c0160d40>] ? sync_page+0x0/0x60 [<c0160d18>] __lock_page+0x88/0xa0 [<c013daf0>] ? wake_bit_function+0x0/0x60 [<c0160eb9>] find_lock_page+0x39/0x70 [<c0162ad7>] filemap_fault+0x137/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280
fcron D 00000000 7168 20797 2257
f5a0fd80 00000046 00000046 00000000 00000000 f6952940 f6952bcc df71112a 00000b25 00000000 f6952bcc c020375e 00000202 00000202 f670a014 f6952940 f5a0fdc8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01db389>] reiserfs_lookup+0x59/0x170 [<c019b923>] ? d_alloc+0x123/0x1b0 [<c019275f>] __lookup_hash+0xcf/0x130 [<c01927e7>] lookup_hash+0x27/0x30 [<c0195cb5>] do_filp_open+0x245/0x7a0 [<c019ed68>] ? alloc_fd+0x28/0xf0 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c019ee18>] ? alloc_fd+0xd8/0xf0 [<c0188879>] do_sys_open+0x59/0xf0 [<c0188979>] sys_open+0x29/0x40 [<c0102e48>] sysenter_do_call+0x12/0x36 Sched Debug Version: v0.09, 2.6.29-10247-g8e2c4f2-dirty #2 now at 12468457.474391 msecs .jiffies : 1216523 .sysctl_sched_latency : 20.000000 .sysctl_sched_min_granularity : 4.000000 .sysctl_sched_wakeup_granularity : 5.000000 .sysctl_sched_child_runs_first : 0.000001 .sysctl_sched_features : 113917
cpu#0, 1830.032 MHz
.nr_running : 1 .load : 1024 .nr_switches : 6163991 .nr_load_updates : 582498 .nr_uninterruptible : 21 .next_balance : 0.000000 .curr->pid : 20784 .clock : 12465239.317891 .cpu_load[0] : 1024 .cpu_load[1] : 1024 .cpu_load[2] : 1024 .cpu_load[3] : 1024 .cpu_load[4] : 1024 .yld_count : 0 .sched_switch : 0 .sched_count : 778698744 .sched_goidle : 307156 .ttwu_count : 0 .ttwu_local : 0 .bkl_count : 275
cfs_rq[0]:
.exec_clock : 1507714.391289 .MIN_vruntime : 0.000001 .min_vruntime : 1444293.051451 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 1 .load : 1024 .nr_spread_over : 35
rt_rq[0]:
.rt_nr_running : 0 .rt_throttled : 0 .rt_time : 0.000000 .rt_runtime : 950.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R dbench 20784 1444293.051451 4569 120 1444293.051451 401016.637062 16471.092414
INFO: task svscan:2223 blocked for more than 480 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000
Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197b70>] ? filldir+0x0/0xd0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197b70>] ? filldir+0x0/0xd0 [<c0197ed8>] sys_getdents+0x68/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36
INFO: lockdep is turned off.
Quoted 19 lines · Alexander Beregalov
Quoted 18 lines · Frederic Weisbecker
Quoted 15 lines · Alexander Beregalov
Hi Frederic
I've applied your patch.
This is UP system.
This warning seems related to your patch.[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>]might_fault+0x52/0xa0but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>]sysfs_readdir+0x56/0x200
Quoted 8 lines
Quoted 3 lines
Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?[Sent in private mail]
This should be related.
System is hung after dbench testing for some time.
Ok, I can reproduce these hangs with dbench. I will investigate on that and try to fix it soon.
Thanks a lot for this report,
Frederic.
Quoted 666 lines
SysRq : Emergency Sync
SysRq : Show Blocked State
task PC stack pid father
events/0 D 00000000 5672 6 2
f7067ee800000046 00000046 00000000 00000000f7056720f70569acb6acece300000b4400000000f70569acc018c1df00000202 00000202f6982a60f7056720f7067f30c0448e3a00000000 00000002 00000000c018c1df000000000018c3b2Call Trace: [<c018c1df>] ? lock_super+0x1f/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c018c1df>] ? lock_super+0x1f/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c018c1df>] ? lock_super+0x1f/0x30 [<c018c1df>] lock_super+0x1f/0x30 [<c018c3b9>] sync_supers+0x69/0xc0 [<c01a922c>] do_sync+0x1c/0x60 [<c013a0e6>] ? worker_thread+0xe6/0x250 [<c01a927d>] do_sync_work+0xd/0x20 [<c013a148>] worker_thread+0x148/0x250 [<c013a0e6>] ? worker_thread+0xe6/0x250 [<c01a9270>] ? do_sync_work+0x0/0x20 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c013a000>] ? worker_thread+0x0/0x250 [<c013d86a>] kthread+0x3a/0x70 [<c013d830>] ? kthread+0x0/0x70 [<c0103677>] kernel_thread_helper+0x7/0x10metalog D 00000000 4936 2183 1
f7161cec00000046 00000046 00000000 00000000f6b8d280f6b8d50c1eba9be400000b4400000000f6b8d50cc020375e00000202 00000202f670a014f6b8d280f7161d34c0448e3a00000000 00000002 00000000c020375e000000000018440cCall Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aaad>] sys_write+0x3d/0x70 [<c0102e48>] sysenter_do_call+0x12/0x36svscan D 00000000 5384 2223 1
f731dd8c00000046 00000046 00000000 00000000f6740000f674028ceaa8d98d00000af900000000f674028cc020375e00000202 00000202f670a014f6740000f731ddd4c0448e3a00000000 00000002 00000000c020375e00000000 00740000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197b70>] ? filldir+0x0/0xd0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197b70>] ? filldir+0x0/0xd0 [<c0197ed8>] sys_getdents+0x68/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36multilog D 00000000 4800 2236 2228
f731bcec00000046 00000046 00000000 00000000f6ba8000f6ba828c50105c2900000b2600000000f6ba828cc020375e00000202 00000202f670a014f6ba8000f731bd34c0448e3a00000000 00000002 00000000c020375e000000000014ce9dCall Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c0173f52>] ? might_fault+0x52/0xa0 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aaad>] sys_write+0x3d/0x70 [<c0102e48>] sysenter_do_call+0x12/0x36agetty D 00000000 5432 2441 1
f73cbba800000046 00000046 00000000 00000000f73dd280f73dd50c6a069cd500000b3300000000f73dd50cc020375e00000202 00000202f670a014f73dd280f73cbbf0c0448e3a00000000 00000002 00000000c020375e00000000 00000246Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c044b07d>] ? _spin_unlock_irq+0x2d/0x50 [<c0124b02>] ? finish_task_switch+0x62/0xb0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c0124aa0>] ? finish_task_switch+0x0/0xb0 [<c044b2df>] ? __reacquire_kernel_lock+0x2f/0x40 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c0332d78>] ? serial8250_tx_empty+0x18/0x50 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0 [<c01b32f0>] do_mpage_readpage+0x340/0x490 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0 [<c016a415>] ? __lru_cache_add+0x65/0xa0 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01b352e>] mpage_readpages+0x9e/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c04480b8>] ? __schedule+0x2f8/0x4f0 [<c01dc710>] ? reiserfs_readpages+0x0/0x20 [<c01dc729>] reiserfs_readpages+0x19/0x20 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200 [<c016980e>] do_page_cache_readahead+0x4e/0x70 [<c0162caf>] filemap_fault+0x30f/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280screen D 00000000 5244 2529 2528
f612fd9000000046 00000046 00000000 00000000f6babde0f6bac06c33c73fab00000b1c00000000f6bac06cc0193bf300000202 00000202f6f9615cf6babde0f612fdd8c0448e3a00000000 00000002 00000000c0193bf3000000000012fed0Call Trace: [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c0193bf3>] ? do_lookup+0xd3/0x1c0 [<c0193bf3>] do_lookup+0xd3/0x1c0 [<c0193e49>] __link_path_walk+0x169/0xcc0 [<c0194b53>] path_walk+0x53/0xb0 [<c0194c8b>] do_path_lookup+0x8b/0x170 [<c01958ea>] user_path_at+0x4a/0x80 [<c01b42ba>] ? dnotify_parent+0x2a/0x80 [<c018e471>] vfs_stat_fd+0x21/0x50 [<c018e571>] vfs_stat+0x11/0x20 [<c018e594>] sys_stat64+0x14/0x30 [<c018abdf>] ? vfs_read+0xff/0x140 [<c0317d70>] ? tty_read+0x0/0xb0 [<c018acdd>] ? sys_read+0x3d/0x70 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36rtorrent D 00000000 4084 2543 2542
f6325d8800000046 00000046 00000000 00000000f6b894a0f6b8972cd79a38dd00000af900000000f6b8972cc020375e00000202 00000202f670a014f6b894a0f6325dd0c0448e3a00000000 00000002 00000000c020375e00000000 00000000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197a80>] ? filldir64+0x0/0xf0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197a80>] ? filldir64+0x0/0xf0 [<c0197e29>] sys_getdents64+0x69/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36bash D 00000000 5788 2546 2529
f585dcd800000046 00000046 00000000 00000000f6b8bde0f6b8c06c188b381a00000b1b00000000f6b8c06cc020375e00000202 00000202f670a014f6b8bde0f585dd20c0448e3a00000000 00000002 00000000c020375e00000000 00000000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01db389>] reiserfs_lookup+0x59/0x170 [<c019b923>] ? d_alloc+0x123/0x1b0 [<c0193cb1>] do_lookup+0x191/0x1c0 [<c0194305>] __link_path_walk+0x625/0xcc0 [<c0194b53>] path_walk+0x53/0xb0 [<c0194c8b>] do_path_lookup+0x8b/0x170 [<c01958ea>] user_path_at+0x4a/0x80 [<c018dda4>] ? cp_new_stat64+0xe4/0x100 [<c018e471>] vfs_stat_fd+0x21/0x50 [<c018e571>] vfs_stat+0x11/0x20 [<c018e594>] sys_stat64+0x14/0x30 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4816 20778 1
f44bbd1c00000046 00000046 00000000 00000000ce105280ce10550cd12c8ac700000af800000000ce10550cc020375e00000202 00000202f670a014ce105280f44bbd64c0448e3a00000000 00000002 00000000c020375e0000000000766a00Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c018570d>] ? check_object+0x13d/0x200 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4660 20779 1
f44b3de400000046 00000046 00000000 00000000ce106720ce1069acd12c836c00000af800000000ce1069acc020375e00000202 00000202f670a014ce106720f44b3e2cc0448e3a00000000 00000002 00000000c020375e00000000 00767400Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c04480b8>] ? __schedule+0x2f8/0x4f0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0133420>] ? process_timeout+0x0/0x10 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4724 20780 1
f6389de400000046 00000046 00000000 00000000ce102940ce102bccd12c753300000af800000000ce102bccc020375e00000202 00000202f670a014ce102940f6389e2cc0448e3a00000000 00000002 00000000c020375e00000000 00766280Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c0169c29>] ? pagevec_lookup_tag+0x29/0x40 [<c0168241>] ? write_cache_pages+0xd1/0x350 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4944 20781 1
f589dc8400000046 00000046 00000000 00000000ce1014a0ce10172cd0f2ddbd00000af800000000ce10172cc020375e00000202 00000202f670a014ce1014a0f589dcccc0448e3a00000000 00000002 00000000c020375e00000000 00766280Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fe400>] journal_begin+0x80/0x110 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e9686>] reiserfs_dirty_inode+0x36/0xb0 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0 [<c01a613c>] __mark_inode_dirty+0x2c/0x180 [<c019c6f5>] file_update_time+0xc5/0xe0 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520 [<c0163980>] ? generic_file_aio_write+0x50/0xd0 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c018a331>] do_sync_write+0xd1/0x110 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c02ca436>] ? copy_to_user+0x36/0x130 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aa6b>] sys_pwrite64+0x7b/0x80 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4832 20782 1
f4617a6400000046 00000046 00000000 00000000ce103de0ce10406cd12ca39300000af800000000ce10406cc020375e00000202 00000202f670a014ce103de0f4617aacc0448e3a00000000 00000002 00000000c020375e00000000 00767900Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c01fe400>] journal_begin+0x80/0x110 [<c01fe4c2>] ? reiserfs_persistent_transaction+0x32/0x90 [<c01fe4d1>] reiserfs_persistent_transaction+0x41/0x90 [<c01df460>] reiserfs_get_block+0x1e0/0x14f0 [<c018dbbe>] ? inode_get_bytes+0x1e/0x50 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c018570d>] ? check_object+0x13d/0x200 [<c01ab2c3>] ? create_empty_buffers+0x33/0xa0 [<c01ad424>] __block_prepare_write+0x1a4/0x380 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01ad7a8>] block_write_begin+0x48/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01ddfbc>] reiserfs_write_begin+0xec/0x200 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01618f5>] generic_file_buffered_write+0x155/0x2c0 [<c0163520>] __generic_file_aio_write_nolock+0x1c0/0x520 [<c0163993>] generic_file_aio_write+0x63/0xd0 [<c018a331>] do_sync_write+0xd1/0x110 [<c0173f52>] ? might_fault+0x52/0xa0 [<c013dab0>] ? autoremove_wake_function+0x0/0x40 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0 [<c018a94c>] vfs_write+0x9c/0x140 [<c02ca436>] ? copy_to_user+0x36/0x130 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0 [<c018aa6b>] sys_pwrite64+0x7b/0x80 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4764 20783 1
f5833e4800000046 00000046 00000000 00000000f73de720f73de9acd12c96e700000af800000000f73de9acc020375e00000202 00000202f670a014f73de720f5833e90c0448e3a00000000 00000002 00000000c020375e00000000 00766280Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4600 20785 1
f460bd1c00000046 00000046 00000000 00000000f6b88000f6b8828cd12c9d1800000af800000000f6b8828cc020375e00000202 00000202f670a014f6b88000f460bd64c0448e3a00000000 00000002 00000000c020375e00000000 00766000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4548 20786 1
f5903d1c00000046 00000046 00000000 00000000f6960000f696028cd12fcd2a00000af800000000f696028cc020375e00000202 00000202f670a014f6960000f5903d64c0448e3a00000000 00000002 00000000c020375e00000000 00766280Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c018570d>] ? check_object+0x13d/0x200 [<c01fba72>] flush_commit_list+0x2d2/0x7c0 [<c0121ab0>] ? __wake_up+0x40/0x50 [<c01fde19>] do_journal_end+0xdc9/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36dbench D 00000000 4704 20787 1
f4557d6000000046 00000046 00000000 00000000f69614a0f696172cd12c7cad00000af800000000f696172cc020375e00000202 00000202f670a014f69614a0f4557da8c0448e3a00000000 00000002 00000000c020375e0000000000767b80Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fb8c5>] flush_commit_list+0x125/0x7c0 [<c01862b1>] ? __slab_alloc+0xc1/0x5f0 [<c02c4f65>] ? __prop_inc_single+0x65/0x80 [<c01fbfc3>] get_list_bitmap+0x63/0xb0 [<c01fdd5f>] do_journal_end+0xd0f/0xf50 [<c0133420>] ? process_timeout+0x0/0x10 [<c01fdffd>] journal_end_sync+0x5d/0x80 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e1e68>] reiserfs_sync_file+0x38/0x90 [<c01a905d>] vfs_fsync+0x6d/0xe0 [<c01a90fd>] do_fsync+0x2d/0x50 [<c01a914d>] sys_fsync+0xd/0x10 [<c0102e48>] sysenter_do_call+0x12/0x36pdflush D 00000000 6080 20790 2
f09bfe3c00000046 00000046 00000000 00000000f7088000f708828cd12caa9600000af800000000f708828cc020375e00000202 00000202f670a014f7088000f09bfe84c0448e3a00000000 00000002 00000000c020375e0000000000766c80Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01fe08b>] do_journal_begin_r+0x6b/0x320 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01fe400>] journal_begin+0x80/0x110 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c01e95fe>] reiserfs_sync_fs+0x3e/0x80 [<c0449203>] ? mutex_lock_nested+0x33/0x40 [<c018c1df>] ? lock_super+0x1f/0x30 [<c01e964d>] reiserfs_write_super+0xd/0x10 [<c018c3d4>] sync_supers+0x84/0xc0 [<c0169330>] ? pdflush+0x0/0x1c0 [<c016860b>] wb_kupdate+0x2b/0x100 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c0169330>] ? pdflush+0x0/0x1c0 [<c01693f8>] pdflush+0xc8/0x1c0 [<c01685e0>] ? wb_kupdate+0x0/0x100 [<c013d86a>] kthread+0x3a/0x70 [<c013d830>] ? kthread+0x0/0x70 [<c0103677>] kernel_thread_helper+0x7/0x10git-daemon D 00000000 6696 20791 2273
f3705ba800000046 00000046 00000000 00000000f66fe720f66fe9ac4a31216d00000afb00000000f66fe9acc020375e00000202 00000202f670a014f66fe720f3705bf0c0448e3a00000000 00000002 00000000c020375e00000000 00000000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c018440c>] ? slab_pad_check+0x3c/0x120 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c01666ee>] ? get_page_from_freelist+0x41e/0x530 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0 [<c01b32f0>] do_mpage_readpage+0x340/0x490 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0 [<c016a415>] ? __lru_cache_add+0x65/0xa0 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80 [<c01b352e>] mpage_readpages+0x9e/0xe0 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01dc710>] ? reiserfs_readpages+0x0/0x20 [<c01dc729>] reiserfs_readpages+0x19/0x20 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200 [<c016980e>] do_page_cache_readahead+0x4e/0x70 [<c0162caf>] filemap_fault+0x30f/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280git-daemon D
f63ede146888 20792 2273
f63ede2000000046 00000000f63ede1400000046f66fa940f66fabcc977b8ab000000afe00000000f66fabccf63ede14c014e52bc057e5a0000003aef63ede70f63ede28c04482c2f63ede34c04482f4f63ede68f63ede3cc0160d95f63ede5cCall Trace: [<c014e52b>] ? trace_hardirqs_on+0xb/0x10 [<c04482c2>] schedule+0x12/0x30 [<c04482f4>] io_schedule+0x14/0x20 [<c0160d95>] sync_page+0x55/0x60 [<c0448881>] __wait_on_bit_lock+0x41/0x80 [<c0160d40>] ? sync_page+0x0/0x60 [<c0160d18>] __lock_page+0x88/0xa0 [<c013daf0>] ? wake_bit_function+0x0/0x60 [<c0160eb9>] find_lock_page+0x39/0x70 [<c0162ad7>] filemap_fault+0x137/0x430 [<c0175f6d>] __do_fault+0x3d/0x360 [<c0176aa0>] handle_mm_fault+0x100/0x590 [<c014159d>] ? down_read_trylock+0x5d/0x70 [<c011aff1>] do_page_fault+0x121/0x280 [<c011aed0>] ? do_page_fault+0x0/0x280 [<c044b40a>] error_code+0x6a/0x70 [<c011aed0>] ? do_page_fault+0x0/0x280fcron D 00000000 7168 20797 2257
f5a0fd8000000046 00000046 00000000 00000000f6952940f6952bccdf71112a00000b2500000000f6952bccc020375e00000202 00000202f670a014f6952940f5a0fdc8c0448e3a00000000 00000002 00000000c020375e00000000 00000000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01db389>] reiserfs_lookup+0x59/0x170 [<c019b923>] ? d_alloc+0x123/0x1b0 [<c019275f>] __lookup_hash+0xcf/0x130 [<c01927e7>] lookup_hash+0x27/0x30 [<c0195cb5>] do_filp_open+0x245/0x7a0 [<c019ed68>] ? alloc_fd+0x28/0xf0 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c019ee18>] ? alloc_fd+0xd8/0xf0 [<c0188879>] do_sys_open+0x59/0xf0 [<c0188979>] sys_open+0x29/0x40 [<c0102e48>] sysenter_do_call+0x12/0x36 Sched Debug Version: v0.09, 2.6.29-10247-g8e2c4f2-dirty #2 now at 12468457.474391 msecs .jiffies : 1216523 .sysctl_sched_latency : 20.000000 .sysctl_sched_min_granularity : 4.000000 .sysctl_sched_wakeup_granularity : 5.000000 .sysctl_sched_child_runs_first : 0.000001 .sysctl_sched_features : 113917cpu#0, 1830.032 MHz
.nr_running : 1 .load : 1024 .nr_switches : 6163991 .nr_load_updates : 582498 .nr_uninterruptible : 21 .next_balance : 0.000000 .curr->pid : 20784 .clock : 12465239.317891 .cpu_load[0] : 1024 .cpu_load[1] : 1024 .cpu_load[2] : 1024 .cpu_load[3] : 1024 .cpu_load[4] : 1024 .yld_count : 0 .sched_switch : 0 .sched_count : 778698744 .sched_goidle : 307156 .ttwu_count : 0 .ttwu_local : 0 .bkl_count : 275cfs_rq[0]:
.exec_clock : 1507714.391289 .MIN_vruntime : 0.000001 .min_vruntime : 1444293.051451 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 1 .load : 1024 .nr_spread_over : 35rt_rq[0]:
.rt_nr_running : 0 .rt_throttled : 0 .rt_time : 0.000000 .rt_runtime : 950.000000runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R dbench 20784 1444293.051451 4569 120 1444293.051451 401016.637062 16471.092414INFO: task svscan:2223 blocked for more than 480 seconds."echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan D 00000000 5384 2223 1
f731dd8c00000046 00000046 00000000 00000000f6740000f674028ceaa8d98d00000af900000000f674028cc020375e00000202 00000202f670a014f6740000f731ddd4c0448e3a00000000 00000002 00000000c020375e00000000 00740000Call Trace: [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0448e3a>] __mutex_lock_common+0x11a/0x430 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c0449203>] mutex_lock_nested+0x33/0x40 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30 [<c020375e>] reiserfs_write_lock+0x1e/0x30 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740 [<c018b46f>] ? file_move+0x1f/0x50 [<c0197b70>] ? filldir+0x0/0xd0 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110 [<c044ae07>] ? _spin_unlock+0x27/0x50 [<c018b48e>] ? file_move+0x3e/0x50 [<c0173f9d>] ? might_fault+0x9d/0xa0 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0 [<c01e2d67>] reiserfs_readdir+0x17/0x20 [<c0197d9e>] vfs_readdir+0x7e/0xa0 [<c0197b70>] ? filldir+0x0/0xd0 [<c0197ed8>] sys_getdents+0x68/0xb0 [<c0102e48>] sysenter_do_call+0x12/0x36INFO: lockdep is turned off.
Quoted 2 lines · Frederic Weisbecker
So the new lock that replaces the bkl here is a per superblock mutex
You have to be very careful with this. Mutexes can be slower than spinlocks (and the new BKL is a spinlock) in some situations, they typically schedule much more etc., which can be costly. So this might have actually made it slower if you're unlucky. There were some experimental changes recently to make mutexes more behave like spinlocks, but I don't think they're fully performance equivalent.
Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?
Quoted 1 line
After this patch, reiserfs suffers from a slight performance regression (for now).
That might be related to the scheduling behaviour. Watch the reschedule counters in vmstat 1
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 4 lines · Andi Kleen
You have to be very careful with this. Mutexes can be slower than spinlocks (and the new BKL is a spinlock) in some situations, they typically schedule much more etc., which can be costly.
Actually, with the new adaptive spinning, that basically shouldn't be true any more. Or rather, you should need some really bad/unlucky situation for it to scheduler more than necessary, and if the locker _acts_ like a spinlock (ie it doesn't block while holding the lock), performance should approach a spinlock.
That said, there are definitely reasons why a mutex can be slower than the BKL, and the whole "BKL gets implicitly dropped at sleep time" is very high on that list of reasons. The sleeping patterns can be _very_ different with a mutex than with a BKL.
Quoted 2 lines
Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?
I do agree that a filesystem should try to avoid sleeping locks if at all possible, especially on the paths that the VM uses for writeback. But on the other hand, I think the issue with reiserfs is just the bad latencies that the BKL can cause, and then it doesn't matter.
Linus
Quoted 7 lines · Linus Torvalds
Quoted 2 lines
Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?
I do agree that a filesystem should try to avoid sleeping locks if at all possible, especially on the paths that the VM uses for writeback. But on the other hand, I think the issue with reiserfs is just the bad latencies that the BKL can cause, and then it doesn't matter.
The main motivator is the tip:core/kill-the-BKL tree: we are working on removing the BKL from all of the kernel, once and forever. We are actually quite close to that end goal: reiser3 was the last big stumbling block and it's great that Frederic is tackling that.
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL auto-dropping on schedule() it would be rather fragile to use spinlocks, and it would take forever to validate the result. Just one codepath missed with having some rare scheduling possibility and we'd have a kernel crash down the road.
Ingo
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. I also got a few reiser3 fs still around, it tended to perform very well
on kernel hacker workloads.
Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.
Quoted 2 lines
auto-dropping on
schedule()it would be rather fragile to use spinlocks, and it would take forever to validate the result.
Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 13 lines · Andi Kleen
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. I also got a few reiser3 fs still around, it tended to perform very well
on kernel hacker workloads.
Quoted 3 lines
Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.
It's not only about performances, latency is also a fair goal here. Moreover, reiserfs is not the only one concerned while dropping its Bkl, this is one big source of contention less for the rest of the system.
And concerning performance, I'm working on it :-)
Frederic.
Quoted 11 lines
Quoted 2 lines
auto-dropping on
schedule()it would be rather fragile to use spinlocks, and it would take forever to validate the result.Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · Andi Kleen
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]
( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )
Quoted 2 lines
[...] I also got a few reiser3 fs still around, it tended to perform very well on kernel hacker workloads.
Then i am sure you must like this patch: it introduces a per superblock lock, splitting up the big BKL serialization. You totally failed to even acknowledge that advantage, maybe you missed that aspect?
For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.
That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.
Frederic reported a slight drop in single-threaded performance, to be expected from a work in progress patch.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
Quoted 8 lines
Quoted 4 lines
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]
( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )
and that is bad? Isn't that more a sign of a certain 'stableness' - which is seriously lacking from more 'popular' filesystems like ext3? btw, does ext3 finally turn barriers on per default or does it still prefer letting data rot in drive caches because of 'performance'?
how many users would ext4 have if fedora&ubuntu would not force it onto the unwashed masses? And is that really a good idea after the amusing episode of the last month?
btw, what are ext3&4 doing when the underlying device does not support barriers (like software raid)?
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
Quoted 8 lines
Quoted 4 lines
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]
( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )
legacy (adj) — A pejorative term used in the computer industry meaning "it works"
http://homepages.tesco.net/~J.deBoynePollard/FGA/legacy-is-not-a-pejorative.html
It's by far the best of all the currently available and stable filesystems for our workload (big honking Cyrus IMAP boxes)
There have been bugfixes and occasional bits and pieces over the years. We applied a couple of patches for a while until they were accepted upstream only a couple of years ago.
It doesn't get much new development because, gee, "it works". Some people like their filesystem to keep just working in a predictable way.
I would be very concerned if people though it was OK to break it just because these shiny new ext4 and btrfs filesystem which are _NOT_YET_READY_ had replaced it.
Ta.
Quoted 8 lines
Quoted 2 lines
[...] I also got a few reiser3 fs still around, it tended to perform very well on kernel hacker workloads.
Then i am sure you must like this patch: it introduces a per superblock lock, splitting up the big BKL serialization. You totally failed to even acknowledge that advantage, maybe you missed that aspect?
Quoted 7 lines
For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.
That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.
Yes - I'm certainly interested in that.
That said, we have a box with 83 reiserfs partitions on it, and which is constrained by IO (main servers really don't need much CPU). Performance is pretty good even now.
So - I'm interested in this patch series, but not at the expense of making reiserfs any less stable. Our customers, funnily enough, like it when our service is stable!
Thanks,
Bron.
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Bron Gondwana
Quoted 7 lines
For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.
That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.
Yes - I'm certainly interested in that.
Quoted 7 lines
That said, we have a box with 83 reiserfs partitions on it, and which is constrained by IO (main servers really don't need much CPU). Performance is pretty good even now.
So - I'm interested in this patch series, but not at the expense of making reiserfs any less stable. Our customers, funnily enough, like it when our service is stable!
Definitely so :-)
Ingo
Quoted 10 lines · Andi Kleen
Quoted 5 lines
Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).
But for a legacy filesystem like reiser3, which depended on the BKL
reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently.
... and based on track record probably still should be :/
--Toby
(Who has no idea why good solid code that works really well magically gets labelled 'legacy' and shelved in favour of flaky, immature code.)
Quoted 23 lines
I also got a few
reiser3 fs still around, it tended to perform very well on kernel hacker workloads.Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.
Quoted 2 lines
auto-dropping on
schedule()it would be rather fragile to use spinlocks, and it would take forever to validate the result.Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.
-Andi
Signature · 5 lines
[email protected] -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Ingo Molnar
Acked-by Peter ZijlstraQuoted 5 lines · Andi Kleen
Quoted 2 lines
After this patch, reiserfs suffers from a slight performance regression (for now).
That might be related to the scheduling behaviour. Watch the reschedule counters in vmstat 1
FYI, Frederic is one of the main upstream tracing developers, and i can assure you that when working on scalability he watches something far more sophisticated than "vmstat 1" ;-)
See the commit below for example.
Thanks,
Ingo
--------------------->
From 8f350bb74293dded6c51870c34a74645b054caba Mon Sep 17 00:00:00 2001
From: Frederic Weisbecker <[email protected]>
Date: Mon, 6 Apr 2009 01:49:33 +0200
Subject: [PATCH] tracing/lockdep: report the time waited for a lock
While trying to optimize the new lock on reiserfs to replace the bkl, I find the lock tracing very useful though it lacks something important for performance (and latency) instrumentation:
the time a task waits for a lock.
That's what this patch implements:
bash-4816 [000] 202.652815: lock_contended: lock_contended: &sb->s_type->i_mutex_key bash-4816 [000] 202.652819: lock_acquired: &rq->lock (0.000 us) <...>-4787 [000] 202.652825: lock_acquired: &rq->lock (0.000 us) <...>-4787 [000] 202.652829: lock_acquired: &rq->lock (0.000 us) bash-4816 [000] 202.652833: lock_acquired: &sb->s_type->i_mutex_key (16.005 us)
As shown above, the "lock acquired" field is followed by the time it has been waiting for the lock. Usually, a lock contended entry is followed by a near lock_acquired entry with a non-zero time waited.
LKML-Reference: <[email protected]>
--- include/trace/lockdep_event_types.h | 23 ++++++++++++++++++----- kernel/lockdep.c | 8 ++++---- 2 files changed, 22 insertions(+), 9 deletions(-)
include/trace/lockdep_event_types.h +18 -5
@@ -32,11 +32,24 @@ TRACE_FORMAT(lock_contended,TP_FMT ("%s" , lock->name) ); -TRACE_FORMAT (lock_acquired, - TP_PROTO(struct lockdep_map *lock, unsigned long ip), -TP_ARGS (lock, ip), -TP_FMT ("%s" , lock->name) - ); +TRACE_EVENT (lock_acquired, + TP_PROTO(struct lockdep_map *lock, unsigned long ip, s64 waittime), + +TP_ARGS (lock, ip, waittime), + +TP_STRUCT__entry ( + __field(const char *, name) + __field(unsigned long, wait_usec) + __field(unsigned long, wait_nsec_rem) + ), +TP_fast_assign ( + __entry->name = lock->name; + __entry->wait_nsec_rem = do_div(waittime, NSEC_PER_USEC); + __entry->wait_usec = (unsigned long) waittime; + ), +TP_printk ("%s (%lu.%03lu us)" , __entry->name, __entry->wait_usec, + __entry->wait_nsec_rem) +);#endif #endif
kernel/lockdep.c +4 -4
@@ -3061,6 +3061,8 @@ found_it:put_lock_stats (stats); } +DEFINE_TRACE (lock_acquired); +static void __lock_acquired (struct lockdep_map *lock,unsigned long ip) {
@@ -3099,6 +3101,8 @@ found_it: hlock->holdtime_stamp = now; } +trace_lock_acquired (lock, ip, waittime); + stats =get_lock_stats (hlock_class (hlock));if (waittime) {if (hlock->read)
@@ -3137,14 +3141,10 @@ void lock_contended(struct lockdep_map *lock, unsigned long ip) }EXPORT_SYMBOL_GPL (lock_contended); -DEFINE_TRACE (lock_acquired); -void lock_acquired (struct lockdep_map *lock,unsigned long ip) {unsigned long flags; -trace_lock_acquired (lock, ip); -if (unlikely (!lock_stat))return ;