Skip to contentSkip to conversation

reiserfs: kill-the-BKL

PATCH In review Frederic Weisbecker 8 participants 18 messages Apr 7, 2009 – Apr 12, 2009 Tracing

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 reiserfs: kill-the-BKL

For the tip:kill-the-BKL tree.

First of all, I'm sorry for such a monolithic patch. I know it doesn't make it easy to review. But I can't really cut it in several subjects because all bits are needed here for the "Bkl to a mutex" conversion.

Although I first thought that half of it was only there for performance optimizations and could also be decoupled, further tests showed me that they all contribute to correctness and not only performance.

This patch is an attempt to remove the Bkl based locking scheme from reiserfs and is intended.
It is a bit inspired from an old attempt by Peter Zijlstra:
http://lkml.indiana.edu/hypermail/linux/kernel/0704.2/2174.html

The bkl is heavily used in this filesystem to prevent from concurrent write accesses on the filesystem.

Reiserfs makes a deep use of the specific properties of the Bkl:

- It can be acqquired recursively by a same task
- It is released on the schedule() calls and reacquired when schedule() returns

The two properties above are a roadmap for the reiserfs write locking so it's very hard to simply replace it with a common mutex.

- We need a recursive-able locking unless we want to restructure several blocks of the code.
- We need to identify the sites where the bkl was implictly relaxed (schedule, wait, sync, etc...) so that we can in turn release and reacquire our new lock explicitly.
Such implicit releases of the lock are often required to let other resources producer/consumer do their job or we can suffer unexpected starvations or deadlocks.

So the new lock that replaces the bkl here is a per superblock mutex with a specific property: it can be acquired recursively by a same task, like the bkl.
For such purpose, we integrate a lock owner and a lock depth field on the superblock information structure.

The first axis on this patch is to turn reiserfs_write_(un)lock() function into a wrapper to manage this mutex. Also some explicit calls to lock_kernel() have been converted to reiserfs_write_lock() helpers.

The second axis is to find the important blocking sites (schedule...(), wait_on_buffer(), sync_dirty_buffer(), etc...) and then apply an explicit release of the write lock on these locations before blocking. Then we can safely wait for those who can give us resources or those who need some. Typically this is a fight between the current writer, the reiserfs workqueue (aka the async commiter) and the pdflush threads.

The third axis is a consequence of the second. The write lock is usually on top of a lock dependency chain which can include the journal lock, the flush lock or the commit lock. So it's dangerous to release and trying to reacquire the write lock while we still hold other locks.

This is fine with the bkl:

T1 T2

lock_kernel()

    mutex_lock(A)
    unlock_kernel()
    // do something
                            lock_kernel()
                                mutex_lock(A) -> already locked by T1
                                schedule() (and then unlock_kernel())
    lock_kernel()
    mutex_unlock(A)
    ....

This is not fine with a mutex:

T1 T2

mutex_lock(write)

    mutex_lock(A)
    mutex_unlock(write)
    // do something
                           mutex_lock(write)
                              mutex_lock(A) -> already locked by T1
                              schedule()

    mutex_lock(write) -> already locked by T2
    deadlock

The solution in this patch is to provide a helper which releases the write lock and sleep a bit if we can't lock a mutex that depend on it. It's another simulation of the bkl behaviour.

The last axis is to locate the fs callbacks that are called with the bkl held, according to Documentation/filesystem/Locking.

Those are:

- reiserfs_remount
- reiserfs_fill_super
- reiserfs_put_super

Reiserfs didn't need to explicitly lock because of the context of these callbacks. But now we must take care of that with the new locking.

After this patch, reiserfs suffers from a slight performance regression (for now). On UP, a high volume write with dd reports an average of 27 MB/s instead of 30 MB/s without the patch applied.

---
 fs/reiserfs/Makefile           |    2 +-
 fs/reiserfs/bitmap.c           |    2 +
 fs/reiserfs/dir.c              |    8 +++
 fs/reiserfs/fix_node.c         |   10 +++
 fs/reiserfs/inode.c            |   23 +++++--
 fs/reiserfs/ioctl.c            |    6 +-
 fs/reiserfs/journal.c          |  134 +++++++++++++++++++++++++++++++---------
 fs/reiserfs/lock.c             |   63 +++++++++++++++++++
 fs/reiserfs/resize.c           |    2 +
 fs/reiserfs/stree.c            |    2 +
 fs/reiserfs/super.c            |   41 ++++++++++++-
 include/linux/reiserfs_fs.h    |   12 ++--
 include/linux/reiserfs_fs_sb.h |    9 +++
 13 files changed, 271 insertions(+), 43 deletions(-)
 create mode 100644 fs/reiserfs/lock.c
fs/reiserfs/Makefile +1 -1
index 7c5ab63..6a9e30c 100644
@@ -7,7 +7,7 @@ obj-$(CONFIG_REISERFS_FS) += reiserfs.o
 reiserfs-objs := bitmap.o do_balan.o namei.o inode.o file.o dir.o fix_node.o \
 		 super.o prints.o objectid.o lbalance.o ibalance.o stree.o \
 		 hashes.o tail_conversion.o journal.o resize.o \
-		 item_ops.o ioctl.o procfs.o xattr.o
+		 item_ops.o ioctl.o procfs.o xattr.o lock.o
 
 ifeq ($(CONFIG_REISERFS_FS_XATTR),y)
 reiserfs-objs += xattr_user.o xattr_trusted.o
fs/reiserfs/bitmap.c +2 -0
index e716161..1470334 100644
@@ -1256,7 +1256,9 @@ struct buffer_head *reiserfs_read_bitmap_block(struct super_block *sb,
 	else {
 		if (buffer_locked(bh)) {
 			PROC_INFO_INC(sb, scan_bitmap.wait);
+			reiserfs_write_unlock(sb);
 			__wait_on_buffer(bh);
+			reiserfs_write_lock(sb);
 		}
 		BUG_ON(!buffer_uptodate(bh));
 		BUG_ON(atomic_read(&bh->b_count) == 0);
fs/reiserfs/dir.c +8 -0
index 67a80d7..6d71aa0 100644
@@ -174,14 +174,22 @@ int reiserfs_readdir_dentry(struct dentry *dentry, void *dirent,
 				// user space buffer is swapped out. At that time
 				// entry can move to somewhere else
 				memcpy(local_buf, d_name, d_reclen);
+
+				/*
+				 * Since filldir might sleep, we can release
+				 * the write lock here for other waiters
+				 */
+				reiserfs_write_unlock(inode->i_sb);
 				if (filldir
 				    (dirent, local_buf, d_reclen, d_off, d_ino,
 				     DT_UNKNOWN) < 0) {
+					reiserfs_write_lock(inode->i_sb);
 					if (local_buf != small_buf) {
 						kfree(local_buf);
 					}
 					goto end;
 				}
+				reiserfs_write_lock(inode->i_sb);
 				if (local_buf != small_buf) {
 					kfree(local_buf);
 				}
fs/reiserfs/fix_node.c +10 -0
index 5e5a4e6..bf5f2cb 100644
@@ -1022,7 +1022,11 @@ static int get_far_parent(struct tree_balance *tb,
 	/* Check whether the common parent is locked. */
 
 	if (buffer_locked(*pcom_father)) {
+
+		/* Release the write lock while the buffer is busy */
+		reiserfs_write_unlock(tb->tb_sb);
 		__wait_on_buffer(*pcom_father);
+		reiserfs_write_lock(tb->tb_sb);
 		if (FILESYSTEM_CHANGED_TB(tb)) {
 			brelse(*pcom_father);
 			return REPEAT_SEARCH;
@@ -1927,7 +1931,9 @@ static int get_direct_parent(struct tree_balance *tb, int h)
 		return REPEAT_SEARCH;
 
 	if (buffer_locked(bh)) {
+		reiserfs_write_unlock(tb->tb_sb);
 		__wait_on_buffer(bh);
+		reiserfs_write_lock(tb->tb_sb);
 		if (FILESYSTEM_CHANGED_TB(tb))
 			return REPEAT_SEARCH;
 	}
@@ -2278,7 +2284,9 @@ static int wait_tb_buffers_until_unlocked(struct tree_balance *tb)
 				    REPEAT_SEARCH : CARRY_ON;
 			}
 #endif
+			reiserfs_write_unlock(tb->tb_sb);
 			__wait_on_buffer(locked);
+			reiserfs_write_lock(tb->tb_sb);
 			if (FILESYSTEM_CHANGED_TB(tb))
 				return REPEAT_SEARCH;
 		}
@@ -2349,7 +2357,9 @@ int fix_nodes(int op_mode, struct tree_balance *tb,
 
 	/* if it possible in indirect_to_direct conversion */
 	if (buffer_locked(tbS0)) {
+		reiserfs_write_unlock(tb->tb_sb);
 		__wait_on_buffer(tbS0);
+		reiserfs_write_lock(tb->tb_sb);
 		if (FILESYSTEM_CHANGED_TB(tb))
 			return REPEAT_SEARCH;
 	}
fs/reiserfs/inode.c +18 -5
index 6fd0f47..88ef0b7 100644
@@ -489,10 +489,14 @@ static int reiserfs_get_blocks_direct_io(struct inode *inode,
 	   disappeared */
 	if (REISERFS_I(inode)->i_flags & i_pack_on_close_mask) {
 		int err;
-		lock_kernel();
+
+		reiserfs_write_lock(inode->i_sb);
+
 		err = reiserfs_commit_for_inode(inode);
 		REISERFS_I(inode)->i_flags &= ~i_pack_on_close_mask;
-		unlock_kernel();
+
+		reiserfs_write_unlock(inode->i_sb);
+
 		if (err < 0)
 			ret = err;
 	}
@@ -616,7 +620,6 @@ int reiserfs_get_block(struct inode *inode, sector_t block,
 	loff_t new_offset =
 	    (((loff_t) block) << inode->i_sb->s_blocksize_bits) + 1;
 
-	/* bad.... */
 	reiserfs_write_lock(inode->i_sb);
 	version = get_inode_item_key_version(inode);
 
@@ -997,10 +1000,14 @@ int reiserfs_get_block(struct inode *inode, sector_t block,
 			if (retval)
 				goto failure;
 		}
-		/* inserting indirect pointers for a hole can take a
-		 ** long time.  reschedule if needed
+		/*
+		 * inserting indirect pointers for a hole can take a
+		 * long time.  reschedule if needed and also release the write
+		 * lock for others.
 		 */
+		reiserfs_write_unlock(inode->i_sb);
 		cond_resched();
+		reiserfs_write_lock(inode->i_sb);
 
 		retval = search_for_position_by_key(inode->i_sb, &key, &path);
 		if (retval == IO_ERROR) {
@@ -2612,7 +2619,10 @@ int reiserfs_prepare_write(struct file *f, struct page *page,
 	int ret;
 	int old_ref = 0;
 
+	reiserfs_write_unlock(inode->i_sb);
 	reiserfs_wait_on_write_block(inode->i_sb);
+	reiserfs_write_lock(inode->i_sb);
+
 	fix_tail_page_for_writing(page);
 	if (reiserfs_transaction_running(inode->i_sb)) {
 		struct reiserfs_transaction_handle *th;
@@ -2762,7 +2772,10 @@ int reiserfs_commit_write(struct file *f, struct page *page,
 	int update_sd = 0;
 	struct reiserfs_transaction_handle *th = NULL;
 
+	reiserfs_write_unlock(inode->i_sb);
 	reiserfs_wait_on_write_block(inode->i_sb);
+	reiserfs_write_lock(inode->i_sb);
+
 	if (reiserfs_transaction_running(inode->i_sb)) {
 		th = current->journal_info;
 	}
fs/reiserfs/ioctl.c +4 -2
index 0ccc3fd..5e40b0c 100644
@@ -141,9 +141,11 @@ long reiserfs_compat_ioctl(struct file *file, unsigned int cmd,
 	default:
 		return -ENOIOCTLCMD;
 	}
-	lock_kernel();
+
+	reiserfs_write_lock(inode->i_sb);
 	ret = reiserfs_ioctl(inode, file, cmd, (unsigned long) compat_ptr(arg));
-	unlock_kernel();
+	reiserfs_write_unlock(inode->i_sb);
+
 	return ret;
 }
 #endif
fs/reiserfs/journal.c +105 -29

297 lines not on this page · raw message

fs/reiserfs/lock.c new file +63 -0

63 lines not on this page · raw message

fs/reiserfs/resize.c +2 -0
index 238e9d9..6a7bfb3 100644
@@ -142,7 +142,9 @@ int reiserfs_resize(struct super_block *s, unsigned long block_count_new)
 
 			set_buffer_uptodate(bh);
 			mark_buffer_dirty(bh);
+			reiserfs_write_unlock(s);
 			sync_dirty_buffer(bh);
+			reiserfs_write_lock(s);
 			// update bitmap_info stuff
 			bitmap[i].free_count = sb_blocksize(sb) * 8 - 1;
 			brelse(bh);
fs/reiserfs/stree.c +2 -0
index d036ee5..6bd99a9 100644
@@ -629,7 +629,9 @@ int search_by_key(struct super_block *sb, const struct cpu_key *key,	/* Key to s
 				search_by_key_reada(sb, reada_bh,
 						    reada_blocks, reada_count);
 			ll_rw_block(READ, 1, &bh);
+			reiserfs_write_unlock(sb);
 			wait_on_buffer(bh);
+			reiserfs_write_lock(sb);
 			if (!buffer_uptodate(bh))
 				goto io_error;
 		} else {
fs/reiserfs/super.c +40 -1

99 lines not on this page · raw message

include/linux/reiserfs_fs.h +7 -5
index 2245c78..f6b7b7b 100644
@@ -52,11 +52,13 @@
 #define REISERFS_IOC32_GETVERSION	FS_IOC32_GETVERSION
 #define REISERFS_IOC32_SETVERSION	FS_IOC32_SETVERSION
 
-/* Locking primitives */
-/* Right now we are still falling back to (un)lock_kernel, but eventually that
-   would evolve into real per-fs locks */
-#define reiserfs_write_lock( sb ) lock_kernel()
-#define reiserfs_write_unlock( sb ) unlock_kernel()
+/*
+ * Locking primitives. The write lock is a per superblock
+ * special mutex that has properties close to the Big Kernel Lock
+ * which was used in the previous locking scheme.
+ */
+void reiserfs_write_lock(struct super_block *s);
+void reiserfs_write_unlock(struct super_block *s);
 
 struct fid;
 
include/linux/reiserfs_fs_sb.h +9 -0
index 5621d87..cec8319 100644
@@ -7,6 +7,8 @@
 #ifdef __KERNEL__
 #include <linux/workqueue.h>
 #include <linux/rwsem.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
 #endif
 
 typedef enum {
@@ -355,6 +357,13 @@ struct reiserfs_sb_info {
 	struct reiserfs_journal *s_journal;	/* pointer to journal information */
 	unsigned short s_mount_state;	/* reiserfs state (valid, invalid) */
 
+	/* Serialize writers access, replace the old bkl */
+	struct mutex lock;
+	/* Owner of the lock (can be recursive) */
+	struct task_struct *lock_owner;
+	/* Depth of the lock, start from -1 like the bkl */
+	int lock_depth;
+
 	/* Comment? -Hans */
 	void (*end_io_handler) (struct buffer_head *, int);
 	hashf_t s_hash_function;	/* pointer to function which is used
Signature · 1 line
1.6.1
#2 lore raw
In reply to Frederic Weisbecker

fs/reiserfs/lock.c line 62

Quoted 11 lines · Frederic Weisbecker
fs/reiserfs/lock.c new file +11 -0
+/*
+ * Utility function to force a BUG if it is called without the superblock
+ * write lock held.  caller is the string printed just before calling BUG()
+ */
+void reiserfs_check_lock_depth(struct super_block *sb, char *caller)
+{
+	struct reiserfs_sb_info *sb_i = REISERFS_SB(sb);
+
+	if (sb_i->lock_depth < 0)
+		reiserfs_panic(sb, "%s called without kernel lock held %d",
+			       caller);

s/kernel lock/sb write lock/

fs/reiserfs/super.c static void reiserfs_put_super(struct super_block *s) · line 512

Quoted 4 lines
fs/reiserfs/super.c +2 -0
+	reiserfs_write_unlock(s);
+	mutex_destroy(&REISERFS_SB(s)->lock);
 	kfree(s->s_fs_info);
 	s->s_fs_info = NULL;

ah, mutex_destroy() - it's a small detail but still nice ;-)

Anyway, it still looks good to me, after the earlier patch i reviewed. Would be nice to have the testing feedback of reiserfs users, and the locking review from anyone who knows this code.

Ingo

#3 lore raw
In reply to Frederic Weisbecker

Hi Frederic

I've applied your patch.
This is UP system.
This warning seems related to your patch.

[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>] might_fault+0x52/0xa0

but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #3 (sysfs_mutex){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c01cd17c>] sysfs_addrm_start+0x2c/0xc0
       [<c01cdd30>] create_dir+0x40/0x80
       [<c01cdd9b>] sysfs_create_dir+0x2b/0x50
       [<c02c3121>] kobject_add_internal+0xc1/0x220
       [<c02c3351>] kobject_add_varg+0x31/0x50
       [<c02c33cc>] kobject_add+0x2c/0x60
       [<c0337a52>] device_add+0xe2/0x540
       [<c01cac02>] add_partition+0x112/0x210
       [<c01cb2ef>] rescan_partitions+0x23f/0x320
       [<c01b0f13>] __blkdev_get+0x153/0x330
       [<c01b10fa>] blkdev_get+0xa/0x10
       [<c01caaca>] register_disk+0x10a/0x130
       [<c02bd5c9>] add_disk+0x109/0x170
       [<c034e85a>] sd_probe_async+0x19a/0x280
       [<c0143900>] async_thread+0xd0/0x230
       [<c013d86a>] kthread+0x3a/0x70
       [<c0103677>] kernel_thread_helper+0x7/0x10
       [<ffffffff>] 0xffffffff

-> #2 (&bdev->bd_mutex){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c01b0df3>] __blkdev_get+0x33/0x330
       [<c01b10fa>] blkdev_get+0xa/0x10
       [<c01b1261>] open_by_devnum+0x21/0x50
       [<c01fed2e>] journal_init+0x22e/0x19a0
       [<c01ea881>] reiserfs_fill_super+0x3a1/0x1050
       [<c018d053>] get_sb_bdev+0x133/0x160
       [<c01e8b21>] get_super_block+0x21/0x30
       [<c018bd7a>] vfs_kern_mount+0x3a/0xa0
       [<c018be39>] do_kern_mount+0x39/0xd0
       [<c01a2063>] do_mount+0x3a3/0x790
       [<c01a24d9>] sys_mount+0x89/0xc0
       [<c06fac3d>] mount_block_root+0xd0/0x261
       [<c06fae27>] mount_root+0x59/0x5f
       [<c06faf1c>] prepare_namespace+0xef/0x171
       [<c06fa4e3>] kernel_init+0xd5/0xee
       [<c0103677>] kernel_thread_helper+0x7/0x10
       [<ffffffff>] 0xffffffff

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c020375e>] reiserfs_write_lock+0x1e/0x30
       [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
       [<c01b32f0>] do_mpage_readpage+0x340/0x490
       [<c01b352e>] mpage_readpages+0x9e/0xe0
       [<c01dc729>] reiserfs_readpages+0x19/0x20
       [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
       [<c016980e>] do_page_cache_readahead+0x4e/0x70
       [<c0162caf>] filemap_fault+0x30f/0x430
       [<c0175f6d>] __do_fault+0x3d/0x360
       [<c0176aa0>] handle_mm_fault+0x100/0x590
       [<c011aff1>] do_page_fault+0x121/0x280
       [<c044b40a>] error_code+0x6a/0x70
       [<c01bf810>] load_elf_binary+0x9d0/0x19f0
       [<c018fde3>] search_binary_handler+0x163/0x2f0
       [<c019018d>] do_execve+0x21d/0x2b0
       [<c01016de>] sys_execve+0x3e/0x70
       [<c0102ec9>] syscall_call+0x7/0xb
       [<ffffffff>] 0xffffffff

-> #0 (&mm->mmap_sem){++++++}:

       [<c014fbde>] __lock_acquire+0xdce/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0173f83>] might_fault+0x83/0xa0
       [<c02ca436>] copy_to_user+0x36/0x130
       [<c0197b24>] filldir64+0xa4/0xf0
       [<c01ccdf6>] sysfs_readdir+0x116/0x200
       [<c0197d9e>] vfs_readdir+0x7e/0xa0
       [<c0197e29>] sys_getdents64+0x69/0xb0
       [<c0102e48>] sysenter_do_call+0x12/0x36
       [<ffffffff>] 0xffffffff

other info that might help us debug this:

2 locks held by udevadm/1239:
#0: (&type->i_mutex_dir_key){+.+.+.}, at: [<c0197d6f>] vfs_readdir+0x4f/0xa0 #1: (sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

stack backtrace:
Pid: 1239, comm: udevadm Not tainted 2.6.29-10247-g8e2c4f2-dirty #2
Call Trace:
 [<c0447c92>] ? printk+0x18/0x1e
 [<c014db5c>] print_circular_bug_tail+0x8c/0xe0
 [<c014fbde>] __lock_acquire+0xdce/0x1160
 [<c014bb1d>] ? put_lock_stats+0xd/0x30
 [<c014ffe4>] lock_acquire+0x74/0x90
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c0173f83>] might_fault+0x83/0xa0
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c02ca436>] copy_to_user+0x36/0x130
 [<c0197b24>] filldir64+0xa4/0xf0
 [<c01ccdf6>] sysfs_readdir+0x116/0x200
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197e29>] sys_getdents64+0x69/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36
#4 lore raw
Quoted 7 lines · Alexander Beregalov

Hi Frederic

I've applied your patch.
This is UP system.
This warning seems related to your patch.

Quoted 8 lines

[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>] might_fault+0x52/0xa0

but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?

Thanks,
Frederic.

Quoted 114 lines

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #3 (sysfs_mutex){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c01cd17c>] sysfs_addrm_start+0x2c/0xc0
       [<c01cdd30>] create_dir+0x40/0x80
       [<c01cdd9b>] sysfs_create_dir+0x2b/0x50
       [<c02c3121>] kobject_add_internal+0xc1/0x220
       [<c02c3351>] kobject_add_varg+0x31/0x50
       [<c02c33cc>] kobject_add+0x2c/0x60
       [<c0337a52>] device_add+0xe2/0x540
       [<c01cac02>] add_partition+0x112/0x210
       [<c01cb2ef>] rescan_partitions+0x23f/0x320
       [<c01b0f13>] __blkdev_get+0x153/0x330
       [<c01b10fa>] blkdev_get+0xa/0x10
       [<c01caaca>] register_disk+0x10a/0x130
       [<c02bd5c9>] add_disk+0x109/0x170
       [<c034e85a>] sd_probe_async+0x19a/0x280
       [<c0143900>] async_thread+0xd0/0x230
       [<c013d86a>] kthread+0x3a/0x70
       [<c0103677>] kernel_thread_helper+0x7/0x10
       [<ffffffff>] 0xffffffff

-> #2 (&bdev->bd_mutex){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c01b0df3>] __blkdev_get+0x33/0x330
       [<c01b10fa>] blkdev_get+0xa/0x10
       [<c01b1261>] open_by_devnum+0x21/0x50
       [<c01fed2e>] journal_init+0x22e/0x19a0
       [<c01ea881>] reiserfs_fill_super+0x3a1/0x1050
       [<c018d053>] get_sb_bdev+0x133/0x160
       [<c01e8b21>] get_super_block+0x21/0x30
       [<c018bd7a>] vfs_kern_mount+0x3a/0xa0
       [<c018be39>] do_kern_mount+0x39/0xd0
       [<c01a2063>] do_mount+0x3a3/0x790
       [<c01a24d9>] sys_mount+0x89/0xc0
       [<c06fac3d>] mount_block_root+0xd0/0x261
       [<c06fae27>] mount_root+0x59/0x5f
       [<c06faf1c>] prepare_namespace+0xef/0x171
       [<c06fa4e3>] kernel_init+0xd5/0xee
       [<c0103677>] kernel_thread_helper+0x7/0x10
       [<ffffffff>] 0xffffffff

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:

       [<c014fb2a>] __lock_acquire+0xd1a/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0448d70>] __mutex_lock_common+0x50/0x430
       [<c0449203>] mutex_lock_nested+0x33/0x40
       [<c020375e>] reiserfs_write_lock+0x1e/0x30
       [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
       [<c01b32f0>] do_mpage_readpage+0x340/0x490
       [<c01b352e>] mpage_readpages+0x9e/0xe0
       [<c01dc729>] reiserfs_readpages+0x19/0x20
       [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
       [<c016980e>] do_page_cache_readahead+0x4e/0x70
       [<c0162caf>] filemap_fault+0x30f/0x430
       [<c0175f6d>] __do_fault+0x3d/0x360
       [<c0176aa0>] handle_mm_fault+0x100/0x590
       [<c011aff1>] do_page_fault+0x121/0x280
       [<c044b40a>] error_code+0x6a/0x70
       [<c01bf810>] load_elf_binary+0x9d0/0x19f0
       [<c018fde3>] search_binary_handler+0x163/0x2f0
       [<c019018d>] do_execve+0x21d/0x2b0
       [<c01016de>] sys_execve+0x3e/0x70
       [<c0102ec9>] syscall_call+0x7/0xb
       [<ffffffff>] 0xffffffff

-> #0 (&mm->mmap_sem){++++++}:

       [<c014fbde>] __lock_acquire+0xdce/0x1160
       [<c014ffe4>] lock_acquire+0x74/0x90
       [<c0173f83>] might_fault+0x83/0xa0
       [<c02ca436>] copy_to_user+0x36/0x130
       [<c0197b24>] filldir64+0xa4/0xf0
       [<c01ccdf6>] sysfs_readdir+0x116/0x200
       [<c0197d9e>] vfs_readdir+0x7e/0xa0
       [<c0197e29>] sys_getdents64+0x69/0xb0
       [<c0102e48>] sysenter_do_call+0x12/0x36
       [<ffffffff>] 0xffffffff

other info that might help us debug this:

2 locks held by udevadm/1239:
#0: (&type->i_mutex_dir_key){+.+.+.}, at: [<c0197d6f>] vfs_readdir+0x4f/0xa0 #1: (sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

stack backtrace:
Pid: 1239, comm: udevadm Not tainted 2.6.29-10247-g8e2c4f2-dirty #2
Call Trace:
 [<c0447c92>] ? printk+0x18/0x1e
 [<c014db5c>] print_circular_bug_tail+0x8c/0xe0
 [<c014fbde>] __lock_acquire+0xdce/0x1160
 [<c014bb1d>] ? put_lock_stats+0xd/0x30
 [<c014ffe4>] lock_acquire+0x74/0x90
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c0173f83>] might_fault+0x83/0xa0
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c02ca436>] copy_to_user+0x36/0x130
 [<c0197b24>] filldir64+0xa4/0xf0
 [<c01ccdf6>] sysfs_readdir+0x116/0x200
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197e29>] sys_getdents64+0x69/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36
#5 lore raw
Quoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Alexander Beregalov

Hi Frederic

I've applied your patch.
This is UP system.
This warning seems related to your patch.

[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>] might_fault+0x52/0xa0

Quoted 7 lines
Quoted 2 lines

but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?

[Sent in private mail]

This should be related.
System is hung after dbench testing for some time.

SysRq : Emergency Sync
SysRq : Show Blocked State
task PC stack pid father
events/0 D 00000000 5672 6 2
f7067ee8 00000046 00000046 00000000 00000000 f7056720 f70569ac b6acece3 00000b44 00000000 f70569ac c018c1df 00000202 00000202 f6982a60 f7056720 f7067f30 c0448e3a 00000000 00000002 00000000 c018c1df 00000000 0018c3b2

Call Trace:
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c018c1df>] lock_super+0x1f/0x30
 [<c018c3b9>] sync_supers+0x69/0xc0
 [<c01a922c>] do_sync+0x1c/0x60
 [<c013a0e6>] ? worker_thread+0xe6/0x250
 [<c01a927d>] do_sync_work+0xd/0x20
 [<c013a148>] worker_thread+0x148/0x250
 [<c013a0e6>] ? worker_thread+0xe6/0x250
 [<c01a9270>] ? do_sync_work+0x0/0x20
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c013a000>] ? worker_thread+0x0/0x250
 [<c013d86a>] kthread+0x3a/0x70
 [<c013d830>] ? kthread+0x0/0x70
 [<c0103677>] kernel_thread_helper+0x7/0x10

metalog D 00000000 4936 2183 1
f7161cec 00000046 00000046 00000000 00000000 f6b8d280 f6b8d50c 1eba9be4 00000b44 00000000 f6b8d50c c020375e 00000202 00000202 f670a014 f6b8d280 f7161d34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0018440c

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aaad>] sys_write+0x3d/0x70
 [<c0102e48>] sysenter_do_call+0x12/0x36

svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c0197ed8>] sys_getdents+0x68/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

multilog D 00000000 4800 2236 2228
f731bcec 00000046 00000046 00000000 00000000 f6ba8000 f6ba828c 50105c29 00000b26 00000000 f6ba828c c020375e 00000202 00000202 f670a014 f6ba8000 f731bd34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0014ce9d

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aaad>] sys_write+0x3d/0x70
 [<c0102e48>] sysenter_do_call+0x12/0x36

agetty D 00000000 5432 2441 1
f73cbba8 00000046 00000046 00000000 00000000 f73dd280 f73dd50c 6a069cd5 00000b33 00000000 f73dd50c c020375e 00000202 00000202 f670a014 f73dd280 f73cbbf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000246

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c044b07d>] ? _spin_unlock_irq+0x2d/0x50
 [<c0124b02>] ? finish_task_switch+0x62/0xb0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c0124aa0>] ? finish_task_switch+0x0/0xb0
 [<c044b2df>] ? __reacquire_kernel_lock+0x2f/0x40
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c0332d78>] ? serial8250_tx_empty+0x18/0x50
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0
 [<c01b32f0>] do_mpage_readpage+0x340/0x490
 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0
 [<c016a415>] ? __lru_cache_add+0x65/0xa0
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01b352e>] mpage_readpages+0x9e/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c04480b8>] ? __schedule+0x2f8/0x4f0
 [<c01dc710>] ? reiserfs_readpages+0x0/0x20
 [<c01dc729>] reiserfs_readpages+0x19/0x20
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
 [<c016980e>] do_page_cache_readahead+0x4e/0x70
 [<c0162caf>] filemap_fault+0x30f/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

screen D 00000000 5244 2529 2528
f612fd90 00000046 00000046 00000000 00000000 f6babde0 f6bac06c 33c73fab 00000b1c 00000000 f6bac06c c0193bf3 00000202 00000202 f6f9615c f6babde0 f612fdd8 c0448e3a 00000000 00000002 00000000 c0193bf3 00000000 0012fed0

Call Trace:
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0193bf3>] do_lookup+0xd3/0x1c0
 [<c0193e49>] __link_path_walk+0x169/0xcc0
 [<c0194b53>] path_walk+0x53/0xb0
 [<c0194c8b>] do_path_lookup+0x8b/0x170
 [<c01958ea>] user_path_at+0x4a/0x80
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c018e471>] vfs_stat_fd+0x21/0x50
 [<c018e571>] vfs_stat+0x11/0x20
 [<c018e594>] sys_stat64+0x14/0x30
 [<c018abdf>] ? vfs_read+0xff/0x140
 [<c0317d70>] ? tty_read+0x0/0xb0
 [<c018acdd>] ? sys_read+0x3d/0x70
 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

rtorrent D 00000000 4084 2543 2542
f6325d88 00000046 00000046 00000000 00000000 f6b894a0 f6b8972c d79a38dd 00000af9 00000000 f6b8972c c020375e 00000202 00000202 f670a014 f6b894a0 f6325dd0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197e29>] sys_getdents64+0x69/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

bash D 00000000 5788 2546 2529
f585dcd8 00000046 00000046 00000000 00000000 f6b8bde0 f6b8c06c 188b381a 00000b1b 00000000 f6b8c06c c020375e 00000202 00000202 f670a014 f6b8bde0 f585dd20 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01db389>] reiserfs_lookup+0x59/0x170
 [<c019b923>] ? d_alloc+0x123/0x1b0
 [<c0193cb1>] do_lookup+0x191/0x1c0
 [<c0194305>] __link_path_walk+0x625/0xcc0
 [<c0194b53>] path_walk+0x53/0xb0
 [<c0194c8b>] do_path_lookup+0x8b/0x170
 [<c01958ea>] user_path_at+0x4a/0x80
 [<c018dda4>] ? cp_new_stat64+0xe4/0x100
 [<c018e471>] vfs_stat_fd+0x21/0x50
 [<c018e571>] vfs_stat+0x11/0x20
 [<c018e594>] sys_stat64+0x14/0x30
 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4816 20778 1
f44bbd1c 00000046 00000046 00000000 00000000 ce105280 ce10550c d12c8ac7 00000af8 00000000 ce10550c c020375e 00000202 00000202 f670a014 ce105280 f44bbd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766a00

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4660 20779 1
f44b3de4 00000046 00000046 00000000 00000000 ce106720 ce1069ac d12c836c 00000af8 00000000 ce1069ac c020375e 00000202 00000202 f670a014 ce106720 f44b3e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767400

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c04480b8>] ? __schedule+0x2f8/0x4f0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4724 20780 1
f6389de4 00000046 00000046 00000000 00000000 ce102940 ce102bcc d12c7533 00000af8 00000000 ce102bcc c020375e 00000202 00000202 f670a014 ce102940 f6389e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c0169c29>] ? pagevec_lookup_tag+0x29/0x40
 [<c0168241>] ? write_cache_pages+0xd1/0x350
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4944 20781 1
f589dc84 00000046 00000046 00000000 00000000 ce1014a0 ce10172c d0f2ddbd 00000af8 00000000 ce10172c c020375e 00000202 00000202 f670a014 ce1014a0 f589dccc c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fe400>] journal_begin+0x80/0x110
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e9686>] reiserfs_dirty_inode+0x36/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c02ca436>] ? copy_to_user+0x36/0x130
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aa6b>] sys_pwrite64+0x7b/0x80
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4832 20782 1
f4617a64 00000046 00000046 00000000 00000000 ce103de0 ce10406c d12ca393 00000af8 00000000 ce10406c c020375e 00000202 00000202 f670a014 ce103de0 f4617aac c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767900

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c01fe400>] journal_begin+0x80/0x110
 [<c01fe4c2>] ? reiserfs_persistent_transaction+0x32/0x90
 [<c01fe4d1>] reiserfs_persistent_transaction+0x41/0x90
 [<c01df460>] reiserfs_get_block+0x1e0/0x14f0
 [<c018dbbe>] ? inode_get_bytes+0x1e/0x50
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01ab2c3>] ? create_empty_buffers+0x33/0xa0
 [<c01ad424>] __block_prepare_write+0x1a4/0x380
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01ad7a8>] block_write_begin+0x48/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01ddfbc>] reiserfs_write_begin+0xec/0x200
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01618f5>] generic_file_buffered_write+0x155/0x2c0
 [<c0163520>] __generic_file_aio_write_nolock+0x1c0/0x520
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c02ca436>] ? copy_to_user+0x36/0x130
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aa6b>] sys_pwrite64+0x7b/0x80
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4764 20783 1
f5833e48 00000046 00000046 00000000 00000000 f73de720 f73de9ac d12c96e7 00000af8 00000000 f73de9ac c020375e 00000202 00000202 f670a014 f73de720 f5833e90 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4600 20785 1
f460bd1c 00000046 00000046 00000000 00000000 f6b88000 f6b8828c d12c9d18 00000af8 00000000 f6b8828c c020375e 00000202 00000202 f670a014 f6b88000 f460bd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4548 20786 1
f5903d1c 00000046 00000046 00000000 00000000 f6960000 f696028c d12fcd2a 00000af8 00000000 f696028c c020375e 00000202 00000202 f670a014 f6960000 f5903d64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4704 20787 1
f4557d60 00000046 00000046 00000000 00000000 f69614a0 f696172c d12c7cad 00000af8 00000000 f696172c c020375e 00000202 00000202 f670a014 f69614a0 f4557da8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767b80

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c01862b1>] ? __slab_alloc+0xc1/0x5f0
 [<c02c4f65>] ? __prop_inc_single+0x65/0x80
 [<c01fbfc3>] get_list_bitmap+0x63/0xb0
 [<c01fdd5f>] do_journal_end+0xd0f/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

pdflush D 00000000 6080 20790 2
f09bfe3c 00000046 00000046 00000000 00000000 f7088000 f708828c d12caa96 00000af8 00000000 f708828c c020375e 00000202 00000202 f670a014 f7088000 f09bfe84 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766c80

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fe400>] journal_begin+0x80/0x110
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e95fe>] reiserfs_sync_fs+0x3e/0x80
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c01e964d>] reiserfs_write_super+0xd/0x10
 [<c018c3d4>] sync_supers+0x84/0xc0
 [<c0169330>] ? pdflush+0x0/0x1c0
 [<c016860b>] wb_kupdate+0x2b/0x100
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c0169330>] ? pdflush+0x0/0x1c0
 [<c01693f8>] pdflush+0xc8/0x1c0
 [<c01685e0>] ? wb_kupdate+0x0/0x100
 [<c013d86a>] kthread+0x3a/0x70
 [<c013d830>] ? kthread+0x0/0x70
 [<c0103677>] kernel_thread_helper+0x7/0x10

git-daemon D 00000000 6696 20791 2273
f3705ba8 00000046 00000046 00000000 00000000 f66fe720 f66fe9ac 4a31216d 00000afb 00000000 f66fe9ac c020375e 00000202 00000202 f670a014 f66fe720 f3705bf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c018440c>] ? slab_pad_check+0x3c/0x120
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c01666ee>] ? get_page_from_freelist+0x41e/0x530
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0
 [<c01b32f0>] do_mpage_readpage+0x340/0x490
 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0
 [<c016a415>] ? __lru_cache_add+0x65/0xa0
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01b352e>] mpage_readpages+0x9e/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01dc710>] ? reiserfs_readpages+0x0/0x20
 [<c01dc729>] reiserfs_readpages+0x19/0x20
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
 [<c016980e>] do_page_cache_readahead+0x4e/0x70
 [<c0162caf>] filemap_fault+0x30f/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

git-daemon D f63ede14 6888 20792 2273
f63ede20 00000046 00000000 f63ede14 00000046 f66fa940 f66fabcc 977b8ab0 00000afe 00000000 f66fabcc f63ede14 c014e52b c057e5a0 000003ae f63ede70 f63ede28 c04482c2 f63ede34 c04482f4 f63ede68 f63ede3c c0160d95 f63ede5c

Call Trace:
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c04482c2>] schedule+0x12/0x30
 [<c04482f4>] io_schedule+0x14/0x20
 [<c0160d95>] sync_page+0x55/0x60
 [<c0448881>] __wait_on_bit_lock+0x41/0x80
 [<c0160d40>] ? sync_page+0x0/0x60
 [<c0160d18>] __lock_page+0x88/0xa0
 [<c013daf0>] ? wake_bit_function+0x0/0x60
 [<c0160eb9>] find_lock_page+0x39/0x70
 [<c0162ad7>] filemap_fault+0x137/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

fcron D 00000000 7168 20797 2257
f5a0fd80 00000046 00000046 00000000 00000000 f6952940 f6952bcc df71112a 00000b25 00000000 f6952bcc c020375e 00000202 00000202 f670a014 f6952940 f5a0fdc8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01db389>] reiserfs_lookup+0x59/0x170
 [<c019b923>] ? d_alloc+0x123/0x1b0
 [<c019275f>] __lookup_hash+0xcf/0x130
 [<c01927e7>] lookup_hash+0x27/0x30
 [<c0195cb5>] do_filp_open+0x245/0x7a0
 [<c019ed68>] ? alloc_fd+0x28/0xf0
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c019ee18>] ? alloc_fd+0xd8/0xf0
 [<c0188879>] do_sys_open+0x59/0xf0
 [<c0188979>] sys_open+0x29/0x40
 [<c0102e48>] sysenter_do_call+0x12/0x36
Sched Debug Version: v0.09, 2.6.29-10247-g8e2c4f2-dirty #2
now at 12468457.474391 msecs
  .jiffies                                 : 1216523
  .sysctl_sched_latency                    : 20.000000
  .sysctl_sched_min_granularity            : 4.000000
  .sysctl_sched_wakeup_granularity         : 5.000000
  .sysctl_sched_child_runs_first           : 0.000001
  .sysctl_sched_features                   : 113917

cpu#0, 1830.032 MHz

  .nr_running                    : 1
  .load                          : 1024
  .nr_switches                   : 6163991
  .nr_load_updates               : 582498
  .nr_uninterruptible            : 21
  .next_balance                  : 0.000000
  .curr->pid                     : 20784
  .clock                         : 12465239.317891
  .cpu_load[0]                   : 1024
  .cpu_load[1]                   : 1024
  .cpu_load[2]                   : 1024
  .cpu_load[3]                   : 1024
  .cpu_load[4]                   : 1024
  .yld_count                     : 0
  .sched_switch                  : 0
  .sched_count                   : 778698744
  .sched_goidle                  : 307156
  .ttwu_count                    : 0
  .ttwu_local                    : 0
  .bkl_count                     : 275

cfs_rq[0]:

  .exec_clock                    : 1507714.391289
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 1444293.051451
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : 0.000000
  .nr_running                    : 1
  .load                          : 1024
  .nr_spread_over                : 35

rt_rq[0]:

  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R dbench 20784 1444293.051451 4569 120 1444293.051451 401016.637062 16471.092414

INFO: task svscan:2223 blocked for more than 480 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c0197ed8>] sys_getdents+0x68/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

INFO: lockdep is turned off.

#6 lore raw
Quoted 19 lines · Alexander Beregalov
Quoted 18 lines · Frederic Weisbecker
Quoted 15 lines · Alexander Beregalov

Hi Frederic

I've applied your patch.
This is UP system.
This warning seems related to your patch.

[ INFO: possible circular locking dependency detected ] 2.6.29-10247-g8e2c4f2-dirty #2
-------------------------------------------------------
udevadm/1239 is trying to acquire lock:
(&mm->mmap_sem){++++++}, at: [<c0173f52>] might_fault+0x52/0xa0

but task is already holding lock:
(sysfs_mutex){+.+.+.}, at: [<c01ccd36>] sysfs_readdir+0x56/0x200

Quoted 8 lines
Quoted 3 lines

Ah, I've never hit it. I'm not sure this is related to my patch but it could be.
Could you send me your config?

[Sent in private mail]

This should be related.
System is hung after dbench testing for some time.

Ok, I can reproduce these hangs with dbench. I will investigate on that and try to fix it soon.

Thanks a lot for this report,
Frederic.

Quoted 666 lines

SysRq : Emergency Sync
SysRq : Show Blocked State
task PC stack pid father
events/0 D 00000000 5672 6 2
f7067ee8 00000046 00000046 00000000 00000000 f7056720 f70569ac b6acece3 00000b44 00000000 f70569ac c018c1df 00000202 00000202 f6982a60 f7056720 f7067f30 c0448e3a 00000000 00000002 00000000 c018c1df 00000000 0018c3b2

Call Trace:
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c018c1df>] lock_super+0x1f/0x30
 [<c018c3b9>] sync_supers+0x69/0xc0
 [<c01a922c>] do_sync+0x1c/0x60
 [<c013a0e6>] ? worker_thread+0xe6/0x250
 [<c01a927d>] do_sync_work+0xd/0x20
 [<c013a148>] worker_thread+0x148/0x250
 [<c013a0e6>] ? worker_thread+0xe6/0x250
 [<c01a9270>] ? do_sync_work+0x0/0x20
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c013a000>] ? worker_thread+0x0/0x250
 [<c013d86a>] kthread+0x3a/0x70
 [<c013d830>] ? kthread+0x0/0x70
 [<c0103677>] kernel_thread_helper+0x7/0x10

metalog D 00000000 4936 2183 1
f7161cec 00000046 00000046 00000000 00000000 f6b8d280 f6b8d50c 1eba9be4 00000b44 00000000 f6b8d50c c020375e 00000202 00000202 f670a014 f6b8d280 f7161d34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0018440c

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aaad>] sys_write+0x3d/0x70
 [<c0102e48>] sysenter_do_call+0x12/0x36

svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c0197ed8>] sys_getdents+0x68/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

multilog D 00000000 4800 2236 2228
f731bcec 00000046 00000046 00000000 00000000 f6ba8000 f6ba828c 50105c29 00000b26 00000000 f6ba828c c020375e 00000202 00000202 f670a014 f6ba8000 f731bd34 c0448e3a 00000000 00000002 00000000 c020375e 00000000 0014ce9d

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e9674>] reiserfs_dirty_inode+0x24/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aaad>] sys_write+0x3d/0x70
 [<c0102e48>] sysenter_do_call+0x12/0x36

agetty D 00000000 5432 2441 1
f73cbba8 00000046 00000046 00000000 00000000 f73dd280 f73dd50c 6a069cd5 00000b33 00000000 f73dd50c c020375e 00000202 00000202 f670a014 f73dd280 f73cbbf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000246

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c044b07d>] ? _spin_unlock_irq+0x2d/0x50
 [<c0124b02>] ? finish_task_switch+0x62/0xb0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c0124aa0>] ? finish_task_switch+0x0/0xb0
 [<c044b2df>] ? __reacquire_kernel_lock+0x2f/0x40
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c0332d78>] ? serial8250_tx_empty+0x18/0x50
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0
 [<c01b32f0>] do_mpage_readpage+0x340/0x490
 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0
 [<c016a415>] ? __lru_cache_add+0x65/0xa0
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01b352e>] mpage_readpages+0x9e/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c04480b8>] ? __schedule+0x2f8/0x4f0
 [<c01dc710>] ? reiserfs_readpages+0x0/0x20
 [<c01dc729>] reiserfs_readpages+0x19/0x20
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
 [<c016980e>] do_page_cache_readahead+0x4e/0x70
 [<c0162caf>] filemap_fault+0x30f/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

screen D 00000000 5244 2529 2528
f612fd90 00000046 00000046 00000000 00000000 f6babde0 f6bac06c 33c73fab 00000b1c 00000000 f6bac06c c0193bf3 00000202 00000202 f6f9615c f6babde0 f612fdd8 c0448e3a 00000000 00000002 00000000 c0193bf3 00000000 0012fed0

Call Trace:
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c0193bf3>] ? do_lookup+0xd3/0x1c0
 [<c0193bf3>] do_lookup+0xd3/0x1c0
 [<c0193e49>] __link_path_walk+0x169/0xcc0
 [<c0194b53>] path_walk+0x53/0xb0
 [<c0194c8b>] do_path_lookup+0x8b/0x170
 [<c01958ea>] user_path_at+0x4a/0x80
 [<c01b42ba>] ? dnotify_parent+0x2a/0x80
 [<c018e471>] vfs_stat_fd+0x21/0x50
 [<c018e571>] vfs_stat+0x11/0x20
 [<c018e594>] sys_stat64+0x14/0x30
 [<c018abdf>] ? vfs_read+0xff/0x140
 [<c0317d70>] ? tty_read+0x0/0xb0
 [<c018acdd>] ? sys_read+0x3d/0x70
 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

rtorrent D 00000000 4084 2543 2542
f6325d88 00000046 00000046 00000000 00000000 f6b894a0 f6b8972c d79a38dd 00000af9 00000000 f6b8972c c020375e 00000202 00000202 f670a014 f6b894a0 f6325dd0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197a80>] ? filldir64+0x0/0xf0
 [<c0197e29>] sys_getdents64+0x69/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

bash D 00000000 5788 2546 2529
f585dcd8 00000046 00000046 00000000 00000000 f6b8bde0 f6b8c06c 188b381a 00000b1b 00000000 f6b8c06c c020375e 00000202 00000202 f670a014 f6b8bde0 f585dd20 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01db389>] reiserfs_lookup+0x59/0x170
 [<c019b923>] ? d_alloc+0x123/0x1b0
 [<c0193cb1>] do_lookup+0x191/0x1c0
 [<c0194305>] __link_path_walk+0x625/0xcc0
 [<c0194b53>] path_walk+0x53/0xb0
 [<c0194c8b>] do_path_lookup+0x8b/0x170
 [<c01958ea>] user_path_at+0x4a/0x80
 [<c018dda4>] ? cp_new_stat64+0xe4/0x100
 [<c018e471>] vfs_stat_fd+0x21/0x50
 [<c018e571>] vfs_stat+0x11/0x20
 [<c018e594>] sys_stat64+0x14/0x30
 [<c02c9d84>] ? trace_hardirqs_on_thunk+0xc/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4816 20778 1
f44bbd1c 00000046 00000046 00000000 00000000 ce105280 ce10550c d12c8ac7 00000af8 00000000 ce10550c c020375e 00000202 00000202 f670a014 ce105280 f44bbd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766a00

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4660 20779 1
f44b3de4 00000046 00000046 00000000 00000000 ce106720 ce1069ac d12c836c 00000af8 00000000 ce1069ac c020375e 00000202 00000202 f670a014 ce106720 f44b3e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767400

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c04480b8>] ? __schedule+0x2f8/0x4f0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4724 20780 1
f6389de4 00000046 00000046 00000000 00000000 ce102940 ce102bcc d12c7533 00000af8 00000000 ce102bcc c020375e 00000202 00000202 f670a014 ce102940 f6389e2c c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c0169c29>] ? pagevec_lookup_tag+0x29/0x40
 [<c0168241>] ? write_cache_pages+0xd1/0x350
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4944 20781 1
f589dc84 00000046 00000046 00000000 00000000 ce1014a0 ce10172c d0f2ddbd 00000af8 00000000 ce10172c c020375e 00000202 00000202 f670a014 ce1014a0 f589dccc c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fe400>] journal_begin+0x80/0x110
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e9686>] reiserfs_dirty_inode+0x36/0xb0
 [<c01a0f7a>] ? mnt_want_write+0x1a/0xc0
 [<c01a613c>] __mark_inode_dirty+0x2c/0x180
 [<c019c6f5>] file_update_time+0xc5/0xe0
 [<c01634eb>] __generic_file_aio_write_nolock+0x18b/0x520
 [<c0163980>] ? generic_file_aio_write+0x50/0xd0
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c02ca436>] ? copy_to_user+0x36/0x130
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aa6b>] sys_pwrite64+0x7b/0x80
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4832 20782 1
f4617a64 00000046 00000046 00000000 00000000 ce103de0 ce10406c d12ca393 00000af8 00000000 ce10406c c020375e 00000202 00000202 f670a014 ce103de0 f4617aac c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767900

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c01fe400>] journal_begin+0x80/0x110
 [<c01fe4c2>] ? reiserfs_persistent_transaction+0x32/0x90
 [<c01fe4d1>] reiserfs_persistent_transaction+0x41/0x90
 [<c01df460>] reiserfs_get_block+0x1e0/0x14f0
 [<c018dbbe>] ? inode_get_bytes+0x1e/0x50
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01ab2c3>] ? create_empty_buffers+0x33/0xa0
 [<c01ad424>] __block_prepare_write+0x1a4/0x380
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01ad7a8>] block_write_begin+0x48/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01ddfbc>] reiserfs_write_begin+0xec/0x200
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01618f5>] generic_file_buffered_write+0x155/0x2c0
 [<c0163520>] __generic_file_aio_write_nolock+0x1c0/0x520
 [<c0163993>] generic_file_aio_write+0x63/0xd0
 [<c018a331>] do_sync_write+0xd1/0x110
 [<c0173f52>] ? might_fault+0x52/0xa0
 [<c013dab0>] ? autoremove_wake_function+0x0/0x40
 [<c01e23ba>] reiserfs_file_write+0x8a/0xa0
 [<c018a94c>] vfs_write+0x9c/0x140
 [<c02ca436>] ? copy_to_user+0x36/0x130
 [<c01e2330>] ? reiserfs_file_write+0x0/0xa0
 [<c018aa6b>] sys_pwrite64+0x7b/0x80
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4764 20783 1
f5833e48 00000046 00000046 00000000 00000000 f73de720 f73de9ac d12c96e7 00000af8 00000000 f73de9ac c020375e 00000202 00000202 f670a014 f73de720 f5833e90 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c01fe9c1>] reiserfs_commit_for_inode+0xa1/0x1e0
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4600 20785 1
f460bd1c 00000046 00000046 00000000 00000000 f6b88000 f6b8828c d12c9d18 00000af8 00000000 f6b8828c c020375e 00000202 00000202 f670a014 f6b88000 f460bd64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4548 20786 1
f5903d1c 00000046 00000046 00000000 00000000 f6960000 f696028c d12fcd2a 00000af8 00000000 f696028c c020375e 00000202 00000202 f670a014 f6960000 f5903d64 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766280

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c018570d>] ? check_object+0x13d/0x200
 [<c01fba72>] flush_commit_list+0x2d2/0x7c0
 [<c0121ab0>] ? __wake_up+0x40/0x50
 [<c01fde19>] do_journal_end+0xdc9/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

dbench D 00000000 4704 20787 1
f4557d60 00000046 00000046 00000000 00000000 f69614a0 f696172c d12c7cad 00000af8 00000000 f696172c c020375e 00000202 00000202 f670a014 f69614a0 f4557da8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00767b80

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fb8c5>] flush_commit_list+0x125/0x7c0
 [<c01862b1>] ? __slab_alloc+0xc1/0x5f0
 [<c02c4f65>] ? __prop_inc_single+0x65/0x80
 [<c01fbfc3>] get_list_bitmap+0x63/0xb0
 [<c01fdd5f>] do_journal_end+0xd0f/0xf50
 [<c0133420>] ? process_timeout+0x0/0x10
 [<c01fdffd>] journal_end_sync+0x5d/0x80
 [<c01feadd>] reiserfs_commit_for_inode+0x1bd/0x1e0
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e1e68>] reiserfs_sync_file+0x38/0x90
 [<c01a905d>] vfs_fsync+0x6d/0xe0
 [<c01a90fd>] do_fsync+0x2d/0x50
 [<c01a914d>] sys_fsync+0xd/0x10
 [<c0102e48>] sysenter_do_call+0x12/0x36

pdflush D 00000000 6080 20790 2
f09bfe3c 00000046 00000046 00000000 00000000 f7088000 f708828c d12caa96 00000af8 00000000 f708828c c020375e 00000202 00000202 f670a014 f7088000 f09bfe84 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00766c80

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01fe08b>] do_journal_begin_r+0x6b/0x320
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01fe400>] journal_begin+0x80/0x110
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c01e95fe>] reiserfs_sync_fs+0x3e/0x80
 [<c0449203>] ? mutex_lock_nested+0x33/0x40
 [<c018c1df>] ? lock_super+0x1f/0x30
 [<c01e964d>] reiserfs_write_super+0xd/0x10
 [<c018c3d4>] sync_supers+0x84/0xc0
 [<c0169330>] ? pdflush+0x0/0x1c0
 [<c016860b>] wb_kupdate+0x2b/0x100
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c0169330>] ? pdflush+0x0/0x1c0
 [<c01693f8>] pdflush+0xc8/0x1c0
 [<c01685e0>] ? wb_kupdate+0x0/0x100
 [<c013d86a>] kthread+0x3a/0x70
 [<c013d830>] ? kthread+0x0/0x70
 [<c0103677>] kernel_thread_helper+0x7/0x10

git-daemon D 00000000 6696 20791 2273
f3705ba8 00000046 00000046 00000000 00000000 f66fe720 f66fe9ac 4a31216d 00000afb 00000000 f66fe9ac c020375e 00000202 00000202 f670a014 f66fe720 f3705bf0 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01df2e6>] reiserfs_get_block+0x66/0x14f0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c018440c>] ? slab_pad_check+0x3c/0x120
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c01666ee>] ? get_page_from_freelist+0x41e/0x530
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c016108e>] ? add_to_page_cache_locked+0x5e/0xe0
 [<c01b32f0>] do_mpage_readpage+0x340/0x490
 [<c01610e9>] ? add_to_page_cache_locked+0xb9/0xe0
 [<c016a415>] ? __lru_cache_add+0x65/0xa0
 [<c0161184>] ? add_to_page_cache_lru+0x74/0x80
 [<c01b352e>] mpage_readpages+0x9e/0xe0
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01dc710>] ? reiserfs_readpages+0x0/0x20
 [<c01dc729>] reiserfs_readpages+0x19/0x20
 [<c01df280>] ? reiserfs_get_block+0x0/0x14f0
 [<c01696ff>] __do_page_cache_readahead+0x13f/0x200
 [<c016980e>] do_page_cache_readahead+0x4e/0x70
 [<c0162caf>] filemap_fault+0x30f/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

git-daemon D f63ede14 6888 20792 2273
f63ede20 00000046 00000000 f63ede14 00000046 f66fa940 f66fabcc 977b8ab0 00000afe 00000000 f66fabcc f63ede14 c014e52b c057e5a0 000003ae f63ede70 f63ede28 c04482c2 f63ede34 c04482f4 f63ede68 f63ede3c c0160d95 f63ede5c

Call Trace:
 [<c014e52b>] ? trace_hardirqs_on+0xb/0x10
 [<c04482c2>] schedule+0x12/0x30
 [<c04482f4>] io_schedule+0x14/0x20
 [<c0160d95>] sync_page+0x55/0x60
 [<c0448881>] __wait_on_bit_lock+0x41/0x80
 [<c0160d40>] ? sync_page+0x0/0x60
 [<c0160d18>] __lock_page+0x88/0xa0
 [<c013daf0>] ? wake_bit_function+0x0/0x60
 [<c0160eb9>] find_lock_page+0x39/0x70
 [<c0162ad7>] filemap_fault+0x137/0x430
 [<c0175f6d>] __do_fault+0x3d/0x360
 [<c0176aa0>] handle_mm_fault+0x100/0x590
 [<c014159d>] ? down_read_trylock+0x5d/0x70
 [<c011aff1>] do_page_fault+0x121/0x280
 [<c011aed0>] ? do_page_fault+0x0/0x280
 [<c044b40a>] error_code+0x6a/0x70
 [<c011aed0>] ? do_page_fault+0x0/0x280

fcron D 00000000 7168 20797 2257
f5a0fd80 00000046 00000046 00000000 00000000 f6952940 f6952bcc df71112a 00000b25 00000000 f6952bcc c020375e 00000202 00000202 f670a014 f6952940 f5a0fdc8 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00000000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01db389>] reiserfs_lookup+0x59/0x170
 [<c019b923>] ? d_alloc+0x123/0x1b0
 [<c019275f>] __lookup_hash+0xcf/0x130
 [<c01927e7>] lookup_hash+0x27/0x30
 [<c0195cb5>] do_filp_open+0x245/0x7a0
 [<c019ed68>] ? alloc_fd+0x28/0xf0
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c019ee18>] ? alloc_fd+0xd8/0xf0
 [<c0188879>] do_sys_open+0x59/0xf0
 [<c0188979>] sys_open+0x29/0x40
 [<c0102e48>] sysenter_do_call+0x12/0x36
Sched Debug Version: v0.09, 2.6.29-10247-g8e2c4f2-dirty #2
now at 12468457.474391 msecs
  .jiffies                                 : 1216523
  .sysctl_sched_latency                    : 20.000000
  .sysctl_sched_min_granularity            : 4.000000
  .sysctl_sched_wakeup_granularity         : 5.000000
  .sysctl_sched_child_runs_first           : 0.000001
  .sysctl_sched_features                   : 113917

cpu#0, 1830.032 MHz

  .nr_running                    : 1
  .load                          : 1024
  .nr_switches                   : 6163991
  .nr_load_updates               : 582498
  .nr_uninterruptible            : 21
  .next_balance                  : 0.000000
  .curr->pid                     : 20784
  .clock                         : 12465239.317891
  .cpu_load[0]                   : 1024
  .cpu_load[1]                   : 1024
  .cpu_load[2]                   : 1024
  .cpu_load[3]                   : 1024
  .cpu_load[4]                   : 1024
  .yld_count                     : 0
  .sched_switch                  : 0
  .sched_count                   : 778698744
  .sched_goidle                  : 307156
  .ttwu_count                    : 0
  .ttwu_local                    : 0
  .bkl_count                     : 275

cfs_rq[0]:

  .exec_clock                    : 1507714.391289
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 1444293.051451
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : 0.000000
  .nr_running                    : 1
  .load                          : 1024
  .nr_spread_over                : 35

rt_rq[0]:

  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
R dbench 20784 1444293.051451 4569 120 1444293.051451 401016.637062 16471.092414

INFO: task svscan:2223 blocked for more than 480 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
svscan D 00000000 5384 2223 1
f731dd8c 00000046 00000046 00000000 00000000 f6740000 f674028c eaa8d98d 00000af9 00000000 f674028c c020375e 00000202 00000202 f670a014 f6740000 f731ddd4 c0448e3a 00000000 00000002 00000000 c020375e 00000000 00740000

Call Trace:
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0448e3a>] __mutex_lock_common+0x11a/0x430
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c0449203>] mutex_lock_nested+0x33/0x40
 [<c020375e>] ? reiserfs_write_lock+0x1e/0x30
 [<c020375e>] reiserfs_write_lock+0x1e/0x30
 [<c01e266a>] reiserfs_readdir_dentry+0x5a/0x740
 [<c018b46f>] ? file_move+0x1f/0x50
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c014ce9d>] ? print_lock_contention_bug+0x1d/0x110
 [<c044ae07>] ? _spin_unlock+0x27/0x50
 [<c018b48e>] ? file_move+0x3e/0x50
 [<c0173f9d>] ? might_fault+0x9d/0xa0
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c0449013>] ? __mutex_lock_common+0x2f3/0x430
 [<c0197d6f>] ? vfs_readdir+0x4f/0xa0
 [<c01e2d67>] reiserfs_readdir+0x17/0x20
 [<c0197d9e>] vfs_readdir+0x7e/0xa0
 [<c0197b70>] ? filldir+0x0/0xd0
 [<c0197ed8>] sys_getdents+0x68/0xb0
 [<c0102e48>] sysenter_do_call+0x12/0x36

INFO: lockdep is turned off.

#7 lore raw
In reply to Frederic Weisbecker
Quoted 2 lines · Frederic Weisbecker

So the new lock that replaces the bkl here is a per superblock mutex

You have to be very careful with this. Mutexes can be slower than spinlocks (and the new BKL is a spinlock) in some situations, they typically schedule much more etc., which can be costly. So this might have actually made it slower if you're unlucky. There were some experimental changes recently to make mutexes more behave like spinlocks, but I don't think they're fully performance equivalent.

Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?

Quoted 1 line

After this patch, reiserfs suffers from a slight performance regression (for now).

That might be related to the scheduling behaviour. Watch the reschedule counters in vmstat 1

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#8 lore raw
Quoted 4 lines · Andi Kleen

You have to be very careful with this. Mutexes can be slower than spinlocks (and the new BKL is a spinlock) in some situations, they typically schedule much more etc., which can be costly.

Actually, with the new adaptive spinning, that basically shouldn't be true any more. Or rather, you should need some really bad/unlucky situation for it to scheduler more than necessary, and if the locker _acts_ like a spinlock (ie it doesn't block while holding the lock), performance should approach a spinlock.

That said, there are definitely reasons why a mutex can be slower than the BKL, and the whole "BKL gets implicitly dropped at sleep time" is very high on that list of reasons. The sleeping patterns can be _very_ different with a mutex than with a BKL.

Quoted 2 lines

Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?

I do agree that a filesystem should try to avoid sleeping locks if at all possible, especially on the paths that the VM uses for writeback. But on the other hand, I think the issue with reiserfs is just the bad latencies that the BKL can cause, and then it doesn't matter.

Linus

#9 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 2 lines

Better would be to use spinlocks if possible. I guess you just would need to find all sleep points and wrap them with lock dropping?

I do agree that a filesystem should try to avoid sleeping locks if at all possible, especially on the paths that the VM uses for writeback. But on the other hand, I think the issue with reiserfs is just the bad latencies that the BKL can cause, and then it doesn't matter.

The main motivator is the tip:core/kill-the-BKL tree: we are working on removing the BKL from all of the kernel, once and forever. We are actually quite close to that end goal: reiser3 was the last big stumbling block and it's great that Frederic is tackling that.

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL auto-dropping on schedule() it would be rather fragile to use spinlocks, and it would take forever to validate the result. Just one codepath missed with having some rare scheduling possibility and we'd have a kernel crash down the road.

Ingo

#10 lore raw
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. I also got a few reiser3 fs still around, it tended to perform very well
on kernel hacker workloads.

Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.

Quoted 2 lines

auto-dropping on schedule() it would be rather fragile to use spinlocks, and it would take forever to validate the result.

Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#11 lore raw
Quoted 13 lines · Andi Kleen
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. I also got a few reiser3 fs still around, it tended to perform very well
on kernel hacker workloads.

Quoted 3 lines

Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.

It's not only about performances, latency is also a fair goal here. Moreover, reiserfs is not the only one concerned while dropping its Bkl, this is one big source of contention less for the rest of the system.

And concerning performance, I'm working on it :-)

Frederic.

Quoted 11 lines
Quoted 2 lines

auto-dropping on schedule() it would be rather fragile to use spinlocks, and it would take forever to validate the result.

Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#12 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]

( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )

Quoted 2 lines

[...] I also got a few reiser3 fs still around, it tended to perform very well on kernel hacker workloads.

Then i am sure you must like this patch: it introduces a per superblock lock, splitting up the big BKL serialization. You totally failed to even acknowledge that advantage, maybe you missed that aspect?

For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.

That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.

Frederic reported a slight drop in single-threaded performance, to be expected from a work in progress patch.

Ingo

#13 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

Quoted 8 lines
Quoted 4 lines

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]

( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )

and that is bad? Isn't that more a sign of a certain 'stableness' - which is seriously lacking from more 'popular' filesystems like ext3? btw, does ext3 finally turn barriers on per default or does it still prefer letting data rot in drive caches because of 'performance'?

how many users would ext4 have if fedora&ubuntu would not force it onto the unwashed masses? And is that really a good idea after the amusing episode of the last month?

btw, what are ext3&4 doing when the underlying device does not support barriers (like software raid)?

#14 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

Quoted 8 lines
Quoted 4 lines

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently. [...]

( Drop the condescending tone please - i very much know that SuSE
installed reiser3 by default for years. It is still a legacy filesystem and no new development has gone into it for years. )

legacy (adj) — A pejorative term used in the computer industry meaning "it works"

http://homepages.tesco.net/~J.deBoynePollard/FGA/legacy-is-not-a-pejorative.html

It's by far the best of all the currently available and stable filesystems for our workload (big honking Cyrus IMAP boxes)

There have been bugfixes and occasional bits and pieces over the years. We applied a couple of patches for a while until they were accepted upstream only a couple of years ago.

It doesn't get much new development because, gee, "it works". Some people like their filesystem to keep just working in a predictable way.

I would be very concerned if people though it was OK to break it just because these shiny new ext4 and btrfs filesystem which are _NOT_YET_READY_ had replaced it.

Ta.

Quoted 8 lines
Quoted 2 lines

[...] I also got a few reiser3 fs still around, it tended to perform very well on kernel hacker workloads.

Then i am sure you must like this patch: it introduces a per superblock lock, splitting up the big BKL serialization. You totally failed to even acknowledge that advantage, maybe you missed that aspect?

Quoted 7 lines

For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.

That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.

Yes - I'm certainly interested in that.

That said, we have a box with 83 reiserfs partitions on it, and which is constrained by IO (main servers really don't need much CPU). Performance is pretty good even now.

So - I'm interested in this patch series, but not at the expense of making reiserfs any less stable. Our customers, funnily enough, like it when our service is stable!

Thanks,

Bron.
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#15 lore raw
Quoted 10 lines · Bron Gondwana
Quoted 7 lines

For example, if you have /home and / on separate reiser3 filesystems, you could see as much as a 200% jump in performance straight away on certain workloads, on a dual-core box.

That big BKL overhead is a real reiser3 scalability problem - especially on reiser3 using servers which are likely to have several filesystems on the same box.

Yes - I'm certainly interested in that.

Quoted 7 lines

That said, we have a box with 83 reiserfs partitions on it, and which is constrained by IO (main servers really don't need much CPU). Performance is pretty good even now.

So - I'm interested in this patch series, but not at the expense of making reiserfs any less stable. Our customers, funnily enough, like it when our service is stable!

Definitely so :-)

Ingo

#16 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen
Quoted 5 lines

Using a mutex seems like the sane choice here. I'd advocate spinlocks for a new filesystem any day (but even there it's a fine choice to have a mutex, if top of the line scalability is not an issue).

But for a legacy filesystem like reiser3, which depended on the BKL

reiser3 is much more widely used in the user base than a lot of "non legacy" file systems. It's very likely it has significantly
more users than ext4 for example. Remember that it was the default file system for a major distribution until very recently.

... and based on track record probably still should be :/

--Toby
(Who has no idea why good solid code that works really well magically gets labelled 'legacy' and shelved in favour of flaky, immature code.)

Quoted 23 lines

I also got a few
reiser3 fs still around, it tended to perform very well on kernel hacker workloads.

Given all that I think the current performance penalties Frederic reports are not acceptable. Dropping BKL is not a cause in itself, but should just improve performance.

Quoted 2 lines

auto-dropping on schedule() it would be rather fragile to use spinlocks, and it would take forever to validate the result.

Not convinced it would be that hard. It could be probably done with some straight forward static code analysis. And after that since there's not much development going anymore it's unlikely to break again.

-Andi

Signature · 5 lines
[email protected] -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Acked-by Peter Zijlstra
#17 lore raw
In reply to Andi Kleen
Quoted 5 lines · Andi Kleen
Quoted 2 lines

After this patch, reiserfs suffers from a slight performance regression (for now).

That might be related to the scheduling behaviour. Watch the reschedule counters in vmstat 1

FYI, Frederic is one of the main upstream tracing developers, and i can assure you that when working on scalability he watches something far more sophisticated than "vmstat 1" ;-)

See the commit below for example.

Thanks,

Ingo

--------------------->
From 8f350bb74293dded6c51870c34a74645b054caba Mon Sep 17 00:00:00 2001
From: Frederic Weisbecker <[email protected]>
Date: Mon, 6 Apr 2009 01:49:33 +0200
Subject: [PATCH] tracing/lockdep: report the time waited for a lock

While trying to optimize the new lock on reiserfs to replace the bkl, I find the lock tracing very useful though it lacks something important for performance (and latency) instrumentation:
the time a task waits for a lock.

That's what this patch implements:

  bash-4816  [000]   202.652815: lock_contended: lock_contended: &sb->s_type->i_mutex_key
  bash-4816  [000]   202.652819: lock_acquired: &rq->lock (0.000 us)
 <...>-4787  [000]   202.652825: lock_acquired: &rq->lock (0.000 us)
 <...>-4787  [000]   202.652829: lock_acquired: &rq->lock (0.000 us)
  bash-4816  [000]   202.652833: lock_acquired: &sb->s_type->i_mutex_key (16.005 us)

As shown above, the "lock acquired" field is followed by the time it has been waiting for the lock. Usually, a lock contended entry is followed by a near lock_acquired entry with a non-zero time waited.

LKML-Reference: <[email protected]>

---
 include/trace/lockdep_event_types.h |   23 ++++++++++++++++++-----
 kernel/lockdep.c                    |    8 ++++----
 2 files changed, 22 insertions(+), 9 deletions(-)
include/trace/lockdep_event_types.h +18 -5
index adccfcd..863f1e4 100644
@@ -32,11 +32,24 @@ TRACE_FORMAT(lock_contended,
 	TP_FMT("%s", lock->name)
 	);
 
-TRACE_FORMAT(lock_acquired,
-	TP_PROTO(struct lockdep_map *lock, unsigned long ip),
-	TP_ARGS(lock, ip),
-	TP_FMT("%s", lock->name)
-	);
+TRACE_EVENT(lock_acquired,
+	TP_PROTO(struct lockdep_map *lock, unsigned long ip, s64 waittime),
+
+	TP_ARGS(lock, ip, waittime),
+
+	TP_STRUCT__entry(
+		__field(const char *, name)
+		__field(unsigned long, wait_usec)
+		__field(unsigned long, wait_nsec_rem)
+	),
+	TP_fast_assign(
+		__entry->name = lock->name;
+		__entry->wait_nsec_rem = do_div(waittime, NSEC_PER_USEC);
+		__entry->wait_usec = (unsigned long) waittime;
+	),
+	TP_printk("%s (%lu.%03lu us)", __entry->name, __entry->wait_usec,
+				       __entry->wait_nsec_rem)
+);
 
 #endif
 #endif
kernel/lockdep.c +4 -4
index b0f0118..c4582a6 100644
@@ -3061,6 +3061,8 @@ found_it:
 	put_lock_stats(stats);
 }
 
+DEFINE_TRACE(lock_acquired);
+
 static void
 __lock_acquired(struct lockdep_map *lock, unsigned long ip)
 {
@@ -3099,6 +3101,8 @@ found_it:
 		hlock->holdtime_stamp = now;
 	}
 
+	trace_lock_acquired(lock, ip, waittime);
+
 	stats = get_lock_stats(hlock_class(hlock));
 	if (waittime) {
 		if (hlock->read)
@@ -3137,14 +3141,10 @@ void lock_contended(struct lockdep_map *lock, unsigned long ip)
 }
 EXPORT_SYMBOL_GPL(lock_contended);
 
-DEFINE_TRACE(lock_acquired);
-
 void lock_acquired(struct lockdep_map *lock, unsigned long ip)
 {
 	unsigned long flags;
 
-	trace_lock_acquired(lock, ip);
-
 	if (unlikely(!lock_stat))
 		return;