Skip to contentSkip to conversation

possible circular locking in reiserfs_unpack

Bug Jarek Poplawski 3 participants 8 messages Sep 5, 2010 – Sep 22, 2010

lore
Share & tools
lore
#0 lore raw

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Jarek P.

[   92.766639] =======================================================
[   92.767222] [ INFO: possible circular locking dependency detected ]
[   92.767222] 2.6.35c #13
[   92.767222] -------------------------------------------------------
[   92.767222] lilo/1606 is trying to acquire lock:
[   92.767222]  (&sb->s_type->i_mutex_key#8){+.+.+.}, at: [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222] 
[   92.767222] but task is already holding lock:
[   92.767222]  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222] 
[   92.767222] which lock already depends on the new lock.
[   92.767222] 
[   92.767222] 
[   92.767222] the existing dependency chain (in reverse order) is:
[   92.767222] 
[   92.767222] -> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[   92.767222]        [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]        [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]        [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]        [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222]        [<d0329e9a>] reiserfs_lookup_privroot+0x2a/0x90 [reiserfs]
[   92.767222]        [<d0316b81>] reiserfs_fill_super+0x941/0xe60 [reiserfs]
[   92.767222]        [<c10b7d17>] get_sb_bdev+0x117/0x170
[   92.767222]        [<d0313e21>] get_super_block+0x21/0x30 [reiserfs]
[   92.767222]        [<c10b74ba>] vfs_kern_mount+0x6a/0x1b0
[   92.767222]        [<c10b7659>] do_kern_mount+0x39/0xe0
[   92.767222]        [<c10cebe0>] do_mount+0x340/0x790
[   92.767222]        [<c10cf0b4>] sys_mount+0x84/0xb0
[   92.767222]        [<c12f25cd>] syscall_call+0x7/0xb
[   92.767222] 
[   92.767222] -> #0 (&sb->s_type->i_mutex_key#8){+.+.+.}:
[   92.767222]        [<c1056186>] __lock_acquire+0x1026/0x1180
[   92.767222]        [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]        [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]        [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]        [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]        [<d0329772>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   92.767222]        [<c10c3228>] vfs_ioctl+0x28/0xa0
[   92.767222]        [<c10c3c5d>] do_vfs_ioctl+0x32d/0x5c0
[   92.767222]        [<c10c3f53>] sys_ioctl+0x63/0x70
[   92.767222]        [<c12f25cd>] syscall_call+0x7/0xb
[   92.767222] 
[   92.767222] other info that might help us debug this:
[   92.767222] 
[   92.767222] 1 lock held by lilo/1606:
[   92.767222]  #0:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222] 
[   92.767222] stack backtrace:
[   92.767222] Pid: 1606, comm: lilo Not tainted 2.6.35c #13
[   92.767222] Call Trace:
[   92.767222]  [<c12ef64a>] ? printk+0x18/0x1e
[   92.767222]  [<c1054212>] print_circular_bug+0xd2/0xe0
[   92.767222]  [<c1056186>] __lock_acquire+0x1026/0x1180
[   92.767222]  [<c1089489>] ? __generic_file_aio_write+0x1c9/0x550
[   92.767222]  [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f0b08>] ? __mutex_lock_common+0x318/0x410
[   92.767222]  [<d032a268>] ? reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222]  [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f0c58>] ? mutex_lock_nested+0x18/0x20
[   92.767222]  [<d0329772>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   92.767222]  [<d0329500>] ? reiserfs_ioctl+0x0/0x320 [reiserfs]
[   92.767222]  [<c10c3228>] vfs_ioctl+0x28/0xa0
[   92.767222]  [<c10c3c5d>] do_vfs_ioctl+0x32d/0x5c0
[   92.767222]  [<c109a428>] ? might_fault+0x88/0x90
[   92.767222]  [<c109a3e2>] ? might_fault+0x42/0x90
[   92.767222]  [<c10b6638>] ? fget_light+0xf8/0x2f0
[   92.767222]  [<c10c3f53>] sys_ioctl+0x63/0x70
[   92.767222]  [<c12f25cd>] syscall_call+0x7/0xb
#1 lore raw
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

Thanks.

Quoted 74 lines
[   92.766639] =======================================================
[   92.767222] [ INFO: possible circular locking dependency detected ]
[   92.767222] 2.6.35c #13
[   92.767222] -------------------------------------------------------
[   92.767222] lilo/1606 is trying to acquire lock:
[   92.767222]  (&sb->s_type->i_mutex_key#8){+.+.+.}, at: [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222] 
[   92.767222] but task is already holding lock:
[   92.767222]  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222] 
[   92.767222] which lock already depends on the new lock.
[   92.767222] 
[   92.767222] 
[   92.767222] the existing dependency chain (in reverse order) is:
[   92.767222] 
[   92.767222] -> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[   92.767222]        [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]        [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]        [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]        [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222]        [<d0329e9a>] reiserfs_lookup_privroot+0x2a/0x90 [reiserfs]
[   92.767222]        [<d0316b81>] reiserfs_fill_super+0x941/0xe60 [reiserfs]
[   92.767222]        [<c10b7d17>] get_sb_bdev+0x117/0x170
[   92.767222]        [<d0313e21>] get_super_block+0x21/0x30 [reiserfs]
[   92.767222]        [<c10b74ba>] vfs_kern_mount+0x6a/0x1b0
[   92.767222]        [<c10b7659>] do_kern_mount+0x39/0xe0
[   92.767222]        [<c10cebe0>] do_mount+0x340/0x790
[   92.767222]        [<c10cf0b4>] sys_mount+0x84/0xb0
[   92.767222]        [<c12f25cd>] syscall_call+0x7/0xb
[   92.767222] 
[   92.767222] -> #0 (&sb->s_type->i_mutex_key#8){+.+.+.}:
[   92.767222]        [<c1056186>] __lock_acquire+0x1026/0x1180
[   92.767222]        [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]        [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]        [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]        [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]        [<d0329772>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   92.767222]        [<c10c3228>] vfs_ioctl+0x28/0xa0
[   92.767222]        [<c10c3c5d>] do_vfs_ioctl+0x32d/0x5c0
[   92.767222]        [<c10c3f53>] sys_ioctl+0x63/0x70
[   92.767222]        [<c12f25cd>] syscall_call+0x7/0xb
[   92.767222] 
[   92.767222] other info that might help us debug this:
[   92.767222] 
[   92.767222] 1 lock held by lilo/1606:
[   92.767222]  #0:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a268>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222] 
[   92.767222] stack backtrace:
[   92.767222] Pid: 1606, comm: lilo Not tainted 2.6.35c #13
[   92.767222] Call Trace:
[   92.767222]  [<c12ef64a>] ? printk+0x18/0x1e
[   92.767222]  [<c1054212>] print_circular_bug+0xd2/0xe0
[   92.767222]  [<c1056186>] __lock_acquire+0x1026/0x1180
[   92.767222]  [<c1089489>] ? __generic_file_aio_write+0x1c9/0x550
[   92.767222]  [<c1056347>] lock_acquire+0x67/0x80
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f083d>] __mutex_lock_common+0x4d/0x410
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f0b08>] ? __mutex_lock_common+0x318/0x410
[   92.767222]  [<d032a268>] ? reiserfs_write_lock+0x28/0x40 [reiserfs]
[   92.767222]  [<c12f0c58>] mutex_lock_nested+0x18/0x20
[   92.767222]  [<d0329450>] ? reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<d0329450>] reiserfs_unpack+0x60/0x110 [reiserfs]
[   92.767222]  [<c12f0c58>] ? mutex_lock_nested+0x18/0x20
[   92.767222]  [<d0329772>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   92.767222]  [<d0329500>] ? reiserfs_ioctl+0x0/0x320 [reiserfs]
[   92.767222]  [<c10c3228>] vfs_ioctl+0x28/0xa0
[   92.767222]  [<c10c3c5d>] do_vfs_ioctl+0x32d/0x5c0
[   92.767222]  [<c109a428>] ? might_fault+0x88/0x90
[   92.767222]  [<c109a3e2>] ? might_fault+0x42/0x90
[   92.767222]  [<c10b6638>] ? fget_light+0xf8/0x2f0
[   92.767222]  [<c10c3f53>] sys_ioctl+0x63/0x70
[   92.767222]  [<c12f25cd>] syscall_call+0x7/0xb
#2 lore raw
Quoted 8 lines · Andrew Morton
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Quoted 4 lines

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

Thanks.

Ah, when you see &REISERFS_SB(s)->lock in a bug report, don't hesitate to blame me :-)

This is a problem resulting from the bkl conversion to a mutex that introduced a lot of new locking dependencies. Most of them have been fixed, but for less tested paths like ioctl, we hear about it later.

Does the following patch fixes the issue?
If so, I'll make a proper changelog and put the appropriate 2.6.33-35 stable tags for the backport.

Thnaks!

fs/reiserfs/ioctl.c +1 -1
index f53505d..679d502 100644
@@ -188,7 +188,7 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 	reiserfs_write_lock(inode->i_sb);
 
 	write_from = inode->i_size & (blocksize - 1);
#3 lore raw
Quoted 11 lines · Frederic Weisbecker
Quoted 10 lines · Andrew Morton
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

I'll try if Frederic's patch doesn't help. But, generally, it seems with this kind of (not too long) lockdep info it should be much faster to have a look of somebody with a basic reiserfs locking knowledge.;-)

Quoted 7 lines
Quoted 2 lines

Thanks.

Ah, when you see &REISERFS_SB(s)->lock in a bug report, don't hesitate to blame me :-)

Quoted 7 lines

This is a problem resulting from the bkl conversion to a mutex that introduced a lot of new locking dependencies. Most of them have been fixed, but for less tested paths like ioctl, we hear about it later.

Does the following patch fixes the issue?
If so, I'll make a proper changelog and put the appropriate 2.6.33-35 stable tags for the backport.

I should be able to test it when back home (within 9 hours).

Thanks,
Jarek P.

Quoted 18 lines

Thnaks!

fs/reiserfs/ioctl.c +1 -1
index f53505d..679d502 100644
@@ -188,7 +188,7 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 	reiserfs_write_lock(inode->i_sb);
 
 	write_from = inode->i_size & (blocksize - 1);
#4 lore raw
In reply to Frederic Weisbecker

Frederic Weisbecker wrote, On 12/23/-28158 08:59 PM:

Quoted 34 lines · Andrew Morton
Quoted 11 lines
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

Thanks.

Ah, when you see &REISERFS_SB(s)->lock in a bug report, don't hesitate to blame me :-)

This is a problem resulting from the bkl conversion to a mutex that introduced a lot of new locking dependencies. Most of them have been fixed, but for less tested paths like ioctl, we hear about it later.

Does the following patch fixes the issue?
If so, I'll make a proper changelog and put the appropriate 2.6.33-35 stable tags for the backport.

Thnaks!

fs/reiserfs/ioctl.c +0 -0
index f53505d..679d502 100644

fs/reiserfs/ioctl.c int reiserfs_unpack(struct inode *inode, struct file *filp) · line 194

Quoted 8 lines
fs/reiserfs/ioctl.c +1 -1
@@ -188,7 +188,7 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 	reiserfs_write_lock(inode->i_sb);
 
 	write_from = inode->i_size & (blocksize - 1);

So, there is still a warning but a bit different now.

Jarek P.

[   67.110273] =======================================================
[   67.110313] [ INFO: possible circular locking dependency detected ]
[   67.110313] 2.6.35.4.4a #3
[   67.110313] -------------------------------------------------------
[   67.110313] lilo/1620 is trying to acquire lock:
[   67.110313]  (&journal->j_mutex){+.+...}, at: [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313] 
[   67.110313] but task is already holding lock:
[   67.110313]  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313] 
[   67.110313] which lock already depends on the new lock.
[   67.110313] 
[   67.110313] 
[   67.110313] the existing dependency chain (in reverse order) is:
[   67.110313] 
[   67.110313] -> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[   67.110313]        [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]        [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]        [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]        [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313]        [<d0325c06>] do_journal_begin_r+0x86/0x340 [reiserfs]
[   67.110313]        [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]        [<d0315be4>] reiserfs_remount+0x224/0x530 [reiserfs]
[   67.110313]        [<c10b6a20>] do_remount_sb+0x60/0x110
[   67.110313]        [<c10cee25>] do_mount+0x625/0x790
[   67.110313]        [<c10cf014>] sys_mount+0x84/0xb0
[   67.110313]        [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313] 
[   67.110313] -> #0 (&journal->j_mutex){+.+...}:
[   67.110313]        [<c10560f6>] __lock_acquire+0x1026/0x1180
[   67.110313]        [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]        [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]        [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]        [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]        [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]        [<d0326271>] reiserfs_persistent_transaction+0x41/0x90 [reiserfs]
[   67.110313]        [<d030d06c>] reiserfs_get_block+0x22c/0x1530 [reiserfs]
[   67.110313]        [<c10db9db>] __block_prepare_write+0x1bb/0x3a0
[   67.110313]        [<c10dbbe6>] block_prepare_write+0x26/0x40
[   67.110313]        [<d030b738>] reiserfs_prepare_write+0x88/0x170 [reiserfs]
[   67.110313]        [<d03294d6>] reiserfs_unpack+0xe6/0x120 [reiserfs]
[   67.110313]        [<d0329782>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   67.110313]        [<c10c3188>] vfs_ioctl+0x28/0xa0
[   67.110313]        [<c10c3bbd>] do_vfs_ioctl+0x32d/0x5c0
[   67.110313]        [<c10c3eb3>] sys_ioctl+0x63/0x70
[   67.110313]        [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313] 
[   67.110313] other info that might help us debug this:
[   67.110313] 
[   67.110313] 2 locks held by lilo/1620:
[   67.110313]  #0:  (&sb->s_type->i_mutex_key#8){+.+.+.}, at: [<d032945a>] reiserfs_unpack+0x6a/0x120 [reiserfs]
[   67.110313]  #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313] 
[   67.110313] stack backtrace:
[   67.110313] Pid: 1620, comm: lilo Not tainted 2.6.35.4.4a #3
[   67.110313] Call Trace:
[   67.110313]  [<c12f9aba>] ? printk+0x18/0x1e
[   67.110313]  [<c1054182>] print_circular_bug+0xd2/0xe0
[   67.110313]  [<c10560f6>] __lock_acquire+0x1026/0x1180
[   67.110313]  [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c1055275>] ? __lock_acquire+0x1a5/0x1180
[   67.110313]  [<c10897ae>] ? mempool_alloc_slab+0xe/0x10
[   67.110313]  [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c1054bd2>] ? mark_held_locks+0x62/0x80
[   67.110313]  [<c10b163d>] ? kmem_cache_alloc+0x7d/0xb0
[   67.110313]  [<c1054e5c>] ? trace_hardirqs_on_caller+0x11c/0x160
[   67.110313]  [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]  [<d0326262>] ? reiserfs_persistent_transaction+0x32/0x90 [reiserfs]
[   67.110313]  [<d0326271>] reiserfs_persistent_transaction+0x41/0x90 [reiserfs]
[   67.110313]  [<d030d06c>] reiserfs_get_block+0x22c/0x1530 [reiserfs]
[   67.110313]  [<c1055275>] ? __lock_acquire+0x1a5/0x1180
[   67.110313]  [<c12fc672>] ? _raw_spin_unlock_irq+0x22/0x50
[   67.110313]  [<c10b163d>] ? kmem_cache_alloc+0x7d/0xb0
[   67.110313]  [<c1054e5c>] ? trace_hardirqs_on_caller+0x11c/0x160
[   67.110313]  [<c102789b>] ? sub_preempt_count+0x7b/0xb0
[   67.110313]  [<c12fc457>] ? _raw_spin_unlock+0x27/0x40
[   67.110313]  [<c10db9db>] __block_prepare_write+0x1bb/0x3a0
[   67.110313]  [<c10dbbe6>] block_prepare_write+0x26/0x40
[   67.110313]  [<d030ce40>] ? reiserfs_get_block+0x0/0x1530 [reiserfs]
[   67.110313]  [<d030b738>] reiserfs_prepare_write+0x88/0x170 [reiserfs]
[   67.110313]  [<d030ce40>] ? reiserfs_get_block+0x0/0x1530 [reiserfs]
[   67.110313]  [<d03294d6>] reiserfs_unpack+0xe6/0x120 [reiserfs]
[   67.110313]  [<d0329782>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   67.110313]  [<d0329510>] ? reiserfs_ioctl+0x0/0x320 [reiserfs]
[   67.110313]  [<c10c3188>] vfs_ioctl+0x28/0xa0
[   67.110313]  [<c10c3bbd>] do_vfs_ioctl+0x32d/0x5c0
[   67.110313]  [<c109a228>] ? might_fault+0x88/0x90
[   67.110313]  [<c109a1e2>] ? might_fault+0x42/0x90
[   67.110313]  [<c10b6588>] ? fget_light+0xf8/0x2f0
[   67.110313]  [<c10c3eb3>] sys_ioctl+0x63/0x70
[   67.110313]  [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313]  [<c12f007b>] ? cookie_v6_check+0x44b/0x630
#5 lore raw
Quoted 42 lines · Jarek Poplawski

Frederic Weisbecker wrote, On 12/23/-28158 08:59 PM:

Quoted 40 lines · Andrew Morton
Quoted 11 lines
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

Thanks.

Ah, when you see &REISERFS_SB(s)->lock in a bug report, don't hesitate to blame me :-)

This is a problem resulting from the bkl conversion to a mutex that introduced a lot of new locking dependencies. Most of them have been fixed, but for less tested paths like ioctl, we hear about it later.

Does the following patch fixes the issue?
If so, I'll make a proper changelog and put the appropriate 2.6.33-35 stable tags for the backport.

Thnaks!

fs/reiserfs/ioctl.c +1 -1
index f53505d..679d502 100644
@@ -188,7 +188,7 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 	reiserfs_write_lock(inode->i_sb);
 
Quoted 7 lines

fs/reiserfs/ioctl.c int reiserfs_unpack(struct inode *inode, struct file *filp) · line 194

Quoted 2 lines
fs/reiserfs/ioctl.c +0 -0
 	write_from = inode->i_size & (blocksize - 1);

So, there is still a warning but a bit different now.

Jarek P.

That's another bug, due to the fact we sometimes recursively acquire the reiserfs big lock.

Anyway, will have a look this evening.

Thanks.

Quoted 98 lines
[   67.110273] =======================================================
[   67.110313] [ INFO: possible circular locking dependency detected ]
[   67.110313] 2.6.35.4.4a #3
[   67.110313] -------------------------------------------------------
[   67.110313] lilo/1620 is trying to acquire lock:
[   67.110313]  (&journal->j_mutex){+.+...}, at: [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313] 
[   67.110313] but task is already holding lock:
[   67.110313]  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313] 
[   67.110313] which lock already depends on the new lock.
[   67.110313] 
[   67.110313] 
[   67.110313] the existing dependency chain (in reverse order) is:
[   67.110313] 
[   67.110313] -> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
[   67.110313]        [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]        [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]        [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]        [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313]        [<d0325c06>] do_journal_begin_r+0x86/0x340 [reiserfs]
[   67.110313]        [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]        [<d0315be4>] reiserfs_remount+0x224/0x530 [reiserfs]
[   67.110313]        [<c10b6a20>] do_remount_sb+0x60/0x110
[   67.110313]        [<c10cee25>] do_mount+0x625/0x790
[   67.110313]        [<c10cf014>] sys_mount+0x84/0xb0
[   67.110313]        [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313] 
[   67.110313] -> #0 (&journal->j_mutex){+.+...}:
[   67.110313]        [<c10560f6>] __lock_acquire+0x1026/0x1180
[   67.110313]        [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]        [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]        [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]        [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]        [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]        [<d0326271>] reiserfs_persistent_transaction+0x41/0x90 [reiserfs]
[   67.110313]        [<d030d06c>] reiserfs_get_block+0x22c/0x1530 [reiserfs]
[   67.110313]        [<c10db9db>] __block_prepare_write+0x1bb/0x3a0
[   67.110313]        [<c10dbbe6>] block_prepare_write+0x26/0x40
[   67.110313]        [<d030b738>] reiserfs_prepare_write+0x88/0x170 [reiserfs]
[   67.110313]        [<d03294d6>] reiserfs_unpack+0xe6/0x120 [reiserfs]
[   67.110313]        [<d0329782>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   67.110313]        [<c10c3188>] vfs_ioctl+0x28/0xa0
[   67.110313]        [<c10c3bbd>] do_vfs_ioctl+0x32d/0x5c0
[   67.110313]        [<c10c3eb3>] sys_ioctl+0x63/0x70
[   67.110313]        [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313] 
[   67.110313] other info that might help us debug this:
[   67.110313] 
[   67.110313] 2 locks held by lilo/1620:
[   67.110313]  #0:  (&sb->s_type->i_mutex_key#8){+.+.+.}, at: [<d032945a>] reiserfs_unpack+0x6a/0x120 [reiserfs]
[   67.110313]  #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<d032a278>] reiserfs_write_lock+0x28/0x40 [reiserfs]
[   67.110313] 
[   67.110313] stack backtrace:
[   67.110313] Pid: 1620, comm: lilo Not tainted 2.6.35.4.4a #3
[   67.110313] Call Trace:
[   67.110313]  [<c12f9aba>] ? printk+0x18/0x1e
[   67.110313]  [<c1054182>] print_circular_bug+0xd2/0xe0
[   67.110313]  [<c10560f6>] __lock_acquire+0x1026/0x1180
[   67.110313]  [<c10562b7>] lock_acquire+0x67/0x80
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c12facad>] __mutex_lock_common+0x4d/0x410
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c1055275>] ? __lock_acquire+0x1a5/0x1180
[   67.110313]  [<c10897ae>] ? mempool_alloc_slab+0xe/0x10
[   67.110313]  [<c12fb0c8>] mutex_lock_nested+0x18/0x20
[   67.110313]  [<d0325bff>] ? do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<d0325bff>] do_journal_begin_r+0x7f/0x340 [reiserfs]
[   67.110313]  [<c1054bd2>] ? mark_held_locks+0x62/0x80
[   67.110313]  [<c10b163d>] ? kmem_cache_alloc+0x7d/0xb0
[   67.110313]  [<c1054e5c>] ? trace_hardirqs_on_caller+0x11c/0x160
[   67.110313]  [<d0325f77>] journal_begin+0x77/0x140 [reiserfs]
[   67.110313]  [<d0326262>] ? reiserfs_persistent_transaction+0x32/0x90 [reiserfs]
[   67.110313]  [<d0326271>] reiserfs_persistent_transaction+0x41/0x90 [reiserfs]
[   67.110313]  [<d030d06c>] reiserfs_get_block+0x22c/0x1530 [reiserfs]
[   67.110313]  [<c1055275>] ? __lock_acquire+0x1a5/0x1180
[   67.110313]  [<c12fc672>] ? _raw_spin_unlock_irq+0x22/0x50
[   67.110313]  [<c10b163d>] ? kmem_cache_alloc+0x7d/0xb0
[   67.110313]  [<c1054e5c>] ? trace_hardirqs_on_caller+0x11c/0x160
[   67.110313]  [<c102789b>] ? sub_preempt_count+0x7b/0xb0
[   67.110313]  [<c12fc457>] ? _raw_spin_unlock+0x27/0x40
[   67.110313]  [<c10db9db>] __block_prepare_write+0x1bb/0x3a0
[   67.110313]  [<c10dbbe6>] block_prepare_write+0x26/0x40
[   67.110313]  [<d030ce40>] ? reiserfs_get_block+0x0/0x1530 [reiserfs]
[   67.110313]  [<d030b738>] reiserfs_prepare_write+0x88/0x170 [reiserfs]
[   67.110313]  [<d030ce40>] ? reiserfs_get_block+0x0/0x1530 [reiserfs]
[   67.110313]  [<d03294d6>] reiserfs_unpack+0xe6/0x120 [reiserfs]
[   67.110313]  [<d0329782>] reiserfs_ioctl+0x272/0x320 [reiserfs]
[   67.110313]  [<d0329510>] ? reiserfs_ioctl+0x0/0x320 [reiserfs]
[   67.110313]  [<c10c3188>] vfs_ioctl+0x28/0xa0
[   67.110313]  [<c10c3bbd>] do_vfs_ioctl+0x32d/0x5c0
[   67.110313]  [<c109a228>] ? might_fault+0x88/0x90
[   67.110313]  [<c109a1e2>] ? might_fault+0x42/0x90
[   67.110313]  [<c10b6588>] ? fget_light+0xf8/0x2f0
[   67.110313]  [<c10c3eb3>] sys_ioctl+0x63/0x70
[   67.110313]  [<c12fca3d>] syscall_call+0x7/0xb
[   67.110313]  [<c12f007b>] ? cookie_v6_check+0x44b/0x630
#6 lore raw
In reply to Jarek Poplawski
Quoted 42 lines · Jarek Poplawski

Frederic Weisbecker wrote, On 12/23/-28158 08:59 PM:

Quoted 40 lines · Andrew Morton
Quoted 11 lines
Quoted 4 lines · On Sun, 5 Sep 2010 13:31:21 +0200 Jarek Poplawski <[email protected]> wrote:

Hi,
I get this warning on every lilo write with 2.6.35.4 and a bit/git later too.

Can you tell us the latest kernel version which did *not* have this bug? That way we can narrow the problem down a bit.

Thanks.

Ah, when you see &REISERFS_SB(s)->lock in a bug report, don't hesitate to blame me :-)

This is a problem resulting from the bkl conversion to a mutex that introduced a lot of new locking dependencies. Most of them have been fixed, but for less tested paths like ioctl, we hear about it later.

Does the following patch fixes the issue?
If so, I'll make a proper changelog and put the appropriate 2.6.33-35 stable tags for the backport.

Thnaks!

fs/reiserfs/ioctl.c +1 -1
index f53505d..679d502 100644
@@ -188,7 +188,7 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 	reiserfs_write_lock(inode->i_sb);
 
Quoted 7 lines

fs/reiserfs/ioctl.c int reiserfs_unpack(struct inode *inode, struct file *filp) · line 194

Quoted 2 lines
fs/reiserfs/ioctl.c +0 -0
 	write_from = inode->i_size & (blocksize - 1);

So, there is still a warning but a bit different now.

Jarek P.

I can reproduce your first case, but not this one.

So, I hope you can give a try to the following fix,

Thanks!

fs/reiserfs/ioctl.c +5 -4
index f53505d..90a757b 100644
@@ -170,7 +170,7 @@ int reiserfs_prepare_write(struct file *f, struct page *page,
 int reiserfs_unpack(struct inode *inode, struct file *filp)
 {
 	int retval = 0;
-	int index;
+	int index, depth;
 	struct page *page;
 	struct address_space *mapping;
 	unsigned long write_from;
@@ -185,11 +185,12 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 		return 0;
 	}
 
+	depth = reiserfs_write_lock_once(inode->i_sb);
+
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
-	reiserfs_write_lock(inode->i_sb);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 
 	write_from = inode->i_size & (blocksize - 1);
 	/* if we are on a block boundary, we are already unpacked.  */
@@ -224,6 +225,6 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 
       out:
 	mutex_unlock(&inode->i_mutex);
-	reiserfs_write_unlock(inode->i_sb);
+	reiserfs_write_unlock_once(inode->i_sb, depth);
 	return retval;
 }
author Tested-by
#7 lore raw
Quoted 7 lines · Frederic Weisbecker

I can reproduce your first case, but not this one.

So, I hope you can give a try to the following fix,

Thanks!

With this patch I don't have any lockdep warnings during lilo writes.

Well done, thanks!

Quoted 38 lines
fs/reiserfs/ioctl.c +5 -4
index f53505d..90a757b 100644
@@ -170,7 +170,7 @@ int reiserfs_prepare_write(struct file *f, struct page *page,
 int reiserfs_unpack(struct inode *inode, struct file *filp)
 {
 	int retval = 0;
-	int index;
+	int index, depth;
 	struct page *page;
 	struct address_space *mapping;
 	unsigned long write_from;
@@ -185,11 +185,12 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 		return 0;
 	}
 
+	depth = reiserfs_write_lock_once(inode->i_sb);
+
 	/* we need to make sure nobody is changing the file size beneath
 	 ** us
 	 */
-	mutex_lock(&inode->i_mutex);
-	reiserfs_write_lock(inode->i_sb);
+	reiserfs_mutex_lock_safe(&inode->i_mutex, inode->i_sb);
 
 	write_from = inode->i_size & (blocksize - 1);
 	/* if we are on a block boundary, we are already unpacked.  */
@@ -224,6 +225,6 @@ int reiserfs_unpack(struct inode *inode, struct file *filp)
 
       out:
 	mutex_unlock(&inode->i_mutex);
-	reiserfs_write_unlock(inode->i_sb);
+	reiserfs_write_unlock_once(inode->i_sb, depth);
 	return retval;
 }