XFS lock warning, 2.6.29-rc8
hi
Here is what i got suddently. But seems everything is working. If any other information required, please let me know.
/dev/md2 on / type xfs (rw,noatime)
[39993.560200]
[39993.560202] =======================================================
[39993.560206] [ INFO: possible circular locking dependency detected ]
[39993.560208] 2.6.29-rc8-home1 #2
[39993.560210] -------------------------------------------------------
[39993.560212] kmail/5077 is trying to acquire lock:
[39993.560214] (&(&ip->i_iolock)->mr_lock){----}, at: [<ffffffff811426aa>]
xfs_ilock+0x27/0x79
[39993.560222]
[39993.560223] but task is already holding lock:
[39993.560225] (&mm->mmap_sem){----}, at: [<ffffffff81095090>]
sys_munmap+0x32/0x59
[39993.560230]
[39993.560231] which lock already depends on the new lock.
[39993.560232]
[39993.560233]
[39993.560234] the existing dependency chain (in reverse order) is:
[39993.560236]
[39993.560236] -> #1 (&mm->mmap_sem){----}:
[39993.560240] [<ffffffff8105b9fc>] __lock_acquire+0x12b0/0x161a
[39993.560244] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560247] [<ffffffff813fc2a5>] down_read+0x34/0x41
[39993.560252] [<ffffffff813ffbfe>] do_page_fault+0x3fa/0x848
[39993.560256] [<ffffffff813fda2f>] page_fault+0x1f/0x30
[39993.560258] [<ffffffff8107e13d>] generic_file_aio_read+0x35f/0x5da
[39993.560268] [<ffffffff811650d8>] xfs_read+0x17d/0x1f5
[39993.560272] [<ffffffff81161646>] xfs_file_aio_read+0x5f/0x61
[39993.560275] [<ffffffff810a88ff>] do_sync_read+0xe7/0x12d
[39993.560284] [<ffffffff810a928c>] vfs_read+0xab/0x134
[39993.560287] [<ffffffff810a93d9>] sys_read+0x47/0x6f
[39993.560289] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
[39993.560293] [<ffffffffffffffff>] 0xffffffffffffffff
[39993.560302]
[39993.560302] -> #0 (&(&ip->i_iolock)->mr_lock){----}:
[39993.560306] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
[39993.560309] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560312] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
[39993.560316] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
[39993.560318] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
[39993.560322] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
[39993.560325] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
[39993.560329] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
[39993.560332] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
[39993.560335] [<ffffffff810ba1c7>] iput+0x61/0x65
[39993.560338] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
[39993.560341] [<ffffffff810b7651>] d_kill+0x40/0x60
[39993.560344] [<ffffffff810b942d>] dput+0x141/0x14e
[39993.560347] [<ffffffff810a9a85>] __fput+0x173/0x198
[39993.560350] [<ffffffff810a9ac2>] fput+0x18/0x1a
[39993.560353] [<ffffffff8109411e>] remove_vma+0x3e/0x63
[39993.560355] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
[39993.560358] [<ffffffff8109509e>] sys_munmap+0x40/0x59
[39993.560361] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
[39993.560363] [<ffffffffffffffff>] 0xffffffffffffffff
[39993.560366]
[39993.560367] other info that might help us debug this:
[39993.560368]
[39993.560370] 1 lock held by kmail/5077:
[39993.560371] #0: (&mm->mmap_sem){----}, at: [<ffffffff81095090>]
sys_munmap+0x32/0x59
[39993.560376]
[39993.560377] stack backtrace:
[39993.560379] Pid: 5077, comm: kmail Not tainted 2.6.29-rc8-home1 #2
[39993.560381] Call Trace:
[39993.560384] [<ffffffff8105a366>] print_circular_bug_tail+0xc5/0xd0
[39993.560388] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
[39993.560391] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560393] [<ffffffff811426aa>] ? xfs_ilock+0x27/0x79
[39993.560396] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
[39993.560399] [<ffffffff811426aa>] ? xfs_ilock+0x27/0x79
[39993.560401] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
[39993.560404] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
[39993.560407] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
[39993.560410] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
[39993.560413] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
[39993.560416] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
[39993.560419] [<ffffffff810ba1c7>] iput+0x61/0x65
[39993.560421] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
[39993.560424] [<ffffffff810b7651>] d_kill+0x40/0x60
[39993.560427] [<ffffffff810b942d>] dput+0x141/0x14e
[39993.560429] [<ffffffff810a9a85>] __fput+0x173/0x198
[39993.560432] [<ffffffff810a9ac2>] fput+0x18/0x1a
[39993.560434] [<ffffffff8109411e>] remove_vma+0x3e/0x63
[39993.560437] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b
[39993.560440] [<ffffffff8109509e>] sys_munmap+0x40/0x59
[39993.560442] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b[ changed cc to [email protected]. [email protected] does not exist. ]
Quoted 4 lines · Denys Fedoryschenko
hi
Here is what i got suddently. But seems everything is working. If any other information required, please let me know.
Known issue.
Quoted 30 lines
[39993.560234] the existing dependency chain (in reverse order) is:
[39993.560236]
[39993.560236] -> #1 (&mm->mmap_sem){----}:
[39993.560240] [<ffffffff8105b9fc>] __lock_acquire+0x12b0/0x161a
[39993.560244] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560247] [<ffffffff813fc2a5>] down_read+0x34/0x41
[39993.560252] [<ffffffff813ffbfe>] do_page_fault+0x3fa/0x848
[39993.560256] [<ffffffff813fda2f>] page_fault+0x1f/0x30
[39993.560258] [<ffffffff8107e13d>] generic_file_aio_read+0x35f/0x5da
[39993.560268] [<ffffffff811650d8>] xfs_read+0x17d/0x1f5
[39993.560272] [<ffffffff81161646>] xfs_file_aio_read+0x5f/0x61
[39993.560275] [<ffffffff810a88ff>] do_sync_read+0xe7/0x12d
[39993.560284] [<ffffffff810a928c>] vfs_read+0xab/0x134
[39993.560287] [<ffffffff810a93d9>] sys_read+0x47/0x6f
[39993.560289] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b
[39993.560293] [<ffffffffffffffff>] 0xffffffffffffffff
[39993.560302]
[39993.560302] -> #0 (&(&ip->i_iolock)->mr_lock){----}:
[39993.560306] [<ffffffff8105b6cb>] __lock_acquire+0xf7f/0x161a
[39993.560309] [<ffffffff8105bdbb>] lock_acquire+0x55/0x71
[39993.560312] [<ffffffff8104ebde>] down_write_nested+0x37/0x46
[39993.560316] [<ffffffff811426aa>] xfs_ilock+0x27/0x79
[39993.560318] [<ffffffff8115b40e>] xfs_inactive+0x14d/0x411
[39993.560322] [<ffffffff8116552f>] xfs_fs_clear_inode+0x83/0x85
[39993.560325] [<ffffffff810ba8f7>] clear_inode+0x8a/0xe3
[39993.560329] [<ffffffff810bb044>] generic_delete_inode+0xf8/0x173
[39993.560332] [<ffffffff810bb0d6>] generic_drop_inode+0x17/0x1d5
[39993.560335] [<ffffffff810ba1c7>] iput+0x61/0x65
[39993.560338] [<ffffffff810b7554>] dentry_iput+0xa9/0xc1
[39993.560341] [<ffffffff810b7651>] d_kill+0x40/0x60Quoted 8 lines
[39993.560344] [<ffffffff810b942d>] dput+0x141/0x14e [39993.560347] [<ffffffff810a9a85>] __fput+0x173/0x198 [39993.560350] [<ffffffff810a9ac2>] fput+0x18/0x1a [39993.560353] [<ffffffff8109411e>] remove_vma+0x3e/0x63 [39993.560355] [<ffffffff8109503c>] do_munmap+0x2e9/0x30b [39993.560358] [<ffffffff8109509e>] sys_munmap+0x40/0x59 [39993.560361] [<ffffffff8100c0db>] system_call_fastpath+0x16/0x1b [39993.560363] [<ffffffffffffffff>] 0xffffffffffffffff
This is a VM problem where it calls fput() with the mmap_sem() held in remove_vma(). It makes the incorrect assumption that filesystems will never use the same lock in the IO path and the inode release path.
This can deadlock if you are really unlucky.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
fput under mmap_sem
Quoted 5 lines · Dave Chinner
This is a VM problem where it calls
fput()with themmap_sem()held inremove_vma(). It makes the incorrect assumption that filesystems will never use the same lock in the IO path and the inode release path.This can deadlock if you are really unlucky.
I really wonder why other filesystems haven't hit this yet. Any chance we can get the fput moved out of mmap_sem to get rid of this class of problems?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: fput under mmap_sem
Quoted 10 lines · On Wednesday 18 March 2009 18:13:13 Christoph Hellwig wrote:
Quoted 5 lines · Dave Chinner
This is a VM problem where it calls
fput()with themmap_sem()held inremove_vma(). It makes the incorrect assumption that filesystems will never use the same lock in the IO path and the inode release path.This can deadlock if you are really unlucky.
I really wonder why other filesystems haven't hit this yet. Any chance we can get the fput moved out of
mmap_semto get rid of this class of problems?
Yes I don't think there is any reason against holding the file refcount a little longer, but it can be pretty nasty to do in practice because of deep call chains and sometimes multiple fputs within a given lock section.
Possibly the easiest and quickest way will be to move aio's deferred __fput into a usable API and try that. If there are any performance problems, then we can try to move those fputs out from the mmap_sem one at a time (eg. I don't expect fput for vma replacement/merging to often cause the refcount to reach 0, and this is one of the harder places; wheras fput for a simple munmap might be more often causing refcount to reach 0 but it should be simpler to move out of mmap_sem if it is a problem).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>