Skip to contentSkip to conversation

lockdep: inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-R} usage.

DISCUSSION Dave Chinner 4 participants 7 messages Jan 15, 2010 – Jan 20, 2010 VFS

lore
Share & tools
lore
#0 lore raw

Just got this on a 2.6.33-rc3 kernel during unmount:

[21819.329256] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-R} usage.
[21819.349943] kswapd0/407 [HC0[0]:SC0[0]:HE1:SE1] takes:
[21819.349943]  (iprune_sem){+++++-}, at: [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943] {RECLAIM_FS-ON-W} state was registered at:
[21819.349943]   [<ffffffff810824c3>] mark_held_locks+0x73/0x90
[21819.349943]   [<ffffffff810825a5>] lockdep_trace_alloc+0xc5/0xd0
[21819.349943]   [<ffffffff811145f1>] kmem_cache_alloc+0x41/0x150
[21819.349943]   [<ffffffff813504f9>] kmem_zone_alloc+0x99/0xe0
[21819.349943]   [<ffffffff8135055e>] kmem_zone_zalloc+0x1e/0x50
[21819.349943]   [<ffffffff81346148>] _xfs_trans_alloc+0x38/0x80
[21819.349943]   [<ffffffff8134634f>] xfs_trans_alloc+0x9f/0xb0
[21819.349943]   [<ffffffff8134b3d0>] xfs_free_eofblocks+0x120/0x290
[21819.349943]   [<ffffffff8134f353>] xfs_inactive+0x103/0x560
[21819.349943]   [<ffffffff8135e6bf>] xfs_fs_clear_inode+0xdf/0x120
[21819.349943]   [<ffffffff81132615>] clear_inode+0xb5/0x140
[21819.349943]   [<ffffffff81132918>] dispose_list+0x38/0x130
[21819.349943]   [<ffffffff81132de3>] invalidate_inodes+0x123/0x170
[21819.349943]   [<ffffffff8111db4e>] generic_shutdown_super+0x4e/0x100
[21819.349943]   [<ffffffff8111dc31>] kill_block_super+0x31/0x50
[21819.349943]   [<ffffffff8111e455>] deactivate_super+0x85/0xa0
[21819.349943]   [<ffffffff81136f8a>] mntput_no_expire+0xca/0x110
[21819.349943]   [<ffffffff81137374>] sys_umount+0x64/0x370
[21819.349943]   [<ffffffff81002fdb>] system_call_fastpath+0x16/0x1b
[21819.349943] irq event stamp: 4151539
[21819.349943] hardirqs last  enabled at (4151539): [<ffffffff81706e04>] _raw_spin_unlock_irqrestore+0x44/0x70
[21819.349943] hardirqs last disabled at (4151538): [<ffffffff81706505>] _raw_spin_lock_irqsave+0x25/0x90
[21819.349943] softirqs last  enabled at (4151312): [<ffffffff8105373b>] __do_softirq+0x18b/0x1e0
[21819.349943] softirqs last disabled at (4150645): [<ffffffff81003e8c>] call_softirq+0x1c/0x50
[21819.349943]
[21819.349943] other info that might help us debug this:
[21819.349943] 1 lock held by kswapd0/407:
[21819.349943]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810e5c5d>] shrink_slab+0x3d/0x180
[21819.349943]
[21819.349943] stack backtrace:
[21819.349943] Pid: 407, comm: kswapd0 Not tainted 2.6.33-rc3-dgc #35
[21819.349943] Call Trace:
[21819.349943]  [<ffffffff81081353>] print_usage_bug+0x183/0x190
[21819.349943]  [<ffffffff81082372>] mark_lock+0x342/0x420
[21819.349943]  [<ffffffff810814c0>] ? check_usage_forwards+0x0/0x100
[21819.349943]  [<ffffffff81083531>] __lock_acquire+0x4d1/0x17a0
[21819.349943]  [<ffffffff8108327f>] ? __lock_acquire+0x21f/0x17a0
[21819.349943]  [<ffffffff810848ce>] lock_acquire+0xce/0x100
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81705592>] down_read+0x52/0x90
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff810e5d4a>] shrink_slab+0x12a/0x180
[21819.349943]  [<ffffffff810e66d6>] kswapd+0x586/0x990
[21819.349943]  [<ffffffff810e35c0>] ? isolate_pages_global+0x0/0x240
[21819.349943]  [<ffffffff8106c7c0>] ? autoremove_wake_function+0x0/0x40
[21819.349943]  [<ffffffff810e6150>] ? kswapd+0x0/0x990
[21819.349943]  [<ffffffff8106c276>] kthread+0x96/0xa0
[21819.349943]  [<ffffffff81003d94>] kernel_thread_helper+0x4/0x10
[21819.349943]  [<ffffffff8170717c>] ? restore_args+0x0/0x30
[21819.349943]  [<ffffffff8106c1e0>] ? kthread+0x0/0xa0
[21819.349943]  [<ffffffff81003d90>] ? kernel_thread_helper+0x0/0x10

I can't work out what the <mumble>RECLAIM_FS<mumble> notations are supposed to mean from the code and they are not documented at all, so I need someone to explain what this means before I can determine if it is a valid warning or not....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#1 lore raw
Quoted 59 lines · Dave Chinner

Just got this on a 2.6.33-rc3 kernel during unmount:

[21819.329256] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-R} usage.
[21819.349943] kswapd0/407 [HC0[0]:SC0[0]:HE1:SE1] takes:
[21819.349943]  (iprune_sem){+++++-}, at: [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943] {RECLAIM_FS-ON-W} state was registered at:
[21819.349943]   [<ffffffff810824c3>] mark_held_locks+0x73/0x90
[21819.349943]   [<ffffffff810825a5>] lockdep_trace_alloc+0xc5/0xd0
[21819.349943]   [<ffffffff811145f1>] kmem_cache_alloc+0x41/0x150
[21819.349943]   [<ffffffff813504f9>] kmem_zone_alloc+0x99/0xe0
[21819.349943]   [<ffffffff8135055e>] kmem_zone_zalloc+0x1e/0x50
[21819.349943]   [<ffffffff81346148>] _xfs_trans_alloc+0x38/0x80
[21819.349943]   [<ffffffff8134634f>] xfs_trans_alloc+0x9f/0xb0
[21819.349943]   [<ffffffff8134b3d0>] xfs_free_eofblocks+0x120/0x290
[21819.349943]   [<ffffffff8134f353>] xfs_inactive+0x103/0x560
[21819.349943]   [<ffffffff8135e6bf>] xfs_fs_clear_inode+0xdf/0x120
[21819.349943]   [<ffffffff81132615>] clear_inode+0xb5/0x140
[21819.349943]   [<ffffffff81132918>] dispose_list+0x38/0x130
[21819.349943]   [<ffffffff81132de3>] invalidate_inodes+0x123/0x170
[21819.349943]   [<ffffffff8111db4e>] generic_shutdown_super+0x4e/0x100
[21819.349943]   [<ffffffff8111dc31>] kill_block_super+0x31/0x50
[21819.349943]   [<ffffffff8111e455>] deactivate_super+0x85/0xa0
[21819.349943]   [<ffffffff81136f8a>] mntput_no_expire+0xca/0x110
[21819.349943]   [<ffffffff81137374>] sys_umount+0x64/0x370
[21819.349943]   [<ffffffff81002fdb>] system_call_fastpath+0x16/0x1b
[21819.349943] irq event stamp: 4151539
[21819.349943] hardirqs last  enabled at (4151539): [<ffffffff81706e04>] _raw_spin_unlock_irqrestore+0x44/0x70
[21819.349943] hardirqs last disabled at (4151538): [<ffffffff81706505>] _raw_spin_lock_irqsave+0x25/0x90
[21819.349943] softirqs last  enabled at (4151312): [<ffffffff8105373b>] __do_softirq+0x18b/0x1e0
[21819.349943] softirqs last disabled at (4150645): [<ffffffff81003e8c>] call_softirq+0x1c/0x50
[21819.349943]
[21819.349943] other info that might help us debug this:
[21819.349943] 1 lock held by kswapd0/407:
[21819.349943]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810e5c5d>] shrink_slab+0x3d/0x180
[21819.349943]
[21819.349943] stack backtrace:
[21819.349943] Pid: 407, comm: kswapd0 Not tainted 2.6.33-rc3-dgc #35
[21819.349943] Call Trace:
[21819.349943]  [<ffffffff81081353>] print_usage_bug+0x183/0x190
[21819.349943]  [<ffffffff81082372>] mark_lock+0x342/0x420
[21819.349943]  [<ffffffff810814c0>] ? check_usage_forwards+0x0/0x100
[21819.349943]  [<ffffffff81083531>] __lock_acquire+0x4d1/0x17a0
[21819.349943]  [<ffffffff8108327f>] ? __lock_acquire+0x21f/0x17a0
[21819.349943]  [<ffffffff810848ce>] lock_acquire+0xce/0x100
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81705592>] down_read+0x52/0x90
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff810e5d4a>] shrink_slab+0x12a/0x180
[21819.349943]  [<ffffffff810e66d6>] kswapd+0x586/0x990
[21819.349943]  [<ffffffff810e35c0>] ? isolate_pages_global+0x0/0x240
[21819.349943]  [<ffffffff8106c7c0>] ? autoremove_wake_function+0x0/0x40
[21819.349943]  [<ffffffff810e6150>] ? kswapd+0x0/0x990
[21819.349943]  [<ffffffff8106c276>] kthread+0x96/0xa0
[21819.349943]  [<ffffffff81003d94>] kernel_thread_helper+0x4/0x10
[21819.349943]  [<ffffffff8170717c>] ? restore_args+0x0/0x30
[21819.349943]  [<ffffffff8106c1e0>] ? kthread+0x0/0xa0
[21819.349943]  [<ffffffff81003d90>] ? kernel_thread_helper+0x0/0x10
Quoted 4 lines

I can't work out what the <mumble>RECLAIM_FS<mumble> notations are supposed to mean from the code and they are not documented at all, so I need someone to explain what this means before I can determine if it is a valid warning or not....

The <mumble>RECLAIM_FS<mumble> bit means that lock (iprune_sem) was taken from reclaim and is also taken over an allocation.

It warns that it might deadlock if that allocation ends up trying to reclaim memory.

#2 lore raw
Quoted 60 lines · Peter Zijlstra
Quoted 59 lines · Dave Chinner

Just got this on a 2.6.33-rc3 kernel during unmount:

[21819.329256] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-R} usage.
[21819.349943] kswapd0/407 [HC0[0]:SC0[0]:HE1:SE1] takes:
[21819.349943]  (iprune_sem){+++++-}, at: [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943] {RECLAIM_FS-ON-W} state was registered at:
[21819.349943]   [<ffffffff810824c3>] mark_held_locks+0x73/0x90
[21819.349943]   [<ffffffff810825a5>] lockdep_trace_alloc+0xc5/0xd0
[21819.349943]   [<ffffffff811145f1>] kmem_cache_alloc+0x41/0x150
[21819.349943]   [<ffffffff813504f9>] kmem_zone_alloc+0x99/0xe0
[21819.349943]   [<ffffffff8135055e>] kmem_zone_zalloc+0x1e/0x50
[21819.349943]   [<ffffffff81346148>] _xfs_trans_alloc+0x38/0x80
[21819.349943]   [<ffffffff8134634f>] xfs_trans_alloc+0x9f/0xb0
[21819.349943]   [<ffffffff8134b3d0>] xfs_free_eofblocks+0x120/0x290
[21819.349943]   [<ffffffff8134f353>] xfs_inactive+0x103/0x560
[21819.349943]   [<ffffffff8135e6bf>] xfs_fs_clear_inode+0xdf/0x120
[21819.349943]   [<ffffffff81132615>] clear_inode+0xb5/0x140
[21819.349943]   [<ffffffff81132918>] dispose_list+0x38/0x130
[21819.349943]   [<ffffffff81132de3>] invalidate_inodes+0x123/0x170
[21819.349943]   [<ffffffff8111db4e>] generic_shutdown_super+0x4e/0x100
[21819.349943]   [<ffffffff8111dc31>] kill_block_super+0x31/0x50
[21819.349943]   [<ffffffff8111e455>] deactivate_super+0x85/0xa0
[21819.349943]   [<ffffffff81136f8a>] mntput_no_expire+0xca/0x110
[21819.349943]   [<ffffffff81137374>] sys_umount+0x64/0x370
[21819.349943]   [<ffffffff81002fdb>] system_call_fastpath+0x16/0x1b
[21819.349943] irq event stamp: 4151539
[21819.349943] hardirqs last  enabled at (4151539): [<ffffffff81706e04>] _raw_spin_unlock_irqrestore+0x44/0x70
[21819.349943] hardirqs last disabled at (4151538): [<ffffffff81706505>] _raw_spin_lock_irqsave+0x25/0x90
[21819.349943] softirqs last  enabled at (4151312): [<ffffffff8105373b>] __do_softirq+0x18b/0x1e0
[21819.349943] softirqs last disabled at (4150645): [<ffffffff81003e8c>] call_softirq+0x1c/0x50
[21819.349943]
[21819.349943] other info that might help us debug this:
[21819.349943] 1 lock held by kswapd0/407:
[21819.349943]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810e5c5d>] shrink_slab+0x3d/0x180
[21819.349943]
[21819.349943] stack backtrace:
[21819.349943] Pid: 407, comm: kswapd0 Not tainted 2.6.33-rc3-dgc #35
[21819.349943] Call Trace:
[21819.349943]  [<ffffffff81081353>] print_usage_bug+0x183/0x190
[21819.349943]  [<ffffffff81082372>] mark_lock+0x342/0x420
[21819.349943]  [<ffffffff810814c0>] ? check_usage_forwards+0x0/0x100
[21819.349943]  [<ffffffff81083531>] __lock_acquire+0x4d1/0x17a0
[21819.349943]  [<ffffffff8108327f>] ? __lock_acquire+0x21f/0x17a0
[21819.349943]  [<ffffffff810848ce>] lock_acquire+0xce/0x100
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81705592>] down_read+0x52/0x90
[21819.349943]  [<ffffffff81132a92>] ? shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff81132a92>] shrink_icache_memory+0x82/0x2b0
[21819.349943]  [<ffffffff810e5d4a>] shrink_slab+0x12a/0x180
[21819.349943]  [<ffffffff810e66d6>] kswapd+0x586/0x990
[21819.349943]  [<ffffffff810e35c0>] ? isolate_pages_global+0x0/0x240
[21819.349943]  [<ffffffff8106c7c0>] ? autoremove_wake_function+0x0/0x40
[21819.349943]  [<ffffffff810e6150>] ? kswapd+0x0/0x990
[21819.349943]  [<ffffffff8106c276>] kthread+0x96/0xa0
[21819.349943]  [<ffffffff81003d94>] kernel_thread_helper+0x4/0x10
[21819.349943]  [<ffffffff8170717c>] ? restore_args+0x0/0x30
[21819.349943]  [<ffffffff8106c1e0>] ? kthread+0x0/0xa0
[21819.349943]  [<ffffffff81003d90>] ? kernel_thread_helper+0x0/0x10
Quoted 7 lines
Quoted 4 lines

I can't work out what the <mumble>RECLAIM_FS<mumble> notations are supposed to mean from the code and they are not documented at all, so I need someone to explain what this means before I can determine if it is a valid warning or not....

The <mumble>RECLAIM_FS<mumble> bit means that lock (iprune_sem) was taken from reclaim and is also taken over an allocation.

So there's an implicit, undocumented requirement that inode reclaim during unmount requires a filesystem to do GFP_NOFS allocation?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 11 lines · Dave Chinner
Quoted 8 lines
Quoted 5 lines

I can't work out what the <mumble>RECLAIM_FS<mumble> notations are supposed to mean from the code and they are not documented at all, so I need someone to explain what this means before I can determine if it is a valid warning or not....

The <mumble>RECLAIM_FS<mumble> bit means that lock (iprune_sem) was taken from reclaim and is also taken over an allocation.

So there's an implicit, undocumented requirement that inode reclaim during unmount requires a filesystem to do GFP_NOFS allocation?

Well, I don't know enough about xfs (of filesystems in generic) to say that with any certainty, but I can imagine inode writeback from the sync that goes with umount to cause issues.

If this inode reclaim is past all that and the filesystem is basically RO, then I don't think so and this could be considered a false positive, in which case we need an annotation for this.

I added hch since he poked at similar reclaim recursions on XFS before and Nick since he thought up this annotation and knows more about filesystems than I do.

#4 lore raw
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Dave Chinner
Quoted 8 lines
Quoted 5 lines

I can't work out what the <mumble>RECLAIM_FS<mumble> notations are supposed to mean from the code and they are not documented at all, so I need someone to explain what this means before I can determine if it is a valid warning or not....

The <mumble>RECLAIM_FS<mumble> bit means that lock (iprune_sem) was taken from reclaim and is also taken over an allocation.

So there's an implicit, undocumented requirement that inode reclaim during unmount requires a filesystem to do GFP_NOFS allocation?

Quoted 7 lines

Well, I don't know enough about xfs (of filesystems in generic) to say that with any certainty, but I can imagine inode writeback from the sync that goes with umount to cause issues.

If this inode reclaim is past all that and the filesystem is basically RO, then I don't think so and this could be considered a false positive, in which case we need an annotation for this.

The issue is that the iprune_sem is held write locked over dispose_list() even though the inodes have been removed from the unused list. While iprune_sem is held write locked, we can't enter shrink_icache_memory because that takes the iprune_sem in read mode. Hence allocation anywhere in the dispose_list path has to be GFP_NOFS to avoid this.

XFS relies on the PF_MEMALLOC flag to clear the __GFP_FS flag in allocations so that the same code paths work in both normal and reclaim situations (like _xfs_trans_alloc), but the unmount path sets no such flag. Setting this flag would avoid the problem, but is messy.

FWIW, I'm not sure why we need to hold the iprune_sem after the inodes have been detached from the unused list in the unmount path. The iprune_sem is there to prevent against concurrent access by the shrink_icache_memory path, so once all the inodes are isolated it seems the iprune_sem is not needed anymore. Of course, this code is a maze of twisty passages, so there's likely to be something I've missed that means that this is the only way it can work....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra

Well, I don't know enough about xfs (of filesystems in generic) to say that with any certainty, but I can imagine inode writeback from the sync that goes with umount to cause issues.

If this inode reclaim is past all that and the filesystem is basically RO, then I don't think so and this could be considered a false positive, in which case we need an annotation for this.

The issue is a bit more complicated. In the unmount case invalidate_inodes() is indeed called after the filesystem is effectively read-only for user origination operations. But there's a miriad of other invalidate_inodes() calls:

- fs/block_dev.c:__invalidate_device()

This gets called from block device codes for various kinds of invalidations. Doesn't make any sense at all to me, but hey..

- fs/ext2/super.c:ext2_remount()

Appears like it's used to check for activate inodes during remount. Very fishy usage, and could just be replaced with a list walk without any I/O

- fs/gfs2/glock.c:gfs2_gl_hash_clear()

No idea.

- fs/gfs2/ops_fstype.c:fill_super()

Tries to kill all inodes in the fill_super error path, looks very fishy.

- fs/ntfs/super.c:ntfs_fill_super()

Failure case of fill_super again, does not look very useful.A

- fs/smbfs/inode.c:smb_invalidate_inodes()

Used when a connection goes bad.

In short we can't generally rely on this only happening on a dead fs.

But in the end we abuse iprune_sem to work around a ref counting problem. As long as we keep a reference to the superblock for each inode on the dispose list the superblock can't go away and there's no need for the lock at all.

#6 lore raw

Hi,

Quoted 21 lines · Christoph Hellwig
Quoted 7 lines · Peter Zijlstra

Well, I don't know enough about xfs (of filesystems in generic) to say that with any certainty, but I can imagine inode writeback from the sync that goes with umount to cause issues.

If this inode reclaim is past all that and the filesystem is basically RO, then I don't think so and this could be considered a false positive, in which case we need an annotation for this.

The issue is a bit more complicated. In the unmount case invalidate_inodes() is indeed called after the filesystem is effectively read-only for user origination operations. But there's a miriad of other invalidate_inodes() calls:

- fs/block_dev.c:__invalidate_device()

This gets called from block device codes for various kinds of invalidations. Doesn't make any sense at all to me, but hey..

- fs/ext2/super.c:ext2_remount()

Quoted 8 lines

Appears like it's used to check for activate inodes during remount. Very fishy usage, and could just be replaced with a list walk without any I/O

- fs/gfs2/glock.c:gfs2_gl_hash_clear()

No idea.

Its rather complicated and all down to using "special" inodes to cache metadata so that GFS2 has two VFS inodes per "real" inode, one as per normal and one just to cache metadata.

This causes a circular dependency between glocks and inodes since we have something like this (in my best ascii art):

gfs2 inode -> iopen glock

           -> inode glock -> metadata inode

So at umount time, historically we've had to invalidate inodes once, and then get rid of the inode glocks which implied a iput() on the metadata inode and then invalidate inodes again to be rid of the metadata inodes.

This has been the source of many problems at umount time. In my -nmw git tree at the moment, there are a couple of patches which are aimed at fixing this issue. The solution is to embed a struct address_space in each glock which caches metadata, rather than a complete inode.

Quoted 5 lines

- fs/gfs2/ops_fstype.c:fill_super()

Tries to kill all inodes in the fill_super error path, looks very fishy.

For the same reason as above.

It should be possible to remove one or even both of these calls now. The two patches in the -nmw tree do the bare minimum really to make the change, but it should be possible to do a bit more clean up in that area now,

Steve.