xfs deadlock on mm-unstable kernel?
372.297234][ T3001] ============================================
[ 372.297530][ T3001] WARNING: possible recursive locking detected
[ 372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted
[ 372.298137][ T3001] --------------------------------------------
[ 372.298436][ T3001] cc1/3001 is trying to acquire lock:
[ 372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710
[ 372.299242][ T3001]
[ 372.299242][ T3001] but task is already holding lock:
[ 372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[ 372.300258][ T3001]
[ 372.300258][ T3001] other info that might help us debug this:
[ 372.300650][ T3001] Possible unsafe locking scenario:
[ 372.300650][ T3001]
[ 372.301031][ T3001] CPU0
[ 372.301231][ T3001] ----
[ 372.301386][ T3001] lock(&xfs_dir_ilock_class);
[ 372.301623][ T3001] lock(&xfs_dir_ilock_class);
[ 372.301860][ T3001]
[ 372.301860][ T3001] *** DEADLOCK ***
[ 372.301860][ T3001]
[ 372.302325][ T3001] May be due to missing lock nesting notation
[ 372.302325][ T3001]
[ 372.302723][ T3001] 3 locks held by cc1/3001:
[ 372.302944][ T3001] #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[ 372.303554][ T3001] #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[ 372.304183][ T3001] #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0
[ 372.304740][ T3001]
[ 372.304740][ T3001] stack backtrace:
[ 372.305031][ T3001] CPU: 6 PID: 3001 Comm: cc1 Not tainted 6.10.0-rc6-00453-g2be3de2b70e6 #64
[ 372.305453][ T3001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 372.305934][ T3001] Call Trace:
[ 372.306109][ T3001] <TASK>
[ 372.306253][ T3001] dump_stack_lvl+0x78/0xe0
[ 372.306483][ T3001] validate_chain+0x519/0x9d0
[ 372.306715][ T3001] ? __pfx_validate_chain+0x10/0x10
[ 372.306971][ T3001] ? hlock_class+0x4e/0x130
[ 372.307195][ T3001] ? mark_lock+0x36/0x3b0
[ 372.307409][ T3001] __lock_acquire+0xacf/0x1540
[ 372.307659][ T3001] lock_acquire+0x165/0x3b0
[ 372.307936][ T3001] ? xfs_reclaim_inode+0x59e/0x710
[ 372.308254][ T3001] ? __pfx_lock_acquire+0x10/0x10
[ 372.308507][ T3001] ? __pfx___might_resched+0x10/0x10
[ 372.308770][ T3001] ? __pfx___lock_release.isra.0+0x10/0x10
[ 372.309061][ T3001] down_write_nested+0x9c/0x200
[ 372.309303][ T3001] ? xfs_reclaim_inode+0x59e/0x710
[ 372.309552][ T3001] ? __pfx_down_write_nested+0x10/0x10
[ 372.309822][ T3001] ? trace_xfs_ilock+0x119/0x190
[ 372.310072][ T3001] xfs_reclaim_inode+0x59e/0x710
[ 372.310320][ T3001] xfs_icwalk_ag+0x833/0xe50
[ 372.310549][ T3001] ? __pfx_xfs_icwalk_ag+0x10/0x10
[ 372.310803][ T3001] ? find_held_lock+0x2d/0x110
[ 372.311037][ T3001] ? xfs_perag_grab_tag+0x143/0x2d0
[ 372.311297][ T3001] ? xfs_perag_grab_tag+0x14d/0x2d0
[ 372.311554][ T3001] ? __pfx_xfs_perag_grab_tag+0x10/0x10
[ 372.311828][ T3001] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 372.312096][ T3001] xfs_icwalk+0x4c/0xd0
[ 372.312306][ T3001] xfs_reclaim_inodes_nr+0x148/0x1f0
[ 372.312571][ T3001] ? __pfx_xfs_reclaim_inodes_nr+0x10/0x10
[ 372.312964][ T3001] super_cache_scan+0x393/0x4e0
[ 372.313267][ T3001] do_shrink_slab+0x33e/0xc90
[ 372.313552][ T3001] shrink_slab_memcg+0x479/0x9c0
[ 372.313852][ T3001] ? shrink_slab_memcg+0x2df/0x9c0
[ 372.314074][ T3001] ? __pfx_shrink_slab_memcg+0x10/0x10
[ 372.314328][ T3001] ? find_held_lock+0x2d/0x110
[ 372.314560][ T3001] ? __pfx_shrink_lruvec+0x10/0x10
[ 372.314811][ T3001] ? __lock_release.isra.0+0x103/0x440
[ 372.315078][ T3001] shrink_slab+0x426/0x520
[ 372.315306][ T3001] ? __pfx_shrink_slab+0x10/0x10
[ 372.315675][ T3001] ? __pfx___might_resched+0x10/0x10
[ 372.315999][ T3001] ? page_counter_calculate_protection+0x2b3/0x5f0
[ 372.316317][ T3001] shrink_node_memcgs+0x4bc/0x8b0
[ 372.316624][ T3001] ? sum_zone_node_page_state+0x75/0xe0
[ 372.316953][ T3001] ? __pfx_shrink_node_memcgs+0x10/0x10
[ 372.317226][ T3001] ? node_page_state+0x2a/0x70
[ 372.317462][ T3001] ? prepare_scan_control+0x64e/0xb30
[ 372.317729][ T3001] shrink_node+0x1c4/0xeb0
[ 372.317988][ T3001] ? zone_reclaimable_pages+0x4ab/0x8b0
[ 372.318266][ T3001] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 372.318620][ T3001] do_try_to_free_pages+0x27b/0x1670
[ 372.318969][ T3001] ? __pfx_throttle_direct_reclaim+0x10/0x10
[ 372.319266][ T3001] try_to_free_pages+0x198/0x280
[ 372.319509][ T3001] ? __pfx_try_to_free_pages+0x10/0x10
[ 372.319777][ T3001] ? __pfx___might_resched+0x10/0x10
[ 372.320039][ T3001] __alloc_pages_slowpath.constprop.0+0x749/0x1460
[ 372.320356][ T3001] ? __pfx___alloc_pages_slowpath.constprop.0+0x10/0x10
[ 372.320697][ T3001] __alloc_pages_noprof+0x497/0x540
[ 372.321004][ T3001] ? __pfx___alloc_pages_noprof+0x10/0x10
[ 372.321293][ T3001] ? __pfx___lock_release.isra.0+0x10/0x10
[ 372.321623][ T3001] ? mark_held_locks+0x96/0xe0
[ 372.321858][ T3001] alloc_pages_bulk_noprof+0x6b7/0x1000
[ 372.322134][ T3001] ? lockdep_init_map_type+0x16d/0x7c0
[ 372.322399][ T3001] ? lockdep_init_map_type+0x16d/0x7c0
[ 372.322665][ T3001] xfs_buf_alloc_pages+0x1b9/0x8d0
[ 372.322917][ T3001] ? __pfx___lock_release.isra.0+0x10/0x10
[ 372.323209][ T3001] xfs_buf_find_insert.isra.0+0xd4/0x390
[ 372.323480][ T3001] ? __pfx_xfs_buf_find_insert.isra.0+0x10/0x10
[ 372.323784][ T3001] xfs_buf_get_map+0x8c2/0xe20
[ 372.324017][ T3001] ? __pfx_xfs_buf_get_map+0x10/0x10
[ 372.324275][ T3001] xfs_buf_read_map+0xbd/0xa40
[ 372.324508][ T3001] ? xfs_da_read_buf+0x1f5/0x300
[ 372.324752][ T3001] ? __pfx_xfs_buf_read_map+0x10/0x10
[ 372.325015][ T3001] xfs_trans_read_buf_map+0x289/0x4d0
[ 372.325281][ T3001] ? xfs_da_read_buf+0x1f5/0x300
[ 372.325523][ T3001] ? __lock_release.isra.0+0x103/0x440
[ 372.325791][ T3001] ? __pfx_xfs_trans_read_buf_map+0x10/0x10
[ 372.326082][ T3001] ? unwind_next_frame+0x501/0x1d80
[ 372.326344][ T3001] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 372.326646][ T3001] xfs_da_read_buf+0x1f5/0x300
[ 372.326881][ T3001] ? __pfx_xfs_da_read_buf+0x10/0x10
[ 372.327143][ T3001] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 372.327445][ T3001] ? kernel_text_address+0xbc/0x150
[ 372.327698][ T3001] ? unwind_get_return_address+0x5e/0xa0
[ 372.327985][ T3001] ? xfs_dir2_leaf_search_hash+0xfc/0x2b0
[ 372.328262][ T3001] ? __pfx_xfs_dir2_leaf_search_hash+0x10/0x10
[ 372.328564][ T3001] ? __pfx_xfs_iread_extents+0x10/0x10
[ 372.328834][ T3001] xfs_dir3_data_read+0x38/0x2a0
[ 372.329192][ T3001] xfs_dir2_leaf_lookup_int+0x464/0xae0
[ 372.329529][ T3001] ? __pfx_xfs_dir2_leaf_lookup_int+0x10/0x10
[ 372.329829][ T3001] ? xfs_bmap_last_offset+0x14c/0x2f0
[ 372.330093][ T3001] ? __pfx_xfs_bmap_last_offset+0x10/0x10
[ 372.330371][ T3001] xfs_dir2_leaf_lookup+0xf0/0x410
[ 372.330621][ T3001] ? __pfx_xfs_dir2_leaf_lookup+0x10/0x10
[ 372.330901][ T3001] ? down_read_nested+0xc0/0x340
[ 372.331146][ T3001] xfs_dir_lookup_args+0xcc/0xe0
[ 372.331391][ T3001] ? __pfx_xfs_dir_lookup_args+0x10/0x10
[ 372.331669][ T3001] xfs_dir_lookup+0x596/0x870
[ 372.331901][ T3001] xfs_lookup+0x185/0x2e0
[ 372.332117][ T3001] ? __pfx_xfs_lookup+0x10/0x10
[ 372.332357][ T3001] xfs_vn_lookup+0x14e/0x1a0
[ 372.332586][ T3001] ? __pfx_xfs_vn_lookup+0x10/0x10
[ 372.332839][ T3001] ? lockdep_init_map_type+0x16d/0x7c0
[ 372.333109][ T3001] __lookup_slow+0x20e/0x3c0
[ 372.333340][ T3001] ? __pfx___lookup_slow+0x10/0x10
[ 372.333594][ T3001] ? __startup_64+0x190/0x550
[ 372.333827][ T3001] ? __startup_64+0x190/0x550
[ 372.334069][ T3001] walk_component+0x2b3/0x500
[ 372.334301][ T3001] path_lookupat+0x117/0x680
[ 372.334530][ T3001] filename_lookup+0x1b0/0x520
[ 372.334765][ T3001] ? __pfx_filename_lookup+0x10/0x10
[ 372.335024][ T3001] ? __lock_acquire+0xacf/0x1540
[ 372.335272][ T3001] ? __pfx___lock_release.isra.0+0x10/0x10
[ 372.335560][ T3001] ? __pfx___might_resched+0x10/0x10
[ 372.335834][ T3001] vfs_statx+0xc9/0x4e0
[ 372.336046][ T3001] ? __pfx_vfs_statx+0x10/0x10
[ 372.336283][ T3001] ? getname_flags.part.0+0xb7/0x440
[ 372.336544][ T3001] vfs_fstatat+0x96/0xc0
[ 372.336755][ T3001] __do_sys_newlstat+0x99/0x100
[ 372.336994][ T3001] ? __pfx___do_sys_newlstat+0x10/0x10
[ 372.337264][ T3001] ? __pfx___lock_release.isra.0+0x10/0x10
[ 372.337552][ T3001] ? syscall_trace_enter+0x152/0x250
[ 372.340527][ T3001] ? syscall_trace_enter+0x152/0x250
[ 372.340822][ T3001] do_syscall_64+0xc1/0x1d0
[ 372.341041][ T3001] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 372.341327][ T3001] RIP: 0033:0x7fdb7b6ecb29
[ 372.341554][ T3001] Code: 64 c7 00 16 00 00 00 b8 ff ff ff ff c3 0f 1f 40 00 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 06 00 00 00 0f 05 <48> 3d 00 f0 ff f0
[ 372.342495][ T3001] RSP: 002b:00007fff978b5ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
[ 372.342903][ T3001] RAX: ffffffffffffffda RBX: 00007fff978b6ff0 RCX: 00007fdb7b6ecb29
[ 372.343284][ T3001] RDX: 00007fff978b5f10 RSI: 00007fff978b5f10 RDI: 00007fff978b5ff0
[ 372.343681][ T3001] RBP: 00007fff978b5fe0 R08: 0000558afac57acb R09: 0000558ae4d344e0
[ 372.344046][ T3001] R10: 00000000a053c1a1 R11: 0000000000000246 R12: 00007fff978b5ff0
[ 372.344414][ T3001] R13: 0000558afac57acc R14: 0000558afac57ac2 R15: 00007fff978b600c
[ 372.344816][ T3001] </TASK>Quoted 18 lines · Alex Shi
372.297234][ T3001] ============================================
[ 372.297530][ T3001] WARNING: possible recursive locking detected [ 372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted [ 372.298137][ T3001] -------------------------------------------- [ 372.298436][ T3001] cc1/3001 is trying to acquire lock: [ 372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710 [ 372.299242][ T3001] [ 372.299242][ T3001] but task is already holding lock: [ 372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60 [ 372.300258][ T3001] [ 372.300258][ T3001] other info that might help us debug this: [ 372.300650][ T3001] Possible unsafe locking scenario: [ 372.300650][ T3001] [ 372.301031][ T3001] CPU0 [ 372.301231][ T3001] ---- [ 372.301386][ T3001] lock(&xfs_dir_ilock_class); [ 372.301623][ T3001] lock(&xfs_dir_ilock_class); [ 372.301860][ T3001]
Quoted 8 lines
[ 372.301860][ T3001] *** DEADLOCK ***
[ 372.301860][ T3001]
[ 372.302325][ T3001] May be due to missing lock nesting notation
[ 372.302325][ T3001]
[ 372.302723][ T3001] 3 locks held by cc1/3001:
[ 372.302944][ T3001] #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[ 372.303554][ T3001] #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[ 372.304183][ T3001] #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0False positive. Inodes above allocation must be actively referenced, and inodes accees by xfs_reclaim_inode() must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim in xfs_reclaim_inode() can deadlock against the refrenced inode locked by the inode lookup code.
Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all MAX_LOCKDEP_SUBCLASSES to tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done from super_cache_scan() paths that would avoid these false positives....
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 35 lines · Dave Chinner
Quoted 18 lines · Alex Shi
372.297234][ T3001] ============================================
[ 372.297530][ T3001] WARNING: possible recursive locking detected [ 372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted [ 372.298137][ T3001] -------------------------------------------- [ 372.298436][ T3001] cc1/3001 is trying to acquire lock: [ 372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710 [ 372.299242][ T3001] [ 372.299242][ T3001] but task is already holding lock: [ 372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60 [ 372.300258][ T3001] [ 372.300258][ T3001] other info that might help us debug this: [ 372.300650][ T3001] Possible unsafe locking scenario: [ 372.300650][ T3001] [ 372.301031][ T3001] CPU0 [ 372.301231][ T3001] ---- [ 372.301386][ T3001] lock(&xfs_dir_ilock_class); [ 372.301623][ T3001] lock(&xfs_dir_ilock_class); [ 372.301860][ T3001]Quoted 8 lines
[ 372.301860][ T3001] *** DEADLOCK *** [ 372.301860][ T3001] [ 372.302325][ T3001] May be due to missing lock nesting notation [ 372.302325][ T3001] [ 372.302723][ T3001] 3 locks held by cc1/3001: [ 372.302944][ T3001] #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500 [ 372.303554][ T3001] #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60 [ 372.304183][ T3001] #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0False positive. Inodes above allocation must be actively referenced, and inodes accees by
xfs_reclaim_inode()must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim inxfs_reclaim_inode()can deadlock against the refrenced inode locked by the inode lookup code.
Quoted 6 lines
Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all
MAX_LOCKDEP_SUBCLASSESto tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done fromsuper_cache_scan()paths that would avoid these false positives....
So the former inode (the one triggering the reclaim) is created and can not be the same as the one in reclaim list. Couldn't we assign it a different lock-class?
My guess would be that you drop the lockdep_set_class() in xfs_setup_inode() and then do it in xfs_iget_cache_miss() before adding it to the tree. So you would have one class initially and then change it once it enters the tree. I guess once the inode is removed from the tree, it goes to kfree().
Quoted 1 line
-Dave.
Sebastian
Quoted 43 lines · Sebastian Andrzej Siewior
Quoted 35 lines · Dave Chinner
Quoted 18 lines · Alex Shi
372.297234][ T3001] ============================================
[ 372.297530][ T3001] WARNING: possible recursive locking detected [ 372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted [ 372.298137][ T3001] -------------------------------------------- [ 372.298436][ T3001] cc1/3001 is trying to acquire lock: [ 372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710 [ 372.299242][ T3001] [ 372.299242][ T3001] but task is already holding lock: [ 372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60 [ 372.300258][ T3001] [ 372.300258][ T3001] other info that might help us debug this: [ 372.300650][ T3001] Possible unsafe locking scenario: [ 372.300650][ T3001] [ 372.301031][ T3001] CPU0 [ 372.301231][ T3001] ---- [ 372.301386][ T3001] lock(&xfs_dir_ilock_class); [ 372.301623][ T3001] lock(&xfs_dir_ilock_class); [ 372.301860][ T3001]Quoted 8 lines
[ 372.301860][ T3001] *** DEADLOCK *** [ 372.301860][ T3001] [ 372.302325][ T3001] May be due to missing lock nesting notation [ 372.302325][ T3001] [ 372.302723][ T3001] 3 locks held by cc1/3001: [ 372.302944][ T3001] #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500 [ 372.303554][ T3001] #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60 [ 372.304183][ T3001] #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0False positive. Inodes above allocation must be actively referenced, and inodes accees by
xfs_reclaim_inode()must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim inxfs_reclaim_inode()can deadlock against the refrenced inode locked by the inode lookup code.Quoted 6 lines
Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all
MAX_LOCKDEP_SUBCLASSESto tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done fromsuper_cache_scan()paths that would avoid these false positives....
Quoted 3 lines
So the former inode (the one triggering the reclaim) is created and can not be the same as the one in reclaim list. Couldn't we assign it a different lock-class?
We've done that in the past. The problem with that is we lose lock ordering verification across reclaim. i.e. inode lock ordering must be the same both above and below reclaim, and changing the lock class loses the ability to verify this.
This is important to us as some code (e.g. extent removal) can be called from both above and below reclaim, and they require the same transaction and inode lock contexts to be held regardless of where they are called from...
-Dave.
Signature · 2 lines
Dave Chinner [email protected]