Skip to contentSkip to conversation

xfs deadlock on mm-unstable kernel?

DISCUSSION Alex Shi 3 participants 4 messages Jul 8, 2024 – Nov 13, 2024 Memory management

lore
Share & tools
lore

Alex Shi

author
#0 lore raw

372.297234][ T3001] ============================================

[  372.297530][ T3001] WARNING: possible recursive locking detected
[  372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted
[  372.298137][ T3001] --------------------------------------------
[  372.298436][ T3001] cc1/3001 is trying to acquire lock:
[  372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710
[  372.299242][ T3001] 
[  372.299242][ T3001] but task is already holding lock:
[  372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.300258][ T3001] 
[  372.300258][ T3001] other info that might help us debug this:
[  372.300650][ T3001]  Possible unsafe locking scenario:
[  372.300650][ T3001] 
[  372.301031][ T3001]        CPU0
[  372.301231][ T3001]        ----
[  372.301386][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301623][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301860][ T3001] 
[  372.301860][ T3001]  *** DEADLOCK ***
[  372.301860][ T3001] 
[  372.302325][ T3001]  May be due to missing lock nesting notation
[  372.302325][ T3001] 
[  372.302723][ T3001] 3 locks held by cc1/3001:
[  372.302944][ T3001]  #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[  372.303554][ T3001]  #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.304183][ T3001]  #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0
[  372.304740][ T3001] 
[  372.304740][ T3001] stack backtrace:
[  372.305031][ T3001] CPU: 6 PID: 3001 Comm: cc1 Not tainted 6.10.0-rc6-00453-g2be3de2b70e6 #64
[  372.305453][ T3001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[  372.305934][ T3001] Call Trace:
[  372.306109][ T3001]  <TASK>
[  372.306253][ T3001]  dump_stack_lvl+0x78/0xe0
[  372.306483][ T3001]  validate_chain+0x519/0x9d0
[  372.306715][ T3001]  ? __pfx_validate_chain+0x10/0x10
[  372.306971][ T3001]  ? hlock_class+0x4e/0x130
[  372.307195][ T3001]  ? mark_lock+0x36/0x3b0
[  372.307409][ T3001]  __lock_acquire+0xacf/0x1540
[  372.307659][ T3001]  lock_acquire+0x165/0x3b0
[  372.307936][ T3001]  ? xfs_reclaim_inode+0x59e/0x710
[  372.308254][ T3001]  ? __pfx_lock_acquire+0x10/0x10
[  372.308507][ T3001]  ? __pfx___might_resched+0x10/0x10
[  372.308770][ T3001]  ? __pfx___lock_release.isra.0+0x10/0x10
[  372.309061][ T3001]  down_write_nested+0x9c/0x200
[  372.309303][ T3001]  ? xfs_reclaim_inode+0x59e/0x710
[  372.309552][ T3001]  ? __pfx_down_write_nested+0x10/0x10
[  372.309822][ T3001]  ? trace_xfs_ilock+0x119/0x190
[  372.310072][ T3001]  xfs_reclaim_inode+0x59e/0x710
[  372.310320][ T3001]  xfs_icwalk_ag+0x833/0xe50
[  372.310549][ T3001]  ? __pfx_xfs_icwalk_ag+0x10/0x10
[  372.310803][ T3001]  ? find_held_lock+0x2d/0x110
[  372.311037][ T3001]  ? xfs_perag_grab_tag+0x143/0x2d0
[  372.311297][ T3001]  ? xfs_perag_grab_tag+0x14d/0x2d0
[  372.311554][ T3001]  ? __pfx_xfs_perag_grab_tag+0x10/0x10
[  372.311828][ T3001]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  372.312096][ T3001]  xfs_icwalk+0x4c/0xd0
[  372.312306][ T3001]  xfs_reclaim_inodes_nr+0x148/0x1f0
[  372.312571][ T3001]  ? __pfx_xfs_reclaim_inodes_nr+0x10/0x10
[  372.312964][ T3001]  super_cache_scan+0x393/0x4e0
[  372.313267][ T3001]  do_shrink_slab+0x33e/0xc90
[  372.313552][ T3001]  shrink_slab_memcg+0x479/0x9c0
[  372.313852][ T3001]  ? shrink_slab_memcg+0x2df/0x9c0
[  372.314074][ T3001]  ? __pfx_shrink_slab_memcg+0x10/0x10
[  372.314328][ T3001]  ? find_held_lock+0x2d/0x110
[  372.314560][ T3001]  ? __pfx_shrink_lruvec+0x10/0x10
[  372.314811][ T3001]  ? __lock_release.isra.0+0x103/0x440
[  372.315078][ T3001]  shrink_slab+0x426/0x520
[  372.315306][ T3001]  ? __pfx_shrink_slab+0x10/0x10
[  372.315675][ T3001]  ? __pfx___might_resched+0x10/0x10
[  372.315999][ T3001]  ? page_counter_calculate_protection+0x2b3/0x5f0
[  372.316317][ T3001]  shrink_node_memcgs+0x4bc/0x8b0
[  372.316624][ T3001]  ? sum_zone_node_page_state+0x75/0xe0
[  372.316953][ T3001]  ? __pfx_shrink_node_memcgs+0x10/0x10
[  372.317226][ T3001]  ? node_page_state+0x2a/0x70
[  372.317462][ T3001]  ? prepare_scan_control+0x64e/0xb30
[  372.317729][ T3001]  shrink_node+0x1c4/0xeb0
[  372.317988][ T3001]  ? zone_reclaimable_pages+0x4ab/0x8b0
[  372.318266][ T3001]  ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[  372.318620][ T3001]  do_try_to_free_pages+0x27b/0x1670
[  372.318969][ T3001]  ? __pfx_throttle_direct_reclaim+0x10/0x10
[  372.319266][ T3001]  try_to_free_pages+0x198/0x280
[  372.319509][ T3001]  ? __pfx_try_to_free_pages+0x10/0x10
[  372.319777][ T3001]  ? __pfx___might_resched+0x10/0x10
[  372.320039][ T3001]  __alloc_pages_slowpath.constprop.0+0x749/0x1460
[  372.320356][ T3001]  ? __pfx___alloc_pages_slowpath.constprop.0+0x10/0x10
[  372.320697][ T3001]  __alloc_pages_noprof+0x497/0x540
[  372.321004][ T3001]  ? __pfx___alloc_pages_noprof+0x10/0x10
[  372.321293][ T3001]  ? __pfx___lock_release.isra.0+0x10/0x10
[  372.321623][ T3001]  ? mark_held_locks+0x96/0xe0
[  372.321858][ T3001]  alloc_pages_bulk_noprof+0x6b7/0x1000
[  372.322134][ T3001]  ? lockdep_init_map_type+0x16d/0x7c0
[  372.322399][ T3001]  ? lockdep_init_map_type+0x16d/0x7c0
[  372.322665][ T3001]  xfs_buf_alloc_pages+0x1b9/0x8d0
[  372.322917][ T3001]  ? __pfx___lock_release.isra.0+0x10/0x10
[  372.323209][ T3001]  xfs_buf_find_insert.isra.0+0xd4/0x390
[  372.323480][ T3001]  ? __pfx_xfs_buf_find_insert.isra.0+0x10/0x10
[  372.323784][ T3001]  xfs_buf_get_map+0x8c2/0xe20
[  372.324017][ T3001]  ? __pfx_xfs_buf_get_map+0x10/0x10
[  372.324275][ T3001]  xfs_buf_read_map+0xbd/0xa40
[  372.324508][ T3001]  ? xfs_da_read_buf+0x1f5/0x300
[  372.324752][ T3001]  ? __pfx_xfs_buf_read_map+0x10/0x10
[  372.325015][ T3001]  xfs_trans_read_buf_map+0x289/0x4d0
[  372.325281][ T3001]  ? xfs_da_read_buf+0x1f5/0x300
[  372.325523][ T3001]  ? __lock_release.isra.0+0x103/0x440
[  372.325791][ T3001]  ? __pfx_xfs_trans_read_buf_map+0x10/0x10
[  372.326082][ T3001]  ? unwind_next_frame+0x501/0x1d80
[  372.326344][ T3001]  ? entry_SYSCALL_64_after_hwframe+0x77/0x7f
[  372.326646][ T3001]  xfs_da_read_buf+0x1f5/0x300
[  372.326881][ T3001]  ? __pfx_xfs_da_read_buf+0x10/0x10
[  372.327143][ T3001]  ? __pfx_stack_trace_consume_entry+0x10/0x10
[  372.327445][ T3001]  ? kernel_text_address+0xbc/0x150
[  372.327698][ T3001]  ? unwind_get_return_address+0x5e/0xa0
[  372.327985][ T3001]  ? xfs_dir2_leaf_search_hash+0xfc/0x2b0
[  372.328262][ T3001]  ? __pfx_xfs_dir2_leaf_search_hash+0x10/0x10
[  372.328564][ T3001]  ? __pfx_xfs_iread_extents+0x10/0x10
[  372.328834][ T3001]  xfs_dir3_data_read+0x38/0x2a0
[  372.329192][ T3001]  xfs_dir2_leaf_lookup_int+0x464/0xae0
[  372.329529][ T3001]  ? __pfx_xfs_dir2_leaf_lookup_int+0x10/0x10
[  372.329829][ T3001]  ? xfs_bmap_last_offset+0x14c/0x2f0
[  372.330093][ T3001]  ? __pfx_xfs_bmap_last_offset+0x10/0x10
[  372.330371][ T3001]  xfs_dir2_leaf_lookup+0xf0/0x410
[  372.330621][ T3001]  ? __pfx_xfs_dir2_leaf_lookup+0x10/0x10
[  372.330901][ T3001]  ? down_read_nested+0xc0/0x340
[  372.331146][ T3001]  xfs_dir_lookup_args+0xcc/0xe0
[  372.331391][ T3001]  ? __pfx_xfs_dir_lookup_args+0x10/0x10
[  372.331669][ T3001]  xfs_dir_lookup+0x596/0x870
[  372.331901][ T3001]  xfs_lookup+0x185/0x2e0
[  372.332117][ T3001]  ? __pfx_xfs_lookup+0x10/0x10
[  372.332357][ T3001]  xfs_vn_lookup+0x14e/0x1a0
[  372.332586][ T3001]  ? __pfx_xfs_vn_lookup+0x10/0x10
[  372.332839][ T3001]  ? lockdep_init_map_type+0x16d/0x7c0
[  372.333109][ T3001]  __lookup_slow+0x20e/0x3c0
[  372.333340][ T3001]  ? __pfx___lookup_slow+0x10/0x10
[  372.333594][ T3001]  ? __startup_64+0x190/0x550
[  372.333827][ T3001]  ? __startup_64+0x190/0x550
[  372.334069][ T3001]  walk_component+0x2b3/0x500
[  372.334301][ T3001]  path_lookupat+0x117/0x680
[  372.334530][ T3001]  filename_lookup+0x1b0/0x520
[  372.334765][ T3001]  ? __pfx_filename_lookup+0x10/0x10
[  372.335024][ T3001]  ? __lock_acquire+0xacf/0x1540
[  372.335272][ T3001]  ? __pfx___lock_release.isra.0+0x10/0x10
[  372.335560][ T3001]  ? __pfx___might_resched+0x10/0x10
[  372.335834][ T3001]  vfs_statx+0xc9/0x4e0
[  372.336046][ T3001]  ? __pfx_vfs_statx+0x10/0x10
[  372.336283][ T3001]  ? getname_flags.part.0+0xb7/0x440
[  372.336544][ T3001]  vfs_fstatat+0x96/0xc0
[  372.336755][ T3001]  __do_sys_newlstat+0x99/0x100
[  372.336994][ T3001]  ? __pfx___do_sys_newlstat+0x10/0x10
[  372.337264][ T3001]  ? __pfx___lock_release.isra.0+0x10/0x10
[  372.337552][ T3001]  ? syscall_trace_enter+0x152/0x250
[  372.340527][ T3001]  ? syscall_trace_enter+0x152/0x250
[  372.340822][ T3001]  do_syscall_64+0xc1/0x1d0
[  372.341041][ T3001]  entry_SYSCALL_64_after_hwframe+0x77/0x7f
[  372.341327][ T3001] RIP: 0033:0x7fdb7b6ecb29
[  372.341554][ T3001] Code: 64 c7 00 16 00 00 00 b8 ff ff ff ff c3 0f 1f 40 00 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 06 00 00 00 0f 05 <48> 3d 00 f0 ff f0
[  372.342495][ T3001] RSP: 002b:00007fff978b5ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
[  372.342903][ T3001] RAX: ffffffffffffffda RBX: 00007fff978b6ff0 RCX: 00007fdb7b6ecb29
[  372.343284][ T3001] RDX: 00007fff978b5f10 RSI: 00007fff978b5f10 RDI: 00007fff978b5ff0
[  372.343681][ T3001] RBP: 00007fff978b5fe0 R08: 0000558afac57acb R09: 0000558ae4d344e0
[  372.344046][ T3001] R10: 00000000a053c1a1 R11: 0000000000000246 R12: 00007fff978b5ff0
[  372.344414][ T3001] R13: 0000558afac57acc R14: 0000558afac57ac2 R15: 00007fff978b600c
[  372.344816][ T3001]  </TASK>
#1 lore raw
Quoted 18 lines · Alex Shi

372.297234][ T3001] ============================================

[  372.297530][ T3001] WARNING: possible recursive locking detected
[  372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted
[  372.298137][ T3001] --------------------------------------------
[  372.298436][ T3001] cc1/3001 is trying to acquire lock:
[  372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710
[  372.299242][ T3001] 
[  372.299242][ T3001] but task is already holding lock:
[  372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.300258][ T3001] 
[  372.300258][ T3001] other info that might help us debug this:
[  372.300650][ T3001]  Possible unsafe locking scenario:
[  372.300650][ T3001] 
[  372.301031][ T3001]        CPU0
[  372.301231][ T3001]        ----
[  372.301386][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301623][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301860][ T3001] 
Quoted 8 lines
[  372.301860][ T3001]  *** DEADLOCK ***
[  372.301860][ T3001] 
[  372.302325][ T3001]  May be due to missing lock nesting notation
[  372.302325][ T3001] 
[  372.302723][ T3001] 3 locks held by cc1/3001:
[  372.302944][ T3001]  #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[  372.303554][ T3001]  #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.304183][ T3001]  #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0

False positive. Inodes above allocation must be actively referenced, and inodes accees by xfs_reclaim_inode() must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim in xfs_reclaim_inode() can deadlock against the refrenced inode locked by the inode lookup code.

Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all MAX_LOCKDEP_SUBCLASSES to tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done from super_cache_scan() paths that would avoid these false positives....

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 35 lines · Dave Chinner
Quoted 18 lines · Alex Shi

372.297234][ T3001] ============================================

[  372.297530][ T3001] WARNING: possible recursive locking detected
[  372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted
[  372.298137][ T3001] --------------------------------------------
[  372.298436][ T3001] cc1/3001 is trying to acquire lock:
[  372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710
[  372.299242][ T3001] 
[  372.299242][ T3001] but task is already holding lock:
[  372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.300258][ T3001] 
[  372.300258][ T3001] other info that might help us debug this:
[  372.300650][ T3001]  Possible unsafe locking scenario:
[  372.300650][ T3001] 
[  372.301031][ T3001]        CPU0
[  372.301231][ T3001]        ----
[  372.301386][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301623][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301860][ T3001] 
Quoted 8 lines
[  372.301860][ T3001]  *** DEADLOCK ***
[  372.301860][ T3001] 
[  372.302325][ T3001]  May be due to missing lock nesting notation
[  372.302325][ T3001] 
[  372.302723][ T3001] 3 locks held by cc1/3001:
[  372.302944][ T3001]  #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[  372.303554][ T3001]  #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.304183][ T3001]  #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0

False positive. Inodes above allocation must be actively referenced, and inodes accees by xfs_reclaim_inode() must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim in xfs_reclaim_inode() can deadlock against the refrenced inode locked by the inode lookup code.

Quoted 6 lines

Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all MAX_LOCKDEP_SUBCLASSES to tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done from super_cache_scan() paths that would avoid these false positives....

So the former inode (the one triggering the reclaim) is created and can not be the same as the one in reclaim list. Couldn't we assign it a different lock-class?
My guess would be that you drop the lockdep_set_class() in xfs_setup_inode() and then do it in xfs_iget_cache_miss() before adding it to the tree. So you would have one class initially and then change it once it enters the tree. I guess once the inode is removed from the tree, it goes to kfree().

Quoted 1 line

-Dave.

Sebastian

#3 lore raw
Quoted 43 lines · Sebastian Andrzej Siewior
Quoted 35 lines · Dave Chinner
Quoted 18 lines · Alex Shi

372.297234][ T3001] ============================================

[  372.297530][ T3001] WARNING: possible recursive locking detected
[  372.297827][ T3001] 6.10.0-rc6-00453-g2be3de2b70e6 #64 Not tainted
[  372.298137][ T3001] --------------------------------------------
[  372.298436][ T3001] cc1/3001 is trying to acquire lock:
[  372.298701][ T3001] ffff88802cb910d8 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_reclaim_inode+0x59e/0x710
[  372.299242][ T3001] 
[  372.299242][ T3001] but task is already holding lock:
[  372.299679][ T3001] ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.300258][ T3001] 
[  372.300258][ T3001] other info that might help us debug this:
[  372.300650][ T3001]  Possible unsafe locking scenario:
[  372.300650][ T3001] 
[  372.301031][ T3001]        CPU0
[  372.301231][ T3001]        ----
[  372.301386][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301623][ T3001]   lock(&xfs_dir_ilock_class);
[  372.301860][ T3001] 
Quoted 8 lines
[  372.301860][ T3001]  *** DEADLOCK ***
[  372.301860][ T3001] 
[  372.302325][ T3001]  May be due to missing lock nesting notation
[  372.302325][ T3001] 
[  372.302723][ T3001] 3 locks held by cc1/3001:
[  372.302944][ T3001]  #0: ffff88800e146078 (&inode->i_sb->s_type->i_mutex_dir_key){++++}-{3:3}, at: walk_component+0x2a5/0x500
[  372.303554][ T3001]  #1: ffff88800e145e58 (&xfs_dir_ilock_class){++++}-{3:3}, at: xfs_ilock_data_map_shared+0x4d/0x60
[  372.304183][ T3001]  #2: ffff8880040190e0 (&type->s_umount_key#48){++++}-{3:3}, at: super_cache_scan+0x82/0x4e0

False positive. Inodes above allocation must be actively referenced, and inodes accees by xfs_reclaim_inode() must have no references and been evicted and destroyed by the VFS. So there is no way that an unreferenced inode being locked for reclaim in xfs_reclaim_inode() can deadlock against the refrenced inode locked by the inode lookup code.

Quoted 6 lines

Unfortunately, we don't have enough lockdep subclasses available to annotate this correctly - we're already using all MAX_LOCKDEP_SUBCLASSES to tell lockdep about all the ways we can nest inode locks. That leaves us no space to add a "reclaim" annotation for locking done from super_cache_scan() paths that would avoid these false positives....

Quoted 3 lines

So the former inode (the one triggering the reclaim) is created and can not be the same as the one in reclaim list. Couldn't we assign it a different lock-class?

We've done that in the past. The problem with that is we lose lock ordering verification across reclaim. i.e. inode lock ordering must be the same both above and below reclaim, and changing the lock class loses the ability to verify this.

This is important to us as some code (e.g. extent removal) can be called from both above and below reclaim, and they require the same transaction and inode lock contexts to be held regardless of where they are called from...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]