xfs lockdep trace after unlink
Subsystems (2)
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low! turning off the locking correctness validator. Please attach the output of /proc/lock_stat to the bug report CPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs] [<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2
lock_stats is at http://codemonkey.org.uk/junk/locks-2013-10-08.txt
Dave
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!
Or lockdep?
Quoted 39 lines
turning off the locking correctness validator. Please attach the output of
/proc/lock_statto the bug reportCPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs]
Quoted 8 lines
[<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2
It's hard to see what in XFS is causing this. You're reading a single block directory, which means we're holding two inode locks here, and then we've done a lookup on a radix tree under rcu_read_lock(). Hence I can't see how we've overrun the lockdep chain depth in the XFS code path. FWIW, it's thrown this warning when calling rcu_read_unlock() here:
struct xfs_perag *
xfs_perag_get(
struct xfs_mount *mp,
xfs_agnumber_t agno)
{
struct xfs_perag *pag;
int ref = 0;
rcu_read_lock();
pag = radix_tree_lookup(&mp->m_perag_tree, agno);
if (pag) {
ASSERT(atomic_read(&pag->pag_ref) >= 0);
ref = atomic_inc_return(&pag->pag_ref);
}Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rcu_read_unlock();
trace_xfs_perag_get(mp, agno, ref, _RET_IP_);
return pag;
}So I'd be looking at RCU or lockdep at this point....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!
I have to ask... What happens when you bump up MAX_LOCKDEP_CHAINS?
There is a patch to avoid an RCU/scheduler/perf deadlock, which may be found below. But this stack doesn't look to me to be matching that deadlock.
Thanx, Paul
Quoted 48 lines
turning off the locking correctness validator. Please attach the output of
/proc/lock_statto the bug reportCPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs] [<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2
Quoted 1 line
lock_statsis at http://codemonkey.org.uk/junk/locks-2013-10-08.txt
------------------------------------------------------------------------
rcu: Break call_rcu() deadlock involving scheduler and perf
Dave Jones got the following lockdep splat:
Quoted 114 lines
======================================================
[ INFO: possible circular locking dependency detected ] 3.12.0-rc3+ #92 Not tainted
-------------------------------------------------------
trinity-child2/15191 is trying to acquire lock:
(&rdp->nocb_wq){......}, at: [<ffffffff8108ff43>]__wake_up+0x23/0x50but task is already holding lock:
(&ctx->lock){-.-...}, at: [<ffffffff81154c19>]perf_event_exit_task+0x109/0x230which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (&ctx->lock){-.-...}:
[<ffffffff810cc243>] lock_acquire+0x93/0x200 [<ffffffff81733f90>] _raw_spin_lock+0x40/0x80 [<ffffffff811500ff>] __perf_event_task_sched_out+0x2df/0x5e0 [<ffffffff81091b83>] perf_event_task_sched_out+0x93/0xa0 [<ffffffff81732052>] __schedule+0x1d2/0xa20 [<ffffffff81732f30>] preempt_schedule_irq+0x50/0xb0 [<ffffffff817352b6>] retint_kernel+0x26/0x30 [<ffffffff813eed04>] tty_flip_buffer_push+0x34/0x50 [<ffffffff813f0504>] pty_write+0x54/0x60 [<ffffffff813e900d>] n_tty_write+0x32d/0x4e0 [<ffffffff813e5838>] tty_write+0x158/0x2d0 [<ffffffff811c4850>] vfs_write+0xc0/0x1f0 [<ffffffff811c52cc>] SyS_write+0x4c/0xa0 [<ffffffff8173d4e4>] tracesys+0xdd/0xe2-> #2 (&rq->lock){-.-.-.}:
[<ffffffff810cc243>] lock_acquire+0x93/0x200 [<ffffffff81733f90>] _raw_spin_lock+0x40/0x80 [<ffffffff810980b2>] wake_up_new_task+0xc2/0x2e0 [<ffffffff81054336>] do_fork+0x126/0x460 [<ffffffff81054696>] kernel_thread+0x26/0x30 [<ffffffff8171ff93>] rest_init+0x23/0x140 [<ffffffff81ee1e4b>] start_kernel+0x3f6/0x403 [<ffffffff81ee1571>] x86_64_start_reservations+0x2a/0x2c [<ffffffff81ee1664>] x86_64_start_kernel+0xf1/0xf4-> #1 (&p->pi_lock){-.-.-.}:
[<ffffffff810cc243>] lock_acquire+0x93/0x200 [<ffffffff8173419b>] _raw_spin_lock_irqsave+0x4b/0x90 [<ffffffff810979d1>] try_to_wake_up+0x31/0x350 [<ffffffff81097d62>] default_wake_function+0x12/0x20 [<ffffffff81084af8>] autoremove_wake_function+0x18/0x40 [<ffffffff8108ea38>] __wake_up_common+0x58/0x90 [<ffffffff8108ff59>] __wake_up+0x39/0x50 [<ffffffff8110d4f8>] __call_rcu_nocb_enqueue+0xa8/0xc0 [<ffffffff81111450>] __call_rcu+0x140/0x820 [<ffffffff81111b8d>] call_rcu+0x1d/0x20 [<ffffffff81093697>] cpu_attach_domain+0x287/0x360 [<ffffffff81099d7e>] build_sched_domains+0xe5e/0x10a0 [<ffffffff81efa7fc>] sched_init_smp+0x3b7/0x47a [<ffffffff81ee1f4e>] kernel_init_freeable+0xf6/0x202 [<ffffffff817200be>] kernel_init+0xe/0x190 [<ffffffff8173d22c>] ret_from_fork+0x7c/0xb0-> #0 (&rdp->nocb_wq){......}:
[<ffffffff810cb7ca>] __lock_acquire+0x191a/0x1be0 [<ffffffff810cc243>] lock_acquire+0x93/0x200 [<ffffffff8173419b>] _raw_spin_lock_irqsave+0x4b/0x90 [<ffffffff8108ff43>] __wake_up+0x23/0x50 [<ffffffff8110d4f8>] __call_rcu_nocb_enqueue+0xa8/0xc0 [<ffffffff81111450>] __call_rcu+0x140/0x820 [<ffffffff81111bb0>] kfree_call_rcu+0x20/0x30 [<ffffffff81149abf>] put_ctx+0x4f/0x70 [<ffffffff81154c3e>] perf_event_exit_task+0x12e/0x230 [<ffffffff81056b8d>] do_exit+0x30d/0xcc0 [<ffffffff8105893c>] do_group_exit+0x4c/0xc0 [<ffffffff810589c4>] SyS_exit_group+0x14/0x20 [<ffffffff8173d4e4>] tracesys+0xdd/0xe2other info that might help us debug this:
Chain exists of: &rdp->nocb_wq --> &rq->lock --> &ctx->lock Possible unsafe locking scenario:CPU0 CPU1 ---- ---- lock(&ctx->lock); lock(&rq->lock); lock(&ctx->lock);
lock(&rdp->nocb_wq);*** DEADLOCK ***
1 lock held by trinity-child2/15191:
#0: (&ctx->lock){-.-...}, at: [<ffffffff81154c19>]perf_event_exit_task+0x109/0x230stack backtrace: CPU: 2 PID: 15191 Comm: trinity-child2 Not tainted 3.12.0-rc3+ #92 ffffffff82565b70 ffff880070c2dbf8 ffffffff8172a363 ffffffff824edf40 ffff880070c2dc38 ffffffff81726741 ffff880070c2dc90 ffff88022383b1c0 ffff88022383aac0 0000000000000000 ffff88022383b188 ffff88022383b1c0 Call Trace: [<ffffffff8172a363>] dump_stack+0x4e/0x82 [<ffffffff81726741>] print_circular_bug+0x200/0x20f [<ffffffff810cb7ca>] __lock_acquire+0x191a/0x1be0 [<ffffffff810c6439>] ? get_lock_stats+0x19/0x60 [<ffffffff8100b2f4>] ? native_sched_clock+0x24/0x80 [<ffffffff810cc243>] lock_acquire+0x93/0x200 [<ffffffff8108ff43>] ? __wake_up+0x23/0x50 [<ffffffff8173419b>] _raw_spin_lock_irqsave+0x4b/0x90 [<ffffffff8108ff43>] ? __wake_up+0x23/0x50 [<ffffffff8108ff43>] __wake_up+0x23/0x50 [<ffffffff8110d4f8>] __call_rcu_nocb_enqueue+0xa8/0xc0 [<ffffffff81111450>] __call_rcu+0x140/0x820 [<ffffffff8109bc8f>] ? local_clock+0x3f/0x50 [<ffffffff81111bb0>] kfree_call_rcu+0x20/0x30
Quoted 8 lines
[<ffffffff81149abf>] put_ctx+0x4f/0x70 [<ffffffff81154c3e>] perf_event_exit_task+0x12e/0x230 [<ffffffff81056b8d>] do_exit+0x30d/0xcc0 [<ffffffff810c9af5>] ? trace_hardirqs_on_caller+0x115/0x1e0 [<ffffffff810c9bcd>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff8105893c>] do_group_exit+0x4c/0xc0 [<ffffffff810589c4>] SyS_exit_group+0x14/0x20 [<ffffffff8173d4e4>] tracesys+0xdd/0xe2
The underlying problem is that perf is invoking call_rcu() with the scheduler locks held, but in NOCB mode, call_rcu() will with high probability invoke the scheduler -- which just might want to use its locks. The reason that call_rcu() needs to invoke the scheduler is to wake up the corresponding rcuo callback-offload kthread, which does the job of starting up a grace period and invoking the callbacks afterwards.
One solution (championed on a related problem by Lai Jiangshan) is to simply defer the wakeup to some point where scheduler locks are no longer held. Since we don't want to unnecessarily incur the cost of such deferral, the task before us is threefold:
1. Determine when it is likely that a relevant scheduler lock is held.
2. Defer the wakeup in such cases.
3. Ensure that all deferred wakeups eventually happen, preferably
sooner rather than later.
We use irqs_disabled_flags() as a proxy for relevant scheduler locks being held. This works because the relevant locks are always acquired with interrupts disabled. We may defer more often than needed, but that is at least safe.
The wakeup deferral is tracked via a new field in the per-CPU and per-RCU-flavor rcu_data structure, namely ->nocb_defer_wakeup.
This flag is checked by the RCU core processing. The __rcu_pending() function now checks this flag, which causes rcu_check_callbacks() to initiate RCU core processing at each scheduling-clock interrupt where this flag is set. Of course this is not sufficient because scheduling-clock interrupts are often turned off (the things we used to be able to count on!). So the flags are also checked on entry to any state that RCU considers to be idle, which includes both NO_HZ_IDLE idle state and NO_HZ_FULL user-mode-execution state.
This approach should allow call_rcu() to be invoked regardless of what locks you might be holding, the key word being "should".
Documentation/RCU/trace.txt +12 -8
@@ -396,14 +396,14 @@ o Each element of the form "3/3 ..>. 0:7 ^0" represents one rcu_node The output of "cat rcu/rcu_sched/rcu_pending" looks as follows: - 0!np=26111 qsp=29 rpq=5386 cbr=1 cng=570 gpc=3674 gps=577 nn=15903 - 1!np=28913 qsp=35 rpq=6097 cbr=1 cng=448 gpc=3700 gps=554 nn=18113 - 2!np=32740 qsp=37 rpq=6202 cbr=0 cng=476 gpc=4627 gps=546 nn=20889 - 3 np=23679 qsp=22 rpq=5044 cbr=1 cng=415 gpc=3403 gps=347 nn=14469 - 4!np=30714 qsp=4 rpq=5574 cbr=0 cng=528 gpc=3931 gps=639 nn=20042 - 5 np=28910 qsp=2 rpq=5246 cbr=0 cng=428 gpc=4105 gps=709 nn=18422 - 6!np=38648 qsp=5 rpq=7076 cbr=0 cng=840 gpc=4072 gps=961 nn=25699 - 7 np=37275 qsp=2 rpq=6873 cbr=0 cng=868 gpc=3416 gps=971 nn=25147 + 0!np=26111 qsp=29 rpq=5386 cbr=1 cng=570 gpc=3674 gps=577 nn=15903 ndw=0 + 1!np=28913 qsp=35 rpq=6097 cbr=1 cng=448 gpc=3700 gps=554 nn=18113 ndw=0 + 2!np=32740 qsp=37 rpq=6202 cbr=0 cng=476 gpc=4627 gps=546 nn=20889 ndw=0 + 3 np=23679 qsp=22 rpq=5044 cbr=1 cng=415 gpc=3403 gps=347 nn=14469 ndw=0 + 4!np=30714 qsp=4 rpq=5574 cbr=0 cng=528 gpc=3931 gps=639 nn=20042 ndw=0 + 5 np=28910 qsp=2 rpq=5246 cbr=0 cng=428 gpc=4105 gps=709 nn=18422 ndw=0 + 6!np=38648 qsp=5 rpq=7076 cbr=0 cng=840 gpc=4072 gps=961 nn=25699 ndw=0 + 7 np=37275 qsp=2 rpq=6873 cbr=0 cng=868 gpc=3416 gps=971 nn=25147 ndw=0 The fields are as follows:
@@ -432,6 +432,10 @@ o "gpc" is the number of times that an old grace period had o "gps" is the number of times that a new grace period had started, but this CPU was not yet aware of it. +o "ndw" is the number of times that a wakeup of an rcuo + callback-offload kthread had to be deferred in order to avoid + deadlock. + o "nn" is the number of times that this CPU needed nothing.
kernel/rcutree.c +20 -4
@@ -362,6 +362,9 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)static void rcu_eqs_enter_common (struct rcu_dynticks *rdtp,long long oldval,bool user) { +struct rcu_state *rsp; +struct rcu_data *rdp; +trace_rcu_dyntick (TPS ("Start" ), oldval, rdtp->dynticks_nesting);if (!user && !is_idle_task (current)) { struct task_struct *idle __maybe_unused =
@@ -373,6 +376,10 @@ static void rcu_eqs_enter_common(struct rcu_dynticks *rdtp, long long oldval, current->pid, current->comm, idle->pid, idle->comm); /* must be idle task! */ } +for_each_rcu_flavor (rsp) { + rdp =this_cpu_ptr (rsp->rda); +do_nocb_deferred_wakeup (rdp); + }rcu_prepare_for_idle (smp_processor_id ());/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */ smp_mb__before_atomic_inc ();/* See above. */
@@ -1908,13 +1915,13 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,* Adopt the RCU callbacks from the specified rcu_state structure's * orphanage. The caller must hold the ->orphan_lock. */ -static void rcu_adopt_orphan_cbs (struct rcu_state *rsp) +static void rcu_adopt_orphan_cbs (struct rcu_state *rsp,unsigned long flags) {int i;struct rcu_data *rdp =__this_cpu_ptr (rsp->rda);/* No-CBs CPUs are handled specially. */ -if (rcu_nocb_adopt_orphan_cbs (rsp, rdp)) +if (rcu_nocb_adopt_orphan_cbs (rsp, rdp, flags))return ;/* Do the accounting first. */
@@ -1993,7 +2000,7 @@ static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)/* Orphan the dead CPU's callbacks, and adopt them if appropriate. */ rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp); -rcu_adopt_orphan_cbs (rsp); +rcu_adopt_orphan_cbs (rsp, flags);/* Remove the outgoing CPU from the masks in the rcu_node hierarchy. */ mask = rdp->grpmask;/* rnp->grplo is constant. */
@@ -2310,6 +2317,9 @@ __rcu_process_callbacks(struct rcu_state *rsp)/* If there are callbacks ready, invoke them. */ if (cpu_has_callbacks_ready_to_invoke (rdp))invoke_rcu_callbacks (rsp, rdp); + +/* Do any needed deferred wakeups of rcuo kthreads. */ +do_nocb_deferred_wakeup (rdp); }/*
@@ -2444,7 +2454,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),if (cpu !=-1 ) rdp =per_cpu_ptr (rsp->rda, cpu); - offline = !__call_rcu_nocb (rdp, head, lazy); + offline = !__call_rcu_nocb (rdp, head, lazy, flags);WARN_ON_ONCE (offline);/* _call_rcu() is illegal on offline CPU; leak the callback. */ local_irq_restore (flags);
@@ -2797,6 +2807,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)return 1 ; } +/* Does this CPU need a deferred NOCB wakeup? */ +if (rcu_nocb_need_deferred_wakeup (rdp)) { + rdp->n_rp_nocb_defer_wakeup++; +return 1 ; + } +/* nothing to do */ rdp->n_rp_need_nothing++;return 0 ;
kernel/rcutree.h +7 -2
@@ -317,6 +317,7 @@ struct rcu_data {unsigned long n_rp_cpu_needs_gp;unsigned long n_rp_gp_completed;unsigned long n_rp_gp_started; +unsigned long n_rp_nocb_defer_wakeup;unsigned long n_rp_need_nothing;/* 6) _rcu_barrier() and OOM callbacks. */
@@ -335,6 +336,7 @@ struct rcu_data {int nocb_p_count_lazy;/* (approximate). */ wait_queue_head_t nocb_wq;/* For nocb kthreads to sleep on. */ struct task_struct *nocb_kthread; +bool nocb_defer_wakeup;/* Defer wakeup of nocb_kthread. */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */ /* 8) RCU CPU stall data. */
@@ -550,9 +552,12 @@ static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq);static void rcu_nocb_gp_cleanup (struct rcu_state *rsp,struct rcu_node *rnp);static void rcu_init_one_nocb (struct rcu_node *rnp);static bool __call_rcu_nocb (struct rcu_data *rdp,struct rcu_head *rhp, -bool lazy); +bool lazy,unsigned long flags);static bool rcu_nocb_adopt_orphan_cbs (struct rcu_state *rsp, -struct rcu_data *rdp); +struct rcu_data *rdp, +unsigned long flags); +static bool rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp); +static void do_nocb_deferred_wakeup (struct rcu_data *rdp);static void rcu_boot_init_nocb_percpu_data (struct rcu_data *rdp);static void rcu_spawn_nocb_kthreads (struct rcu_state *rsp);static void rcu_kick_nohz_cpu (int cpu);
kernel/rcutree_plugin.h +45 -10
119 lines not on this page · raw message
kernel/rcutree_trace.c +2 -1
@@ -364,9 +364,10 @@ static void print_one_rcu_pending(struct seq_file *m, struct rcu_data *rdp) rdp->n_rp_report_qs, rdp->n_rp_cb_ready, rdp->n_rp_cpu_needs_gp); -seq_printf (m,"gpc=%ld gps=%ld nn=%ld\n" , +seq_printf (m,"gpc=%ld gps=%ld nn=%ld ndw%ld\n" , rdp->n_rp_gp_completed, rdp->n_rp_gp_started, + rdp->n_rp_nocb_defer_wakeup, rdp->n_rp_need_nothing); }
Quoted 67 lines · Dave Chinner
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!Or lockdep?
Quoted 39 lines
turning off the locking correctness validator. Please attach the output of
/proc/lock_statto the bug reportCPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs]Quoted 8 lines
[<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2It's hard to see what in XFS is causing this. You're reading a single block directory, which means we're holding two inode locks here, and then we've done a lookup on a radix tree under
rcu_read_lock(). Hence I can't see how we've overrun the lockdep chain depth in the XFS code path. FWIW, it's thrown this warning when callingrcu_read_unlock()here:struct
xfs_perag*
xfs_perag_get(
structxfs_mount*mp,
xfs_agnumber_tagno)
Quoted 11 lines
{ struct xfs_perag *pag; int ref = 0; rcu_read_lock(); pag = radix_tree_lookup(&mp->m_perag_tree, agno); if (pag) { ASSERT(atomic_read(&pag->pag_ref) >= 0); ref = atomic_inc_return(&pag->pag_ref); }Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rcu_read_unlock();
Would xfs be holding one of the scheduler's rq or pi locks at this point? That could result in deadlock.
But I doubt that this is the problem, unless radix_tree_lookup() grabs one and returns with it held. Otherwise, you have interrupts disabled throughout the RCU read-side critical section, and thus are guaranteed to take the lockless fastpath through rcu_read_unlock(). (As opposed to merely having extremely high probability of taking that fastpath.)
Thanx, Paul
Quoted 4 lines
trace_xfs_perag_get(mp, agno, ref, _RET_IP_);
return pag;
}Quoted 9 lines
So I'd be looking at RCU or lockdep at this point....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Paul E. McKenney
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!I have to ask... What happens when you bump up
MAX_LOCKDEP_CHAINS?
We eat up a even more memory I guess.
kernel/lockdep_internals.h:#define MAX_LOCKDEP_CHAINS_BITS 15
kernel/lockdep_internals.h:#define MAX_LOCKDEP_CHAINS (1UL << MAX_LOCKDEP_CHAINS_BITS)
I had to increase MAX_LOCKDEP_ENTRIES on my local tree already because I was hitting that limit.
Quoted 3 lines
There is a patch to avoid an RCU/scheduler/perf deadlock, which may be found below. But this stack doesn't look to me to be matching that deadlock.
I'll add that to my tree until it gets mainlined.
thanks,
Dave
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 78 lines · Paul E. McKenney
Quoted 77 lines · Dave Chinner
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!Or lockdep?
Quoted 39 lines
turning off the locking correctness validator. Please attach the output of
/proc/lock_statto the bug reportCPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs]Quoted 8 lines
[<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2It's hard to see what in XFS is causing this. You're reading a single block directory, which means we're holding two inode locks here, and then we've done a lookup on a radix tree under
rcu_read_lock(). Hence I can't see how we've overrun the lockdep chain depth in the XFS code path. FWIW, it's thrown this warning when callingrcu_read_unlock()here:struct
xfs_perag*
xfs_perag_get(
structxfs_mount*mp,
xfs_agnumber_tagno){ struct xfs_perag *pag; int ref = 0; rcu_read_lock(); pag = radix_tree_lookup(&mp->m_perag_tree, agno); if (pag) { ASSERT(atomic_read(&pag->pag_ref) >= 0); ref = atomic_inc_return(&pag->pag_ref); }
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rcu_read_unlock();Would xfs be holding one of the scheduler's rq or pi locks at this point? That could result in deadlock.
XFS doesn't have any hooks into the scheduler at all. So if there is a problem with scheduler locks, then it's been leaked by the scheduler or something intimately familiar with the scheduler...
Quoted 2 lines
But I doubt that this is the problem, unless
radix_tree_lookup()grabs one and returns with it held.
Same thing - if the radix tree code returns with a scheduler lock held then there's a bug in the scheduler somewhere...
Quoted 4 lines
Otherwise, you have interrupts disabled
throughout the RCU read-side critical section, and thus are guaranteed to take the lockless fastpath throughrcu_read_unlock(). (As opposed to merely having extremely high probability of taking that fastpath.)
XFS doesn't disable interrupts anywhere itself, so I'm assuming that you are talking about something that is done internally in the rcu_read_lock()/unlock() calls?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 79 lines · Dave Chinner
Quoted 78 lines · Paul E. McKenney
Quoted 77 lines · Dave Chinner
Quoted 4 lines · Dave Jones
I was deleting a kernel tree, when this happened.. RCU, or xfs ?
BUG: MAX_LOCKDEP_CHAINS too low!Or lockdep?
Quoted 39 lines
turning off the locking correctness validator. Please attach the output of
/proc/lock_statto the bug reportCPU: 2 PID: 8109 Comm: rm Not tainted 3.12.0-rc4+ #96 ffffffff824bc2e0 ffff880026f416a8 ffffffff8172d798 41e619a2e5098827 ffff880026f41768 ffffffff810cc91f 0000000000000002 000010a02977b643 ffff880026f416d8 0000000000000212 ffff880026f416f8 ffffffff810c7329 Call Trace: [<ffffffff8172d798>] dump_stack+0x4e/0x82 [<ffffffff810cc91f>] __lock_acquire+0x1b7f/0x1be0 [<ffffffff810c7329>] ? get_lock_stats+0x19/0x60 [<ffffffff810c7f5d>] ? lock_release_holdtime.part.29+0x9d/0x160 [<ffffffff810cd133>] lock_acquire+0x93/0x200 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff817373c0>] _raw_spin_lock+0x40/0x80 [<ffffffff81097d0a>] ? try_to_wake_up+0x22a/0x350 [<ffffffff81097d0a>] try_to_wake_up+0x22a/0x350 [<ffffffff81097ea2>] default_wake_function+0x12/0x20 [<ffffffff81084c38>] autoremove_wake_function+0x18/0x40 [<ffffffff81090083>] ? __wake_up+0x23/0x50 [<ffffffff8108eb78>] __wake_up_common+0x58/0x90 [<ffffffff81090099>] __wake_up+0x39/0x50 [<ffffffff8110dc38>] rcu_report_qs_rsp+0x48/0x70 [<ffffffff8110f7d4>] rcu_report_unblock_qs_rnp+0x84/0x90 [<ffffffff81117fcf>] ? rcu_read_unlock_special+0x9f/0x4e0 [<ffffffff81118264>] rcu_read_unlock_special+0x334/0x4e0 [<ffffffff810c720f>] ? trace_hardirqs_off_caller+0x1f/0xc0 [<ffffffff8107f16c>] __rcu_read_unlock+0x8c/0x90 [<ffffffffa027084e>] xfs_perag_get+0xde/0x2a0 [xfs] [<ffffffffa0270775>] ? xfs_perag_get+0x5/0x2a0 [xfs] [<ffffffffa01f72f6>] _xfs_buf_find+0xd6/0x480 [xfs] [<ffffffffa01f780a>] xfs_buf_get_map+0x2a/0x260 [xfs] [<ffffffffa01f8fcc>] xfs_buf_read_map+0x2c/0x200 [xfs] [<ffffffffa0281019>] xfs_trans_read_buf_map+0x4b9/0xa70 [xfs] [<ffffffffa0245578>] xfs_da_read_buf+0xb8/0x340 [xfs] [<ffffffff810ca84b>] ? mark_held_locks+0xbb/0x140 [<ffffffffa024a3b9>] xfs_dir3_block_read+0x39/0x80 [xfs] [<ffffffffa024a440>] xfs_dir2_block_lookup_int+0x40/0x260 [xfs] [<ffffffffa024b49d>] xfs_dir2_block_removename+0x3d/0x390 [xfs] [<ffffffffa022d0da>] ? xfs_bmap_last_offset+0x4a/0xa0 [xfs]Quoted 8 lines
[<ffffffffa024991c>] xfs_dir_removename+0x11c/0x180 [xfs] [<ffffffffa0262085>] xfs_remove+0x2e5/0x510 [xfs] [<ffffffffa020ab8b>] xfs_vn_unlink+0x4b/0x90 [xfs] [<ffffffff811d4fd0>] vfs_unlink+0x90/0x100 [<ffffffff811d51bf>] do_unlinkat+0x17f/0x240 [<ffffffff81010b45>] ? syscall_trace_enter+0x145/0x2a0 [<ffffffff811d830b>] SyS_unlinkat+0x1b/0x40 [<ffffffff817408e4>] tracesys+0xdd/0xe2It's hard to see what in XFS is causing this. You're reading a single block directory, which means we're holding two inode locks here, and then we've done a lookup on a radix tree under
rcu_read_lock(). Hence I can't see how we've overrun the lockdep chain depth in the XFS code path. FWIW, it's thrown this warning when callingrcu_read_unlock()here:struct
xfs_perag*
xfs_perag_get(
structxfs_mount*mp,
xfs_agnumber_tagno){ struct xfs_perag *pag; int ref = 0; rcu_read_lock(); pag = radix_tree_lookup(&mp->m_perag_tree, agno); if (pag) { ASSERT(atomic_read(&pag->pag_ref) >= 0); ref = atomic_inc_return(&pag->pag_ref); }
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rcu_read_unlock();Would xfs be holding one of the scheduler's rq or pi locks at this point? That could result in deadlock.
XFS doesn't have any hooks into the scheduler at all. So if there is a problem with scheduler locks, then it's been leaked by the scheduler or something intimately familiar with the scheduler...
OK, that is what I thought, but had to ask.
Quoted 6 lines
Quoted 2 lines
But I doubt that this is the problem, unless
radix_tree_lookup()grabs one and returns with it held.Same thing - if the radix tree code returns with a scheduler lock held then there's a bug in the scheduler somewhere...
Quoted 8 lines
Quoted 4 lines
Otherwise, you have interrupts disabled
throughout the RCU read-side critical section, and thus are guaranteed to take the lockless fastpath throughrcu_read_unlock(). (As opposed to merely having extremely high probability of taking that fastpath.)XFS doesn't disable interrupts anywhere itself, so I'm assuming that you are talking about something that is done internally in the rcu_read_lock()/unlock() calls?
No, it is just that the scheduler locks of concern are irq-disabled spinlocks. So unless interrupts are disabled your cannot (or at least really should not) be holding them.
In short, looks like something else is going on here.
Thanx, Paul