sched: spinlock recursion in migrate_swap_stop
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following spew:
[ 1738.758667] BUG: spinlock recursion on CPU#23, migration/23/328 [ 1738.761291] lock: 0xffff8801f93b0c30, .magic: dead4ead, .owner: migration/23/328, .owner_cpu: 23 [ 1738.762733] CPU: 23 PID: 328 Comm: migration/23 Not tainted 3.15.0-rc5-next-20140512-sasha-00019-ga20bc00-dirty #456 [ 1738.764081] ffff8801f93b0c30 ffff8805505efbc8 ffffffff9453e1ec 0000000000000004 [ 1738.764914] ffff8805505cb000 ffff8805505efbe8 ffffffff94531095 ffff8801f93b0c30 [ 1738.765120] ffffffff9583485e ffff8805505efc08 ffffffff945310c0 ffff8801f93b0c30 [ 1738.765120] Call Trace: [ 1738.768386] dump_stack (lib/dump_stack.c:52) [ 1738.768386] spin_dump (kernel/locking/spinlock_debug.c:68 (discriminator 6)) [ 1738.768386] spin_bug (kernel/locking/spinlock_debug.c:76) [ 1738.768386] do_raw_spin_lock (kernel/locking/spinlock_debug.c:84 kernel/locking/spinlock_debug.c:135) [ 1738.768386] _raw_spin_lock_nested (kernel/locking/spinlock.c:362 (discriminator 2)) [ 1738.768386] ? migrate_swap_stop (arch/x86/include/asm/paravirt.h:804 kernel/sched/sched.h:1406 kernel/sched/core.c:1099) [ 1738.768386] ? _raw_spin_lock (kernel/locking/spinlock.c:152) [ 1738.768386] ? migrate_swap_stop (kernel/sched/sched.h:1393 kernel/sched/core.c:1097) [ 1738.768386] migrate_swap_stop (arch/x86/include/asm/paravirt.h:804 kernel/sched/sched.h:1406 kernel/sched/core.c:1099) [ 1738.768386] ? queue_stop_cpus_work (include/linux/cpumask.h:108 include/linux/cpumask.h:174 kernel/stop_machine.c:375) [ 1738.768386] multi_cpu_stop (kernel/stop_machine.c:225) [ 1738.768386] ? queue_stop_cpus_work (kernel/stop_machine.c:171) [ 1738.768386] cpu_stopper_thread (kernel/stop_machine.c:498) [ 1738.768386] ? put_lock_stats.isra.12 (arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254) [ 1738.768386] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:160 kernel/locking/spinlock.c:191) [ 1738.768386] ? __this_cpu_preempt_check (lib/smp_processor_id.c:63) [ 1738.768386] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) [ 1738.768386] smpboot_thread_fn (kernel/smpboot.c:160) [ 1738.768386] ? __smpboot_create_thread (kernel/smpboot.c:105) [ 1738.768386] kthread (kernel/kthread.c:210) [ 1738.768386] ? complete (kernel/sched/completion.c:35) [ 1738.768386] ? kthread_create_on_node (kernel/kthread.c:176) [ 1738.768386] ret_from_fork (arch/x86/kernel/entry_64.S:553) [ 1738.768386] ? kthread_create_on_node (kernel/kthread.c:176)
Thanks,
Sasha
ping? It seems to be easy enough to reproduce on -next, I'd be happy to try debug patches/fixes.
Thanks,
Sasha
Quoted 41 lines · Sasha Levin
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following spew:
[ 1738.758667] BUG: spinlock recursion on CPU#23, migration/23/328 [ 1738.761291] lock: 0xffff8801f93b0c30, .magic: dead4ead, .owner: migration/23/328, .owner_cpu: 23 [ 1738.762733] CPU: 23 PID: 328 Comm: migration/23 Not tainted 3.15.0-rc5-next-20140512-sasha-00019-ga20bc00-dirty #456 [ 1738.764081] ffff8801f93b0c30 ffff8805505efbc8 ffffffff9453e1ec 0000000000000004 [ 1738.764914] ffff8805505cb000 ffff8805505efbe8 ffffffff94531095 ffff8801f93b0c30 [ 1738.765120] ffffffff9583485e ffff8805505efc08 ffffffff945310c0 ffff8801f93b0c30 [ 1738.765120] Call Trace: [ 1738.768386] dump_stack (lib/dump_stack.c:52) [ 1738.768386] spin_dump (kernel/locking/spinlock_debug.c:68 (discriminator 6)) [ 1738.768386] spin_bug (kernel/locking/spinlock_debug.c:76) [ 1738.768386] do_raw_spin_lock (kernel/locking/spinlock_debug.c:84 kernel/locking/spinlock_debug.c:135) [ 1738.768386] _raw_spin_lock_nested (kernel/locking/spinlock.c:362 (discriminator 2)) [ 1738.768386] ? migrate_swap_stop (arch/x86/include/asm/paravirt.h:804 kernel/sched/sched.h:1406 kernel/sched/core.c:1099) [ 1738.768386] ? _raw_spin_lock (kernel/locking/spinlock.c:152) [ 1738.768386] ? migrate_swap_stop (kernel/sched/sched.h:1393 kernel/sched/core.c:1097) [ 1738.768386] migrate_swap_stop (arch/x86/include/asm/paravirt.h:804 kernel/sched/sched.h:1406 kernel/sched/core.c:1099) [ 1738.768386] ? queue_stop_cpus_work (include/linux/cpumask.h:108 include/linux/cpumask.h:174 kernel/stop_machine.c:375) [ 1738.768386] multi_cpu_stop (kernel/stop_machine.c:225) [ 1738.768386] ? queue_stop_cpus_work (kernel/stop_machine.c:171) [ 1738.768386] cpu_stopper_thread (kernel/stop_machine.c:498) [ 1738.768386] ? put_lock_stats.isra.12 (arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254) [ 1738.768386] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:160 kernel/locking/spinlock.c:191) [ 1738.768386] ? __this_cpu_preempt_check (lib/smp_processor_id.c:63) [ 1738.768386] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) [ 1738.768386] smpboot_thread_fn (kernel/smpboot.c:160) [ 1738.768386] ? __smpboot_create_thread (kernel/smpboot.c:105) [ 1738.768386] kthread (kernel/kthread.c:210) [ 1738.768386] ? complete (kernel/sched/completion.c:35) [ 1738.768386] ? kthread_create_on_node (kernel/kthread.c:176) [ 1738.768386] ret_from_fork (arch/x86/kernel/entry_64.S:553) [ 1738.768386] ? kthread_create_on_node (kernel/kthread.c:176)Thanks,
Sasha
Peter Zijlstra
maintainerQuoted 2 lines · Sasha Levin
ping? It seems to be easy enough to reproduce on -next, I'd be happy to try debug patches/fixes.
Does this fuzzing you do also include hotplug? If so, does disabling that make this problem go away?
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Sasha Levin
ping? It seems to be easy enough to reproduce on -next, I'd be happy to try debug patches/fixes.
Does this fuzzing you do also include hotplug? If so, does disabling that make this problem go away?
There were no hotplug operations going on when this happens, so it seems unrelated.
Thanks,
Sasha
Quoted 10 lines · Sasha Levin
Quoted 7 lines · Peter Zijlstra
Quoted 7 lines · Sasha Levin
Quoted 2 lines
Quoted 2 lines
ping? It seems to be easy enough to reproduce on -next, I'd be happy to try debug patches/fixes.
Does this fuzzing you do also include hotplug? If so, does disabling that make this problem go away?
There were no hotplug operations going on when this happens, so it seems unrelated.
I've added a small test:
kernel/sched/core.c +1 -0
@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out;trace_sched_swap_numa (cur, arg.src_cpu, p, arg.dst_cpu); +BUG_ON (cur == p); ret =stop_two_cpus (arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:
Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Thanks,
Sasha
Peter Zijlstra
maintainerQuoted 28 lines · Sasha Levin
Quoted 10 lines · Sasha Levin
Quoted 7 lines · Peter Zijlstra
Quoted 7 lines · Sasha Levin
Quoted 2 lines
Quoted 2 lines
ping? It seems to be easy enough to reproduce on -next, I'd be happy to try debug patches/fixes.
Does this fuzzing you do also include hotplug? If so, does disabling that make this problem go away?
There were no hotplug operations going on when this happens, so it seems unrelated.
I've added a small test:
kernel/sched/core.c +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out;trace_sched_swap_numa (cur, arg.src_cpu, p, arg.dst_cpu); +BUG_ON (cur == p); ret =stop_two_cpus (arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:
Quoted 2 lines
Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Oi, good call that, lemme go stare.
Peter Zijlstra
maintainerQuoted 10 lines · Peter Zijlstra
Quoted 8 lines · Sasha Levin
+++
b/kernel/sched/core.c(quoted diff) +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out; trace_sched_swap_numa(cur, arg.src_cpu, p, arg.dst_cpu); + BUG_ON(cur == p); ret = stop_two_cpus(arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg);
Quoted 7 lines
Quoted 5 lines
(quoted diff) +0
-0out:Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Oi, good call that, lemme go stare.
I think something simple like this should be sufficient to avoid the problem of selecting oneself as a flip target.
--- kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/fair.c +2 -0
@@ -1115,6 +1115,8 @@ static void task_numa_compare(struct task_numa_env *env, cur =ACCESS_ONCE (dst_rq->curr);if (cur->pid ==0 )/* idle */ cur =NULL ; +if (cur == env->p) + goto unlock;/* * "imp" is the fault differential for the source task between the
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
Quoted 12 lines · Sasha Levin
Quoted 11 lines
Quoted 11 lines
+++
b/kernel/sched/core.c(quoted diff) +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out; trace_sched_swap_numa(cur, arg.src_cpu, p, arg.dst_cpu); + BUG_ON(cur == p); ret = stop_two_cpus(arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Oi, good call that, lemme go stare.
I think something simple like this should be sufficient to avoid the problem of selecting oneself as a flip target.
Why would that happen in the first place?
Thanks,
Sasha
Peter Zijlstra
maintainerQuoted 14 lines · Sasha Levin
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
Quoted 12 lines · Sasha Levin
Quoted 11 lines
Quoted 11 lines
+++
b/kernel/sched/core.c(quoted diff) +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out; trace_sched_swap_numa(cur, arg.src_cpu, p, arg.dst_cpu); + BUG_ON(cur == p); ret = stop_two_cpus(arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Oi, good call that, lemme go stare.
I think something simple like this should be sufficient to avoid the problem of selecting oneself as a flip target.
Why would that happen in the first place?
We do all that with preemption enabled, because its big and expensive, so its entirely possible that current (env->p) got moved around while we were doing it, at which point we'll look at it as a possible dst, while its already our src.
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Sasha Levin
Quoted 19 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 9 lines · Sasha Levin
Quoted 8 lines
Quoted 8 lines
+++
b/kernel/sched/core.c(quoted diff) +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out; trace_sched_swap_numa(cur, arg.src_cpu, p, arg.dst_cpu); + BUG_ON(cur == p); ret = stop_two_cpus(arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg);Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +0
-0out:Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
Oi, good call that, lemme go stare.
I think something simple like this should be sufficient to avoid the problem of selecting oneself as a flip target.
Quoted 6 lines
Quoted 1 line
Why would that happen in the first place?
We do all that with preemption enabled, because its big and expensive, so its entirely possible that current (
env->p) got moved around while we were doing it, at which point we'll look at it as a possible dst, while its already our src.
Seems to be working fine now, thanks!
Thanks,
Sasha
kernel/sched/core.c int migrate_swap(struct task_struct *cur, struct task_struct *p) · line 1159
Quoted 9 lines
kernel/sched/core.c +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out;trace_sched_swap_numa (cur, arg.src_cpu, p, arg.dst_cpu); +BUG_ON (cur == p);
I am not sure how this check can ever be successful because at the start of this function migrate_swap() we have
if (arg.src_cpu == arg.dst_cpu)
goto out;
if cur is actually p; then should the above condition should always be successul right?
Or am I missing something?
Quoted 8 lines
kernel/sched/core.c +0
-0ret =stop_two_cpus (arg.dst_cpu, arg.src_cpu, migrate_swap_stop, &arg); out:Which seems to get hit. This sounds like a race with task moving to other cpu maybe?
--
Thanks and Regards
Srikar Dronamraju
Peter Zijlstra
maintainerQuoted 13 lines · Srikar Dronamraju
kernel/sched/core.c int migrate_swap(struct task_struct *cur, struct task_struct *p) · line 1159
Quoted 9 lines
kernel/sched/core.c +1
-0@@ -1154,6 +1156,7 @@ int migrate_swap(struct task_struct *cur, struct task_struct *p) goto out;trace_sched_swap_numa (cur, arg.src_cpu, p, arg.dst_cpu); +BUG_ON (cur == p);I am not sure how this check can ever be successful because at the start of this function
migrate_swap()we have
Quoted 8 lines
if (arg.src_cpu == arg.dst_cpu)
goto out;if cur is actually p; then should the above condition should always be successul right?
Or am I missing something?
Yeah, current might have migrated while we were looking for a target.