cpuset: fix possible deadlock in async_rebuild_sched_domains
PATCH RESEND 1/1 cpuset: fix possible deadlock in async_rebuild_sched_domains
Lockdep reported some possible circular locking info when we tested cpuset on NUMA/fake NUMA box.
=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.29-rc1-00224-ga652504 #111
-------------------------------------------------------
bash/2968 is trying to acquire lock:
(events){--..}, at: [<ffffffff8024c8cd>] flush_work+0x24/0xd8
but task is already holding lock:
(cgroup_mutex){--..}, at: [<ffffffff8026ad1e>] cgroup_lock_live_group+0x12/0x29
which lock already depends on the new lock.
......
-------------------------------------------------------
Steps to reproduce:
# mkdir /dev/cpuset # mount -t cpuset xxx /dev/cpuset # mkdir /dev/cpuset/0 # echo 0 > /dev/cpuset/0/cpus # echo 0 > /dev/cpuset/0/mems # echo 1 > /dev/cpuset/0/memory_migrate # cat /dev/zero > /dev/null & # echo $! > /dev/cpuset/0/tasks
This is because async_rebuild_sched_domains has the following lock sequence:run_workqueue(async_rebuild_sched_domains)
-> do_rebuild_sched_domains -> cgroup_lock
But, attaching tasks when memory_migrate is set has following:cgroup_lock_live_group(cgroup_tasks_write)
-> do_migrate_pages -> flush_work
This patch fixes it by using a separate workqueue thread.
--- kernel/cpuset.c | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-)
kernel/cpuset.c +12 -1
@@ -61,6 +61,14 @@#include <linux/cgroup.h> /* +* Workqueue for cpuset related tasks. +* +* Using kevent workqueue may cause deadlock when memory_migrate +* is set. So we create a separate workqueue thread for cpuset. +*/ +static struct workqueue_struct *cpuset_wq; + +/* * Tracks how many cpusets are currently defined in system. * When there is only one cpuset (the root cpuset) we can * short circuit some hooks.
@@ -831,7 +839,7 @@ static DECLARE_WORK(rebuild_sched_domains_work, do_rebuild_sched_domains);*/ static void async_rebuild_sched_domains (void ) { -schedule_work (&rebuild_sched_domains_work); +queue_work (cpuset_wq, &rebuild_sched_domains_work); }/*
@@ -2111,6 +2119,9 @@ void __init cpuset_init_smp(void)hotcpu_notifier (cpuset_track_online_cpus,0 );hotplug_memory_notifier (cpuset_track_online_nodes,10 ); + + cpuset_wq =create_singlethread_workqueue ("cpuset" ); +BUG_ON (!cpuset_wq); }/**
Signature · 1 line
1.6.0.3
Ingo Molnar
AppliedQuoted 2 lines · Miao Xie
Lockdep reported some possible circular locking info when we tested cpuset on NUMA/fake NUMA box.
I have applied your original patch to tip/sched/urgent two days ago, thanks!
Ingo
Quoted 27 lines · On Tue, 20 Jan 2009 11:10:54 +0800 Miao Xie <[email protected]> wrote:
Lockdep reported some possible circular locking info when we tested cpuset on NUMA/fake NUMA box.
=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.29-rc1-00224-ga652504 #111
-------------------------------------------------------
bash/2968 is trying to acquire lock:
(events){--..}, at: [<ffffffff8024c8cd>]flush_work+0x24/0xd8but task is already holding lock:
(cgroup_mutex){--..}, at: [<ffffffff8026ad1e>]cgroup_lock_live_group+0x12/0x29which lock already depends on the new lock.
......
-------------------------------------------------------Steps to reproduce:
# mkdir /dev/cpuset # mount -t cpuset xxx /dev/cpuset # mkdir /dev/cpuset/0 # echo 0 > /dev/cpuset/0/cpus # echo 0 > /dev/cpuset/0/mems # echo 1 > /dev/cpuset/0/memory_migrate # cat /dev/zero > /dev/null & # echo $! > /dev/cpuset/0/tasks
Quoted 7 lines
This is because
async_rebuild_sched_domainshas the following lock sequence:run_workqueue(async_rebuild_sched_domains)-> do_rebuild_sched_domains -> cgroup_lockBut, attaching tasks when
memory_migrateis set has following:cgroup_lock_live_group(cgroup_tasks_write)-> do_migrate_pages -> flush_work
Where is this flush_work() call? lru_add_drain_all()->schedule_on_each_cpu()?
If so, and if that is the only such callsite then we could/should rework this code to use work_on_cpu(), if we manage to fix that thing.
It would be somewhat inefficient. It would be better if work_on_cpu() were to take a cpumask argument, and avoid blocking behind each CPU one at a time. But first things first.
Quoted 1 line
This patch fixes it by using a separate workqueue thread.
<wonders when RESERVED_PIDS became a logarithm>