Skip to contentSkip to conversation

workqueue: move lockdep annotations up to destroy_workqueue()

PATCH In review Amerigo Wang 3 participants 14 messages Mar 31, 2010 – Apr 2, 2010 Workqueue

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 workqueue: move lockdep annotations up to destroy_workqueue()

This fixes a lockdep warning when invoking destroy_workqueue(), because the lockdep annotations are invoked under cpu_add_remove_lock.

So, move the lockdep annotations before taking cpu_add_remove_lock in destroy_workqueue(), this will not affect the original purpose of adding them for destroy_workqueue() etc.

However, it will affect another caller of cleanup_workqueue_thread(), that is, workqueue_cpu_callback(). This should be fine, because there are no other cases than cpu hotplug could call it.

Cc Oleg Nesterov <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Rusty Russell <[email protected]>

---

kernel/workqueue.c +3 -3
index dee4865..0f050e2 100644
@@ -1025,9 +1025,6 @@ static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)
 	if (cwq->thread == NULL)
 		return;
 
-	lock_map_acquire(&cwq->wq->lockdep_map);
-	lock_map_release(&cwq->wq->lockdep_map);
-
 	flush_cpu_workqueue(cwq);
 	/*
 	 * If the caller is CPU_POST_DEAD and cwq->worklist was not empty,
@@ -1055,6 +1052,9 @@ void destroy_workqueue(struct workqueue_struct *wq)
 	const struct cpumask *cpu_map = wq_cpu_map(wq);
 	int cpu;
 
+	lock_map_acquire(&wq->lockdep_map);
+	lock_map_release(&wq->lockdep_map);
+
 	cpu_maps_update_begin();
 	spin_lock(&workqueue_lock);
 	list_del(&wq->list);
#1 lore raw
Quoted 3 lines · Amerigo Wang

This fixes a lockdep warning when invoking destroy_workqueue(), because the lockdep annotations are invoked under cpu_add_remove_lock.

Confused. Why does lockdep complains?

Quoted 7 lines

So, move the lockdep annotations before taking cpu_add_remove_lock in destroy_workqueue(), this will not affect the original purpose of adding them for destroy_workqueue() etc.

However, it will affect another caller of cleanup_workqueue_thread(), that is, workqueue_cpu_callback(). This should be fine, because there are no other cases than cpu hotplug could call it.

OK, but nobody should take cpu_maps_update_begin() under wq->lockdep_map, in particular work->func() must not.

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Oleg.

author
#2 lore raw
Quoted 5 lines · Oleg Nesterov
Quoted 2 lines · Amerigo Wang

This fixes a lockdep warning when invoking destroy_workqueue(), because the lockdep annotations are invoked under cpu_add_remove_lock.

Confused. Why does lockdep complains?

Because lockdep annotations are added to prevent other locks are taken before calling these functions.

Quoted 9 lines
Quoted 8 lines

So, move the lockdep annotations before taking cpu_add_remove_lock in destroy_workqueue(), this will not affect the original purpose of adding them for destroy_workqueue() etc.

However, it will affect another caller of cleanup_workqueue_thread(), that is, workqueue_cpu_callback(). This should be fine, because there are no other cases than cpu hotplug could call it.

Quoted 8 lines

OK, but nobody should take cpu_maps_update_begin() under wq->lockdep_map, in particular work->func() must not.

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Sure, below is the whole warning. Please teach me how this is valid.

Thanks.

Mar 31 16:15:00 dhcp-66-70-5 kernel: bonding: bond0: released all slaves
Mar 31 16:15:00 dhcp-66-70-5 kernel:
Mar 31 16:15:00 dhcp-66-70-5 kernel: =======================================================
Mar 31 16:15:00 dhcp-66-70-5 kernel: [ INFO: possible circular locking dependency detected ]
Mar 31 16:15:00 dhcp-66-70-5 kernel: 2.6.34-rc3 #90
Mar 31 16:15:00 dhcp-66-70-5 kernel: -------------------------------------------------------
Mar 31 16:15:00 dhcp-66-70-5 kernel: modprobe/5264 is trying to acquire lock:
Mar 31 16:15:00 dhcp-66-70-5 kernel:  ((bond_dev->name)){+.+...}, at: [<ffffffff8108524a>] cleanup_workqueue_thread+0x2b/0x10b
Mar 31 16:15:00 dhcp-66-70-5 kernel:
Mar 31 16:15:00 dhcp-66-70-5 kernel: but task is already holding lock:
Mar 31 16:15:00 dhcp-66-70-5 kernel:  (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27
Mar 31 16:15:00 dhcp-66-70-5 kernel:
Mar 31 16:15:00 dhcp-66-70-5 kernel: which lock already depends on the new lock.
Mar 31 16:15:01 dhcp-66-70-5 kernel:
Mar 31 16:15:01 dhcp-66-70-5 kernel:
Mar 31 16:15:01 dhcp-66-70-5 kernel: the existing dependency chain (in reverse order) is:
Mar 31 16:15:01 dhcp-66-70-5 kernel:
Mar 31 16:15:01 dhcp-66-70-5 kernel: -> #3 (cpu_add_remove_lock){+.+.+.}:
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810853cd>] destroy_workqueue+0x41/0x107
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b
Mar 31 16:15:02 dhcp-66-70-5 kernel:
Mar 31 16:15:02 dhcp-66-70-5 kernel: -> #2 (rtnl_mutex){+.+.+.}:
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffffa0836779>] bond_mii_monitor+0x39f/0x74b [bonding]
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108654f>] worker_thread+0x2da/0x46c
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108b1ea>] kthread+0xdd/0xec
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff81004894>] kernel_thread_helper+0x4/0x10
Mar 31 16:15:02 dhcp-66-70-5 kernel:
Mar 31 16:15:02 dhcp-66-70-5 kernel: -> #1 ((&(&bond->mii_work)->work)){+.+...}:
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81086542>] worker_thread+0x2cd/0x46c
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8108b1ea>] kthread+0xdd/0xec
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81004894>] kernel_thread_helper+0x4/0x10
Mar 31 16:15:03 dhcp-66-70-5 kernel:
Mar 31 16:15:03 dhcp-66-70-5 kernel: -> #0 ((bond_dev->name)){+.+...}:
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a6696>] validate_chain+0xaee/0x1540
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b
Mar 31 16:15:03 dhcp-66-70-5 kernel:
Mar 31 16:15:03 dhcp-66-70-5 kernel: other info that might help us debug this:
Mar 31 16:15:03 dhcp-66-70-5 kernel:
Mar 31 16:15:03 dhcp-66-70-5 kernel: 2 locks held by modprobe/5264:
Mar 31 16:15:03 dhcp-66-70-5 kernel:  #0:  (rtnl_mutex){+.+.+.}, at: [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
Mar 31 16:15:03 dhcp-66-70-5 kernel:  #1:  (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27
Mar 31 16:15:03 dhcp-66-70-5 kernel:
Mar 31 16:15:03 dhcp-66-70-5 kernel: stack backtrace:
Mar 31 16:15:03 dhcp-66-70-5 kernel: Pid: 5264, comm: modprobe Not tainted 2.6.34-rc3 #90
Mar 31 16:15:03 dhcp-66-70-5 kernel: Call Trace:
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810a50c1>] print_circular_bug+0x169/0x17e
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810a6696>] validate_chain+0xaee/0x1540
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8108524a>] ? cleanup_workqueue_thread+0x2b/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8108524a>] ? cleanup_workqueue_thread+0x2b/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffffa0833a19>] ? bond_remove_proc_entry+0x0/0x109 [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff81003a8c>] ? sysret_check+0x27/0x62
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff810a3268>] ? trace_hardirqs_on_caller+0x264/0x29f
Mar 31 16:15:03 dhcp-66-70-5 kernel:  [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b
maintainer
#3 lore raw

Hello, guys.

Quoted 9 lines · Cong Wang
Quoted 7 lines

OK, but nobody should take cpu_maps_update_begin() under wq->lockdep_map, in particular work->func() must not.

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Sure, below is the whole warning. Please teach me how this is valid.

I still have some trouble interpreting lockdep warnings. Please correct me if I get something wrong.

Quoted 5 lines

modprobe/5264 is trying to acquire lock:
((bond_dev->name)){+.+...}, at: [<ffffffff8108524a>] cleanup_workqueue_thread+0x2b/0x10b

but task is already holding lock:
(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27

This (cpu hotplug -> wq) is the expected sequence. Plug cpu hotplugging and then flush cpu workqueues.

Quoted 1 line

which lock already depends on the new lock.

But lockdep says the other way around has already happened.

Quoted 18 lines

the existing dependency chain (in reverse order) is:

-> #3 (cpu_add_remove_lock){+.+.+.}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
       [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27
       [<ffffffff810853cd>] destroy_workqueue+0x41/0x107
       [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
       [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
       [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
       [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
       [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
       [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
       [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
       [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
       [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b

This is bond_uninit() calling destroy_workqueue() but I don't get how this thread would be already holding wq lock.

Quoted 11 lines

-> #2 (rtnl_mutex){+.+.+.}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
       [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
       [<ffffffffa0836779>] bond_mii_monitor+0x39f/0x74b [bonding]
       [<ffffffff8108654f>] worker_thread+0x2da/0x46c
       [<ffffffff8108b1ea>] kthread+0xdd/0xec
       [<ffffffff81004894>] kernel_thread_helper+0x4/0x10
Quoted 7 lines

-> #1 ((&(&bond->mii_work)->work)){+.+...}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff81086542>] worker_thread+0x2cd/0x46c
       [<ffffffff8108b1ea>] kthread+0xdd/0xec
       [<ffffffff81004894>] kernel_thread_helper+0x4/0x10

These two are form a workqueue worker thread and I don't quite understand why they are here.

Quoted 15 lines

-> #0 ((bond_dev->name)){+.+...}:

       [<ffffffff810a6696>] validate_chain+0xaee/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
       [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
       [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
       [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
       [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
       [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
       [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
       [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
       [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
       [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
       [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b

This seems to be from the original thread of frame#3. It's grabbing wq lock here but the problem is that the lock will be released immediately, so bond_dev->name (the wq) can't be held by the time it reaches frame#3. How is this dependency chain completed? Is it somehow transitive through rtnl_mutex?

Quoted 5 lines

other info that might help us debug this:

2 locks held by modprobe/5264:
#0: (rtnl_mutex){+.+.+.}, at: [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
#1: (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27

Isn't there a circular dependency here? bonding_exit() calls destroy_workqueue() under rtnl_mutex but destroy_workqueue() should flush works which could be trying to grab rtnl_lock. Or am I completely misunderstanding locking here?

Thanks.

Signature · 1 line
tejun
author
#4 lore raw
Quoted 47 lines · Tejun Heo

Hello, guys.

Quoted 8 lines · Cong Wang
Quoted 7 lines

OK, but nobody should take cpu_maps_update_begin() under wq->lockdep_map, in particular work->func() must not.

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Sure, below is the whole warning. Please teach me how this is valid.

I still have some trouble interpreting lockdep warnings. Please correct me if I get something wrong.

Quoted 5 lines

modprobe/5264 is trying to acquire lock:
((bond_dev->name)){+.+...}, at: [<ffffffff8108524a>] cleanup_workqueue_thread+0x2b/0x10b

but task is already holding lock:
(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27

This (cpu hotplug -> wq) is the expected sequence. Plug cpu hotplugging and then flush cpu workqueues.

Quoted 1 line

which lock already depends on the new lock.

But lockdep says the other way around has already happened.

Quoted 18 lines

the existing dependency chain (in reverse order) is:

-> #3 (cpu_add_remove_lock){+.+.+.}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
       [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27
       [<ffffffff810853cd>] destroy_workqueue+0x41/0x107
       [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
       [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
       [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
       [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
       [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
       [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
       [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
       [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
       [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b
Quoted 2 lines

This is bond_uninit() calling destroy_workqueue() but I don't get how this thread would be already holding wq lock.

destroy_workqueue() does hold wq lock and then releases it.

Quoted 39 lines
Quoted 12 lines

-> #2 (rtnl_mutex){+.+.+.}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
       [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
       [<ffffffffa0836779>] bond_mii_monitor+0x39f/0x74b [bonding]
       [<ffffffff8108654f>] worker_thread+0x2da/0x46c
       [<ffffffff8108b1ea>] kthread+0xdd/0xec
       [<ffffffff81004894>] kernel_thread_helper+0x4/0x10
Quoted 7 lines

-> #1 ((&(&bond->mii_work)->work)){+.+...}:

       [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff81086542>] worker_thread+0x2cd/0x46c
       [<ffffffff8108b1ea>] kthread+0xdd/0xec
       [<ffffffff81004894>] kernel_thread_helper+0x4/0x10

These two are form a workqueue worker thread and I don't quite understand why they are here.

Quoted 15 lines

-> #0 ((bond_dev->name)){+.+...}:

       [<ffffffff810a6696>] validate_chain+0xaee/0x1540
       [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
       [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
       [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
       [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
       [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
       [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
       [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
       [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
       [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
       [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
       [<ffffffffa0845ca6>] bonding_exit+0x5a/0x76 [bonding]
       [<ffffffff810b7749>] sys_delete_module+0x306/0x3b1
       [<ffffffff81003a5b>] system_call_fastpath+0x16/0x1b
Quoted 5 lines

This seems to be from the original thread of frame#3. It's grabbing wq lock here but the problem is that the lock will be released immediately, so bond_dev->name (the wq) can't be held by the time it reaches frame#3. How is this dependency chain completed? Is it somehow transitive through rtnl_mutex?

wq lock is held *after* cpu_add_remove_lock, lockdep also said this, the process is trying to hold wq lock while having cpu_add_remove_lock.

Quoted 11 lines
Quoted 6 lines

other info that might help us debug this:

2 locks held by modprobe/5264:
#0: (rtnl_mutex){+.+.+.}, at: [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
#1: (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff810631d1>] cpu_maps_update_begin+0x1e/0x27

Isn't there a circular dependency here? bonding_exit() calls destroy_workqueue() under rtnl_mutex but destroy_workqueue() should flush works which could be trying to grab rtnl_lock. Or am I completely misunderstanding locking here?

Sure, that is why I sent another patch for bonding. :)

After this patch, another lockdep warning appears, it is exactly what you expect.

Thanks.

maintainer
#5 lore raw

Hello,

Quoted 8 lines · Cong Wang
Quoted 5 lines

This seems to be from the original thread of frame#3. It's grabbing wq lock here but the problem is that the lock will be released immediately, so bond_dev->name (the wq) can't be held by the time it reaches frame#3. How is this dependency chain completed? Is it somehow transitive through rtnl_mutex?

wq lock is held *after* cpu_add_remove_lock, lockdep also said this, the process is trying to hold wq lock while having cpu_add_remove_lock.

Yeah yeah, I'm just failing to see how the other direction is completed. ie. where does the kernel try to grab cpu_add_remove_lock *after* grabbing wq lock?

Quoted 6 lines
Quoted 4 lines

Isn't there a circular dependency here? bonding_exit() calls destroy_workqueue() under rtnl_mutex but destroy_workqueue() should flush works which could be trying to grab rtnl_lock. Or am I completely misunderstanding locking here?

Sure, that is why I sent another patch for bonding. :)

Ah... great. :-)

Quoted 2 lines

After this patch, another lockdep warning appears, it is exactly what you expect.

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

Thanks.

Signature · 1 line
tejun
author
#6 lore raw
Quoted 23 lines · Tejun Heo

Hello,

Quoted 7 lines · Cong Wang
Quoted 5 lines

This seems to be from the original thread of frame#3. It's grabbing wq lock here but the problem is that the lock will be released immediately, so bond_dev->name (the wq) can't be held by the time it reaches frame#3. How is this dependency chain completed? Is it somehow transitive through rtnl_mutex?

wq lock is held *after* cpu_add_remove_lock, lockdep also said this, the process is trying to hold wq lock while having cpu_add_remove_lock.

Yeah yeah, I'm just failing to see how the other direction is completed. ie. where does the kernel try to grab cpu_add_remove_lock *after* grabbing wq lock?

Quoted 5 lines
Quoted 4 lines

Isn't there a circular dependency here? bonding_exit() calls destroy_workqueue() under rtnl_mutex but destroy_workqueue() should flush works which could be trying to grab rtnl_lock. Or am I completely misunderstanding locking here?

Sure, that is why I sent another patch for bonding. :)

Ah... great. :-)

Quoted 8 lines
Quoted 2 lines

After this patch, another lockdep warning appears, it is exactly what you expect.

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

I thought this is obvious.

Here it is:

void destroy_workqueue(struct workqueue_struct *wq)

{
        const struct cpumask *cpu_map = wq_cpu_map(wq);
        int cpu;

        cpu_maps_update_begin();        <----------------- Hold cpu_add_remove_lock here
        spin_lock(&workqueue_lock);
        list_del(&wq->list);
        spin_unlock(&workqueue_lock);

        for_each_cpu(cpu, cpu_map)
                cleanup_workqueue_thread(per_cpu_ptr(wq->cpu_wq, cpu));  <------ See below
        cpu_maps_update_done();        <----------------- Release cpu_add_remove_lock here

...
static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)

{
        /*
         * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
         * cpu_add_remove_lock protects cwq->thread.
         */
        if (cwq->thread == NULL)
                return;

        lock_map_acquire(&cwq->wq->lockdep_map); <-------------- Lockdep complains here.
        lock_map_release(&cwq->wq->lockdep_map);

...

Am I missing something??

Thanks.

maintainer
#7 lore raw

Hello,

Quoted 40 lines · Cong Wang
Quoted 5 lines

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

I thought this is obvious.

Here it is:

void destroy_workqueue(struct workqueue_struct *wq)

{
        const struct cpumask *cpu_map = wq_cpu_map(wq);
        int cpu;

        cpu_maps_update_begin();        <----------------- Hold
cpu_add_remove_lock here
        spin_lock(&workqueue_lock);
        list_del(&wq->list);
        spin_unlock(&workqueue_lock);

        for_each_cpu(cpu, cpu_map)
                cleanup_workqueue_thread(per_cpu_ptr(wq->cpu_wq, cpu)); 
<------ See below
        cpu_maps_update_done();        <----------------- Release

cpu_add_remove_lock here

...
static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)

{
        /*
         * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
         * cpu_add_remove_lock protects cwq->thread.
         */
        if (cwq->thread == NULL)
                return;

        lock_map_acquire(&cwq->wq->lockdep_map); <-------------- Lockdep
complains here.
        lock_map_release(&cwq->wq->lockdep_map);
Quoted 1 line

...

Yeap, the above is cpu_add_remove_lock -> wq->lockdep_map dependency. I can see that but I'm failing to see where the dependency the other direction is created.

Thanks.

Signature · 1 line
tejun
author
#8 lore raw
Quoted 39 lines · Tejun Heo

Hello,

Quoted 36 lines · Cong Wang
Quoted 5 lines

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

I thought this is obvious.

Here it is:

void destroy_workqueue(struct workqueue_struct *wq)

{
        const struct cpumask *cpu_map = wq_cpu_map(wq);
        int cpu;

        cpu_maps_update_begin();        <----------------- Hold
cpu_add_remove_lock here
        spin_lock(&workqueue_lock);
        list_del(&wq->list);
        spin_unlock(&workqueue_lock);

        for_each_cpu(cpu, cpu_map)
                cleanup_workqueue_thread(per_cpu_ptr(wq->cpu_wq, cpu)); 
<------ See below
        cpu_maps_update_done();        <----------------- Release

cpu_add_remove_lock here

...
static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)

{
        /*
         * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
         * cpu_add_remove_lock protects cwq->thread.
         */
        if (cwq->thread == NULL)
                return;
Quoted 9 lines
Quoted 4 lines
        lock_map_acquire(&cwq->wq->lockdep_map); <-------------- Lockdep
complains here.
        lock_map_release(&cwq->wq->lockdep_map);

...

Yeap, the above is cpu_add_remove_lock -> wq->lockdep_map dependency. I can see that but I'm failing to see where the dependency the other direction is created.

Hmm, it looks like I misunderstand lock_map_acquire()? From the changelog, I thought it was added to complain its caller is holding a lock when invoking it, thus cpu_add_remove_lock is not an exception.

Thanks!

author
#9 lore raw
Quoted 45 lines · Cong Wang
Quoted 44 lines · Tejun Heo

Hello,

Quoted 40 lines · Cong Wang
Quoted 5 lines

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

I thought this is obvious.

Here it is:

void destroy_workqueue(struct workqueue_struct *wq)

{
        const struct cpumask *cpu_map = wq_cpu_map(wq);
        int cpu;

        cpu_maps_update_begin();        <----------------- Hold
cpu_add_remove_lock here
        spin_lock(&workqueue_lock);
        list_del(&wq->list);
        spin_unlock(&workqueue_lock);

        for_each_cpu(cpu, cpu_map)
                cleanup_workqueue_thread(per_cpu_ptr(wq->cpu_wq, 
cpu)); <------ See below
        cpu_maps_update_done();        <----------------- Release

cpu_add_remove_lock here

...
static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)

{
        /*
         * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
         * cpu_add_remove_lock protects cwq->thread.
         */
        if (cwq->thread == NULL)
                return;

        lock_map_acquire(&cwq->wq->lockdep_map); <-------------- Lockdep
complains here.
        lock_map_release(&cwq->wq->lockdep_map);

...

Quoted 9 lines
Quoted 4 lines

Yeap, the above is cpu_add_remove_lock -> wq->lockdep_map dependency. I can see that but I'm failing to see where the dependency the other direction is created.

Hmm, it looks like I misunderstand lock_map_acquire()? From the changelog, I thought it was added to complain its caller is holding a lock when invoking it, thus cpu_add_remove_lock is not an exception.

Oh, I see, wq->lockdep_map is acquired again in run_workqueue(), so I was wrong. :) I think you and Oleg are right, the lockdep warning is not irrelevant.

Sorry for the noise, ignore this patch please.

Thanks.

author
#10 lore raw
Quoted 52 lines · Cong Wang
Quoted 51 lines · Cong Wang
Quoted 49 lines · Tejun Heo

Hello,

Quoted 40 lines · Cong Wang
Quoted 5 lines

Hmmm... can you please try to see whether this circular locking warning involving wq->lockdep_map is reproducible w/ the bonding locking fixed? I still can't see where wq -> cpu_add_remove_lock dependency is created.

I thought this is obvious.

Here it is:

void destroy_workqueue(struct workqueue_struct *wq)

{
        const struct cpumask *cpu_map = wq_cpu_map(wq);
        int cpu;

        cpu_maps_update_begin();        <----------------- Hold
cpu_add_remove_lock here
        spin_lock(&workqueue_lock);
        list_del(&wq->list);
        spin_unlock(&workqueue_lock);

        for_each_cpu(cpu, cpu_map)
                cleanup_workqueue_thread(per_cpu_ptr(wq->cpu_wq, 
cpu)); <------ See below
        cpu_maps_update_done();        <----------------- Release

cpu_add_remove_lock here

...
static void cleanup_workqueue_thread(struct cpu_workqueue_struct *cwq)

{
        /*
         * Our caller is either destroy_workqueue() or CPU_POST_DEAD,
         * cpu_add_remove_lock protects cwq->thread.
         */
        if (cwq->thread == NULL)
                return;

        lock_map_acquire(&cwq->wq->lockdep_map); <-------------- 
Lockdep
complains here.
        lock_map_release(&cwq->wq->lockdep_map);
Quoted 1 line

...

Yeap, the above is cpu_add_remove_lock -> wq->lockdep_map dependency. I can see that but I'm failing to see where the dependency the other direction is created.

Quoted 8 lines
Quoted 4 lines

Hmm, it looks like I misunderstand lock_map_acquire()? From the changelog, I thought it was added to complain its caller is holding a lock when invoking it, thus cpu_add_remove_lock is not an exception.

Oh, I see, wq->lockdep_map is acquired again in run_workqueue(), so I was wrong. :) I think you and Oleg are right, the lockdep warning is not irrelevant.

Oops, typo, I meant "is irrelevant." ;)

maintainer
#11 lore raw
In reply to Cong Wang

Hello,

Quoted 4 lines · Cong Wang
Quoted 4 lines

Hmm, it looks like I misunderstand lock_map_acquire()? From the changelog, I thought it was added to complain its caller is holding a lock when invoking it, thus cpu_add_remove_lock is not an exception.

Oh, that just tells the code is trying to grab a pseudo lock. It's not really a lock but to lockdep it looks like one and lockdep can use it to compute problem cases.

Quoted 3 lines

Oh, I see, wq->lockdep_map is acquired again in run_workqueue(), so I was wrong. :) I think you and Oleg are right, the lockdep warning is not irrelevant.

Yeah, I think the circular dependency you reported on wq->lockdep_map is completed only through dependency through rtnl_mutex. If you fix rtnl_mutex locking, it should go away too.

Thanks.

Signature · 1 line
tejun
#12 lore raw
In reply to Cong Wang
Quoted 7 lines · Cong Wang
Quoted 5 lines

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Sure, below is the whole warning. Please teach me how this is valid.

Oh, I can never understand the output from lockdep, it is much more clever than me ;)

But at first glance,

Quoted 10 lines
Mar 31 16:15:02 dhcp-66-70-5 kernel: -> #2 (rtnl_mutex){+.+.+.}:
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffffa0836779>] bond_mii_monitor+0x39f/0x74b [bonding]
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108654f>] worker_thread+0x2da/0x46c
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108b1ea>] kthread+0xdd/0xec
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff81004894>] kernel_thread_helper+0x4/0x10

OK, so work->func() takes rtnl_mutex.

This means it is not safe to do flush_workqueue() or destroy_workqueue() under rtnl_lock(). This is known fact.

Quoted 4 lines
Mar 31 16:15:03 dhcp-66-70-5 kernel: -> #0 ((bond_dev->name)){+.+...}:
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a6696>] validate_chain+0xaee/0x1540
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Quoted 8 lines
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43

However, rtnl_link_unregister() takes rtnl_mutex and then bond_uninit() does cleanup_workqueue_thread().

So, looks like this warning is valid, this path can deadlock if destroy_workqueue() is called when bond->mii_work is queued.

Lockdep decided to blaim cpu_add_remove_lock in this chain.

Oleg.

author
#13 lore raw
Quoted 41 lines · Oleg Nesterov
Quoted 5 lines · Cong Wang
Quoted 4 lines

I must have missed something, but it seems to me this patch tries to supress the valid warning.

Could you please clarify?

Sure, below is the whole warning. Please teach me how this is valid.

Oh, I can never understand the output from lockdep, it is much more clever than me ;)

But at first glance,

Quoted 10 lines
Mar 31 16:15:02 dhcp-66-70-5 kernel: -> #2 (rtnl_mutex){+.+.+.}:
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a6bc1>] validate_chain+0x1019/0x1540
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff815523f8>] mutex_lock_nested+0x64/0x4e9
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8147af16>] rtnl_lock+0x1e/0x27
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffffa0836779>] bond_mii_monitor+0x39f/0x74b [bonding]
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108654f>] worker_thread+0x2da/0x46c
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff8108b1ea>] kthread+0xdd/0xec
Mar 31 16:15:02 dhcp-66-70-5 kernel:        [<ffffffff81004894>] kernel_thread_helper+0x4/0x10

OK, so work->func() takes rtnl_mutex.

This means it is not safe to do flush_workqueue() or destroy_workqueue() under rtnl_lock(). This is known fact.

Quoted 4 lines
Mar 31 16:15:03 dhcp-66-70-5 kernel: -> #0 ((bond_dev->name)){+.+...}:
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a6696>] validate_chain+0xaee/0x1540
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810a7e75>] __lock_acquire+0xd8d/0xe55
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff810aa3a4>] lock_acquire+0x160/0x1af
Quoted 8 lines
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085278>] cleanup_workqueue_thread+0x59/0x10b
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81085428>] destroy_workqueue+0x9c/0x107
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffffa0839d32>] bond_uninit+0x524/0x58a [bonding]
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8146967b>] rollback_registered_many+0x205/0x2e3
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff81469783>] unregister_netdevice_many+0x2a/0x75
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147ada3>] __rtnl_kill_links+0x8b/0x9d
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147adea>] __rtnl_link_unregister+0x35/0x72
Mar 31 16:15:03 dhcp-66-70-5 kernel:        [<ffffffff8147b293>] rtnl_link_unregister+0x2c/0x43
Quoted 5 lines

However, rtnl_link_unregister() takes rtnl_mutex and then bond_uninit() does cleanup_workqueue_thread().

So, looks like this warning is valid, this path can deadlock if destroy_workqueue() is called when bond->mii_work is queued.

Yeah, this is right.

Quoted 4 lines

Lockdep decided to blaim cpu_add_remove_lock in this chain.

Yes, this is what makes me confused. ;)

Thanks!