Skip to contentSkip to conversation

2.6.25rc7 lockdep trace

DISCUSSION Dave Jones 5 participants 28 messages Mar 28, 2008 – Jun 12, 2008

Subsystems (4)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

I see this every time I shut down.

Dave

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.25-0.161.rc7.fc9.i686 #1
-------------------------------------------------------
NetworkManager/2308 is trying to acquire lock:
(events){--..}, at: [flush_workqueue+0/133] flush_workqueue+0x0/0x85

but task is already holding lock:
(rtnl_mutex){--..}, at: [rtnetlink_rcv+18/38] rtnetlink_rcv+0x12/0x26

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (rtnl_mutex){--..}:

        [__lock_acquire+2713/3089] __lock_acquire+0xa99/0xc11
        [lock_acquire+106/144] lock_acquire+0x6a/0x90
        [mutex_lock_nested+219/625] mutex_lock_nested+0xdb/0x271
        [rtnl_lock+15/17] rtnl_lock+0xf/0x11
        [linkwatch_event+8/34] linkwatch_event+0x8/0x22
        [run_workqueue+211/417] run_workqueue+0xd3/0x1a1
        [worker_thread+182/194] worker_thread+0xb6/0xc2
        [kthread+59/97] kthread+0x3b/0x61
        [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
        [<ffffffff>] 0xffffffff

-> #1 ((linkwatch_work).work){--..}:

        [__lock_acquire+2713/3089] __lock_acquire+0xa99/0xc11
        [lock_acquire+106/144] lock_acquire+0x6a/0x90
        [run_workqueue+205/417] run_workqueue+0xcd/0x1a1
        [worker_thread+182/194] worker_thread+0xb6/0xc2
        [kthread+59/97] kthread+0x3b/0x61
        [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
        [<ffffffff>] 0xffffffff

-> #0 (events){--..}:

        [__lock_acquire+2488/3089] __lock_acquire+0x9b8/0xc11
        [lock_acquire+106/144] lock_acquire+0x6a/0x90
        [flush_workqueue+68/133] flush_workqueue+0x44/0x85
        [flush_scheduled_work+13/15] flush_scheduled_work+0xd/0xf
        [<d096d80a>] tulip_down+0x20/0x1a3 [tulip]
        [<d096e2b5>] tulip_close+0x24/0xd6 [tulip]
        [dev_close+82/111] dev_close+0x52/0x6f
        [dev_change_flags+159/338] dev_change_flags+0x9f/0x152
        [do_setlink+586/764] do_setlink+0x24a/0x2fc
        [rtnl_setlink+226/230] rtnl_setlink+0xe2/0xe6
        [rtnetlink_rcv_msg+418/444] rtnetlink_rcv_msg+0x1a2/0x1bc
        [netlink_rcv_skb+48/134] netlink_rcv_skb+0x30/0x86
        [rtnetlink_rcv+30/38] rtnetlink_rcv+0x1e/0x26
        [netlink_unicast+439/533] netlink_unicast+0x1b7/0x215
        [netlink_sendmsg+600/613] netlink_sendmsg+0x258/0x265
        [sock_sendmsg+222/249] sock_sendmsg+0xde/0xf9
        [sys_sendmsg+319/402] sys_sendmsg+0x13f/0x192
        [sys_socketcall+363/390] sys_socketcall+0x16b/0x186
        [syscall_call+7/11] syscall_call+0x7/0xb
        [<ffffffff>] 0xffffffff

other info that might help us debug this:

1 lock held by NetworkManager/2308:
#0: (rtnl_mutex){--..}, at: [rtnetlink_rcv+18/38] rtnetlink_rcv+0x12/0x26

 stack backtrace:
 Pid: 2308, comm: NetworkManager Not tainted 2.6.25-0.161.rc7.fc9.i686 #1
  [print_circular_bug_tail+91/102] print_circular_bug_tail+0x5b/0x66
  [print_circular_bug_entry+57/67] ? print_circular_bug_entry+0x39/0x43
  [__lock_acquire+2488/3089] __lock_acquire+0x9b8/0xc11
  [_spin_unlock_irq+34/47] ? _spin_unlock_irq+0x22/0x2f
  [lock_acquire+106/144] lock_acquire+0x6a/0x90
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_workqueue+68/133] flush_workqueue+0x44/0x85
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_scheduled_work+13/15] flush_scheduled_work+0xd/0xf
  [<d096d80a>] tulip_down+0x20/0x1a3 [tulip]
  [trace_hardirqs_on+233/266] ? trace_hardirqs_on+0xe9/0x10a
  [dev_deactivate+177/222] ? dev_deactivate+0xb1/0xde
  [<d096e2b5>] tulip_close+0x24/0xd6 [tulip]
  [dev_close+82/111] dev_close+0x52/0x6f
  [dev_change_flags+159/338] dev_change_flags+0x9f/0x152
  [do_setlink+586/764] do_setlink+0x24a/0x2fc
  [_read_unlock+29/32] ? _read_unlock+0x1d/0x20
  [rtnl_setlink+226/230] rtnl_setlink+0xe2/0xe6
  [rtnl_setlink+0/230] ? rtnl_setlink+0x0/0xe6
  [rtnetlink_rcv_msg+418/444] rtnetlink_rcv_msg+0x1a2/0x1bc
  [rtnetlink_rcv_msg+0/444] ? rtnetlink_rcv_msg+0x0/0x1bc
  [netlink_rcv_skb+48/134] netlink_rcv_skb+0x30/0x86
  [rtnetlink_rcv+30/38] rtnetlink_rcv+0x1e/0x26
  [netlink_unicast+439/533] netlink_unicast+0x1b7/0x215
  [netlink_sendmsg+600/613] netlink_sendmsg+0x258/0x265
  [sock_sendmsg+222/249] sock_sendmsg+0xde/0xf9
  [autoremove_wake_function+0/51] ? autoremove_wake_function+0x0/0x33
  [native_sched_clock+181/209] ? native_sched_clock+0xb5/0xd1
  [sched_clock+8/11] ? sched_clock+0x8/0xb
  [lock_release_holdtime+26/277] ? lock_release_holdtime+0x1a/0x115
  [fget_light+142/185] ? fget_light+0x8e/0xb9
  [copy_from_user+57/289] ? copy_from_user+0x39/0x121
  [verify_iovec+64/111] ? verify_iovec+0x40/0x6f
  [sys_sendmsg+319/402] sys_sendmsg+0x13f/0x192
  [sys_recvmsg+366/379] ? sys_recvmsg+0x16e/0x17b
  [check_object+304/388] ? check_object+0x130/0x184
  [check_object+304/388] ? check_object+0x130/0x184
  [kmem_cache_free+186/207] ? kmem_cache_free+0xba/0xcf
  [trace_hardirqs_on+233/266] ? trace_hardirqs_on+0xe9/0x10a
  [d_free+59/77] ? d_free+0x3b/0x4d
  [d_free+59/77] ? d_free+0x3b/0x4d
  [sys_socketcall+363/390] sys_socketcall+0x16b/0x186
  [syscall_call+7/11] syscall_call+0x7/0xb
  =======================
Signature · 1 line
http://www.codemonkey.org.uk
author
#1 lore raw
Quoted 1 line · Dave Jones

I see this every time I shut down.

Something that may be relevant.. There are two NICs in this machine. An onboard e100, which is in use, and there's a PCI tulip card in there which doesn't even have a cable plugged into it.

The not-in-use card seems to be the trigger.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
maintainer
#2 lore raw
In reply to Dave Jones

From: Dave Jones <[email protected]>
Date: Thu, 27 Mar 2008 20:00:13 -0400

Quoted 6 lines
 stack backtrace:
 Pid: 2308, comm: NetworkManager Not tainted 2.6.25-0.161.rc7.fc9.i686 #1
  [print_circular_bug_tail+91/102] print_circular_bug_tail+0x5b/0x66
  [print_circular_bug_entry+57/67] ? print_circular_bug_entry+0x39/0x43
  [__lock_acquire+2488/3089] __lock_acquire+0x9b8/0xc11
  [_spin_unlock_irq+34/47] ? _spin_unlock_irq+0x22/0x2f
Quoted 8 lines
  [lock_acquire+106/144] lock_acquire+0x6a/0x90
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_workqueue+68/133] flush_workqueue+0x44/0x85
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_scheduled_work+13/15] flush_scheduled_work+0xd/0xf
  [<d096d80a>] tulip_down+0x20/0x1a3 [tulip]
  [trace_hardirqs_on+233/266] ? trace_hardirqs_on+0xe9/0x10a
  [dev_deactivate+177/222] ? dev_deactivate+0xb1/0xde

Yes, see for example:

http://www.mail-archive.com/[email protected]/msg31718.html

You can't flush a workqueue in the device close handler exactly because of this locking conflict.

Nobody has come up with a suitable way to fix this yet.

maintainer
#3 lore raw
Quoted 16 lines
Quoted 15 lines
 stack backtrace:
 Pid: 2308, comm: NetworkManager Not tainted 2.6.25-0.161.rc7.fc9.i686 #1
  [print_circular_bug_tail+91/102] print_circular_bug_tail+0x5b/0x66
  [print_circular_bug_entry+57/67] ? print_circular_bug_entry+0x39/0x43
  [__lock_acquire+2488/3089] __lock_acquire+0x9b8/0xc11
  [_spin_unlock_irq+34/47] ? _spin_unlock_irq+0x22/0x2f
  [lock_acquire+106/144] lock_acquire+0x6a/0x90
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_workqueue+68/133] flush_workqueue+0x44/0x85
  [flush_workqueue+0/133] ? flush_workqueue+0x0/0x85
  [flush_scheduled_work+13/15] flush_scheduled_work+0xd/0xf
  [<d096d80a>] tulip_down+0x20/0x1a3 [tulip]
  [trace_hardirqs_on+233/266] ? trace_hardirqs_on+0xe9/0x10a
  [dev_deactivate+177/222] ? dev_deactivate+0xb1/0xde
Quoted 8 lines

Yes, see for example:

http://www.mail-archive.com/[email protected]/msg31718.html

You can't flush a workqueue in the device close handler exactly because of this locking conflict.

Nobody has come up with a suitable way to fix this yet.

Maybe we should check which schedule_work users actually lock the rtnl within the work function and move them to a uses-rtnl-in-work workqueue so that everybody else can have rtnl around flush.

Depending on how many users there are that might not be feasible, but I have so far only seen linkwatch_event() lock the rtnl within the work function and everybody else seems to want to use the rtnl around the flushing.

johannes

maintainer
#4 lore raw
Quoted 9 lines
Quoted 5 lines

You can't flush a workqueue in the device close handler exactly because of this locking conflict.

Nobody has come up with a suitable way to fix this yet.

Maybe we should check which schedule_work users actually lock the rtnl within the work function and move them to a uses-rtnl-in-work workqueue so that everybody else can have rtnl around flush.

On the other hand, most drivers don't actually care that their work has run, they just care that it won't run in the future after they give up resources or similar, hence they can and should use cancel_work_sync() which doesn't suffer from the deadlock. But that needs actual inspection because it does change behaviour from "run and wait for it if scheduled" to "cancel if scheduled".

johannes

maintainer
#7 lore raw
In reply to Johannes Berg

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 01:54:09 +0100

Quoted 3 lines

Maybe we should check which schedule_work users actually lock the rtnl within the work function and move them to a uses-rtnl-in-work workqueue so that everybody else can have rtnl around flush.

The issue is that linkwatch goes: workqueue lock --> RTNL

And thus taking workqueue lock within RTNL is deadlock prone no matter where you do it.

maintainer
#5 lore raw
In reply to Johannes Berg

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 02:01:25 +0100

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

You can't flush a workqueue in the device close handler exactly because of this locking conflict.

Nobody has come up with a suitable way to fix this yet.

Maybe we should check which schedule_work users actually lock the rtnl within the work function and move them to a uses-rtnl-in-work workqueue so that everybody else can have rtnl around flush.

Quoted 6 lines

On the other hand, most drivers don't actually care that their work has run, they just care that it won't run in the future after they give up resources or similar, hence they can and should use cancel_work_sync() which doesn't suffer from the deadlock. But that needs actual inspection because it does change behaviour from "run and wait for it if scheduled" to "cancel if scheduled".

I don't see how you can not race with the transition from scheduled to "executing" without taking the runqueue lock for the testing.

And it is crucial that the workqueue function doesn't execute "accidently" due to such a race before the module and thus the workqueue code is about to get potentially unloaded.

maintainer
#8 lore raw
In reply to David Miller

[merging your two mails]

Quoted 4 lines

The issue is that linkwatch goes: workqueue lock --> RTNL

And thus taking workqueue lock within RTNL is deadlock prone no matter where you do it.

Which lock are you referring to here? There is no real runqueue lock other than a spin-lock for the list. I think you may be misunderstanding the lockdep output in a way I didn't anticipated when adding lockdep debugging for workqueue stuff.

The actual work function is _not_ running with any locks held! The fact that you see lockdep output there is because we tell lockdep we're holding two locks, namely the "work" and the "workqueue". These are fake, they aren't actually locks.

More importantly, those "locks" are locked and unlocked for every _run_ of the struct work function, not around the whole runqueue manipulation, that part is done atomically without global locks.

So what you have is essentially
list_for_each_work

	lock(workqueue)
	lock(struct work)
	work->f();
	unlock(struct work)
	unlock(workqueue)

where both those locks are really only telling lockdep there are locks to get it to warn about the deadlock scenario. Unfortunately Andrew mangled the two patches into one when committing, so I can't point to the changelog, but here are two links to the two patches adding this:
http://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.23-rc1/2.6.23-rc1-mm2/broken-out/workqueue-debug-flushing-deadlocks-with-lockdep.patch
http://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.23-rc1/2.6.23-rc1-mm2/broken-out/workqueue-debug-work-related-deadlocks-with-lockdep.patch

I think the "runqueue lock" you're referring to is the fake "lock(workqueue)" which isn't around the whole runqueue manipulation but only a lockdep helper to debug exactly the deadlock problem we're talking about. I'm open to renaming it, but I don't think lockdep supports formatting its output differently.

Quoted 3 lines

I don't see how you can not race with the transition from scheduled to "executing" without taking the runqueue lock for the testing.

When you call cancel_work_sync(), the work struct will be grabbed by the code (really __cancel_work_timer) and removed from the queue. That just operates on bits and a spinlock, not locks held across the struct work function execution, and ensures it is race-free without needing any such locks.

Quoted 4 lines

And it is crucial that the workqueue function doesn't execute "accidently" due to such a race before the module and thus the workqueue code is about to get potentially unloaded.

That is exactly what cancel_work_sync() guarantees, but not more. On contrary, flush_workqueue() also guarantees that the work is run one last time if it was scheduled at the time of the flush_workqueue() call.

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

johannes

#9 lore raw

Johannes Berg wrote, On 03/29/2008 11:02 AM:

...

Quoted 5 lines

When you call cancel_work_sync(), the work struct will be grabbed by the code (really __cancel_work_timer) and removed from the queue. That just operates on bits and a spinlock, not locks held across the struct work function execution, and ensures it is race-free without needing any such locks

...

Quoted 2 lines

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

These issues are so seldom now that I forget these details each time

"after use", so maybe I miss something again, but shouldn't this rather read something like this?:

cancel_work_sync() _is_ safe to run while holding the RTNL against works which don't take RTNL.

Regards,
Jarek P.

maintainer
#10 lore raw
Quoted 12 lines
Quoted 6 lines

When you call cancel_work_sync(), the work struct will be grabbed by the code (really __cancel_work_timer) and removed from the queue. That just operates on bits and a spinlock, not locks held across the struct work function execution, and ensures it is race-free without needing any such locks

...

Quoted 2 lines

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

Quoted 7 lines

These issues are so seldom now that I forget these details each time

"after use", so maybe I miss something again, but shouldn't this rather read something like this?:

cancel_work_sync() _is_ safe to run while holding the RTNL against works which don't take RTNL.

Yes, indeed, I should have said that.

johannes

#6 lore raw
In reply to David Miller
Quoted 21 lines · David Miller

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 02:01:25 +0100

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

You can't flush a workqueue in the device close handler exactly because of this locking conflict.

Nobody has come up with a suitable way to fix this yet.

Maybe we should check which schedule_work users actually lock the rtnl within the work function and move them to a uses-rtnl-in-work workqueue so that everybody else can have rtnl around flush.

Quoted 6 lines

On the other hand, most drivers don't actually care that their work has run, they just care that it won't run in the future after they give up resources or similar, hence they can and should use cancel_work_sync() which doesn't suffer from the deadlock. But that needs actual inspection because it does change behaviour from "run and wait for it if scheduled" to "cancel if scheduled".

Quoted 3 lines

I don't see how you can not race with the transition from scheduled to "executing" without taking the runqueue lock for the testing.

Yes, cancel_work_sync() takes cwq->lock but this is fine (unless it is buggy ;) Please note that run_workqueue() drops this lock before calling work->func().

If the caller of cancel_work_sync(work) doesn't share locks with work->func() we can't deadlock, even if there are other pending/running work_structs which need the same locks as the caller (say, RTNL).

But, perhaps, you mean wq->lockdep_map? As Johannes pointed out this lock is fake, but I think this doesn't matter, from the correctness POV it is "real" lock. What does matter is that cancel_work_sync() doesn't use this lock at all.

(again, Johannes has already explained this all).

Quoted 4 lines

And it is crucial that the workqueue function doesn't execute "accidently" due to such a race before the module and thus the workqueue code is about to get potentially unloaded.

Which race? Unless explicitly queued afterwards, work->func() can't execute after return from cancel_work_sync(work).

David, I think you misunderstood Johannes, or perhaps I missed something.

Oleg.

maintainer
#11 lore raw
In reply to Johannes Berg

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 11:02:28 +0100

Quoted 2 lines

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

So in theory we should be able to safely transform flush_scheduled_work() calls in network driver close methods into cancel_work_sync()?

Could someone prepare that patch?

Thanks.

maintainer
#12 lore raw
Quoted 9 lines · David Miller

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 11:02:28 +0100

Quoted 2 lines

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

So in theory we should be able to safely transform flush_scheduled_work() calls in network driver close methods into cancel_work_sync()?

Yes, more precisely cancel_work_sync() for each work struct the driver uses, unless the driver actually requires that the work runs before it goes down.

johannes

maintainer
#13 lore raw

From: Johannes Berg <[email protected]>
Date: Fri, 04 Apr 2008 16:48:12 +0200

I'm attempting to bring solving this problem back from the dead.

Please help!

Quoted 7 lines · David Miller
Quoted 6 lines

From: Johannes Berg <[email protected]>
Date: Sat, 29 Mar 2008 11:02:28 +0100

Quoted 2 lines

However, as I just tried to explain, cancel_work_sync() _is_ safe to run while holding the RTNL because it doesn't need any runqueue lock.

Quoted 7 lines
Quoted 3 lines

So in theory we should be able to safely transform flush_scheduled_work() calls in network driver close methods into cancel_work_sync()?

Yes, more precisely cancel_work_sync() for each work struct the driver uses, unless the driver actually requires that the work runs before it goes down.

Ok, I did an audit of all the drivers under drivers/net that invoke flush_scheduled_work(). Here is my audit analysis and the patch I came up with. The only deadlock'y case I couldn't fix right now is the cassini driver, see below for details and the patch:

8139too: Calls flush_scheduled_work() in device remove method, OK.
libertas: Calls flush_scheduled_work() in device probe and remove methods, OK.
bnx2: Uses special polling mdelay() sequence instead of calling
flush_scheduled_work() in close method, converted to cancel_work_sync().
cassini: Does flush_scheduled_work() from ->change_mtu() method, which
also holds RTNL semaphore. Seems tricky to fix as it's trying to schedule work and then immediately wait for it to complete synchronously before returning from the ->change_mtu method. Maybe calling cas_reset_task() directly would work, but this could conflict with reset tasks scheduled in other contexts. NOT FIXED
e1000e: Does flush_scheduled_work() from device remove method, OK.
ehea_main: Does flush_scheduled_work() in ->stop method, converted to

	   cancel_work_sync().

baycom_epp: Does flush_scheduled_work() from ->stop method, converted
to cancel_delayed_work()/cancel_work_sync() sequence.
ibm_newemac: Calls flush_scheduled_work() from device remove method, OK.
igb: Calls flush_scheduled_work() from device remove method, OK. irda/mcs7780: Calls flush_scheduled_work() from USB disconnect, OK.
iseries_veth: Calls flush_scheduled_work() from device disconnect and module
exit, OK.
ixgbe: Calls flush_scheduled_work() from device remove method, OK.
mv643xx_eth: Calls flush_scheduled_work() from device remove method, OK.
myri10ge: Calls flush_scheduled_work() from device remove method, OK.
netxen: Has it's own hand-grown scheme to work around the flush_scheduled_work()
deadlock. Using a workqueue and flush_workqueue().
niu: Calls flush_scheduled_work() only from suspend handler, OK. phy/phy.c: Avoids flush_scheduled_work() calls to avoid deadlock, uses
cancel_work_sync() instead.
r8169: Calls flush_scheduled_work() from device remove method, OK.
s2io: Calls flush_scheduled_work() from device remove method, OK.
sis190: Calls flush_scheduled_work() from device remove method, OK. skge.c: Calls flush_scheduled_work() from device remove method, OK.
smc911x: Uses a similar polling loop like bnx2 did, converted to

	   cancel_work_sync().

smc91x: Same as smc911x
sungem: Calls flush_scheduled_work() only from suspend and device
remove method, OK.
tg3: Calls flush_scheduled_work() only from suspend and device remove
method, OK.
tulip_core: Calls flush_scheduled_work() from tulip_down(), converted to

	    cancel_work_sync().
kaweth: Calls flush_scheduled_work() from kaweth_kill_urbs() which can
	run from the ->stop() handler, converted to
	cancel_delayed_work/cancel_work_sync sequence.

usbnet: Only calls flush_scheduled_work() from disconnect handler, OK.
hostap: Calls flush_scheduled_work() from ->stop, converted to
cancel_work_sync()

drivers/net/bnx2.c +1 -6
index 4b46e68..48ed410 100644
@@ -5907,12 +5907,7 @@ bnx2_close(struct net_device *dev)
 	struct bnx2 *bp = netdev_priv(dev);
 	u32 reset_code;
 
-	/* Calling flush_scheduled_work() may deadlock because
-	 * linkwatch_event() may be on the workqueue and it will try to get
-	 * the rtnl_lock which we are holding.
-	 */
-	while (bp->in_reset_task)
-		msleep(1);
+	cancel_work_sync(&bp->reset_task);
 
 	bnx2_disable_int_sync(bp);
 	bnx2_napi_disable(bp);
drivers/net/ehea/ehea_main.c +2 -1
index faae01d..075fd54 100644
@@ -2605,7 +2605,8 @@ static int ehea_stop(struct net_device *dev)
 	if (netif_msg_ifdown(port))
 		ehea_info("disabling port %s", dev->name);
 
-	flush_scheduled_work();
+	cancel_work_sync(&port->reset_task);
+
 	mutex_lock(&port->port_lock);
 	netif_stop_queue(dev);
 	port_napi_disable(port);
drivers/net/hamradio/baycom_epp.c +2 -1
index dde9c7e..b36ae41 100644
@@ -959,7 +959,8 @@ static int epp_close(struct net_device *dev)
 	unsigned char tmp[1];
 
 	bc->work_running = 0;
-	flush_scheduled_work();
+	cancel_delayed_work(&bc->run_work);
+	cancel_work_sync(&bc->run_work.work);
 	bc->stat = EPP_DCDBIT;
 	tmp[0] = 0;
 	pp->ops->epp_write_addr(pp, tmp, 1, 0);
drivers/net/smc911x.c +1 -9
index 4e28002..5c71098 100644
@@ -1531,16 +1531,8 @@ static int smc911x_close(struct net_device *dev)
 	if (lp->phy_type != 0) {
 		/* We need to ensure that no calls to
 		 * smc911x_phy_configure are pending.
-
-		 * flush_scheduled_work() cannot be called because we
-		 * are running with the netlink semaphore held (from
-		 * devinet_ioctl()) and the pending work queue
-		 * contains linkwatch_event() (scheduled by
-		 * netif_carrier_off() above). linkwatch_event() also
-		 * wants the netlink semaphore.
 		 */
-		while (lp->work_pending)
-			schedule();
+		cancel_work_sync(&lp->phy_configure);
 		smc911x_phy_powerdown(dev, lp->mii.phy_id);
 	}
 
drivers/net/smc91x.c +1 -8
index a188e33..5dbe4d3 100644
@@ -1016,15 +1016,8 @@ static void smc_phy_powerdown(struct net_device *dev)
 
 	/* We need to ensure that no calls to smc_phy_configure are
 	   pending.
-
-	   flush_scheduled_work() cannot be called because we are
-	   running with the netlink semaphore held (from
-	   devinet_ioctl()) and the pending work queue contains
-	   linkwatch_event() (scheduled by netif_carrier_off()
-	   above). linkwatch_event() also wants the netlink semaphore.
 	*/
-	while(lp->work_pending)
-		yield();
+	cancel_work_sync(&lp->phy_configure);
 
 	bmcr = smc_phy_read(dev, phy, MII_BMCR);
 	smc_phy_write(dev, phy, MII_BMCR, bmcr | BMCR_PDOWN);
drivers/net/tulip/tulip_core.c +1 -1
index 55670b5..af8d2c4 100644
@@ -731,7 +731,7 @@ static void tulip_down (struct net_device *dev)
 	void __iomem *ioaddr = tp->base_addr;
 	unsigned long flags;
 
-	flush_scheduled_work();
+	cancel_work_sync(&tp->media_work);
 
 #ifdef CONFIG_TULIP_NAPI
 	napi_disable(&tp->napi);
drivers/net/usb/kaweth.c +2 -1
index 0dcfc03..d776bcf 100644
@@ -706,7 +706,8 @@ static void kaweth_kill_urbs(struct kaweth_device *kaweth)
 	usb_kill_urb(kaweth->rx_urb);
 	usb_kill_urb(kaweth->tx_urb);
 
-	flush_scheduled_work();
+	cancel_delayed_work(&kaweth->lowmem_work);
+	cancel_work_sync(&kaweth->lowmem_work.work);
 
 	/* a scheduled work may have resubmitted,
 	   we hit them again */
drivers/net/wireless/hostap/hostap_main.c +5 -1
index 20d387f..57b800f 100644
@@ -682,7 +682,11 @@ static int prism2_close(struct net_device *dev)
 		netif_device_detach(dev);
 	}
 
-	flush_scheduled_work();
+	cancel_work_sync(&local->reset_queue);
+	cancel_work_sync(&local->set_multicast_list_queue);
+	cancel_work_sync(&local->set_tim_queue);
+	cancel_work_sync(&local->info_queue);
+	cancel_work_sync(&local->comms_qual_update);
 
 	module_put(local->hw_module);
 
#14 lore raw

On 11-06-2008 07:40, David Miller wrote:
...

Quoted 6 lines

Ok, I did an audit of all the drivers under drivers/net that invoke flush_scheduled_work(). Here is my audit analysis and the patch I came up with. The only deadlock'y case I couldn't fix right now is the cassini driver, see below for details and the patch:

8139too: Calls flush_scheduled_work() in device remove method, OK.

I have some doubt here: this rtl8139_thred() work function seems to rearm if netif_running(). Is it guaranteed a dev is down while running this flush_scheduled_work()?

Regards,
Jarek P.

maintainer
#15 lore raw

From: Jarek Poplawski <[email protected]>
Date: Wed, 11 Jun 2008 07:08:47 +0000

Quoted 7 lines · David Miller

...

Quoted 5 lines

Ok, I did an audit of all the drivers under drivers/net that invoke flush_scheduled_work(). Here is my audit analysis and the patch I came up with. The only deadlock'y case I couldn't fix right now is the cassini driver, see below for details and the patch:

Quoted 5 lines
Quoted 1 line

8139too: Calls flush_scheduled_work() in device remove method, OK.

I have some doubt here: this rtl8139_thred() work function seems to rearm if netif_running(). Is it guaranteed a dev is down while running this flush_scheduled_work()?

Indeed, perhaps it should do the flush_scheduled_work() after the unregister_netdev().

#16 lore raw
In reply to David Miller

On 11-06-2008 07:40, David Miller wrote:
...

Quoted 2 lines

ehea_main: Does flush_scheduled_work() in ->stop method, converted to

	   cancel_work_sync().

Looks like this work could be rearmed afterwards from ehea_qp_aff_irq_handler() (and maybe a few other places).

Jarek P.

#19 lore raw
In reply to David Miller

On 11-06-2008 07:40, David Miller wrote:
...

drivers/net/usb/kaweth.c static void kaweth_kill_urbs(struct kaweth_device *kaweth) · line 711

Quoted 12 lines
drivers/net/usb/kaweth.c +2 -1
index 0dcfc03..d776bcf 100644
@@ -706,7 +706,8 @@ static void kaweth_kill_urbs(struct kaweth_device *kaweth)
 	usb_kill_urb(kaweth->rx_urb);
 	usb_kill_urb(kaweth->tx_urb);
 
-	flush_scheduled_work();
+	cancel_delayed_work(&kaweth->lowmem_work);
+	cancel_work_sync(&kaweth->lowmem_work.work);
 

Why not cancel_delayed_work_sync()?

Jarek P.

#21 lore raw
In reply to David Miller

On 11-06-2008 07:40, David Miller wrote:
...

Quoted 7 lines

cassini: Does flush_scheduled_work() from ->change_mtu() method, which
also holds RTNL semaphore. Seems tricky to fix as it's trying to schedule work and then immediately wait for it to complete synchronously before returning from the ->change_mtu method. Maybe calling cas_reset_task() directly would work, but this could conflict with reset tasks scheduled in other contexts. NOT FIXED

Could you give an example of such a conflict? (I wonder how it's avoided with current solution.)

Jarek P.

maintainer
#20 lore raw
In reply to Jarek Poplawski

From: Jarek Poplawski <[email protected]>
Date: Wed, 11 Jun 2008 10:40:27 +0000

Quoted 10 lines · David Miller

...

Quoted 8 lines
drivers/net/usb/kaweth.c +0 -0
index 0dcfc03..d776bcf 100644
@@ -706,7 +706,8 @@ static void kaweth_kill_urbs(struct kaweth_device *kaweth)
 	usb_kill_urb(kaweth->rx_urb);
 	usb_kill_urb(kaweth->tx_urb);
 
Quoted 6 lines

drivers/net/usb/kaweth.c static void kaweth_kill_urbs(struct kaweth_device *kaweth) · line 711

Quoted 4 lines
drivers/net/usb/kaweth.c +2 -1
-	flush_scheduled_work();
+	cancel_delayed_work(&kaweth->lowmem_work);
+	cancel_work_sync(&kaweth->lowmem_work.work);
 

Why not cancel_delayed_work_sync()?

I simply didn't know it existed :-) I've updated my patch to use cancel_delayed_work_sync() as appropriate, thanks!

maintainer
#17 lore raw
In reply to Jarek Poplawski

From: Jarek Poplawski <[email protected]>
Date: Wed, 11 Jun 2008 09:36:45 +0000

Quoted 7 lines · David Miller

...

Quoted 2 lines

ehea_main: Does flush_scheduled_work() in ->stop method, converted to

	   cancel_work_sync().

Looks like this work could be rearmed afterwards from ehea_qp_aff_irq_handler() (and maybe a few other places).

Good catch, but this bug exists both before and after my changes :-)

It is certainly something which should be reported to the EHEA developers. Could you please do this?

maintainer
#22 lore raw
In reply to David Miller

From: David Miller <[email protected]>
Date: Tue, 10 Jun 2008 22:40:23 -0700 (PDT)

Quoted 1 line

I'm attempting to bring solving this problem back from the dead.

Here is my current patch with a proper changelog.

Any objections to my pushing this to Linus?

net: Eliminate flush_scheduled_work() calls while RTNL is held.

If the RTNL is held when we invoke flush_scheduled_work() we could deadlock. One such case is linkwatch, it is a workqueue which tries to grab the RTNL semaphore.

The most common case are net driver ->stop() methods. The simplest conversion is to instead use cancel_{delayed_}work_sync() explicitly on the various workqueues the driver uses.

This is an OK transformation because these workqueues are doing things like resetting the chip, restarting link negotiation, and so forth. And if we're bringing down the device, we're about to turn the chip off and reset it anways. So if we cancel a pending work event, that's fine here.

Some drivers were working around this deadlock by using a msleep() polling loop of some sort, and those cases are converted to instead use cancel_{delayed_}work_sync() as well.

drivers/net/bnx2.c +1 -6
index 4b46e68..48ed410 100644
@@ -5907,12 +5907,7 @@ bnx2_close(struct net_device *dev)
 	struct bnx2 *bp = netdev_priv(dev);
 	u32 reset_code;
 
-	/* Calling flush_scheduled_work() may deadlock because
-	 * linkwatch_event() may be on the workqueue and it will try to get
-	 * the rtnl_lock which we are holding.
-	 */
-	while (bp->in_reset_task)
-		msleep(1);
+	cancel_work_sync(&bp->reset_task);
 
 	bnx2_disable_int_sync(bp);
 	bnx2_napi_disable(bp);
drivers/net/ehea/ehea_main.c +2 -1
index faae01d..075fd54 100644
@@ -2605,7 +2605,8 @@ static int ehea_stop(struct net_device *dev)
 	if (netif_msg_ifdown(port))
 		ehea_info("disabling port %s", dev->name);
 
-	flush_scheduled_work();
+	cancel_work_sync(&port->reset_task);
+
 	mutex_lock(&port->port_lock);
 	netif_stop_queue(dev);
 	port_napi_disable(port);
drivers/net/hamradio/baycom_epp.c +1 -1
index dde9c7e..00bc7fb 100644
@@ -959,7 +959,7 @@ static int epp_close(struct net_device *dev)
 	unsigned char tmp[1];
 
 	bc->work_running = 0;
-	flush_scheduled_work();
+	cancel_delayed_work_sync(&bc->run_work);
 	bc->stat = EPP_DCDBIT;
 	tmp[0] = 0;
 	pp->ops->epp_write_addr(pp, tmp, 1, 0);
drivers/net/smc911x.c +1 -9
index 4e28002..5c71098 100644
@@ -1531,16 +1531,8 @@ static int smc911x_close(struct net_device *dev)
 	if (lp->phy_type != 0) {
 		/* We need to ensure that no calls to
 		 * smc911x_phy_configure are pending.
-
-		 * flush_scheduled_work() cannot be called because we
-		 * are running with the netlink semaphore held (from
-		 * devinet_ioctl()) and the pending work queue
-		 * contains linkwatch_event() (scheduled by
-		 * netif_carrier_off() above). linkwatch_event() also
-		 * wants the netlink semaphore.
 		 */
-		while (lp->work_pending)
-			schedule();
+		cancel_work_sync(&lp->phy_configure);
 		smc911x_phy_powerdown(dev, lp->mii.phy_id);
 	}
 
drivers/net/smc91x.c +1 -8
index a188e33..5dbe4d3 100644
@@ -1016,15 +1016,8 @@ static void smc_phy_powerdown(struct net_device *dev)
 
 	/* We need to ensure that no calls to smc_phy_configure are
 	   pending.
-
-	   flush_scheduled_work() cannot be called because we are
-	   running with the netlink semaphore held (from
-	   devinet_ioctl()) and the pending work queue contains
-	   linkwatch_event() (scheduled by netif_carrier_off()
-	   above). linkwatch_event() also wants the netlink semaphore.
 	*/
-	while(lp->work_pending)
-		yield();
+	cancel_work_sync(&lp->phy_configure);
 
 	bmcr = smc_phy_read(dev, phy, MII_BMCR);
 	smc_phy_write(dev, phy, MII_BMCR, bmcr | BMCR_PDOWN);
drivers/net/tulip/tulip_core.c +1 -1
index 55670b5..af8d2c4 100644
@@ -731,7 +731,7 @@ static void tulip_down (struct net_device *dev)
 	void __iomem *ioaddr = tp->base_addr;
 	unsigned long flags;
 
-	flush_scheduled_work();
+	cancel_work_sync(&tp->media_work);
 
 #ifdef CONFIG_TULIP_NAPI
 	napi_disable(&tp->napi);
drivers/net/usb/kaweth.c +1 -1
index 0dcfc03..7c66b05 100644
@@ -706,7 +706,7 @@ static void kaweth_kill_urbs(struct kaweth_device *kaweth)
 	usb_kill_urb(kaweth->rx_urb);
 	usb_kill_urb(kaweth->tx_urb);
 
-	flush_scheduled_work();
+	cancel_delayed_work_sync(&kaweth->lowmem_work);
 
 	/* a scheduled work may have resubmitted,
 	   we hit them again */
drivers/net/wireless/hostap/hostap_main.c +5 -1
index 20d387f..57b800f 100644
@@ -682,7 +682,11 @@ static int prism2_close(struct net_device *dev)
 		netif_device_detach(dev);
 	}
 
-	flush_scheduled_work();
+	cancel_work_sync(&local->reset_queue);
+	cancel_work_sync(&local->set_multicast_list_queue);
+	cancel_work_sync(&local->set_tim_queue);
+	cancel_work_sync(&local->info_queue);
+	cancel_work_sync(&local->comms_qual_update);
 
 	module_put(local->hw_module);
 
#25 lore raw
In reply to David Miller

On 11-06-2008 07:40, David Miller wrote:
...

Quoted 5 lines
drivers/net/bnx2.c +0 -0
index 4b46e68..48ed410 100644

drivers/net/bnx2.c bnx2_close(struct net_device *dev) · line 5910

Quoted 10 lines
drivers/net/bnx2.c +1 -6
@@ -5907,12 +5907,7 @@ bnx2_close(struct net_device *dev)
 	struct bnx2 *bp = netdev_priv(dev);
 	u32 reset_code;
 
-	/* Calling flush_scheduled_work() may deadlock because
-	 * linkwatch_event() may be on the workqueue and it will try to get
-	 * the rtnl_lock which we are holding.
-	 */
-	while (bp->in_reset_task)
-		msleep(1);
+	cancel_work_sync(&bp->reset_task);

It seems after this change bp->in_reset_task could be removed totally from the code.

...

Quoted 9 lines
drivers/net/smc911x.c +0 -1
index 4e28002..5c71098 100644
@@ -1531,16 +1531,8 @@ static int smc911x_close(struct net_device *dev)
 	if (lp->phy_type != 0) {
 		/* We need to ensure that no calls to
 		 * smc911x_phy_configure are pending.
-

drivers/net/smc911x.c static int smc911x_close(struct net_device *dev) · line 1535

Quoted 10 lines
drivers/net/smc911x.c +1 -8
-		 * flush_scheduled_work() cannot be called because we
-		 * are running with the netlink semaphore held (from
-		 * devinet_ioctl()) and the pending work queue
-		 * contains linkwatch_event() (scheduled by
-		 * netif_carrier_off() above). linkwatch_event() also
-		 * wants the netlink semaphore.
 		 */
-		while (lp->work_pending)
-			schedule();
+		cancel_work_sync(&lp->phy_configure);

... likewise lp->work_pending,

Quoted 11 lines
drivers/net/smc911x.c +0 -0
 		smc911x_phy_powerdown(dev, lp->mii.phy_id);
 	}
 
drivers/net/smc91x.c +0 -0
index a188e33..5dbe4d3 100644
@@ -1016,15 +1016,8 @@ static void smc_phy_powerdown(struct net_device *dev)
 
 	/* We need to ensure that no calls to smc_phy_configure are
 	   pending.

drivers/net/smc91x.c static void smc_phy_powerdown(struct net_device *dev) · line 1020

Quoted 10 lines
drivers/net/smc91x.c +1 -8
-
-	   flush_scheduled_work() cannot be called because we are
-	   running with the netlink semaphore held (from
-	   devinet_ioctl()) and the pending work queue contains
-	   linkwatch_event() (scheduled by netif_carrier_off()
-	   above). linkwatch_event() also wants the netlink semaphore.
 	*/
-	while(lp->work_pending)
-		yield();
+	cancel_work_sync(&lp->phy_configure);

... likewise lp->work_pending.

...

drivers/net/wireless/hostap/hostap_main.c static int prism2_close(struct net_device *dev) · line 688

Quoted 13 lines
drivers/net/wireless/hostap/hostap_main.c +4 -1
index 20d387f..57b800f 100644
@@ -682,7 +682,11 @@ static int prism2_close(struct net_device *dev)
 		netif_device_detach(dev);
 	}
 
-	flush_scheduled_work();
+	cancel_work_sync(&local->reset_queue);
+	cancel_work_sync(&local->set_multicast_list_queue);
+	cancel_work_sync(&local->set_tim_queue);
+	cancel_work_sync(&local->info_queue);

local->info_queue probably needs some #ifndef like in hostap_info_init() or could be skipped here at all (waiting for remove).

drivers/net/wireless/hostap/hostap_main.c static int prism2_close(struct net_device *dev) · line 691

Quoted 3 lines
drivers/net/wireless/hostap/hostap_main.c +1 -0
+	cancel_work_sync(&local->comms_qual_update);
 
 	module_put(local->hw_module);

Regards,
Jarek P.

#18 lore raw
In reply to David Miller
Quoted 8 lines · David Miller

From: Jarek Poplawski <[email protected]>
Date: Wed, 11 Jun 2008 09:36:45 +0000

Quoted 5 lines · David Miller

...

Quoted 2 lines

ehea_main: Does flush_scheduled_work() in ->stop method, converted to

	   cancel_work_sync().
Quoted 8 lines
Quoted 2 lines

Looks like this work could be rearmed afterwards from ehea_qp_aff_irq_handler() (and maybe a few other places).

Good catch, but this bug exists both before and after my changes :-)

It is certainly something which should be reported to the EHEA developers. Could you please do this?

I hope this RE & CC will suffice.

Regards,
Jarek P.

maintainer
#26 lore raw
In reply to Jarek Poplawski

From: Jarek Poplawski <[email protected]>
Date: Thu, 12 Jun 2008 06:13:34 +0000

Quoted 2 lines

It seems after this change bp->in_reset_task could be removed totally from the code.

...

Quoted 1 line

... likewise lp->work_pending,

...

Quoted 1 line

... likewise lp->work_pending.

...

Quoted 2 lines

local->info_queue probably needs some #ifndef like in hostap_info_init() or could be skipped here at all (waiting for remove).

Thanks for the feedback, here's a new patch with your suggestions added.

net: Eliminate flush_scheduled_work() calls while RTNL is held.

If the RTNL is held when we invoke flush_scheduled_work() we could deadlock. One such case is linkwatch, it is a workqueue which tries to grab the RTNL semaphore.

The most common case are net driver ->stop() methods. The simplest conversion is to instead use cancel_{delayed_}work_sync() explicitly on the various workqueues the driver uses.

This is an OK transformation because these workqueues are doing things like resetting the chip, restarting link negotiation, and so forth. And if we're bringing down the device, we're about to turn the chip off and reset it anways. So if we cancel a pending work event, that's fine here.

Some drivers were working around this deadlock by using a msleep() polling loop of some sort, and those cases are converted to instead use cancel_{delayed_}work_sync() as well.

drivers/net/bnx2.c +1 -8
index 4b46e68..367b6d4 100644
@@ -5724,14 +5724,12 @@ bnx2_reset_task(struct work_struct *work)
 	if (!netif_running(bp->dev))
 		return;
 
-	bp->in_reset_task = 1;
 	bnx2_netif_stop(bp);
 
 	bnx2_init_nic(bp);
 
 	atomic_set(&bp->intr_sem, 1);
 	bnx2_netif_start(bp);
-	bp->in_reset_task = 0;
 }
 
 static void
@@ -5907,12 +5905,7 @@ bnx2_close(struct net_device *dev)
 	struct bnx2 *bp = netdev_priv(dev);
 	u32 reset_code;
 
-	/* Calling flush_scheduled_work() may deadlock because
-	 * linkwatch_event() may be on the workqueue and it will try to get
-	 * the rtnl_lock which we are holding.
-	 */
-	while (bp->in_reset_task)
-		msleep(1);
+	cancel_work_sync(&bp->reset_task);
 
 	bnx2_disable_int_sync(bp);
 	bnx2_napi_disable(bp);
drivers/net/bnx2.h +0 -1
index 1eaf5bb..2377cc1 100644
@@ -6656,7 +6656,6 @@ struct bnx2 {
 	int			current_interval;
 	struct			timer_list timer;
 	struct work_struct	reset_task;
-	int			in_reset_task;
 
 	/* Used to synchronize phy accesses. */
 	spinlock_t		phy_lock;
drivers/net/ehea/ehea_main.c +2 -1
index faae01d..075fd54 100644
@@ -2605,7 +2605,8 @@ static int ehea_stop(struct net_device *dev)
 	if (netif_msg_ifdown(port))
 		ehea_info("disabling port %s", dev->name);
 
-	flush_scheduled_work();
+	cancel_work_sync(&port->reset_task);
+
 	mutex_lock(&port->port_lock);
 	netif_stop_queue(dev);
 	port_napi_disable(port);
drivers/net/hamradio/baycom_epp.c +1 -1
index dde9c7e..00bc7fb 100644
@@ -959,7 +959,7 @@ static int epp_close(struct net_device *dev)
 	unsigned char tmp[1];
 
 	bc->work_running = 0;
-	flush_scheduled_work();
+	cancel_delayed_work_sync(&bc->run_work);
 	bc->stat = EPP_DCDBIT;
 	tmp[0] = 0;
 	pp->ops->epp_write_addr(pp, tmp, 1, 0);
drivers/net/smc911x.c +5 -19
index 4e28002..e2ee91a 100644
@@ -136,7 +136,6 @@ struct smc911x_local {
 
 	/* work queue */
 	struct work_struct phy_configure;
-	int work_pending;
 
 	int tx_throttle;
 	spinlock_t lock;
@@ -960,11 +959,11 @@ static void smc911x_phy_configure(struct work_struct *work)
 	 * We should not be called if phy_type is zero.
 	 */
 	if (lp->phy_type == 0)
-		 goto smc911x_phy_configure_exit_nolock;
+		return;
 
 	if (smc911x_phy_reset(dev, phyaddr)) {
 		printk("%s: PHY reset timed out\n", dev->name);
-		goto smc911x_phy_configure_exit_nolock;
+		return;
 	}
 	spin_lock_irqsave(&lp->lock, flags);
 
@@ -1033,8 +1032,6 @@ static void smc911x_phy_configure(struct work_struct *work)
 
 smc911x_phy_configure_exit:
 	spin_unlock_irqrestore(&lp->lock, flags);
-smc911x_phy_configure_exit_nolock:
-	lp->work_pending = 0;
 }
 
 /*
@@ -1356,11 +1353,8 @@ static void smc911x_timeout(struct net_device *dev)
 	 * smc911x_phy_configure() calls msleep() which calls schedule_timeout()
 	 * which calls schedule().	 Hence we use a work queue.
 	 */
-	if (lp->phy_type != 0) {
-		if (schedule_work(&lp->phy_configure)) {
-			lp->work_pending = 1;
-		}
-	}
+	if (lp->phy_type != 0)
+		schedule_work(&lp->phy_configure);
 
 	/* We can accept TX packets again */
 	dev->trans_start = jiffies;
@@ -1531,16 +1525,8 @@ static int smc911x_close(struct net_device *dev)
 	if (lp->phy_type != 0) {
 		/* We need to ensure that no calls to
 		 * smc911x_phy_configure are pending.
-
-		 * flush_scheduled_work() cannot be called because we
-		 * are running with the netlink semaphore held (from
-		 * devinet_ioctl()) and the pending work queue
-		 * contains linkwatch_event() (scheduled by
-		 * netif_carrier_off() above). linkwatch_event() also
-		 * wants the netlink semaphore.
 		 */
-		while (lp->work_pending)
-			schedule();
+		cancel_work_sync(&lp->phy_configure);
 		smc911x_phy_powerdown(dev, lp->mii.phy_id);
 	}
 
drivers/net/smc91x.c +3 -14
index a188e33..f2051b2 100644
@@ -1016,15 +1016,8 @@ static void smc_phy_powerdown(struct net_device *dev)
 
 	/* We need to ensure that no calls to smc_phy_configure are
 	   pending.
-
-	   flush_scheduled_work() cannot be called because we are
-	   running with the netlink semaphore held (from
-	   devinet_ioctl()) and the pending work queue contains
-	   linkwatch_event() (scheduled by netif_carrier_off()
-	   above). linkwatch_event() also wants the netlink semaphore.
 	*/
-	while(lp->work_pending)
-		yield();
+	cancel_work_sync(&lp->phy_configure);
 
 	bmcr = smc_phy_read(dev, phy, MII_BMCR);
 	smc_phy_write(dev, phy, MII_BMCR, bmcr | BMCR_PDOWN);
@@ -1161,7 +1154,6 @@ static void smc_phy_configure(struct work_struct *work)
 smc_phy_configure_exit:
 	SMC_SELECT_BANK(lp, 2);
 	spin_unlock_irq(&lp->lock);
-	lp->work_pending = 0;
 }
 
 /*
@@ -1389,11 +1381,8 @@ static void smc_timeout(struct net_device *dev)
 	 * smc_phy_configure() calls msleep() which calls schedule_timeout()
 	 * which calls schedule().  Hence we use a work queue.
 	 */
-	if (lp->phy_type != 0) {
-		if (schedule_work(&lp->phy_configure)) {
-			lp->work_pending = 1;
-		}
-	}
+	if (lp->phy_type != 0)
+		schedule_work(&lp->phy_configure);
 
 	/* We can accept TX packets again */
 	dev->trans_start = jiffies;
drivers/net/tulip/tulip_core.c +1 -1
index 55670b5..af8d2c4 100644
@@ -731,7 +731,7 @@ static void tulip_down (struct net_device *dev)
 	void __iomem *ioaddr = tp->base_addr;
 	unsigned long flags;
 
-	flush_scheduled_work();
+	cancel_work_sync(&tp->media_work);
 
 #ifdef CONFIG_TULIP_NAPI
 	napi_disable(&tp->napi);
drivers/net/usb/kaweth.c +1 -1
index 0dcfc03..7c66b05 100644
@@ -706,7 +706,7 @@ static void kaweth_kill_urbs(struct kaweth_device *kaweth)
 	usb_kill_urb(kaweth->rx_urb);
 	usb_kill_urb(kaweth->tx_urb);
 
-	flush_scheduled_work();
+	cancel_delayed_work_sync(&kaweth->lowmem_work);
 
 	/* a scheduled work may have resubmitted,
 	   we hit them again */
drivers/net/wireless/hostap/hostap_main.c +7 -1
index 20d387f..f7aec93 100644
@@ -682,7 +682,13 @@ static int prism2_close(struct net_device *dev)
 		netif_device_detach(dev);
 	}
 
-	flush_scheduled_work();
+	cancel_work_sync(&local->reset_queue);
+	cancel_work_sync(&local->set_multicast_list_queue);
+	cancel_work_sync(&local->set_tim_queue);
+#ifndef PRISM2_NO_STATION_MODES
+	cancel_work_sync(&local->info_queue);
+#endif
+	cancel_work_sync(&local->comms_qual_update);
 
 	module_put(local->hw_module);
 
maintainer
#23 lore raw
In reply to David Miller

Technically,

Quoted 2 lines

If the RTNL is held when we invoke flush_scheduled_work() we could deadlock. One such case is linkwatch, it is a workqueue which tries

                                            that ^^^^^^^^^
Quoted 3 lines

The most common case are net driver ->stop() methods. The simplest conversion is to instead use cancel_{delayed_}work_sync() explicitly on the various workqueues the driver uses.

                   and that ^^^^^^^^^^
Quoted 1 line

This is an OK transformation because these workqueues are doing things

                                    and that ^^^^^^^^^^

should read "work struct" because it all uses the global workqueue.

I haven't looked at the drivers in more detail (sorry) but the patch itself looks fine to me (but I wouldn't have caught something like Jarek did with the now-unused variables.)

johannes

#27 lore raw
In reply to David Miller

On Thu, Jun 12, 2008 at 12:01:53AM -0700, David Miller wrote:
...

Quoted 2 lines

net: Eliminate flush_scheduled_work() calls while RTNL is held.

...

Very nice!

BTW, I guess this cassini fix with direct calling the work function (and maybe moving flush_ after unregister_netdev() in 8139too, like in a few other places) could be re-considered here as well.

Jarek P.

maintainer
#24 lore raw
In reply to Johannes Berg

From: Johannes Berg <[email protected]>
Date: Thu, 12 Jun 2008 09:20:34 +0200

Quoted 9 lines
Quoted 2 lines

If the RTNL is held when we invoke flush_scheduled_work() we could deadlock. One such case is linkwatch, it is a workqueue which tries

                                            that ^^^^^^^^^
Quoted 3 lines

The most common case are net driver ->stop() methods. The simplest conversion is to instead use cancel_{delayed_}work_sync() explicitly on the various workqueues the driver uses.

                   and that ^^^^^^^^^^
Quoted 8 lines
Quoted 1 line

This is an OK transformation because these workqueues are doing things

                                    and that ^^^^^^^^^^

should read "work struct" because it all uses the global workqueue.

I haven't looked at the drivers in more detail (sorry) but the patch itself looks fine to me (but I wouldn't have caught something like Jarek did with the now-unused variables.)

I've made those corrections to the commit message, thanks for the feedback.

Since Jarek appears to be happy with the patch at this point I'll commit this and sned it to Linus in my next push of bug fixes.