Skip to contentSkip to conversation

2.6.21 hang in cancel_rearming_delayed_workqueue()

BUG Jason Wessel 2 participants 3 messages May 25, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

There is a problem with the calling cancel_rearming_delayed_work if the timer was not yet active.

I see this problem when netpoll_cleanup() is called without having done any work because it had not processed any packets yet. The problem appears to be a result of the loop check while(!cancel_delayed_work(dwork)). This endlessly loops because del_timer_sync() can return 0 or 1 for success which is passed back as a result to the final invariant check for the loop. In this particular case zero will always be returned because the timer is not active.

It is possible that the problem exists else where, but I thought I would ask if this is expected?

#0 del_timer_sync (timer=0xc7ed90f8) at kernel/timer.c:530
#1 0xc012f08e in cancel_rearming_delayed_workqueue (wq=0xc7fee800,
dwork=0xc7ed90e8) at include/linux/workqueue.h:201
#2 0xc012f0af in cancel_rearming_delayed_work (dwork=0x20)
at kernel/workqueue.c:680
#3 0xc0312f78 in netpoll_cleanup (np=0xc880bf40) at net/core/netpoll.c:784

Possible fix.

kernel/workqueue.c +0 -0
Index: linux-2.6.21/kernel/workqueue.c
===================================================================
@@ -666,7 +666,7 @@ EXPORT_SYMBOL(flush_scheduled_work);

void cancel_rearming_delayed_workqueue(struct workqueue_struct *wq,
struct delayed_work *dwork)
{
- while (!cancel_delayed_work(dwork))
+ while (cancel_delayed_work(dwork) > 0)

              flush_workqueue(wq);
}

EXPORT_SYMBOL(cancel_rearming_delayed_workqueue);

Thanks,
Jason.

#1 lore raw
Quoted 30 lines · Jason Wessel

There is a problem with the calling cancel_rearming_delayed_work if the timer was not yet active.

I see this problem when netpoll_cleanup() is called without having done any work because it had not processed any packets yet. The problem appears to be a result of the loop check
while(!cancel_delayed_work(dwork)). This endlessly loops because
del_timer_sync() can return 0 or 1 for success which is passed back as a result to the final invariant check for the loop. In this particular case zero will always be returned because the timer is not active.

It is possible that the problem exists else where, but I thought I would ask if this is expected?

#0 del_timer_sync (timer=0xc7ed90f8) at kernel/timer.c:530
#1 0xc012f08e in cancel_rearming_delayed_workqueue (wq=0xc7fee800,
dwork=0xc7ed90e8) at include/linux/workqueue.h:201
#2 0xc012f0af in cancel_rearming_delayed_work (dwork=0x20)
at kernel/workqueue.c:680
#3 0xc0312f78 in netpoll_cleanup (np=0xc880bf40) at net/core/netpoll.c:784

Possible fix.

Index: linux-2.6.21/kernel/workqueue.c
===================================================================

kernel/workqueue.c EXPORT_SYMBOL(flush_scheduled_work); · line 672

Quoted 8 lines
kernel/workqueue.c +1 -1
@@ -666,7 +666,7 @@ EXPORT_SYMBOL(flush_scheduled_work);
 void cancel_rearming_delayed_workqueue(struct workqueue_struct *wq,
                                     struct delayed_work *dwork)
 {
-       while (!cancel_delayed_work(dwork))
+       while (cancel_delayed_work(dwork) > 0)
              flush_workqueue(wq);
 }
 EXPORT_SYMBOL(cancel_rearming_delayed_workqueue);

It's very optimistic change...

I wonder, how this all could work so long (or how it is supposed to work now without breaking other callers) with (almost) reversed condition?

According to this comment:

" * cancel_rearming_delayed_workqueue - reliably kill off a delayed
work whose handler rearms the delayed work."

So, it cannot be used in netpoll_cleanup() if there is no rearming during this cancel at all. This is a tricky behaviour of course, and is changed in 2.6.22-rc.

Regards,
Jarek P.

#2 lore raw
Quoted 10 lines · Jarek Poplawski
Quoted 6 lines · Jason Wessel

There is a problem with the calling cancel_rearming_delayed_work if the timer was not yet active.

It is possible that the problem exists else where, but I thought I would ask if this is expected?

Quoted 8 lines

" * cancel_rearming_delayed_workqueue - reliably kill off a delayed
work whose handler rearms the delayed work."

So, it cannot be used in netpoll_cleanup() if there is no rearming during this cancel at all. This is a tricky behaviour of course, and is changed in 2.6.22-rc.

I had read that as well. It seems this the API for the cancel is restricted such that you cannot call it if there is nothing to cancel.

This means the netpoll code is incorrect and the simple fix is attached. With the change the netpoll_cleanup works correctly in 2.6.21.

Thanks,
Jason.

Do not call cancel_rearming_delayed_work() if there is no pending work.

---
 net/core/netpoll.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
net/core/netpoll.c +4 -2
Index: linux-2.6.21/net/core/netpoll.c
===================================================================
@@ -781,8 +781,10 @@ void netpoll_cleanup(struct netpoll *np)
 			if (atomic_dec_and_test(&npinfo->refcnt)) {
 				skb_queue_purge(&npinfo->arp_tx);
 				skb_queue_purge(&npinfo->txq);
-				cancel_rearming_delayed_work(&npinfo->tx_work);
-				flush_scheduled_work();
+				if (delayed_work_pending(&npinfo->tx_work)) {
+					cancel_rearming_delayed_work(&npinfo->tx_work);
+					flush_scheduled_work();
+				}
 
 				kfree(npinfo);
 			}