workqueue destruction BUG_ON
Tejun, all,
In our testing with iwlwifi, we keep running into this BUG_ON:
BUG_ON(cwq->nr_active);
in destroy_workqueue(). This is quite unhelpful, and since the code flushes the workqueue I really don't see how this could be happening, unless maybe there's cross-talk between this and other workqueues?
johannes
Tejun Heo
maintainerHello,
Quoted 7 lines · Johannes Berg
In our testing with iwlwifi, we keep running into this
BUG_ON:BUG_ON(cwq->nr_active);in
destroy_workqueue(). This is quite unhelpful, and since the code flushes the workqueue I really don't see how this could be happening, unless maybe there's cross-talk between this and other workqueues?
Flushing the workqueue doesn't guarantee that the workqueue stays empty. It just flushes the currently pending works. If there are requeueing works or something else is queueing new works, workqueue won't be empty. The check is new. Previously, workqueue code didn't check whether the workqueue is actually empty. Now that the worklist is shared, we need such check in place. There was also a similar report on ath9k. I think it's most likely that something is queueing works on a dying wokrqueue in the wireless common code. I'll prep a debug patch to print out some details.
Thanks.
Signature · 1 line
tejun
Hi,
Quoted 10 lines
Quoted 7 lines
In our testing with iwlwifi, we keep running into this
BUG_ON:BUG_ON(cwq->nr_active);in
destroy_workqueue(). This is quite unhelpful, and since the code flushes the workqueue I really don't see how this could be happening, unless maybe there's cross-talk between this and other workqueues?Flushing the workqueue doesn't guarantee that the workqueue stays empty.
True.
Quoted 7 lines
It just flushes the currently pending works. If there are requeueing works or something else is queueing new works, workqueue won't be empty. The check is new. Previously, workqueue code didn't check whether the workqueue is actually empty. Now that the worklist is shared, we need such check in place. There was also a similar report on ath9k. I think it's most likely that something is queueing works on a dying wokrqueue in the wireless common code.
I think in my iwlwifi case it's actually destroying the iwlwifi workqueue (not sure why it even exists though).
Quoted 2 lines
I'll prep a
debug patch to print out some details.
That'd be helpful, thanks!
johannes
Tejun Heo
maintainerHello,
Quoted 2 lines · Johannes Berg
I think in my iwlwifi case it's actually destroying the iwlwifi workqueue (not sure why it even exists though).
Yeah, I'm planning on auditing each workqueue and remove unnecessary ones.
Quoted 3 lines
Quoted 1 line
I'll prep a debug patch to print out some details.
That'd be helpful, thanks!
Can you please apply the following patch and report the result?
Thanks.
Quoted 1 line
From
492a242b75b0abae3b1c17b4a654ab9ef67e612dMon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 24 Aug 2010 14:22:47 +0200
Subject: [PATCH] workqueue: improve destroy_workqueue() debuggability
Now that the worklist is global, having works pending after wq destruction can easily lead to oops and destroy_workqueue() have several BUG_ON()s to catch these cases. Unfortunately, BUG_ON() doesn't tell much about how the work became pending after the final flush_workqueue().
This patch adds WQ_DYING which is set before the final flush begins and WARN_ON_ONCE() is triggered if a work is requested to be queued on a dying workqueue and the request is ignored. This clearly indicates which caller is trying to queue a work on a dying workqueue and keeps the system working in most cases.
Locking rule comment is updated such that the 'I' rule includes modifying the field from destruction path.
--- include/linux/workqueue.h | 2 ++ kernel/workqueue.c | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletions(-)
include/linux/workqueue.h +2 -0
@@ -241,6 +241,8 @@ enum {WQ_HIGHPRI =1 <<4 ,/* high priority */ WQ_CPU_INTENSIVE =1 <<5 ,/* cpu instensive workqueue */ +WQ_DYING =1 <<6 ,/* internal: workqueue is dying */ +WQ_MAX_ACTIVE =512 ,/* I like 512, better ideas? */ WQ_MAX_UNBOUND_PER_CPU =4 ,/* 4 * #cpus for unbound wq */ WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,
kernel/workqueue.c +6 -1
@@ -87,7 +87,8 @@ enum {/* * Structure fields follow one of the following exclusion rules. * -* I: Set during initialization and read-only afterwards. +* I: Modifiable by initialization/destruction paths and read-only for +* everyone else. * * P: Preemption protected. Disabling preemption is enough and should * only be modified and accessed from the local cpu.
@@ -944,6 +945,9 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,debug_work_activate (work); +if (WARN_ON_ONCE (wq->flags &WQ_DYING )) +return ; +/* determine gcwq to use */ if (!(wq->flags &WQ_UNBOUND )) {struct global_cwq *last_gcwq;
@@ -2828,6 +2832,7 @@ void destroy_workqueue(struct workqueue_struct *wq) {unsigned int cpu; + wq->flags |=WQ_DYING ;flush_workqueue (wq);/*
Signature · 1 line
1.7.1
Quoted 8 lines · Tejun Heo
Can you please apply the following patch and report the result?
Thanks.
From
492a242b75b0abae3b1c17b4a654ab9ef67e612dMon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 24 Aug 2010 14:22:47 +0200
Subject: [PATCH] workqueue: improvedestroy_workqueue()debuggability
Thanks, will do. I couldn't reproduce right away, but I had it quite frequently yesterday so I'll go back to what I was testing yesterday.
johannes
Tejun,
Quoted 4 lines
From
492a242b75b0abae3b1c17b4a654ab9ef67e612dMon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 24 Aug 2010 14:22:47 +0200
Subject: [PATCH] workqueue: improvedestroy_workqueue()debuggability
With this patch applied I still get just the BUG_ON:
[ 500.874185] ------------[ cut here ]------------ [ 500.875212] kernel BUG at kernel/workqueue.c:2849! [ 500.876224] invalid opcode: 0000 [#1] PREEMPT SMP [ 500.877256] last sysfs file: /sys/module/iwlagn/refcnt [ 500.878267] CPU 1 [ 500.878281] Modules linked in: iwlagn(-) ... [ 500.880074] [ 500.880074] Pid: 2167, comm: modprobe Not tainted 2.6.36-rc1-wl-p2p-64225-g61066fb-dirty #18 JPTR/Satellite U505 [ 500.880074] RIP: 0010:[<ffffffff81078aac>] [<ffffffff81078aac>] destroy_workqueue+0x19c/0x1e0 [ 500.880074] RSP: 0018:ffff88012587fd58 EFLAGS: 00010286 [ 500.880074] RAX: 000000000000003c RBX: 0000000000000002 RCX: 000000000000005a [ 500.880074] RDX: ffff880132783d80 RSI: 0000000000000004 RDI: 00000000ffffffff [ 500.880074] RBP: ffff88012587fd68 R08: 0000000000000000 R09: 0000000000000000 [ 500.880074] R10: 0000000000000003 R11: 0000000000000000 R12: ffff880123709240 [ 500.880074] R13: ffff880133aa6240 R14: ffff880124423258 R15: 0000000000000282 [ 500.880074] FS: 00007fcf136b2700(0000) GS:ffff88002cc00000(0000) knlGS:0000000000000000 [ 500.880074] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 500.880074] CR2: 00000000006e3aa4 CR3: 00000001231fc000 CR4: 00000000000006e0 [ 500.880074] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 500.880074] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 500.880074] Process modprobe (pid: 2167, threadinfo ffff88012587e000, task ffff8801336c4780) [ 500.880074] Stack: [ 500.880074] ffff880124422840 ffff880133aa61b0 ffff88012587fda8 ffffffffa040bc30 [ 500.880074] <0> ffff88012587fd98 ffff880133aa6240 ffff880133aa6530 ffffffffa0417980 [ 500.880074] <0> ffff880133aa61b0 0000000001c014b8 ffff88012587fdd8 ffffffff812421b4 [ 500.880074] Call Trace: [ 500.880074] [<ffffffffa040bc30>] iwl_pci_remove+0x18c/0x21f [iwlagn] [ 500.880074] [<ffffffff812421b4>] pci_device_remove+0x54/0x120 [ 500.880074] [<ffffffff812d4ab5>] __device_release_driver+0x75/0xe0 [ 500.880074] [<ffffffff812d4bf8>] driver_detach+0xd8/0xe0 [ 500.880074] [<ffffffff812d3a58>] bus_remove_driver+0x88/0xe0 [ 500.880074] [<ffffffff812d5282>] driver_unregister+0x62/0xa0 [ 500.880074] [<ffffffff81242514>] pci_unregister_driver+0x44/0xc0 [ 500.880074] [<ffffffffa040ba9d>] iwl_exit+0x15/0x1c [iwlagn] [ 500.880074] [<ffffffff810a1e32>] sys_delete_module+0x1a2/0x280 [ 500.880074] [<ffffffff8100c1b2>] system_call_fastpath+0x16/0x1b [ 500.880074] Code: 19 00 39 05 77 81 81 00 7f 06 41 8b 0c 24 eb b4 83 f8 04 89 c6 77 23 41 8b 0c 24 e9 28 ff ff ff 0f 0b eb fe 31 d2 e9 43 ff ff ff <0f> 0b eb fe 0f 0b eb fe 83 f8 04 89 c6 76 dd 41 8b 04 24 e9 c6 [ 500.880074] RIP [<ffffffff81078aac>] destroy_workqueue+0x19c/0x1e0 [ 500.880074] RSP <ffff88012587fd58> [ 500.931585] ---[ end trace 06397bea564a72fe ]---
johannes
Tejun Heo
maintainerHello,
Quoted 6 lines · Johannes Berg
Quoted 4 lines
From
492a242b75b0abae3b1c17b4a654ab9ef67e612dMon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 24 Aug 2010 14:22:47 +0200
Subject: [PATCH] workqueue: improvedestroy_workqueue()debuggabilityWith this patch applied I still get just the
BUG_ON:
Hmmm... weird.
Quoted 2 lines
[ 500.874185] ------------[ cut here ]------------ [ 500.875212] kernel BUG at kernel/workqueue.c:2849!
Are you sure you're running the patched kernel? With the patch applied, the BUG_ON() wouldn't be on line 2849 (on both rc1 and 2).
Thanks.
Signature · 1 line
tejun
Quoted 5 lines · Tejun Heo
Quoted 2 lines
[ 500.874185] ------------[ cut here ]------------ [ 500.875212] kernel BUG at kernel/workqueue.c:2849!Are you sure you're running the patched kernel? With the patch applied, the
BUG_ON()wouldn't be on line 2849 (on both rc1 and 2).
Yes:
void destroy_workqueue(struct workqueue_struct *wq)
{
unsigned int cpu;
wq->flags |= WQ_DYING;
flush_workqueue(wq);
/*
* wq list is used to freeze wq, remove from list after
* flushing is complete in case freeze races us.
*/
spin_lock(&workqueue_lock);
list_del(&wq->list);
spin_unlock(&workqueue_lock);
/* sanity check */
for_each_cwq_cpu(cpu, wq) {
struct cpu_workqueue_struct *cwq = get_cwq(cpu, wq);
int i;
for (i = 0; i < WORK_NR_COLORS; i++)
BUG_ON(cwq->nr_in_flight[i]);
2849: BUG_ON(cwq->nr_active);
BUG_ON(!list_empty(&cwq->delayed_works));Applying the patch reported some offset, but the kernel is just rc1 + wireless stuff.
johannes
Tejun Heo
maintainerHello,
Quoted 36 lines · Johannes Berg
Quoted 5 lines · Tejun Heo
Quoted 2 lines
[ 500.874185] ------------[ cut here ]------------ [ 500.875212] kernel BUG at kernel/workqueue.c:2849!Are you sure you're running the patched kernel? With the patch applied, the
BUG_ON()wouldn't be on line 2849 (on both rc1 and 2).Yes:
void
destroy_workqueue(struct workqueue_struct *wq){ unsigned int cpu; wq->flags |= WQ_DYING; flush_workqueue(wq); /* * wq list is used to freeze wq, remove from list after * flushing is complete in case freeze races us. */ spin_lock(&workqueue_lock); list_del(&wq->list); spin_unlock(&workqueue_lock); /* sanity check */ for_each_cwq_cpu(cpu, wq) { struct cpu_workqueue_struct *cwq = get_cwq(cpu, wq); int i; for (i = 0; i < WORK_NR_COLORS; i++) BUG_ON(cwq->nr_in_flight[i]); 2849: BUG_ON(cwq->nr_active); BUG_ON(!list_empty(&cwq->delayed_works));
Quoted 2 lines
Applying the patch reported some offset, but the kernel is just rc1 + wireless stuff.
I see, thanks for verifying. I probably got confused about the line number. Hmm... weird. I'll prep further debug patch but can you please tell me what you did to trigger the bug?
Thanks.
Signature · 1 line
tejun
Quoted 3 lines · Tejun Heo
I see, thanks for verifying. I probably got confused about the line number. Hmm... weird. I'll prep further debug patch but can you please tell me what you did to trigger the bug?
Not really sure. I think I need to trigger a firmware restart in iwlwifi and then try to unload the module.
johannes
Tejun Heo
maintainerQuoted 6 lines · Johannes Berg
Quoted 3 lines
I see, thanks for verifying. I probably got confused about the line number. Hmm... weird. I'll prep further debug patch but can you please tell me what you did to trigger the bug?
Not really sure. I think I need to trigger a firmware restart in iwlwifi and then try to unload the module.
Does the following patch fix the problem?
Thanks.
include/linux/workqueue.h +9 -7
@@ -25,18 +25,20 @@ typedef void (*work_func_t)(struct work_struct *work);enum {WORK_STRUCT_PENDING_BIT =0 ,/* work item is pending execution */ -WORK_STRUCT_CWQ_BIT =1 ,/* data points to cwq */ -WORK_STRUCT_LINKED_BIT =2 ,/* next work is linked to this one */ +WORK_STRUCT_DELAYED_BIT =1 ,/* work item is delayed */ +WORK_STRUCT_CWQ_BIT =2 ,/* data points to cwq */ +WORK_STRUCT_LINKED_BIT =3 ,/* next work is linked to this one */ #ifdef CONFIG_DEBUG_OBJECTS_WORK -WORK_STRUCT_STATIC_BIT =3 ,/* static initializer (debugobjects) */ -WORK_STRUCT_COLOR_SHIFT =4 ,/* color for workqueue flushing */ +WORK_STRUCT_STATIC_BIT =4 ,/* static initializer (debugobjects) */ +WORK_STRUCT_COLOR_SHIFT =5 ,/* color for workqueue flushing */ #else -WORK_STRUCT_COLOR_SHIFT =3 ,/* color for workqueue flushing */ +WORK_STRUCT_COLOR_SHIFT =4 ,/* color for workqueue flushing */ #endif WORK_STRUCT_COLOR_BITS =4 ,WORK_STRUCT_PENDING =1 <<WORK_STRUCT_PENDING_BIT , +WORK_STRUCT_DELAYED =1 <<WORK_STRUCT_DELAYED_BIT ,WORK_STRUCT_CWQ =1 <<WORK_STRUCT_CWQ_BIT ,WORK_STRUCT_LINKED =1 <<WORK_STRUCT_LINKED_BIT ,#ifdef CONFIG_DEBUG_OBJECTS_WORK
@@ -59,8 +61,8 @@ enum {/* * Reserve 7 bits off of cwq pointer w/ debugobjects turned -* off. This makes cwqs aligned to 128 bytes which isn't too -* excessive while allowing 15 workqueue flush colors. +* off. This makes cwqs aligned to 256 bytes and allows 15 +* workqueue flush colors. */ WORK_STRUCT_FLAG_BITS =WORK_STRUCT_COLOR_SHIFT + WORK_STRUCT_COLOR_BITS,
kernel/workqueue.c +20 -10
@@ -941,6 +941,7 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,struct global_cwq *gcwq;struct cpu_workqueue_struct *cwq;struct list_head *worklist; +unsigned int work_flags;unsigned long flags;debug_work_activate (work);
@@ -990,14 +991,17 @@ static void __queue_work(unsigned int cpu, struct workqueue_struct *wq,BUG_ON (!list_empty (&work->entry)); cwq->nr_in_flight[cwq->work_color]++; + work_flags =work_color_to_flags (cwq->work_color);if (likely (cwq->nr_active < cwq->max_active)) { cwq->nr_active++; worklist =gcwq_determine_ins_pos (gcwq, cwq); - }else + }else { + work_flags |=WORK_STRUCT_DELAYED ; worklist = &cwq->delayed_works; + } -insert_work (cwq, work, worklist,work_color_to_flags (cwq->work_color)); +insert_work (cwq, work, worklist, work_flags);spin_unlock_irqrestore (&gcwq->lock, flags); }
@@ -1666,6 +1670,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)struct list_head *pos =gcwq_determine_ins_pos (cwq->gcwq, cwq);move_linked_works (work, pos,NULL ); +__clear_bit (WORK_STRUCT_DELAYED_BIT ,work_data_bits (work)); cwq->nr_active++; }
@@ -1673,6 +1678,7 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)* cwq_dec_nr_in_flight - decrement cwq's nr_in_flight * @cwq: cwq of interest * @color: color of work which left the queue +* @delayed: for a delayed work * * A work either has completed or is removed from pending queue, * decrement nr_in_flight of its cwq and handle workqueue flushing.
@@ -1680,19 +1686,22 @@ static void cwq_activate_first_delayed(struct cpu_workqueue_struct *cwq)* CONTEXT: * spin_lock_irq(gcwq->lock). */ -static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color) +static void cwq_dec_nr_in_flight (struct cpu_workqueue_struct *cwq,int color, +bool delayed) {/* ignore uncolored works */ if (color ==WORK_NO_COLOR )return ; cwq->nr_in_flight[color]--; - cwq->nr_active--; -if (!list_empty (&cwq->delayed_works)) { -/* one down, submit a delayed one */ -if (cwq->nr_active < cwq->max_active) -cwq_activate_first_delayed (cwq); +if (!delayed) { + cwq->nr_active--; +if (!list_empty (&cwq->delayed_works)) { +/* one down, submit a delayed one */ +if (cwq->nr_active < cwq->max_active) +cwq_activate_first_delayed (cwq); + } }/* is flush in progress and are we at the flushing tip? */
@@ -1823,7 +1832,7 @@ __acquires(&gcwq->lock)hlist_del_init (&worker->hentry); worker->current_work =NULL ; worker->current_cwq =NULL ; -cwq_dec_nr_in_flight (cwq, work_color); +cwq_dec_nr_in_flight (cwq, work_color, false); }/**
@@ -2388,7 +2397,8 @@ static int try_to_grab_pending(struct work_struct *work)debug_work_deactivate (work);list_del_init (&work->entry);cwq_dec_nr_in_flight (get_work_cwq (work), -get_work_color (work)); +get_work_color (work), + *work_data_bits (work) &WORK_STRUCT_DELAYED ); ret =1 ; } }
Quoted 9 lines · Tejun Heo
Quoted 6 lines · Johannes Berg
Quoted 3 lines
I see, thanks for verifying. I probably got confused about the line number. Hmm... weird. I'll prep further debug patch but can you please tell me what you did to trigger the bug?
Not really sure. I think I need to trigger a firmware restart in iwlwifi and then try to unload the module.
Does the following patch fix the problem?
Not sure. I can't seem to reproduce the exact conditions (firmware crash) right now... However I just ran into the WARN_ON_ONCE() your previous patch added as well, so that might explain the ath9k problem...
johannes
Tejun Heo
maintainerHello,
Quoted 6 lines · Johannes Berg
Quoted 5 lines · Tejun Heo
Quoted 4 lines · Johannes Berg
Quoted 3 lines
I see, thanks for verifying. I probably got confused about the line number. Hmm... weird. I'll prep further debug patch but can you please tell me what you did to trigger the bug?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Not really sure. I think I need to trigger a firmware restart in iwlwifi and then try to unload the module.
Does the following patch fix the problem?
Not sure. I can't seem to reproduce the exact conditions (firmware crash) right now... However I just ran into the
WARN_ON_ONCE()your previous patch added as well, so that might explain the ath9k problem...
Can you please post the log of the WARN_ON_ONCE()? That's something which needs to be fixed from the caller.
Thanks.
Signature · 1 line
tejun
Quoted 6 lines · Tejun Heo
Quoted 3 lines
Not sure. I can't seem to reproduce the exact conditions (firmware crash) right now... However I just ran into the
WARN_ON_ONCE()your previous patch added as well, so that might explain the ath9k problem...Can you please post the log of the
WARN_ON_ONCE()? That's something which needs to be fixed from the caller.
Yeah, I know, there's a timer that never gets cancelled, I'll fix it.
johannes
Quoted 5 lines · Tejun Heo
Quoted 2 lines
[ 500.874185] ------------[ cut here ]------------ [ 500.875212] kernel BUG at kernel/workqueue.c:2849!Are you sure you're running the patched kernel? With the patch applied, the
BUG_ON()wouldn't be on line 2849 (on both rc1 and 2).
And this http://paste.pocoo.org/raw/253920/ is the diff between 2.6.35-rc1 and my current workqueue.c
johannes