core: workqueue: BUG_ON on workqueue recursion
PATCH 1/1 core: workqueue: BUG_ON on workqueue recursion
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
Since the system becomes unstable, change this to a BUG_ON instead.
--- kernel/workqueue.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/workqueue.c +1 -1
@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +BUG_ON (cwq->thread == current);spin_lock_irq (&cwq->lock);if (!list_empty (&cwq->worklist) || cwq->current_work !=NULL ) {
Signature · 1 line
1.6.0.4
Quoted 7 lines · Simon Kagstrom
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
Since the system becomes unstable, change this to a
BUG_ONinstead.
I agree with this patch. We are going to deadlock anyway, if the condition is true the caller is cwq->current_work, this means flush_cpu_workqueue() will insert the barrier and hang.
However,
kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 485
Quoted 6 lines
kernel/workqueue.c +1
-1@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +BUG_ON (cwq->thread == current);
Another option is change the code to do
if (WARN_ON(cwq->thread == current)) return;
This gives the kernel chance to survive after the warning.
What do you think?
Oleg.
Tejun Heo
maintainerHello,
Quoted 24 lines · Oleg Nesterov
Quoted 7 lines · Simon Kagstrom
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
Since the system becomes unstable, change this to a
BUG_ONinstead.I agree with this patch. We are going to deadlock anyway, if the condition is true the caller is
cwq->current_work, this meansflush_cpu_workqueue()will insert the barrier and hang.However,
kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 485
Quoted 6 lines
kernel/workqueue.c +1
-1@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +BUG_ON (cwq->thread == current);Another option is change the code to do
Quoted 6 lines
if (WARN_ON(cwq->thread == current)) return;This gives the kernel chance to survive after the warning.
What do you think?
Yeah, I like this one better too. Even solely for debugging, WARN_ON() is better as often users don't have reliable ways to gather kernel log after a BUG_ON().
Thanks.
Signature · 1 line
tejun
PATCH v2 1/1 core: workqueue: return on workqueue recursion
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.
---
ChangeLog:
* Instead of BUG_ON, warn and return on recursive calls as suggested by Oleg Nesterov and Tejun Hao
kernel/workqueue.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/workqueue.c +2 -1
@@ -482,7 +482,8 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +if (WARN_ON (cwq->thread == current)) +return 1 ;spin_lock_irq (&cwq->lock);if (!list_empty (&cwq->worklist) || cwq->current_work !=NULL ) {
Signature · 1 line
1.6.0.4
Re: [PATCH v2] core: workqueue: return on workqueue recursion
Quoted 11 lines · Simon Kagstrom
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.
Signed-off-by Simon Kagstrom <[email protected]>
Quoted 25 lines
---
ChangeLog:
* Instead ofBUG_ON, warn and return on recursive calls as suggested by Oleg Nesterov and Tejun Haokernel/workqueue.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)kernel/workqueue.c +2
-2@@ -482,7 +482,8 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +if (WARN_ON (cwq->thread == current)) +return 1 ;spin_lock_irq (&cwq->lock);if (!list_empty (&cwq->worklist) || cwq->current_work !=NULL ) { --1.6.0.4
Re: [PATCH v2] core: workqueue: return on workqueue recursion
In reply to Simon KagstromQuoted 10 lines · Simon Kagstrom
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.
Signed-off-by Simon Kagstrom <[email protected]>
applied to wq tree. Will push out when the merge window opens.
Thanks.
Signature · 1 line
tejun
Quoted 6 lines · Simon Kagstrom
When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.
Since the system becomes unstable, change this to a
BUG_ONinstead.
For design view, we should disallow this recursion when using workqueue.
I like BUG_ON. But it is not a fatal end usually when it happens, most developers would like to let system go on.
Quoted 19 lines
Signed-off-by Simon Kagstrom <[email protected]>--- kernel/workqueue.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/workqueue.c +1
-1@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)int active =0 ;struct wq_barrier barr; -WARN_ON (cwq->thread == current); +BUG_ON (cwq->thread == current);spin_lock_irq (&cwq->lock);if (!list_empty (&cwq->worklist) || cwq->current_work !=NULL ) {