Skip to contentSkip to conversation

core: workqueue: BUG_ON on workqueue recursion

PATCH Superseded Simon Kagstrom 4 participants 7 messages Feb 3, 2010 – Feb 12, 2010 Workqueue

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 core: workqueue: BUG_ON on workqueue recursion

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

Since the system becomes unstable, change this to a BUG_ON instead.

---
 kernel/workqueue.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/workqueue.c +1 -1
index dee4865..e617d29 100644
@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	BUG_ON(cwq->thread == current);
 
 	spin_lock_irq(&cwq->lock);
 	if (!list_empty(&cwq->worklist) || cwq->current_work != NULL) {
Signature · 1 line
1.6.0.4
#1 lore raw
Quoted 7 lines · Simon Kagstrom

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

Since the system becomes unstable, change this to a BUG_ON instead.

I agree with this patch. We are going to deadlock anyway, if the condition is true the caller is cwq->current_work, this means flush_cpu_workqueue() will insert the barrier and hang.

However,

kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 485

Quoted 6 lines
kernel/workqueue.c +1 -1
@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	BUG_ON(cwq->thread == current);

Another option is change the code to do

	if (WARN_ON(cwq->thread == current))
		return;

This gives the kernel chance to survive after the warning.

What do you think?

Oleg.

maintainer
#2 lore raw

Hello,

Quoted 24 lines · Oleg Nesterov
Quoted 7 lines · Simon Kagstrom

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

Since the system becomes unstable, change this to a BUG_ON instead.

I agree with this patch. We are going to deadlock anyway, if the condition is true the caller is cwq->current_work, this means flush_cpu_workqueue() will insert the barrier and hang.

However,

kernel/workqueue.c static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq) · line 485

Quoted 6 lines
kernel/workqueue.c +1 -1
@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	BUG_ON(cwq->thread == current);

Another option is change the code to do

Quoted 6 lines
	if (WARN_ON(cwq->thread == current))
		return;

This gives the kernel chance to survive after the warning.

What do you think?

Yeah, I like this one better too. Even solely for debugging, WARN_ON() is better as often users don't have reliable ways to gather kernel log after a BUG_ON().

Thanks.

Signature · 1 line
tejun
#3 lore raw

PATCH v2 1/1 core: workqueue: return on workqueue recursion

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.

---
ChangeLog:
* Instead of BUG_ON, warn and return on recursive calls as suggested by Oleg Nesterov and Tejun Hao

 kernel/workqueue.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/workqueue.c +2 -1
index dee4865..49f8fa7 100644
@@ -482,7 +482,8 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	if (WARN_ON(cwq->thread == current))
+		return 1;
 
 	spin_lock_irq(&cwq->lock);
 	if (!list_empty(&cwq->worklist) || cwq->current_work != NULL) {
Signature · 1 line
1.6.0.4
Acked-by on v2 1/1
#4 lore raw

Re: [PATCH v2] core: workqueue: return on workqueue recursion

Quoted 11 lines · Simon Kagstrom

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.

Quoted 25 lines

---
ChangeLog:
* Instead of BUG_ON, warn and return on recursive calls as suggested by Oleg Nesterov and Tejun Hao

 kernel/workqueue.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/workqueue.c +2 -2
index dee4865..49f8fa7 100644
@@ -482,7 +482,8 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	if (WARN_ON(cwq->thread == current))
+		return 1;
 
 	spin_lock_irq(&cwq->lock);
 	if (!list_empty(&cwq->worklist) || cwq->current_work != NULL) {
--

1.6.0.4

maintainer Applied
#5 lore raw

Re: [PATCH v2] core: workqueue: return on workqueue recursion

In reply to Simon Kagstrom
Quoted 10 lines · Simon Kagstrom

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

The system becomes unstable since the workqueue barrier locks the workqueue. This patch instead returns if the workqueue is flushed recursively, which keeps the workqueue alive but warns.

applied to wq tree. Will push out when the merge window opens.

Thanks.

Signature · 1 line
tejun
Acked-by on v1 1/1
#6 lore raw
In reply to Simon Kagstrom
Quoted 6 lines · Simon Kagstrom

When the workqueue is flushed from workqueue context (recursively), the system enters a strange state where things at random (dependent on the global workqueue) start misbehaving. For example, for us the console and logins locks up while the web server continues running.

Since the system becomes unstable, change this to a BUG_ON instead.

For design view, we should disallow this recursion when using workqueue.

I like BUG_ON. But it is not a fatal end usually when it happens, most developers would like to let system go on.

Quoted 19 lines
---
 kernel/workqueue.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/workqueue.c +1 -1
index dee4865..e617d29 100644
@@ -482,7 +482,7 @@ static int flush_cpu_workqueue(struct cpu_workqueue_struct *cwq)
 	int active = 0;
 	struct wq_barrier barr;
 
-	WARN_ON(cwq->thread == current);
+	BUG_ON(cwq->thread == current);
 
 	spin_lock_irq(&cwq->lock);
 	if (!list_empty(&cwq->worklist) || cwq->current_work != NULL) {