Skip to contentSkip to conversation

WARN_ON_ONCE() in process_one_work()?

DISCUSSION Paul E. McKenney 2 participants 6 messages Jun 20, 2018 – Jul 3, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello!

I have hit this WARN_ON_ONCE() in process_one_work:

	WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) &&
		     raw_smp_processor_id() != pool->cpu);

This looks like it is my rcu_gp workqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).

Is this related to the recent addition of WQ_MEM_RECLAIM? Either way, what should I do to further debug this?

Thanx, Paul

[  828.791694] WARNING: CPU: 2 PID: 3 at /home/paulmck/public_git/linux-rcu/kernel/workqueue.c:2069 process_one_work+0x49/0x3c0
[  828.794850] Modules linked in:
[  828.795748] CPU: 2 PID: 3 Comm: rcu_gp Not tainted 4.18.0-rc1+ #1
[  828.797436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[  828.800059] Workqueue:            (null) (rcu_gp)
[  828.801373] RIP: 0010:process_one_work+0x49/0x3c0
[  828.802690] Code: 00 00 00 00 4c 0f 44 e0 49 8b 44 24 08 44 8b a8 00 01 00 00 41 83 e5 20 f6 45 10 04 75 0e 65 8b 05 6c 99 79 68 3b 45 04 74 02 <0f> 0b 48 ba eb 83 b5 80 46 86 c8 61 48 0f af d6 48 c1 ea 3a 48 8b
[  828.807871] RSP: 0000:ffff8a3c400e7e70 EFLAGS: 00010002
[  828.809314] RAX: 0000000000000002 RBX: ffff88acdfc206e0 RCX: ffff88acdfc206e0
[  828.811307] RDX: ffff88acdfc206d8 RSI: ffff88acdfc197b8 RDI: ffff88acde80c6c0
[  828.813253] RBP: ffff88acdfc206c0 R08: 0000000000000000 R09: 0000000000000000
[  828.815239] R10: ffff8a3c400e7e48 R11: 0000000000000000 R12: ffff88acdfc24900
[  828.817073] R13: 0000000000000000 R14: ffff88acde80c6c0 R15: ffff88acde80c6e8
[  828.819071] FS:  0000000000000000(0000) GS:ffff88acdfd00000(0000) knlGS:0000000000000000
[  828.821361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  828.822971] CR2: 00000000000000b0 CR3: 000000001040a000 CR4: 00000000000006e0
[  828.824935] Call Trace:
[  828.825653]  rescuer_thread+0x201/0x320
[  828.826744]  kthread+0xf3/0x130
[  828.827618]  ? process_one_work+0x3c0/0x3c0
[  828.828758]  ? kthread_destroy_worker+0x40/0x40
[  828.830019]  ret_from_fork+0x35/0x40
[  828.831028] ---[ end trace 4197eb078bf0edac ]---
maintainer
#1 lore raw

Hello, Paul.

Sorry about the late reply.

Quoted 11 lines · Paul E. McKenney

I have hit this WARN_ON_ONCE() in process_one_work:

	WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) &&
		     raw_smp_processor_id() != pool->cpu);

This looks like it is my rcu_gp workqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).

Is this related to the recent addition of WQ_MEM_RECLAIM? Either way, what should I do to further debug this?

Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?

Thanks.

kernel/cpu.c +9 -0
index 0db8938fbb23..81caab9643b2 100644
@@ -79,6 +79,15 @@ static struct lockdep_map cpuhp_state_up_map =
 static struct lockdep_map cpuhp_state_down_map =
 	STATIC_LOCKDEP_MAP_INIT("cpuhp_state-down", &cpuhp_state_down_map);
 
+int cpuhp_current_state(int cpu)
+{
+	return per_cpu_ptr(&cpuhp_state, cpu)->state;
+}
+
+int cpuhp_target_state(int cpu)
+{
+	return per_cpu_ptr(&cpuhp_state, cpu)->target;
+}
 
 static inline void cpuhp_lock_acquire(bool bringup)
 {
kernel/workqueue.c +11 -1
index 78b192071ef7..365cf6342808 100644
@@ -1712,6 +1712,9 @@ static struct worker *alloc_worker(int node)
 	return worker;
 }
 
+int cpuhp_current_state(int cpu);
+int cpuhp_target_state(int cpu);
+
 /**
  * worker_attach_to_pool() - attach a worker to a pool
  * @worker: worker to be attached
@@ -1724,13 +1727,20 @@ static struct worker *alloc_worker(int node)
 static void worker_attach_to_pool(struct worker *worker,
 				   struct worker_pool *pool)
 {
+	int ret;
+
 	mutex_lock(&wq_pool_attach_mutex);
 
 	/*
 	 * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any
 	 * online CPUs.  It'll be re-applied when any of the CPUs come up.
 	 */
-	set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask);
+	ret = set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask);
+	if (ret && pool->cpu >= 0 && worker->rescue_wq)
+		printk("XXX rescuer failed to attach: ret=%d pool=%d this_cpu=%d target_cpu=%d cpuhp_state=%d chuhp_target=%d\n",
+		       ret, pool->id, raw_smp_processor_id(), pool->cpu,
+		       cpuhp_current_state(pool->cpu),
+		       cpuhp_target_state(pool->cpu));
 
 	/*
 	 * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
#2 lore raw
Quoted 14 lines · Tejun Heo

Hello, Paul.

Sorry about the late reply.

Quoted 9 lines · Paul E. McKenney

I have hit this WARN_ON_ONCE() in process_one_work:

	WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) &&
		     raw_smp_processor_id() != pool->cpu);

This looks like it is my rcu_gp workqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).

Quoted 6 lines
Quoted 2 lines

Is this related to the recent addition of WQ_MEM_RECLAIM? Either way, what should I do to further debug this?

Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?

I will apply this, but be advised that I have not seen that WARN_ON_ONCE() trigger since. :-/

Thanx, Paul

Quoted 59 lines

Thanks.

kernel/cpu.c +9 -0
index 0db8938fbb23..81caab9643b2 100644
@@ -79,6 +79,15 @@ static struct lockdep_map cpuhp_state_up_map =
 static struct lockdep_map cpuhp_state_down_map =
 	STATIC_LOCKDEP_MAP_INIT("cpuhp_state-down", &cpuhp_state_down_map);
 
+int cpuhp_current_state(int cpu)
+{
+	return per_cpu_ptr(&cpuhp_state, cpu)->state;
+}
+
+int cpuhp_target_state(int cpu)
+{
+	return per_cpu_ptr(&cpuhp_state, cpu)->target;
+}
 
 static inline void cpuhp_lock_acquire(bool bringup)
 {
kernel/workqueue.c +11 -1
index 78b192071ef7..365cf6342808 100644
@@ -1712,6 +1712,9 @@ static struct worker *alloc_worker(int node)
 	return worker;
 }
 
+int cpuhp_current_state(int cpu);
+int cpuhp_target_state(int cpu);
+
 /**
  * worker_attach_to_pool() - attach a worker to a pool
  * @worker: worker to be attached
@@ -1724,13 +1727,20 @@ static struct worker *alloc_worker(int node)
 static void worker_attach_to_pool(struct worker *worker,
 				   struct worker_pool *pool)
 {
+	int ret;
+
 	mutex_lock(&wq_pool_attach_mutex);
 
 	/*
 	 * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any
 	 * online CPUs.  It'll be re-applied when any of the CPUs come up.
 	 */
-	set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask);
+	ret = set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask);
+	if (ret && pool->cpu >= 0 && worker->rescue_wq)
+		printk("XXX rescuer failed to attach: ret=%d pool=%d this_cpu=%d target_cpu=%d cpuhp_state=%d chuhp_target=%d\n",
+		       ret, pool->id, raw_smp_processor_id(), pool->cpu,
+		       cpuhp_current_state(pool->cpu),
+		       cpuhp_target_state(pool->cpu));
 
 	/*
 	 * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
#3 lore raw
Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Tejun Heo

Hello, Paul.

Sorry about the late reply.

Quoted 11 lines · Paul E. McKenney

I have hit this WARN_ON_ONCE() in process_one_work:

	WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) &&
		     raw_smp_processor_id() != pool->cpu);

This looks like it is my rcu_gp workqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).

Is this related to the recent addition of WQ_MEM_RECLAIM? Either way, what should I do to further debug this?

Quoted 6 lines
Quoted 3 lines

Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?

I will apply this, but be advised that I have not seen that WARN_ON_ONCE() trigger since. :-/

But I get a build error:

kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference to cpuhp_target_state' workqueue.c:(.text+0x647): undefined reference to cpuhp_current_state' /home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failed

My guess is that the worker_attach_to_pool() code needs to change so as to invoke cpuhp_current_state() and cpuhp_target_state() only in CONFIG_HOTPLUG_CPU=y kernels, but I figured I should defer to you.

Thanx, Paul

maintainer
#4 lore raw

Hello, Paul.

Quoted 5 lines · Paul E. McKenney
Quoted 2 lines

I will apply this, but be advised that I have not seen that WARN_ON_ONCE() trigger since. :-/

But I get a build error:

Quoted 8 lines

kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference to cpuhp_target_state' workqueue.c:(.text+0x647): undefined reference to cpuhp_current_state' /home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failed

My guess is that the worker_attach_to_pool() code needs to change so as to invoke cpuhp_current_state() and cpuhp_target_state() only in CONFIG_HOTPLUG_CPU=y kernels, but I figured I should defer to you.

Just to clarify, the bug happened on a kernel with cpuhp enabled, right?

Thanks.

Signature · 1 line
tejun
#5 lore raw
Quoted 13 lines · Tejun Heo

Hello, Paul.

Quoted 10 lines · Paul E. McKenney
Quoted 2 lines

I will apply this, but be advised that I have not seen that WARN_ON_ONCE() trigger since. :-/

But I get a build error:

kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference to cpuhp_target_state' workqueue.c:(.text+0x647): undefined reference to cpuhp_current_state' /home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failed

Quoted 6 lines
Quoted 3 lines

My guess is that the worker_attach_to_pool() code needs to change so as to invoke cpuhp_current_state() and cpuhp_target_state() only in CONFIG_HOTPLUG_CPU=y kernels, but I figured I should defer to you.

Just to clarify, the bug happened on a kernel with cpuhp enabled, right?

It seems to be a bit more complicated than that, despite my initial confidence that I knew what was going on. The build failed for these rcutorture scenarios:

SRCU-N: (No idea!)

CONFIG_RCU_TRACE=n
CONFIG_SMP=y
CONFIG_NR_CPUS=4
CONFIG_HOTPLUG_CPU=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=n

SRCU-t:  (Expected, CONFIG_SMP=n)
SRCU-u:  (Expected, CONFIG_SMP=n)
TASKS02:  (Expected, CONFIG_SMP=n)
TASKS03:  (Expected, CONFIG_HOTPLUG_CPU=n)
TINY01:  (Expected, CONFIG_SMP=n)
TINY02:  (Expected, CONFIG_SMP=n)
TREE01:  (No idea!)

CONFIG_SMP=y
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_RCU_TRACE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_MAXSMP=y
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_RCU_NOCB_CPU=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_RCU_BOOST=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y

TREE02 didn't fail despite having CONFIG_HOTPLUG_CPU=n!

TREE03:  (Expected, CONFIG_HOTPLUG_CPU=n)
TREE04:  (Expected, CONFIG_HOTPLUG_CPU=n)

TREE07: (No idea!)

	CONFIG_NR_CPUS=16
	CONFIG_PREEMPT_NONE=y
	CONFIG_PREEMPT_VOLUNTARY=n
	CONFIG_PREEMPT=n
	CONFIG_HZ_PERIODIC=n
	CONFIG_NO_HZ_IDLE=n
	CONFIG_NO_HZ_FULL=y
	CONFIG_RCU_FAST_NO_HZ=n
	CONFIG_RCU_TRACE=y
	CONFIG_HOTPLUG_CPU=y
	CONFIG_RCU_FANOUT=2
	CONFIG_RCU_FANOUT_LEAF=2
	CONFIG_DEBUG_LOCK_ALLOC=n
	CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
	CONFIG_RCU_EXPERT=y
TREE08:  (Expected, CONFIG_HOTPLUG_CPU=n)
TREE09:  (Expected, CONFIG_SMP=n)

Thoughts?

Thanx, Paul