WARN_ON_ONCE() in process_one_work()?
Subsystems (2)
Hello!
I have hit this WARN_ON_ONCE() in process_one_work:
WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu);
This looks like it is my rcu_gp workqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).
Is this related to the recent addition of WQ_MEM_RECLAIM? Either way, what should I do to further debug this?
Thanx, Paul
[ 828.791694] WARNING: CPU: 2 PID: 3 at /home/paulmck/public_git/linux-rcu/kernel/workqueue.c:2069 process_one_work+0x49/0x3c0 [ 828.794850] Modules linked in: [ 828.795748] CPU: 2 PID: 3 Comm: rcu_gp Not tainted 4.18.0-rc1+ #1 [ 828.797436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 828.800059] Workqueue: (null) (rcu_gp) [ 828.801373] RIP: 0010:process_one_work+0x49/0x3c0 [ 828.802690] Code: 00 00 00 00 4c 0f 44 e0 49 8b 44 24 08 44 8b a8 00 01 00 00 41 83 e5 20 f6 45 10 04 75 0e 65 8b 05 6c 99 79 68 3b 45 04 74 02 <0f> 0b 48 ba eb 83 b5 80 46 86 c8 61 48 0f af d6 48 c1 ea 3a 48 8b [ 828.807871] RSP: 0000:ffff8a3c400e7e70 EFLAGS: 00010002 [ 828.809314] RAX: 0000000000000002 RBX: ffff88acdfc206e0 RCX: ffff88acdfc206e0 [ 828.811307] RDX: ffff88acdfc206d8 RSI: ffff88acdfc197b8 RDI: ffff88acde80c6c0 [ 828.813253] RBP: ffff88acdfc206c0 R08: 0000000000000000 R09: 0000000000000000 [ 828.815239] R10: ffff8a3c400e7e48 R11: 0000000000000000 R12: ffff88acdfc24900 [ 828.817073] R13: 0000000000000000 R14: ffff88acde80c6c0 R15: ffff88acde80c6e8 [ 828.819071] FS: 0000000000000000(0000) GS:ffff88acdfd00000(0000) knlGS:0000000000000000 [ 828.821361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 828.822971] CR2: 00000000000000b0 CR3: 000000001040a000 CR4: 00000000000006e0 [ 828.824935] Call Trace: [ 828.825653] rescuer_thread+0x201/0x320 [ 828.826744] kthread+0xf3/0x130 [ 828.827618] ? process_one_work+0x3c0/0x3c0 [ 828.828758] ? kthread_destroy_worker+0x40/0x40 [ 828.830019] ret_from_fork+0x35/0x40 [ 828.831028] ---[ end trace 4197eb078bf0edac ]---
Tejun Heo
maintainerHello, Paul.
Sorry about the late reply.
Quoted 11 lines · Paul E. McKenney
I have hit this
WARN_ON_ONCE()inprocess_one_work:WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu);This looks like it is my
rcu_gpworkqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).Is this related to the recent addition of
WQ_MEM_RECLAIM? Either way, what should I do to further debug this?
Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?
Thanks.
kernel/cpu.c +9 -0
@@ -79,6 +79,15 @@ static struct lockdep_map cpuhp_state_up_map =static struct lockdep_map cpuhp_state_down_map =STATIC_LOCKDEP_MAP_INIT ("cpuhp_state-down" , &cpuhp_state_down_map); +int cpuhp_current_state (int cpu) +{ +return per_cpu_ptr (&cpuhp_state, cpu)->state; +} + +int cpuhp_target_state (int cpu) +{ +return per_cpu_ptr (&cpuhp_state, cpu)->target; +}static inline void cpuhp_lock_acquire (bool bringup) {
kernel/workqueue.c +11 -1
@@ -1712,6 +1712,9 @@ static struct worker *alloc_worker(int node)return worker; } +int cpuhp_current_state (int cpu); +int cpuhp_target_state (int cpu); +/** * worker_attach_to_pool() - attach a worker to a pool * @worker: worker to be attached
@@ -1724,13 +1727,20 @@ static struct worker *alloc_worker(int node)static void worker_attach_to_pool (struct worker *worker,struct worker_pool *pool) { +int ret; +mutex_lock (&wq_pool_attach_mutex);/* * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any * online CPUs. It'll be re-applied when any of the CPUs come up. */ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); + ret =set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); +if (ret && pool->cpu >=0 && worker->rescue_wq) +printk ("XXX rescuer failed to attach: ret=%d pool=%d this_cpu=%d target_cpu=%d cpuhp_state=%d chuhp_target=%d\n" , + ret, pool->id,raw_smp_processor_id (), pool->cpu, +cpuhp_current_state (pool->cpu), +cpuhp_target_state (pool->cpu));/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
Quoted 14 lines · Tejun Heo
Hello, Paul.
Sorry about the late reply.
Quoted 9 lines · Paul E. McKenney
I have hit this
WARN_ON_ONCE()inprocess_one_work:WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu);This looks like it is my
rcu_gpworkqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).
Quoted 6 lines
Quoted 2 lines
Is this related to the recent addition of
WQ_MEM_RECLAIM? Either way, what should I do to further debug this?Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?
I will apply this, but be advised that I have not seen that WARN_ON_ONCE() trigger since. :-/
Thanx, Paul
Quoted 59 lines
Thanks.
kernel/cpu.c +9
-0@@ -79,6 +79,15 @@ static struct lockdep_map cpuhp_state_up_map =static struct lockdep_map cpuhp_state_down_map =STATIC_LOCKDEP_MAP_INIT ("cpuhp_state-down" , &cpuhp_state_down_map); +int cpuhp_current_state (int cpu) +{ +return per_cpu_ptr (&cpuhp_state, cpu)->state; +} + +int cpuhp_target_state (int cpu) +{ +return per_cpu_ptr (&cpuhp_state, cpu)->target; +}static inline void cpuhp_lock_acquire (bool bringup) {kernel/workqueue.c +11
-1@@ -1712,6 +1712,9 @@ static struct worker *alloc_worker(int node)return worker; } +int cpuhp_current_state (int cpu); +int cpuhp_target_state (int cpu); +/** * worker_attach_to_pool() - attach a worker to a pool * @worker: worker to be attached @@ -1724,13 +1727,20 @@ static struct worker *alloc_worker(int node)static void worker_attach_to_pool (struct worker *worker,struct worker_pool *pool) { +int ret; +mutex_lock (&wq_pool_attach_mutex);/* * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any * online CPUs. It'll be re-applied when any of the CPUs come up. */ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); + ret =set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); +if (ret && pool->cpu >=0 && worker->rescue_wq) +printk ("XXX rescuer failed to attach: ret=%d pool=%d this_cpu=%d target_cpu=%d cpuhp_state=%d chuhp_target=%d\n" , + ret, pool->id,raw_smp_processor_id (), pool->cpu, +cpuhp_current_state (pool->cpu), +cpuhp_target_state (pool->cpu));/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Tejun Heo
Hello, Paul.
Sorry about the late reply.
Quoted 11 lines · Paul E. McKenney
I have hit this
WARN_ON_ONCE()inprocess_one_work:WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu);This looks like it is my
rcu_gpworkqueue (see splat below), and it appears to be intermittent. This happens on rcutorture scenario SRCU-N, which does random CPU-hotplug operations (in case that helps).Is this related to the recent addition of
WQ_MEM_RECLAIM? Either way, what should I do to further debug this?
Quoted 6 lines
Quoted 3 lines
Hmm... I checked the code paths but couldn't spot anything suspicious. Can you please apply the following patch and see whether it triggers before hitting the warn and if so report what it says?
I will apply this, but be advised that I have not seen that
WARN_ON_ONCE()trigger since. :-/
But I get a build error:
kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference to cpuhp_target_state' workqueue.c:(.text+0x647): undefined reference to cpuhp_current_state' /home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failed
My guess is that the worker_attach_to_pool() code needs to change so as to invoke cpuhp_current_state() and cpuhp_target_state() only in CONFIG_HOTPLUG_CPU=y kernels, but I figured I should defer to you.
Thanx, Paul
Tejun Heo
maintainerHello, Paul.
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines
I will apply this, but be advised that I have not seen that
WARN_ON_ONCE()trigger since. :-/But I get a build error:
Quoted 8 lines
kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference tocpuhp_target_state' workqueue.c:(.text+0x647): undefined reference tocpuhp_current_state'/home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failedMy guess is that the
worker_attach_to_pool()code needs to change so as to invokecpuhp_current_state()andcpuhp_target_state()only inCONFIG_HOTPLUG_CPU=ykernels, but I figured I should defer to you.
Just to clarify, the bug happened on a kernel with cpuhp enabled, right?
Thanks.
Signature · 1 line
tejun
Quoted 13 lines · Tejun Heo
Hello, Paul.
Quoted 10 lines · Paul E. McKenney
Quoted 2 lines
I will apply this, but be advised that I have not seen that
WARN_ON_ONCE()trigger since. :-/But I get a build error:
kernel/workqueue.o: In function `worker_attach_to_pool':
workqueue.c:(.text+0x63c): undefined reference tocpuhp_target_state' workqueue.c:(.text+0x647): undefined reference tocpuhp_current_state'/home/paulmck/public_git/linux-rcu/Makefile:1015: recipe for target 'vmlinux' failed
Quoted 6 lines
Quoted 3 lines
My guess is that the
worker_attach_to_pool()code needs to change so as to invokecpuhp_current_state()andcpuhp_target_state()only inCONFIG_HOTPLUG_CPU=ykernels, but I figured I should defer to you.Just to clarify, the bug happened on a kernel with cpuhp enabled, right?
It seems to be a bit more complicated than that, despite my initial confidence that I knew what was going on. The build failed for these rcutorture scenarios:
SRCU-N: (No idea!)
CONFIG_RCU_TRACE=n
CONFIG_SMP=y
CONFIG_NR_CPUS=4
CONFIG_HOTPLUG_CPU=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=n
SRCU-t: (Expected, CONFIG_SMP=n) SRCU-u: (Expected, CONFIG_SMP=n) TASKS02: (Expected, CONFIG_SMP=n) TASKS03: (Expected, CONFIG_HOTPLUG_CPU=n) TINY01: (Expected, CONFIG_SMP=n) TINY02: (Expected, CONFIG_SMP=n) TREE01: (No idea!)
CONFIG_SMP=y
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
CONFIG_HZ_PERIODIC=n
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_RCU_TRACE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_MAXSMP=y
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_RCU_NOCB_CPU=y
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_RCU_BOOST=n
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y
TREE02 didn't fail despite having CONFIG_HOTPLUG_CPU=n!
TREE03: (Expected, CONFIG_HOTPLUG_CPU=n) TREE04: (Expected, CONFIG_HOTPLUG_CPU=n)
TREE07: (No idea!)
CONFIG_NR_CPUS=16 CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY=n CONFIG_PREEMPT=n CONFIG_HZ_PERIODIC=n CONFIG_NO_HZ_IDLE=n CONFIG_NO_HZ_FULL=y CONFIG_RCU_FAST_NO_HZ=n CONFIG_RCU_TRACE=y CONFIG_HOTPLUG_CPU=y CONFIG_RCU_FANOUT=2 CONFIG_RCU_FANOUT_LEAF=2 CONFIG_DEBUG_LOCK_ALLOC=n CONFIG_DEBUG_OBJECTS_RCU_HEAD=n CONFIG_RCU_EXPERT=y
TREE08: (Expected, CONFIG_HOTPLUG_CPU=n) TREE09: (Expected, CONFIG_SMP=n)
Thoughts?
Thanx, Paul