workqueue: Fix migrate_disable hotplug changes vs kworker affinity
cover workqueue: Fix migrate_disable hotplug changes vs kworker affinity
Hi folks,
This should fix the issue reported by Qian at [1]. Dietmar mentioned some other issue with hotplug & deadline tasks, but that's being investigated by someone else ATM.
I would like to mention I suspect this bug comes straight from $hell, as I've never ever had to fight off so many (mostly unrelated) issues while looking into it. Distro kernel being mangled, git tree corruption, periods of heisenbugism...
Cheers,
Valentin
[1]: https://lore.kernel.org/r/[email protected]
Valentin Schneider (2):
stop_machine: Add caller debug info to queue_stop_cpus_work
workqueue: Fix affinity of kworkers attached during late hotplug
kernel/stop_machine.c | 1 + kernel/workqueue.c | 24 +++++++++++++++++------- 2 files changed, 18 insertions(+), 7 deletions(-)
--
2.27.0
PATCH 1/2 stop_machine: Add caller debug info to queue_stop_cpus_work
Most callsites were covered by commit
a8b62fd08505 ("stop_machine: Add function and caller debug info")
but this skipped queue_stop_cpus_work(). Add caller debug info to it.
--- kernel/stop_machine.c | 1 + 1 file changed, 1 insertion(+)
kernel/stop_machine.c +1 -0
@@ -409,6 +409,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask, work->fn = fn; work->arg = arg; work->done = done; + work->caller = _RET_IP_;if (cpu_stop_queue_work (cpu, work)) queued = true; }
Signature · 1 line
2.27.0
[tip: sched/core] stop_machine: Add caller debug info to queue_stop_cpus_work
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2a2f80ff63bc36a874ed569bcaef932a8fe43514 Gitweb: https://git.kernel.org/tip/2a2f80ff63bc36a874ed569bcaef932a8fe43514 Author: Valentin Schneider <[email protected]> AuthorDate: Thu, 10 Dec 2020 16:38:29 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 23 Mar 2021 16:01:58 +01:00
stop_machine: Add caller debug info to queue_stop_cpus_work
Most callsites were covered by commit
a8b62fd08505 ("stop_machine: Add function and caller debug info")
but this skipped queue_stop_cpus_work(). Add caller debug info to it.
--- kernel/stop_machine.c | 1 + 1 file changed, 1 insertion(+)
kernel/stop_machine.c +1 -0
@@ -409,6 +409,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask, work->fn = fn; work->arg = arg; work->done = done; + work->caller = _RET_IP_;if (cpu_stop_queue_work (cpu, work)) queued = true; }
PATCH 2/2 workqueue: Fix affinity of kworkers attached during late hotplug
In reply to Valentin SchneiderPer-CPU kworkers forcefully migrated away by hotplug via workqueue_offline_cpu() can end up spawning more kworkers via
manage_workers() -> maybe_create_worker()
Workers created at this point will be bound using
pool->attrs->cpumask
which in this case is wrong, as the hotplug state machine already migrated all pinned kworkers away from this CPU. This ends up triggering the BUG_ON condition is sched_cpu_dying() (i.e. there's a kworker enqueued on the dying rq).
Special-case workers being attached to DISASSOCIATED pools and bind them to cpu_active_mask, mimicking them being present when workqueue_offline_cpu() was invoked.
--- kernel/workqueue.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
kernel/workqueue.c +17 -7
@@ -1848,19 +1848,29 @@ static void worker_attach_to_pool(struct worker *worker, {mutex_lock (&wq_pool_attach_mutex); -/* -* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any -* online CPUs. It'll be re-applied when any of the CPUs come up. -*/ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); -/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains * stable across this function. See the comments above the flag * definition for details. +* +* Worker might get attached to a pool *after* workqueue_offline_cpu() +* was run - e.g. created by manage_workers() from a kworker which was +* forcefully moved away by hotplug. Kworkers created from this point on +* need to have their affinity changed as if they were present during +* workqueue_offline_cpu(). +* +* This will be resolved in rebind_workers(). */ -if (pool->flags &POOL_DISASSOCIATED ) +if (pool->flags &POOL_DISASSOCIATED ) { worker->flags |=WORKER_UNBOUND ; +set_cpus_allowed_ptr (worker->task, cpu_active_mask); + }else { +/* +* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any +* online CPUs. It'll be re-applied when any of the CPUs come up. +*/ +set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); + }list_add_tail (&worker->node, &pool->workers); worker->pool = pool;
Signature · 1 line
2.27.0
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
Hi Valentin,
Quoted 14 lines · Valentin Schneider
Per-CPU kworkers forcefully migrated away by hotplug via
workqueue_offline_cpu()can end up spawning more kworkers via
manage_workers()->maybe_create_worker()Workers created at this point will be bound using
pool->attrs->cpumaskwhich in this case is wrong, as the hotplug state machine already migrated all pinned kworkers away from this CPU. This ends up triggering the
BUG_ONcondition issched_cpu_dying()(i.e. there's a kworker enqueued on the dying rq).
Quoted 6 lines
Special-case workers being attached to DISASSOCIATED pools and bind them to
cpu_active_mask, mimicking them being present whenworkqueue_offline_cpu()was invoked.Fixes 06249738a41a ("workqueue: Manually break affinity on hotplug")
Isn't the problem introduced by 1cf12e0 ("sched/hotplug: Consolidate task migration on CPU unplug") ?
Previously we had:
AP_WORKQUEUE_ONLINE -> set POOL_DISASSOCIATED
...
TEARDOWN_CPU -> clear CPU in cpu_online_mask
| |-AP_SCHED_STARTING -> migrate_tasks() |
AP_OFFLINE
worker_attach_to_pool(), is "protected" by the cpu_online_mask in set_cpus_allowed_ptr(). IIUC, now, the tasks being migrated before the cpu_online_mask is actually flipped, there's a window, between CPUHP_AP_SCHED_WAIT_EMPTY and CPUHP_TEARDOWN_CPU where a kworker can wake-up a new one, for the hotunplugged pool that wouldn't be caught by the hotunplug migration.
Quoted 32 lines
Reported-by Qian Cai <[email protected]>Signed-off-by Valentin Schneider <[email protected]>--- kernel/workqueue.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)kernel/workqueue.c +8
-6@@ -1848,19 +1848,29 @@ static void worker_attach_to_pool(struct worker *worker, {mutex_lock (&wq_pool_attach_mutex); -/* -* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any -* online CPUs. It'll be re-applied when any of the CPUs come up. -*/ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); -/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains * stable across this function. See the comments above the flag * definition for details. +* +* Worker might get attached to a pool *after* workqueue_offline_cpu() +* was run - e.g. created by manage_workers() from a kworker which was +* forcefully moved away by hotplug. Kworkers created from this point on +* need to have their affinity changed as if they were present during +* workqueue_offline_cpu(). +* +* This will be resolved in rebind_workers().
kernel/workqueue.c static void worker_attach_to_pool(struct worker *worker, · line 1871
Quoted 10 lines
kernel/workqueue.c +7
-1*/ -if (pool->flags &POOL_DISASSOCIATED ) +if (pool->flags &POOL_DISASSOCIATED ) { worker->flags |=WORKER_UNBOUND ; +set_cpus_allowed_ptr (worker->task, cpu_active_mask); + }else { +/* +* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any +* online CPUs. It'll be re-applied when any of the CPUs come up. +*/
Does this comment still stand ? IIUC, we should always be in the POOL_DISASSOCIATED case if the CPU from cpumask is offline. Unless a pool->attrs->cpumask can have several CPUs. In that case maybe we should check for the cpu_active_mask here too ?
Signature · 1 line
Vincent
Quoted 8 lines
kernel/workqueue.c +2
-0+set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); + }list_add_tail (&worker->node, &pool->workers); worker->pool = pool;Signature · 1 line
2.27.0
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
Quoted 35 lines · Vincent Donnefort
Hi Valentin,
Quoted 14 lines · Valentin Schneider
Per-CPU kworkers forcefully migrated away by hotplug via
workqueue_offline_cpu()can end up spawning more kworkers via
manage_workers()->maybe_create_worker()Workers created at this point will be bound using
pool->attrs->cpumaskwhich in this case is wrong, as the hotplug state machine already migrated all pinned kworkers away from this CPU. This ends up triggering the
BUG_ONcondition issched_cpu_dying()(i.e. there's a kworker enqueued on the dying rq).Quoted 6 lines
Special-case workers being attached to DISASSOCIATED pools and bind them to
cpu_active_mask, mimicking them being present whenworkqueue_offline_cpu()was invoked.Fixes 06249738a41a ("workqueue: Manually break affinity on hotplug")Isn't the problem introduced by 1cf12e0 ("sched/hotplug: Consolidate task migration on CPU unplug") ?
Previously we had:
AP_WORKQUEUE_ONLINE-> setPOOL_DISASSOCIATED
...
TEARDOWN_CPU-> clear CPU incpu_online_mask| |-AP_SCHED_STARTING -> migrate_tasks() |
Quoted 8 lines
AP_OFFLINE
worker_attach_to_pool(), is "protected" by thecpu_online_maskinset_cpus_allowed_ptr(). IIUC, now, the tasks being migrated before thecpu_online_maskis actually flipped, there's a window, betweenCPUHP_AP_SCHED_WAIT_EMPTYandCPUHP_TEARDOWN_CPUwhere a kworker can wake-up a new one, for the hotunplugged pool that wouldn't be caught by the hotunplug migration.
Yes, very much so, however the commit Valentin picked was supposed to preemptively fix this. So we can consider this a fix for the fix.
But I don't mind an alternative or perhaps even second Fixes tag on this.
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
In reply to Vincent DonnefortQuoted 14 lines · Vincent Donnefort
kernel/workqueue.c static void worker_attach_to_pool(struct worker *worker, · line 1871
Quoted 8 lines · Valentin Schneider
kernel/workqueue.c +7
-0+if (pool->flags &POOL_DISASSOCIATED ) { worker->flags |=WORKER_UNBOUND ; +set_cpus_allowed_ptr (worker->task, cpu_active_mask); + }else { +/* +* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any +* online CPUs. It'll be re-applied when any of the CPUs come up. +*/ Does this comment still stand ? IIUC, we should always be in the
POOL_DISASSOCIATEDcase if the CPU from cpumask is offline. Unless apool->attrs->cpumaskcan have several CPUs. In that case maybe we should check for thecpu_active_maskhere too ?
IIUC it can be a numa mask, and would still be valid in that case.
Quoted 2 lines
Quoted 2 lines
kernel/workqueue.c +2
-0+set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); + }
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
In reply to Vincent DonnefortHi Vincent,
Quoted 16 lines · Vincent Donnefort
Hi Valentin,
Quoted 1 line · Valentin Schneider
Fixes 06249738a41a ("workqueue: Manually break affinity on hotplug")Isn't the problem introduced by 1cf12e0 ("sched/hotplug: Consolidate task migration on CPU unplug") ?
Previously we had:
AP_WORKQUEUE_ONLINE-> setPOOL_DISASSOCIATED
...
TEARDOWN_CPU-> clear CPU incpu_online_mask| |-AP_SCHED_STARTING -> migrate_tasks() |
Quoted 9 lines
AP_OFFLINE
worker_attach_to_pool(), is "protected" by thecpu_online_maskinset_cpus_allowed_ptr(). IIUC, now, the tasks being migrated before thecpu_online_maskis actually flipped, there's a window, betweenCPUHP_AP_SCHED_WAIT_EMPTYandCPUHP_TEARDOWN_CPUwhere a kworker can wake-up a new one, for the hotunplugged pool that wouldn't be caught by the hotunplug migration.
You're right, the splat should only happen with that other commit. That said, this fix complements the one referred to in Fixes:, which is the "logic" I went for.
Quoted 38 lines
Quoted 38 lines
Reported-by Qian Cai <[email protected]>Signed-off-by Valentin Schneider <[email protected]>--- kernel/workqueue.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)kernel/workqueue.c +11
-7@@ -1848,19 +1848,29 @@ static void worker_attach_to_pool(struct worker *worker, {mutex_lock (&wq_pool_attach_mutex); -/* -* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any -* online CPUs. It'll be re-applied when any of the CPUs come up. -*/ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); -/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains * stable across this function. See the comments above the flag * definition for details. +* +* Worker might get attached to a pool *after* workqueue_offline_cpu() +* was run - e.g. created by manage_workers() from a kworker which was +* forcefully moved away by hotplug. Kworkers created from this point on +* need to have their affinity changed as if they were present during +* workqueue_offline_cpu(). +* +* This will be resolved in rebind_workers(). */ -if (pool->flags &POOL_DISASSOCIATED ) +if (pool->flags &POOL_DISASSOCIATED ) { worker->flags |=WORKER_UNBOUND ; +set_cpus_allowed_ptr (worker->task, cpu_active_mask); + }else {
Quoted 8 lines
kernel/workqueue.c static void worker_attach_to_pool(struct worker *worker, · line 1871
Quoted 4 lines
kernel/workqueue.c +4
-0+/* +* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any +* online CPUs. It'll be re-applied when any of the CPUs come up. +*/ Does this comment still stand ? IIUC, we should always be in the
POOL_DISASSOCIATEDcase if the CPU from cpumask is offline. Unless apool->attrs->cpumaskcan have several CPUs.
AIUI that should the case for unbound pools
Quoted 1 line
In that case maybe we should check for the
cpu_active_maskhere too ?
Looking at it again, I think we might need to.
IIUC you can end up with pools bound to a single NUMA node (?). In that case, say the last CPU of a node is going down, then:
workqueue_offline_cpu()
wq_update_unbound_numa()
alloc_unbound_pwq()
get_unbound_pool()
would still pick that node, because it doesn't look at the online / active mask. And at this point, we would affine the
kworkers to that node, and we're back to having kworkers enqueued on a (!active, online) CPU that is going down...
The annoying thing is we can't just compare attrs->cpumask with cpu_active_mask, because workqueue_offline_cpu() happens a few steps below sched_cpu_deactivate() (CPUHP_AP_ACTIVE):
CPUHP_ONLINE -> CPUHP_AP_ACTIVE # CPU X is !active
# Some new kworker gets created here
worker_attach_to_pool()
!cpumask_subset(attrs->cpumask, cpu_active_mask)
-> affine worker to active CPUs CPUHP_AP_ACTIVE -> CPUHP_ONLINE # CPU X is active # Nothing will ever correct the kworker's affinity :(
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
Quoted 12 lines · Valentin Schneider
Quoted 1 line
In that case maybe we should check for the
cpu_active_maskhere too ?Looking at it again, I think we might need to.
IIUC you can end up with pools bound to a single NUMA node (?). In that case, say the last CPU of a node is going down, then:
workqueue_offline_cpu()
wq_update_unbound_numa()
alloc_unbound_pwq()
get_unbound_pool()
Quoted 4 lines
would still pick that node, because it doesn't look at the online / active mask. And at this point, we would affine the
kworkers to that node, and we're back to having kworkers enqueued on a (!active, online) CPU that is going down...
Assuming a node covers at least 2 CPUs, that can't actually happen per is_cpu_allowed().
Re: [PATCH 2/2] workqueue: Fix affinity of kworkers attached during late hotplug
Quoted 13 lines · Valentin Schneider
Quoted 12 lines · Valentin Schneider
Quoted 1 line
In that case maybe we should check for the
cpu_active_maskhere too ?Looking at it again, I think we might need to.
IIUC you can end up with pools bound to a single NUMA node (?). In that case, say the last CPU of a node is going down, then:
workqueue_offline_cpu()
wq_update_unbound_numa()
alloc_unbound_pwq()
get_unbound_pool()
Quoted 7 lines
Quoted 4 lines
would still pick that node, because it doesn't look at the online / active mask. And at this point, we would affine the
kworkers to that node, and we're back to having kworkers enqueued on a (!active, online) CPU that is going down...Assuming a node covers at least 2 CPUs, that can't actually happen per
is_cpu_allowed().
Yes indeed, my bad, no problem here.
Quoted 7 lines · Valentin Schneider
Valentin Schneider (2):
stop_machine: Add caller debug info toqueue_stop_cpus_work
workqueue: Fix affinity of kworkers attached during late hotplugkernel/stop_machine.c | 1 + kernel/workqueue.c | 24 +++++++++++++++++------- 2 files changed, 18 insertions(+), 7 deletions(-)
TJ, would you be okay if I take these through the sched tree that contain the migrate_disable() patches, such that problem and fix stay together?