cpu/hotplug: Prevent offlining the last domain housekeeping CPU
PATCH v2 1/1 cpu/hotplug: Prevent offlining the last domain housekeeping CPU
From: Guopeng Zhang <[email protected]>
Commit 38685e2a0476 ("cpu/hotplug: Don't offline the last non-isolated CPU") kept at least one HK_TYPE_DOMAIN CPU online by selecting a CPU other than the one being offlined. Commit de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"") moved the check into _cpu_down(), but lost that exclusion.
The replacement cpumask_any_and() check still sees the outgoing CPU in cpu_online_mask. The target therefore satisfies the check when it is the last online CPU in the HK_TYPE_DOMAIN mask.
On a system booted with domain isolation, a root-initiated attempt to offline the last HK_TYPE_DOMAIN CPU leaves scheduler-domain rebuilds with an empty span. With panic_on_warn=0 and warn_limit=0, this first triggered the WARN_ON() in build_sched_domains(). The kernel then crashed with a general protection fault in build_perf_domains():
WARNING: ... at build_sched_domains+0x443/0xa60
...
Oops: general protection fault ... RIP: 0010:build_perf_domains+0x40/0x230 Call Trace: <TASK> partition_sched_domains_locked+0x3d9/0x710 partition_sched_domains+0x30/0x40 cpuset_reset_sched_domains+0x25/0x40 sched_cpu_deactivate+0x2f2/0x300 cpuhp_invoke_callback+0x1a4/0x780 cpuhp_thread_fun+0x1b4/0x230 </TASK>
Use cpumask_any_and_but() to exclude the outgoing CPU and return -EBUSY when no other online HK_TYPE_DOMAIN CPU remains.
Skip this check when tasks_frozen is set because CPU freeze deliberately takes all non-primary CPUs offline.
The sysfs CPU-offline path was tested on a 32-vCPU Ubuntu 24.04 guest booted with:
isolcpus=domain,0,3-31
The unpatched kernel warned in build_sched_domains() and then crashed with a general protection fault in build_perf_domains(). With this patch, the request was rejected with -EBUSY, the target CPU remained online, and no scheduler-domain warning was observed.
---
Changes in v2:
- Explain that 38685e2a0476 excluded the outgoing CPU and that de715325cc47 lost that exclusion when moving the check.
- Report the observed WARN_ON() followed by a GPF in build_perf_domains(), and include the relevant splat.
- Skip the regular-hotplug guard while tasks_frozen is set, as suggested by Bradley.
- State the root and domain-isolation requirements.
- Retest the frozen path with the same isolcpus setting. The new guard no longer returns -EBUSY.
v1: https://lore.kernel.org/all/[email protected]/
kernel/cpu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/cpu.c +6 -4
@@ -1415,11 +1415,13 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,cpus_write_lock ();/* -* Keep at least one housekeeping cpu onlined to avoid generating -* an empty sched_domain span. +* Keep at least one HK_TYPE_DOMAIN CPU online during regular hotplug +* to avoid generating an empty sched_domain span. */ -if (cpumask_any_and (cpu_online_mask, -housekeeping_cpumask (HK_TYPE_DOMAIN )) >= nr_cpu_ids) { +if (!tasks_frozen && +cpumask_any_and_but (cpu_online_mask, +housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu) >= nr_cpu_ids) { ret = -EBUSY ; goto out; }
Signature · 1 line
2.43.0
Quoted 43 lines · Guopeng Zhang
From: Guopeng Zhang <[email protected]>
Commit
38685e2a0476("cpu/hotplug: Don't offline the last non-isolated CPU") kept at least oneHK_TYPE_DOMAINCPU online by selecting a CPU other than the one being offlined. Commitde715325cc47("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"") moved the check into_cpu_down(), but lost that exclusion.The replacement
cpumask_any_and()check still sees the outgoing CPU incpu_online_mask. The target therefore satisfies the check when it is the last online CPU in theHK_TYPE_DOMAINmask.On a system booted with domain isolation, a root-initiated attempt to offline the last
HK_TYPE_DOMAINCPU leaves scheduler-domain rebuilds with an empty span. With panic_on_warn=0 and warn_limit=0, this first triggered theWARN_ON()inbuild_sched_domains(). The kernel then crashed with a general protection fault inbuild_perf_domains():WARNING: ... at
build_sched_domains+0x443/0xa60
...Oops: general protection fault ... RIP: 0010:build_perf_domains+0x40/0x230 Call Trace: <TASK> partition_sched_domains_locked+0x3d9/0x710 partition_sched_domains+0x30/0x40 cpuset_reset_sched_domains+0x25/0x40 sched_cpu_deactivate+0x2f2/0x300 cpuhp_invoke_callback+0x1a4/0x780 cpuhp_thread_fun+0x1b4/0x230 </TASK>Use
cpumask_any_and_but()to exclude the outgoing CPU and return-EBUSYwhen no other onlineHK_TYPE_DOMAINCPU remains.Skip this check when
tasks_frozenis set because CPU freeze deliberately takes all non-primary CPUs offline.The sysfs CPU-offline path was tested on a 32-vCPU Ubuntu 24.04 guest booted with:
isolcpus=domain,0,3-31
Quoted 7 lines
The unpatched kernel warned in
build_sched_domains()and then crashed with a general protection fault inbuild_perf_domains(). With this patch, the request was rejected with-EBUSY, the target CPU remained online, and no scheduler-domain warning was observed.Fixes de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"")Suggested-by Bradley Morgan <[email protected]>
Quoted 31 lines
Signed-off-by Guopeng Zhang <[email protected]>---
Changes in v2:
- Explain that38685e2a0476excluded the outgoing CPU and thatde715325cc47lost that exclusion when moving the check.
- Report the observedWARN_ON()followed by a GPF inbuild_perf_domains(), and include the relevant splat.
- Skip the regular-hotplug guard whiletasks_frozenis set, as suggested by Bradley.
- State the root and domain-isolation requirements.
- Retest the frozen path with the same isolcpus setting. The new guard no longer returns-EBUSY.v1: https://lore.kernel.org/all/[email protected]/
kernel/cpu.c| 10 ++++++----1 file changed, 6 insertions(+), 4 deletions(-)kernel/cpu.c +2
-2@@ -1415,11 +1415,13 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,cpus_write_lock ();/* -* Keep at least one housekeeping cpu onlined to avoid generating -* an empty sched_domain span. +* Keep at least one HK_TYPE_DOMAIN CPU online during regular hotplug +* to avoid generating an empty sched_domain span.
kernel/cpu.c static int __ref _cpu_down(unsigned int cpu, int tasks_frozen, · line 1427
Quoted 10 lines
kernel/cpu.c +4
-2*/ -if (cpumask_any_and (cpu_online_mask, -housekeeping_cpumask (HK_TYPE_DOMAIN )) >= nr_cpu_ids) { +if (!tasks_frozen && +cpumask_any_and_but (cpu_online_mask, +housekeeping_cpumask (HK_TYPE_DOMAIN ), + cpu) >= nr_cpu_ids) { ret = -EBUSY ; goto out; }
Thanks!