Skip to contentSkip to conversation

cpu/hotplug: Prevent offlining the last domain housekeeping CPU

PATCH In review Guopeng Zhang 2 participants 2 messages Aug 11 CPU hotplug

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 cpu/hotplug: Prevent offlining the last domain housekeeping CPU

From: Guopeng Zhang <[email protected]>

Commit 38685e2a0476 ("cpu/hotplug: Don't offline the last non-isolated CPU") kept at least one HK_TYPE_DOMAIN CPU online by selecting a CPU other than the one being offlined. Commit de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"") moved the check into _cpu_down(), but lost that exclusion.

The replacement cpumask_any_and() check still sees the outgoing CPU in cpu_online_mask. The target therefore satisfies the check when it is the last online CPU in the HK_TYPE_DOMAIN mask.

On a system booted with domain isolation, a root-initiated attempt to offline the last HK_TYPE_DOMAIN CPU leaves scheduler-domain rebuilds with an empty span. With panic_on_warn=0 and warn_limit=0, this first triggered the WARN_ON() in build_sched_domains(). The kernel then crashed with a general protection fault in build_perf_domains():

WARNING: ... at build_sched_domains+0x443/0xa60
...

 Oops: general protection fault ...
 RIP: 0010:build_perf_domains+0x40/0x230
 Call Trace:
  <TASK>
  partition_sched_domains_locked+0x3d9/0x710
  partition_sched_domains+0x30/0x40
  cpuset_reset_sched_domains+0x25/0x40
  sched_cpu_deactivate+0x2f2/0x300
  cpuhp_invoke_callback+0x1a4/0x780
  cpuhp_thread_fun+0x1b4/0x230
  </TASK>

Use cpumask_any_and_but() to exclude the outgoing CPU and return -EBUSY when no other online HK_TYPE_DOMAIN CPU remains.

Skip this check when tasks_frozen is set because CPU freeze deliberately takes all non-primary CPUs offline.

The sysfs CPU-offline path was tested on a 32-vCPU Ubuntu 24.04 guest booted with:

    isolcpus=domain,0,3-31

The unpatched kernel warned in build_sched_domains() and then crashed with a general protection fault in build_perf_domains(). With this patch, the request was rejected with -EBUSY, the target CPU remained online, and no scheduler-domain warning was observed.

Fixes de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"")

---
Changes in v2:
- Explain that 38685e2a0476 excluded the outgoing CPU and that de715325cc47 lost that exclusion when moving the check.
- Report the observed WARN_ON() followed by a GPF in build_perf_domains(), and include the relevant splat.
- Skip the regular-hotplug guard while tasks_frozen is set, as suggested by Bradley.
- State the root and domain-isolation requirements.
- Retest the frozen path with the same isolcpus setting. The new guard no longer returns -EBUSY.

v1: https://lore.kernel.org/all/[email protected]/

 kernel/cpu.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/cpu.c +6 -4
index b3c8553d7bd6..8ffbfc20b8fe 100644
@@ -1415,11 +1415,13 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,
 	cpus_write_lock();
 
 	/*
-	 * Keep at least one housekeeping cpu onlined to avoid generating
-	 * an empty sched_domain span.
+	 * Keep at least one HK_TYPE_DOMAIN CPU online during regular hotplug
+	 * to avoid generating an empty sched_domain span.
 	 */
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (!tasks_frozen &&
+	    cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {
 		ret = -EBUSY;
 		goto out;
 	}
Signature · 1 line
2.43.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 43 lines · Guopeng Zhang

From: Guopeng Zhang <[email protected]>

Commit 38685e2a0476 ("cpu/hotplug: Don't offline the last non-isolated CPU") kept at least one HK_TYPE_DOMAIN CPU online by selecting a CPU other than the one being offlined. Commit de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"") moved the check into _cpu_down(), but lost that exclusion.

The replacement cpumask_any_and() check still sees the outgoing CPU in cpu_online_mask. The target therefore satisfies the check when it is the last online CPU in the HK_TYPE_DOMAIN mask.

On a system booted with domain isolation, a root-initiated attempt to offline the last HK_TYPE_DOMAIN CPU leaves scheduler-domain rebuilds with an empty span. With panic_on_warn=0 and warn_limit=0, this first triggered the WARN_ON() in build_sched_domains(). The kernel then crashed with a general protection fault in build_perf_domains():

WARNING: ... at build_sched_domains+0x443/0xa60
...

Oops: general protection fault ...
RIP: 0010:build_perf_domains+0x40/0x230
Call Trace:
 <TASK>
 partition_sched_domains_locked+0x3d9/0x710
 partition_sched_domains+0x30/0x40
 cpuset_reset_sched_domains+0x25/0x40
 sched_cpu_deactivate+0x2f2/0x300
 cpuhp_invoke_callback+0x1a4/0x780
 cpuhp_thread_fun+0x1b4/0x230
 </TASK>

Use cpumask_any_and_but() to exclude the outgoing CPU and return -EBUSY when no other online HK_TYPE_DOMAIN CPU remains.

Skip this check when tasks_frozen is set because CPU freeze deliberately takes all non-primary CPUs offline.

The sysfs CPU-offline path was tested on a 32-vCPU Ubuntu 24.04 guest booted with:

   isolcpus=domain,0,3-31
Quoted 7 lines

The unpatched kernel warned in build_sched_domains() and then crashed with a general protection fault in build_perf_domains(). With this patch, the request was rejected with -EBUSY, the target CPU remained online, and no scheduler-domain warning was observed.

Fixes de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"")
Quoted 31 lines

---
Changes in v2:
- Explain that 38685e2a0476 excluded the outgoing CPU and that de715325cc47 lost that exclusion when moving the check.
- Report the observed WARN_ON() followed by a GPF in build_perf_domains(), and include the relevant splat.
- Skip the regular-hotplug guard while tasks_frozen is set, as suggested by Bradley.
- State the root and domain-isolation requirements.
- Retest the frozen path with the same isolcpus setting. The new guard no longer returns -EBUSY.

v1: https://lore.kernel.org/all/[email protected]/

kernel/cpu.c | 10 ++++++----

1 file changed, 6 insertions(+), 4 deletions(-)
kernel/cpu.c +2 -2
index b3c8553d7bd6..8ffbfc20b8fe 100644
@@ -1415,11 +1415,13 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,
 	cpus_write_lock();
 
 	/*
-	 * Keep at least one housekeeping cpu onlined to avoid generating
-	 * an empty sched_domain span.
+	 * Keep at least one HK_TYPE_DOMAIN CPU online during regular hotplug
+	 * to avoid generating an empty sched_domain span.

kernel/cpu.c static int __ref _cpu_down(unsigned int cpu, int tasks_frozen, · line 1427

Quoted 10 lines
kernel/cpu.c +4 -2
 	 */
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (!tasks_frozen &&
+	    cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {
 		ret = -EBUSY;
 		goto out;
 	}

Thanks!