Skip to contentSkip to conversation

cpu/hotplug: Prevent offlining the last domain housekeeping CPU

PATCH Superseded Guopeng Zhang 2 participants 3 messages Jul 18 – Jul 20 CPU hotplug

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpu/hotplug: Prevent offlining the last domain housekeeping CPU

From: Guopeng Zhang <[email protected]>

_cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online. However, cpumask_any_and() includes the CPU being offlined. The target therefore satisfies the check when it is the last online CPU in that housekeeping mask.

Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next scheduler-domain rebuild passes an empty span to build_sched_domains() and triggers its WARN_ON().

Use cpumask_any_and_but() to look for another online HK_TYPE_DOMAIN CPU and reject the offline request with -EBUSY if none exists.

Testing:
Tested on a 32-vCPU Ubuntu 24.04 guest with:

    isolcpus=domain,0,3-31

On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered a kernel panic.

With this patch, the same operation was rejected with -EBUSY (errno 16), the target CPU remained online, and no scheduler-domain warning was observed.

Fixes de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"")
---
 kernel/cpu.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/cpu.c +3 -2
index b3c8553d7bd6..81e961964e59 100644
@@ -1418,8 +1418,9 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,
 	 * Keep at least one housekeeping cpu onlined to avoid generating
 	 * an empty sched_domain span.
 	 */
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {
 		ret = -EBUSY;
 		goto out;
 	}
Signature · 1 line
2.43.0
#1 lore raw

Hi Guopeng,

Quoted 4 lines · Guopeng Zhang

_cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online. However, cpumask_any_and() includes the CPU being offlined. The target therefore satisfies the check when it is the last online CPU in that housekeeping mask.

The bug is real. Note the code before the revert got this right, 38685e2a0476
("cpu/hotplug: Don't offline the last non-isolated CPU") had

	for_each_cpu_and(cpu, cpu_online_mask, housekeeping_cpumask(HK_TYPE_DOMAIN)) {
		if (cpu != work.cpu)

and the exclusion got lost when de715325cc47 moved the check into _cpu_down(). please say that in the changelog, it makes the Fixes: tag obvious.

Quoted 3 lines

Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next scheduler-domain rebuild passes an empty span to build_sched_domains() and triggers its WARN_ON().

[...]

Quoted 2 lines

On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered a kernel panic.

WARN_ON() or panic, pick one. if that was panic_on_warn, say so. if it kept going and crashed later, paste the splat. The changelog should say what actually happens on a default config.

kernel/cpu.c static int __ref _cpu_down(unsigned int cpu, int tasks_frozen, · line 1423

Quoted 5 lines
kernel/cpu.c +3 -2
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {

The check itself is correct for the sysfs path. cpumask_any_and_but() returns >= nr_cpu_ids when nothing is left, and excluding a cpu that is not in the housekeeping mask is harmless. Placement is fine too, de715325cc47 put this under cpus_write_lock() specifically to serialize against cpuset partition updates, so it should stay where it is.

but this breaks suspend. _cpu_down() is also the frozen path, freeze_secondary_cpus() does

	error = _cpu_down(cpu, 1, CPUHP_OFFLINE);

for every CPU except the boot CPU. With your own cmdline (HK_TYPE_DOMAIN = {1,2}, primary = CPU0) suspend now dies at the last housekeeping CPU:

Error taking CPU1 down: -16
Non-boot CPUs are not disabled

The old check passed there because it counted the outgoing CPU. I did not check whether the frozen path can hit the same empty span WARN today, but turning a suspend that used to work into -EBUSY is a regression either way. do e.g:

	if (!tasks_frozen &&
	    cpumask_any_and_but(cpu_online_mask,
				housekeeping_cpumask(HK_TYPE_DOMAIN),
				cpu) >= nr_cpu_ids) {

and test suspend/resume with the same isolcpus cmdline for v2, not just the sysfs offline.

The Fixes: tag LGTM, the revert is where the exclusion went missing. If de715325cc47 is in a released kernel by now, CC stable. iirc cpu hotplug is root only and this needs isolcpus on the cmdline, so put that in the impact line and let the stable folks judge.
Thanks!

#2 lore raw
Quoted 7 lines · Bradley Morgan

Hi Guopeng,

Quoted 4 lines · Guopeng Zhang

_cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online. However, cpumask_any_and() includes the CPU being offlined. The target therefore satisfies the check when it is the last online CPU in that housekeeping mask.

Hi Bradley,

Thanks for the detailed review. I verified the points you raised.

Quoted 5 lines

The bug is real. Note the code before the revert got this right, 38685e2a0476
("cpu/hotplug: Don't offline the last non-isolated CPU") had

Quoted 7 lines
	for_each_cpu_and(cpu, cpu_online_mask, housekeeping_cpumask(HK_TYPE_DOMAIN)) {
		if (cpu != work.cpu)

and the exclusion got lost when de715325cc47 moved the check into _cpu_down(). please say that in the changelog, it makes the Fixes: tag obvious.

Agreed. I will update the changelog accordingly.

Quoted 4 lines
Quoted 3 lines

Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next scheduler-domain rebuild passes an empty span to build_sched_domains() and triggers its WARN_ON().

Quoted 9 lines

[...]

Quoted 2 lines

On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered a kernel panic.

WARN_ON() or panic, pick one. if that was panic_on_warn, say so. if it kept going and crashed later, paste the splat. The changelog should say what actually happens on a default config.

The panic was not caused by panic_on_warn. After the empty-span warning, the scheduler-domain rebuild continued and hit a general protection fault in build_perf_domains(), which was followed by a kernel panic.

The relevant splat is:

[   69.172549] Oops: general protection fault, probably for non-canonical address 0xfc894905e5c15114: 0000 [#1] SMP NOPTI
[   69.173143] CPU: 2 UID: 0 PID: 29 Comm: cpuhp/2 Tainted: G        W   E       7.2.0-rc3-next-20260716-sdaf+ #4 PREEMPT(lazy) 
[   69.173966] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
[   69.174306] Hardware name: Red Hat KVM, BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[   69.174944] RIP: 0010:build_perf_domains+0x40/0x230
[   69.175312] Code: fd 53 e8 83 e1 8e 00 48 c7 c2 80 7c 20 92 48 98 48 8b 04 c5 80 0d 57 90 4c 8b a4 10 48 10 00 00 8b 05 6c 9f 4e 02 85 c0 75 3a <49> 8b 84 24 c0 07 00 00 49 c7 84 24 c0 07 00 00 00 00 00 00 48 85
[   69.176881] RSP: 0018:ffffd1be401efc38 EFLAGS: 00010246
[   69.177496] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   69.178328] RDX: 0000000000000040 RSI: 0000000000000020 RDI: ffff88ed8813ab30
[   69.178821] RBP: ffff88ed8813ab30 R08: 0000000000000001 R09: ffffffff80000000
[   69.179226] R10: 0000000000000000 R11: 0000000000000000 R12: fc894905e5c14954
[   69.179622] R13: 0000000000000008 R14: 0000000000000000 R15: ffff88ed8813a690
[   69.180017] FS:  0000000000000000(0000) GS:ffff88f9bad3e000(0000) knlGS:0000000000000000
[   69.180465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   69.180786] CR2: 00007f6cfb06a010 CR3: 0000000115c49001 CR4: 0000000000772ef0
[   69.181182] PKRU: 55555554
[   69.181340] Call Trace:
[   69.181484]  <TASK>
[   69.181608]  partition_sched_domains_locked+0x3d9/0x710
[   69.182013]  ? partition_sched_domains_locked+0x16d/0x710
[   69.182418]  partition_sched_domains+0x30/0x40
[   69.182769]  rebuild_sched_domains_locked+0xe3/0x110
[   69.183151]  cpuset_handle_hotplug+0x578/0xbc0
[   69.183507]  sched_cpu_deactivate+0x1dc/0x300
[   69.183856]  ? cpuhp_thread_fun+0x5a/0x230
[   69.184187]  cpuhp_invoke_callback+0x1a4/0x780
[   69.184542]  ? cpuhp_thread_fun+0x5a/0x230
[   69.184872]  cpuhp_thread_fun+0x1b4/0x230
[   69.185202]  ? __pfx_smpboot_thread_fn+0x10/0x10
[   69.185562]  smpboot_thread_fn+0xf3/0x220
[   69.185889]  kthread+0xe6/0x120
[   69.186168]  ? __pfx_kthread+0x10/0x10
[   69.186482]  ret_from_fork+0x29f/0x330
[   69.186793]  ? __pfx_kthread+0x10/0x10
[   69.187102]  ret_from_fork_asm+0x1a/0x30
[   69.187424]  </TASK>

I will describe this sequence more accurately in the v2 changelog.

Quoted 15 lines

kernel/cpu.c static int __ref _cpu_down(unsigned int cpu, int tasks_frozen, · line 1423

Quoted 5 lines
kernel/cpu.c +3 -2
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {

The check itself is correct for the sysfs path. cpumask_any_and_but() returns >= nr_cpu_ids when nothing is left, and excluding a cpu that is not in the housekeeping mask is harmless. Placement is fine too, de715325cc47 put this under cpus_write_lock() specifically to serialize against cpuset partition updates, so it should stay where it is.

but this breaks suspend. _cpu_down() is also the frozen path, freeze_secondary_cpus() does

Quoted 8 lines
	error = _cpu_down(cpu, 1, CPUHP_OFFLINE);

for every CPU except the boot CPU. With your own cmdline (HK_TYPE_DOMAIN = {1,2}, primary = CPU0) suspend now dies at the last housekeeping CPU:

Error taking CPU1 down: -16
Non-boot CPUs are not disabled

Confirmed. I tested suspend with the same isolcpus configuration and reproduced this failure.

Quoted 5 lines

The old check passed there because it counted the outgoing CPU. I did not check whether the frozen path can hit the same empty span WARN today, but turning a suspend that used to work into -EBUSY is a regression either way. do e.g:

Quoted 7 lines
	if (!tasks_frozen &&
	    cpumask_any_and_but(cpu_online_mask,
				housekeeping_cpumask(HK_TYPE_DOMAIN),
				cpu) >= nr_cpu_ids) {

and test suspend/resume with the same isolcpus cmdline for v2, not just the sysfs offline.

I will update the patch as suggested and retest both the sysfs CPU offline path and the suspend/resume path with the same isolcpus command line before sending v2.

Quoted 5 lines

The Fixes: tag LGTM, the revert is where the exclusion went missing. If de715325cc47 is in a released kernel by now, CC stable. iirc cpu hotplug is root only and this needs isolcpus on the cmdline, so put that in the impact line and let the stable folks judge.

I will add the root-only and isolcpus requirements to the impact description. I will also check whether de715325cc47 is present in a released kernel and Cc stable if appropriate.

Thanks again for catching the suspend regression.

Thanks,
Guopeng