Skip to content

RFC sched: Handle CPU freeze without active domain housekeeping CPUs

Guopeng Zhang cgroups lkml 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest3 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3sched: Preserve user affinity across frozen CPU fallbackinclude/linux/sched/hotplug.h kernel/cpu.c kernel/sched/core.c +1 more+185 -70 replies
2/3sched: Allow isolated CPUs as a last resort during CPU freezekernel/sched/core.c+9 -20 replies
3/3sched/topology: Tear down domains without active domain housekeeping CPUsinclude/linux/cpuset.h kernel/cgroup/cpuset.c kernel/sched/topology.c+15 -70 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/3] sched: Handle CPU freeze without active domain housekeeping CPUs

From: Guopeng Zhang <[email protected]>

During suspend, freeze_secondary_cpus() takes every CPU except the primary offline. With domain isolation, the primary CPU can be excluded from HK_TYPE_DOMAIN. Offlining the last domain housekeeping CPU then leaves the frozen hotplug path without an active CPU in that mask.

This exposes three related problems. The frozen cpuset callback asks for one fallback scheduler domain even though its span is empty. A user task restricted to the CPUs being frozen can exhaust select_fallback_rq() because the normal fallback mask has no active CPU. Finally, forcing such a task onto a temporary CPU clears user_cpus_ptr, so the temporary affinity can survive after the CPUs are brought back online.

This series preserves and restores user affinity across the frozen CPU fallback, permits an isolated but task-capable active CPU as a temporary last resort, and explicitly tears down scheduler domains while no active HK_TYPE_DOMAIN CPU exists.

I am sending this as an RFC because the first patch adds scheduler state for the freeze transaction and collects affected tasks during thaw. In particular, feedback would be useful on the task tracking and restore scheme, the use of task_cpu_possible_mask() as the temporary fallback, and the explicit zero-domain request passed to partition_sched_domains().

Reproduction
============

The problem was reproduced on a 32-CPU x86 physical machine booted with:

    isolcpus=domain,0,3-31

This leaves CPUs 1 and 2 in HK_TYPE_DOMAIN while the suspend primary is CPU0. On an unpatched 7.2.0-rc3-next-20260716 kernel, with panic_on_warn=0 and warn_limit=0, the following processor-stage suspend test crashed the machine:

echo deep > /sys/power/mem_sleep
echo processors > /sys/power/pm_test
echo mem > /sys/power/state

After CPU2 had been taken offline, rebuilding the scheduler domains for CPU1 produced an empty span warning, followed by a general protection fault in build_perf_domains():

    smpboot: CPU 2 is now offline
    ------------[ cut here ]------------
    WARNING: kernel/sched/topology.c:3093 at build_sched_domains+0x443/0xa60
    ...
    Call Trace:
     partition_sched_domains_locked+0x2dd/0x710
     partition_sched_domains+0x30/0x40
     cpuset_reset_sched_domains+0x25/0x40
     sched_cpu_deactivate+0x2f2/0x300
    ...
    Oops: general protection fault, probably for non-canonical address ...
    RIP: 0010:build_perf_domains+0x40/0x230
    Call Trace:
     partition_sched_domains_locked+0x3d9/0x710
     partition_sched_domains+0x30/0x40
     cpuset_reset_sched_domains+0x25/0x40
     sched_cpu_deactivate+0x2f2/0x300

Kdump captured a vmcore after the fault. This was not a panic_on_warn failure.

Testing
=======

With this series applied, the processor-stage test was repeated while the shell was restricted to the two domain housekeeping CPUs:

taskset -pc 1,2 $$
echo deep > /sys/power/mem_sleep
echo processors > /sys/power/pm_test
echo mem > /sys/power/state
ret=$?
echo none > /sys/power/pm_test
echo "return=$ret"
taskset -pc $$
cat /sys/devices/system/cpu/online

The fallback path was exercised:

process 7271 (bash) no longer affine to cpu1

The test completed successfully and the requested affinity and online CPU mask were restored:

return=0
pid 7271's current affinity list: 1,2
0-31

A real S3 cycle was then tested with:

echo deep > /sys/power/mem_sleep
echo none > /sys/power/pm_test
rtcwake -m no -s 10
echo mem > /sys/power/state
ret=$?
echo "return=$ret"
taskset -pc $$
cat /sys/devices/system/cpu/online

The machine entered S3, resumed, and brought CPUs 1-31 back online. The result was again:

return=0
pid 7271's current affinity list: 1,2
0-31

Neither test produced a scheduler-domain warning, GPF, BUG, Oops, or panic.

Each patch was build-tested in sequence with x86_64_defconfig. The final series completed a full x86_64 build. The affected objects were also build-tested with CONFIG_CPUSETS=n and CONFIG_CPU_ISOLATION=y.

Guopeng Zhang (3):
sched: Preserve user affinity across frozen CPU fallback
sched: Allow isolated CPUs as a last resort during CPU freeze sched/topology: Tear down domains without active domain housekeeping
CPUs

 include/linux/cpuset.h        |   7 +-
 include/linux/sched/hotplug.h |   2 +
 kernel/cgroup/cpuset.c        |   6 +-
 kernel/cpu.c                  |   2 +
 kernel/sched/core.c           | 191 ++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h          |   4 +-
 kernel/sched/topology.c       |   9 +-
 7 files changed, 207 insertions(+), 14 deletions(-)

base-commit: 0718283ab28bc3907e10b61a6b4be6fefa1cbb2f

Signature · 1 line
2.43.0