Skip to contentSkip to conversation

sched_ext: idle: Don't stack a sync wakee behind a higher-class task

PATCH New Hang Zhengyang 1 participant 3 messages Nov 20

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover sched_ext: idle: Don't stack a sync wakee behind a higher-class task

In sched_ext, the default idle CPU selection can put a sync wakee on the waker's CPU, even when the waker is an RT or DL task that keeps running. The wakee then waits there, sometimes for more than a second, while other CPUs are idle. Patch 1 stops this for higher-class wakers. Patch 2 adds a selftest that fails without patch 1.

Tested on sched_ext/for-7.5 (c6cc52a72055) in an 8 vCPU KVM guest, non-debug kernel, booting each kernel twice:

                               before               after
  RT waker, pipe               16/16 on CPU 0,      idle CPU,
                               169 ms - 1.5 s       <= 0.6 ms
  same, ALWAYS_ENQ_IMMED       16/16 late by        <= 0.6 ms
                               16-483 ms
  partial, fair waker          16/16 on CPU 0,      <= 0.6 ms
                               half ~930 ms late
  sync_wake_rt, 8/3/2 vCPUs    fails                passes
  other sched_ext selftests    no change            no change

With KASAN and lockdep on v7.4-rc2, sync_wake_rt fails without the fix and passes with it, with no warnings. If other tasks keep all CPUs busy, the test passes on both kernels, because the shortcut is only used when some CPU is idle. On bare metal I could only run an unpatched distro v7.0 kernel, because of Secure Boot. It has the same
problem: the reader waited 168 ms or more.

The cost: after a sync wakeup from a higher-class task, the wakee no longer runs on the waker's CPU. WF_SYNC doesn't promise this anyway. In a pipe ping-pong between an RT task and an EXT task, a round trip now takes ~9.8 us instead of ~6.5 us, because the EXT task stays on its own CPU. CFS also takes ~10 us here. The difference can be bigger on large non-SMT machines [1]. That series detects pairs of tasks that wake each other. If fair adopts it, we could keep the shortcut for such pairs.

There is no Fixes tag. The shortcut has been there since sched_ext was added, and this patch changes where a task runs, not whether the kernel works correctly.

Hui Su's patch "sched_ext: idle: Honor SCX_PICK_IDLE_CORE in CPU selection shortcuts" changes the line just before this code. Whichever patch is applied second will need a small rebase.

I used Claude Code to find the issue, write the patches and run the tests. I reproduced the issue on bare metal and reviewed the patches.

[1] https://lore.kernel.org/all/[email protected]/

---
Hang Zhengyang (2):
sched_ext: idle: Don't stack a sync wakee behind a higher-class task selftests/sched_ext: Test sync wakeups from an RT task

 kernel/sched/ext/idle.c                          |  11 +
 tools/testing/selftests/sched_ext/Makefile       |   1 +
 tools/testing/selftests/sched_ext/sync_wake_rt.c | 257 +++++++++++++++++++++++
 3 files changed, 269 insertions(+)

---
base-commit: c6cc52a72055ef9775b39937d02e39ca358ba042
change-id: 20261120-scx-fix75-a86a6d31eceb

Best regards,

Signature · 1 line
Hang Zhengyang <[email protected]>
#1 lore raw

PATCH 1/2 sched_ext: idle: Don't stack a sync wakee behind a higher-class task

On a sync wakeup, scx_select_cpu_dfl() can return the waker's CPU. It does this when the waker's local DSQ is empty and some CPU is idle, because the waker is expected to give up its CPU soon.

This is wrong when the waker's CPU runs an RT or DL task, or a fair task in partial mode. The BPF scheduler can't preempt such a task. The wakee waits on the local DSQ until that task blocks, or until the ext dl_server runs the wakee. Meanwhile, other CPUs are idle.

The waker may not sleep at all. pipe_write() and sock_def_readable() always set WF_SYNC, even if the writer keeps running.

This affects every BPF scheduler that leaves CPU selection to the kernel, or that passes SCX_WAKE_SYNC to scx_bpf_select_cpu_dfl() or scx_bpf_select_cpu_and(). In an 8 vCPU guest with scx_simple, an RT task on CPU 0 wrote to a pipe and kept spinning. In 16 of 16 wakeups, the reader ran on CPU 0, 169 ms to 1.5 s later. With an eventfd write instead, which is not a sync wakeup, the reader ran on an idle CPU within 0.1 ms.

SCX_OPS_ALWAYS_ENQ_IMMED doesn't fix this. It re-enqueues the wakee, but in the same test the wakee still ran 16 to 483 ms late. The fair class doesn't have this problem. wake_affine_idle() also picks the waker's CPU, but select_idle_sibling() only uses it if it is idle.

Skip the shortcut when a task of a higher class than EXT is running, or about to run, on the waker's CPU. rq->next_class shows this, and rq_is_open() uses the same check for SCX_ENQ_IMMED. The wakee then goes to an idle CPU, as in any other wakeup. Sync wakeups between EXT tasks don't change, unless a higher-class task is about to run on the waker's CPU.

---
 kernel/sched/ext/idle.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
kernel/sched/ext/idle.c +11 -0
index 922fbc3065cc..bf4dd8cb4255 100644
@@ -534,6 +534,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
 	 * If WAKE_SYNC, try to migrate the wakee to the waker's CPU.
 	 */
 	if (wake_flags & SCX_WAKE_SYNC) {
+		const struct sched_class *next_class;
 		int waker_node;
 
 		/*
@@ -559,9 +560,19 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
 		 * insufficient as the local DSQ of the waker could have tasks
 		 * piled up on it even if there is an idle core elsewhere on
 		 * the system.
+		 *
+		 * Also don't put @p behind a task of a higher class (RT, DL,
+		 * or fair in partial mode). The BPF scheduler can't preempt
+		 * that task, and it may keep the CPU for a long time. @p would
+		 * wait on the local DSQ until that task stops, or until the
+		 * ext dl_server runs @p. ->next_class also shows a higher-class
+		 * task that is about to run here. Reading it without the rq
+		 * lock is fine, since this is only a hint.
 		 */
+		next_class = READ_ONCE(cpu_rq(cpu)->next_class);
 		waker_node = scx_cpu_node_if_enabled(cpu);
 		if (!(current->flags & PF_EXITING) &&
+		    !sched_class_above(next_class, &ext_sched_class) &&
 		    cpu_rq(cpu)->scx.local_dsq.nr == 0 &&
 		    (!(flags & SCX_PICK_IDLE_IN_NODE) || (waker_node == node)) &&
 		    !cpumask_empty(idle_cpumask(waker_node)->cpu)) {
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/2 selftests/sched_ext: Test sync wakeups from an RT task

In reply to Hang Zhengyang

Add the sync_wake_rt test. It loads the minimal scheduler, so the kernel picks the CPU for each wakeup.

In each round, a reader task first runs on one CPU. Then it blocks on a pipe and may run on any usable CPU. An RT task, pinned to that first CPU, writes to the pipe and spins until the reader has run, for at most 300 ms.

The test fails if the reader runs on the RT task's CPU in more than 5 of 10 rounds. It is skipped if fewer than two CPUs are usable. It checks where the reader runs, not how long it waits, because the wait depends on the state of the ext dl_server.

---
 tools/testing/selftests/sched_ext/Makefile       |   1 +
 tools/testing/selftests/sched_ext/sync_wake_rt.c | 257 +++++++++++++++++++++++
 2 files changed, 258 insertions(+)
tools/testing/selftests/sched_ext/Makefile +1 -0
index 44083f1c6596..8d2c5b5b0070 100644
@@ -201,6 +201,7 @@ auto-test-targets :=			\
 	test_example			\
 	total_bw			\
 	cyclic_kick_wait		\
+	sync_wake_rt			\
 
 testcase-targets := $(addsuffix .o,$(addprefix $(SCXOBJ_DIR)/,$(auto-test-targets)))
 
tools/testing/selftests/sched_ext/sync_wake_rt.c new file +257 -0

257 lines not on this page · raw message

Signature · 1 line
2.53.0