Skip to contentSkip to conversation

sched/deadline: Server stops while fair tasks are still runnable

PATCH Applied Gabriele Monaco 4 participants 8 messages Jan 12 – Jan 13 Scheduler

lore
Share & tools
lore
#0 lore raw

The boost model in [1] spotted cases with fair tasks running after the server stopped. I believe this means that it's equally possible for those tasks to starve without the server even trying to start.

There are 2 related situation that can result in this:

1. dl_server_idle is set to true when the CPU is really idle
2. a fair task wakes up (nothing happens since the server is still active)
3. dl_server_timer stops the server since it's idle, although it isn't really

```
    <idle>-0    d.h3.  2.521220: event_boost:     -12: idle x dl_replenish -> idle
    <idle>-0    d.h3.  2.521227: event_laxity:    -12: zero_laxity_wait x dl_replenish_idle -> idle_wait
 thread3-3-580  d..4.  3.338067: event_boost:     -12: idle x dl_server_resume_throttled -> throttled
 thread3-3-580  d..3.  3.338076: sched_wakeup:    ksoftirqd/12:118 [120] CPU:012
 thread5-5-582  d.h2.  3.471237: event_boost:     -12: throttled x dl_server_stop -> stopped (final)
 thread5-5-582  d.h2.  3.471240: event_laxity:    -12: idle_wait x dl_server_stop -> stopped (final)
ktimers/12-117  d..3.  3.538088: error_boost:     -12: event sched_switch_in not expected in the state stopped
ktimers/12-117  d..2.  3.538089: sched_switch:    ktimers/12:117 [98] S ==> ksoftirqd/12:118 [120]
```

1. dl_server_timer runs when the CPU is idle but after a fair task wakes up
2. the call to update_curr sets dl_server_idle since it's only checking rq->curr (which is indeed idle)
3. the server is stopped within the same timer call although a fair task just woke up

```
ktimers/13-125  d.s52  7.309878: event_boost:    -12: stopped x dl_server_start -> ready
ktimers/13-125  d.s52  7.309878: event_laxity:   -12: stopped x dl_server_start -> zero_laxity_wait
ktimers/13-125  d.s42  7.309879: sched_wakeup:   kworker/u519:2:5559 [120] CPU:012
    <idle>-0    d.h3.  7.309889: event_boost:    -12: ready x dl_replenish -> ready
    <idle>-0    d.h3.  7.309889: event_laxity:   -12: zero_laxity_wait x dl_replenish_idle -> idle_wait
    <idle>-0    d.h3.  7.309890: event_boost:    -12: ready x dl_server_stop -> stopped (final)
    <idle>-0    d.h3.  7.309891: event_laxity:   -12: idle_wait x dl_server_stop -> stopped (final)
    <idle>-0    d..3.  7.309895: error_boost:    -12: event sched_switch_in not expected in the state stopped
    <idle>-0    d..2.  7.309896: sched_switch:   swapper/12:0 [120] R ==> kworker/u519:2:5559 [120]

```

In both cases, if no new fair task wakes up, the one that woke up right before stopping the server could starve. That's a problem isn't it?

I tried a quick solution to the first case by clearing dl_server_idle every time the server started (even if it's still active, that is at every fair wakeup) and I think the second case could be avoided by relying on idle_cpu() instead of looking only on rq->curr, taking also waking tasks into account (not tried).

Is this a real issue or is the monitor just too sensitive?

Thanks,
Gabriele

[1] - https://lore.kernel.org/lkml/[email protected]

maintainer
#1 lore raw

Hi Gabriele,

Quoted 41 lines · Gabriele Monaco

The boost model in [1] spotted cases with fair tasks running after the server stopped. I believe this means that it's equally possible for those tasks to starve without the server even trying to start.

There are 2 related situation that can result in this:

1. dl_server_idle is set to true when the CPU is really idle
2. a fair task wakes up (nothing happens since the server is still active)
3. dl_server_timer stops the server since it's idle, although it isn't really

```
    <idle>-0    d.h3.  2.521220: event_boost:     -12: idle x dl_replenish -> idle
    <idle>-0    d.h3.  2.521227: event_laxity:    -12: zero_laxity_wait x dl_replenish_idle -> idle_wait
 thread3-3-580  d..4.  3.338067: event_boost:     -12: idle x dl_server_resume_throttled -> throttled
 thread3-3-580  d..3.  3.338076: sched_wakeup:    ksoftirqd/12:118 [120] CPU:012
 thread5-5-582  d.h2.  3.471237: event_boost:     -12: throttled x dl_server_stop -> stopped (final)
 thread5-5-582  d.h2.  3.471240: event_laxity:    -12: idle_wait x dl_server_stop -> stopped (final)
ktimers/12-117  d..3.  3.538088: error_boost:     -12: event sched_switch_in not expected in the state stopped
ktimers/12-117  d..2.  3.538089: sched_switch:    ktimers/12:117 [98] S ==> ksoftirqd/12:118 [120]
```

1. dl_server_timer runs when the CPU is idle but after a fair task wakes up
2. the call to update_curr sets dl_server_idle since it's only checking rq->curr (which is indeed idle)
3. the server is stopped within the same timer call although a fair task just woke up

```
ktimers/13-125  d.s52  7.309878: event_boost:    -12: stopped x dl_server_start -> ready
ktimers/13-125  d.s52  7.309878: event_laxity:   -12: stopped x dl_server_start -> zero_laxity_wait
ktimers/13-125  d.s42  7.309879: sched_wakeup:   kworker/u519:2:5559 [120] CPU:012
    <idle>-0    d.h3.  7.309889: event_boost:    -12: ready x dl_replenish -> ready
    <idle>-0    d.h3.  7.309889: event_laxity:   -12: zero_laxity_wait x dl_replenish_idle -> idle_wait
    <idle>-0    d.h3.  7.309890: event_boost:    -12: ready x dl_server_stop -> stopped (final)
    <idle>-0    d.h3.  7.309891: event_laxity:   -12: idle_wait x dl_server_stop -> stopped (final)
    <idle>-0    d..3.  7.309895: error_boost:    -12: event sched_switch_in not expected in the state stopped
    <idle>-0    d..2.  7.309896: sched_switch:   swapper/12:0 [120] R ==> kworker/u519:2:5559 [120]

```
Quoted 2 lines

In both cases, if no new fair task wakes up, the one that woke up right before stopping the server could starve. That's a problem isn't it?

Yeah, think so. :/

Looks like we clear dl_defer_idle only while updating the fair task entity (which didn't happen yet in both cases above).

Quoted 4 lines

I tried a quick solution to the first case by clearing dl_server_idle every time the server started (even if it's still active, that is at every fair wakeup) and I think the second case could be avoided by relying on idle_cpu() instead of looking only on rq->curr, taking also waking tasks into account (not tried).

Not sure if you want to wait for Peter to chime in, but if you have patches we can definitely take a look. :)

Thanks,
Juri

#2 lore raw

PATCH 1/1 sched/deadline: Fix server stopping with runnable tasks

The deadline server can currently stop due to idle although fair tasks are runnable. This happens essentially when:

* the server is set to idle, a task wakes up, the server stops
* a task wakes up, the server sets itself to idle and stops right away

Address both cases by clearing the server idle flag whenever a fair task wakes up and accounting also for pending tasks in the definition of idle.

---

This is really quick and dirty but seems to fix it according to the models. I also updated the chart to show what is happening here. I added the same event (server_resume) to the model.

 kernel/sched/deadline.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
kernel/sched/deadline.c +8 -4
index 138136742871..1070e93350df 100644
@@ -1376,7 +1376,7 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, int
 
 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {
-	bool idle = rq->curr == rq->idle;
+	bool idle = rq->curr == rq->idle && !rq->nr_running && !rq->ttwu_pending;
 	s64 scaled_delta_exec;
 
 	if (unlikely(delta_exec <= 0)) {
@@ -1560,8 +1560,8 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  * | 8 |       B:zero_laxity-wait       |     |    |
  * |   |                                | <---+    |
  * |   +--------------------------------+          |
- * |     |              ^     ^           2        |
- * |     | 7            | 2   +--------------------+
+ * |     |              ^         ^       2        |
+ * |     | 7            | 2, 1    +----------------+
  * |     v              |
  * |   +-------------+  |
  * +-- | C:idle-wait | -+
@@ -1606,8 +1606,11 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  *   dl_defer_idle = 0
  *
  *
- * [1] A->B, A->D
+ * [1] A->B, A->D, C->B
  * dl_server_start()
+ *   dl_defer_idle = 0;
+ *   if (dl_server_active)
+ *     return; // [B]
  *   dl_server_active = 1;
  *   enqueue_dl_entity()
  *     update_dl_entity(WAKEUP)
@@ -1741,6 +1744,7 @@ void dl_server_start(struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = dl_se->rq;
 
+	dl_se->dl_defer_idle = 0;
 	if (!dl_server(dl_se) || dl_se->dl_server_active)
 		return;
 

base-commit: 0f61b1860cc3f52aef9036d7235ed1f017632193

Signature · 1 line
2.52.0
maintainer
#3 lore raw

Re: [PATCH] sched/deadline: Fix server stopping with runnable tasks

Quoted 24 lines · Gabriele Monaco

The deadline server can currently stop due to idle although fair tasks are runnable. This happens essentially when:

* the server is set to idle, a task wakes up, the server stops
* a task wakes up, the server sets itself to idle and stops right away

Address both cases by clearing the server idle flag whenever a fair task wakes up and accounting also for pending tasks in the definition of idle.

---

This is really quick and dirty but seems to fix it according to the models. I also updated the chart to show what is happening here. I added the same event (server_resume) to the model.

 kernel/sched/deadline.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
kernel/sched/deadline.c +0 -0
index 138136742871..1070e93350df 100644

kernel/sched/deadline.c update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, int · line 1379

Quoted 5 lines
kernel/sched/deadline.c +1 -1
@@ -1376,7 +1376,7 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, int
 
 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {
-	bool idle = rq->curr == rq->idle;
+	bool idle = rq->curr == rq->idle && !rq->nr_running && !rq->ttwu_pending;

This is idle_cpu(), perhaps we can lift that thing into sched.h or so.

Quoted 28 lines
kernel/sched/deadline.c +6 -3
 	s64 scaled_delta_exec;
 
 	if (unlikely(delta_exec <= 0)) {
@@ -1560,8 +1560,8 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  * | 8 |       B:zero_laxity-wait       |     |    |
  * |   |                                | <---+    |
  * |   +--------------------------------+          |
- * |     |              ^     ^           2        |
- * |     | 7            | 2   +--------------------+
+ * |     |              ^         ^       2        |
+ * |     | 7            | 2, 1    +----------------+
  * |     v              |
  * |   +-------------+  |
  * +-- | C:idle-wait | -+
@@ -1606,8 +1606,11 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  *   dl_defer_idle = 0
  *
  *
- * [1] A->B, A->D
+ * [1] A->B, A->D, C->B
  * dl_server_start()
+ *   dl_defer_idle = 0;
+ *   if (dl_server_active)
+ *     return; // [B]
  *   dl_server_active = 1;
  *   enqueue_dl_entity()
  *     update_dl_entity(WAKEUP)

kernel/sched/deadline.c void dl_server_start(struct sched_dl_entity *dl_se) · line 1749

Quoted 6 lines
kernel/sched/deadline.c +1 -0
@@ -1741,6 +1744,7 @@ void dl_server_start(struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = dl_se->rq;
 
+	dl_se->dl_defer_idle = 0;
 	if (!dl_server(dl_se) || dl_se->dl_server_active)
 		return;

Yeah, this seems sensible.

Let me pick it up and do that idle_cpu() thing.

#4 lore raw

Re: [PATCH] sched/deadline: Fix server stopping with runnable tasks

Quoted 9 lines · Peter Zijlstra
Quoted 6 lines

 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {

(quoted diff) +1 -1
-	bool idle = rq->curr == rq->idle;
+	bool idle = rq->curr == rq->idle && !rq->nr_running && !rq-
Quoted 1 line

ttwu_pending;

This is idle_cpu(), perhaps we can lift that thing into sched.h or so.

Yeah, that's what I meant by quick and dirty.. I have idle_cpu() on the model side (and have them matching simplifies a lot of things).

I just wasn't sure if we wanted a function call in there. But we can probably do something nicer without it too.

Thanks,
Gabriele

maintainer
#5 lore raw

Re: [PATCH] sched/deadline: Fix server stopping with runnable tasks

Quoted 8 lines · Gabriele Monaco
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines

 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {

(quoted diff) +1 -1
-	bool idle = rq->curr == rq->idle;
+	bool idle = rq->curr == rq->idle && !rq->nr_running && !rq-
Quoted 1 line

ttwu_pending;

Quoted 8 lines
Quoted 2 lines

This is idle_cpu(), perhaps we can lift that thing into sched.h or so.

Yeah, that's what I meant by quick and dirty.. I have idle_cpu() on the model side (and have them matching simplifies a lot of things).

I just wasn't sure if we wanted a function call in there. But we can probably do something nicer without it too.

I'm test building this...

---

include/linux/sched.h +0 -1
index bf96a7d595e2..524bdc0ca53f 100644
@@ -1869,7 +1869,6 @@ static inline int task_nice(const struct task_struct *p)
 extern int can_nice(const struct task_struct *p, const int nice);
 extern int task_curr(const struct task_struct *p);
 extern int idle_cpu(int cpu);
-extern int available_idle_cpu(int cpu);
 extern int sched_setscheduler(struct task_struct *, int, const struct sched_param *);
 extern int sched_setscheduler_nocheck(struct task_struct *, int, const struct sched_param *);
 extern void sched_set_fifo(struct task_struct *p);
kernel/sched/deadline.c +8 -4
index 80c9559a3e30..d06b29ae961e 100644
@@ -1420,7 +1420,7 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, int
 
 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {
-	bool idle = rq->curr == rq->idle;
+	bool idle = idle_rq(rq);
 	s64 scaled_delta_exec;
 
 	if (unlikely(delta_exec <= 0)) {
@@ -1603,8 +1603,8 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  * | 8 |       B:zero_laxity-wait       |     |    |
  * |   |                                | <---+    |
  * |   +--------------------------------+          |
- * |     |              ^     ^           2        |
- * |     | 7            | 2   +--------------------+
+ * |     |              ^         ^       2        |
+ * |     | 7            | 2, 1    +----------------+
  * |     v              |
  * |   +-------------+  |
  * +-- | C:idle-wait | -+
@@ -1649,8 +1649,11 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  *   dl_defer_idle = 0
  *
  *
- * [1] A->B, A->D
+ * [1] A->B, A->D, C->B
  * dl_server_start()
+ *   dl_defer_idle = 0;
+ *   if (dl_server_active)
+ *     return; // [B]
  *   dl_server_active = 1;
  *   enqueue_dl_entity()
  *     update_dl_entity(WAKEUP)
@@ -1784,6 +1787,7 @@ void dl_server_start(struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = dl_se->rq;
 
+	dl_se->dl_defer_idle = 0;
 	if (!dl_server(dl_se) || dl_se->dl_server_active)
 		return;
 
kernel/sched/sched.h +22 -0
index 58c9d244f12b..ea791550af0e 100644
@@ -1380,6 +1380,28 @@ static inline u32 sched_rng(void)
 #define cpu_curr(cpu)		(cpu_rq(cpu)->curr)
 #define raw_rq()		raw_cpu_ptr(&runqueues)
 
+static inline bool idle_rq(struct rq *rq)
+{
+	return rq->curr == rq->idle && !rq->nr_running && !rq->ttwu_pending;
+}
+
+/**
+ * available_idle_cpu - is a given CPU idle for enqueuing work.
+ * @cpu: the CPU in question.
+ *
+ * Return: 1 if the CPU is currently idle. 0 otherwise.
+ */
+static inline bool available_idle_cpu(int cpu)
+{
+	if (!idle_rq(cpu_rq(cpu)))
+		return 0;
+
+	if (vcpu_is_preempted(cpu))
+		return 0;
+
+	return 1;
+}
+
 #ifdef CONFIG_SCHED_PROXY_EXEC
 static inline void rq_set_donor(struct rq *rq, struct task_struct *t)
 {
kernel/sched/syscalls.c +1 -29
index 0496dc29ed0f..cb337de679b8 100644
@@ -180,35 +180,7 @@ int task_prio(const struct task_struct *p)
  */
 int idle_cpu(int cpu)
 {
-	struct rq *rq = cpu_rq(cpu);
-
-	if (rq->curr != rq->idle)
-		return 0;
-
-	if (rq->nr_running)
-		return 0;
-
-	if (rq->ttwu_pending)
-		return 0;
-
-	return 1;
-}
-
-/**
- * available_idle_cpu - is a given CPU idle for enqueuing work.
- * @cpu: the CPU in question.
- *
- * Return: 1 if the CPU is currently idle. 0 otherwise.
- */
-int available_idle_cpu(int cpu)
-{
-	if (!idle_cpu(cpu))
-		return 0;
-
-	if (vcpu_is_preempted(cpu))
-		return 0;
-
-	return 1;
+	return idle_rq(cpu_rq(cpu));
 }
 
 /**
maintainer
#6 lore raw

Re: [PATCH] sched/deadline: Fix server stopping with runnable tasks

In reply to Gabriele Monaco
Quoted 8 lines · Gabriele Monaco
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines

 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {

(quoted diff) +1 -1
-	bool idle = rq->curr == rq->idle;
+	bool idle = rq->curr == rq->idle && !rq->nr_running && !rq-
Quoted 1 line

ttwu_pending;

Quoted 8 lines
Quoted 2 lines

This is idle_cpu(), perhaps we can lift that thing into sched.h or so.

Yeah, that's what I meant by quick and dirty.. I have idle_cpu() on the model side (and have them matching simplifies a lot of things).

I just wasn't sure if we wanted a function call in there. But we can probably do something nicer without it too.

Thanks for the patch (looks sensible to me too) and nice catch from the model!

#7 lore raw

[tip: sched/urgent] sched/deadline: Fix server stopping with runnable tasks

In reply to Gabriele Monaco
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     ca1e8eede4fc68ce85a9fdce1a6c13ad64933318
Gitweb:        https://git.kernel.org/tip/ca1e8eede4fc68ce85a9fdce1a6c13ad64933318
Author:        Gabriele Monaco <[email protected]>
AuthorDate:    Tue, 13 Jan 2026 09:52:01 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 13 Jan 2026 11:37:52 +01:00

sched/deadline: Fix server stopping with runnable tasks

The deadline server can currently stop due to idle although fair tasks are runnable. This happens essentially when:

* the server is set to idle, a task wakes up, the server stops
* a task wakes up, the server sets itself to idle and stops right away

Address both cases by clearing the server idle flag whenever a fair task wakes up and accounting also for pending tasks in the definition of idle.

Fixes f5a538c07df2 ("sched/deadline: Fix dl_server stop condition")
---
 kernel/sched/deadline.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
kernel/sched/deadline.c +9 -4
index e3efc40..b5c19b1 100644
@@ -1420,7 +1420,7 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, int 
 
 static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 delta_exec)
 {
-	bool idle = rq->curr == rq->idle;
+	bool idle = idle_rq(rq);
 	s64 scaled_delta_exec;
 
 	if (unlikely(delta_exec <= 0)) {
@@ -1603,8 +1603,8 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  * | 8 |       B:zero_laxity-wait       |     |    |
  * |   |                                | <---+    |
  * |   +--------------------------------+          |
- * |     |              ^     ^           2        |
- * |     | 7            | 2   +--------------------+
+ * |     |              ^         ^       2        |
+ * |     | 7            | 2, 1    +----------------+
  * |     v              |
  * |   +-------------+  |
  * +-- | C:idle-wait | -+
@@ -1649,8 +1649,11 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  *   dl_defer_idle = 0
  *
  *
- * [1] A->B, A->D
+ * [1] A->B, A->D, C->B
  * dl_server_start()
+ *   dl_defer_idle = 0;
+ *   if (dl_server_active)
+ *     return; // [B]
  *   dl_server_active = 1;
  *   enqueue_dl_entity()
  *     update_dl_entity(WAKEUP)
@@ -1759,6 +1762,7 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
  *   "B:zero_laxity-wait" -> "C:idle-wait"        [label="7:dl_server_update_idle"]
  *   "B:zero_laxity-wait" -> "D:running"          [label="3:dl_server_timer"]
  *   "C:idle-wait" -> "A:init"                    [label="8:dl_server_timer"]
+ *   "C:idle-wait" -> "B:zero_laxity-wait"        [label="1:dl_server_start"]
  *   "C:idle-wait" -> "B:zero_laxity-wait"        [label="2:dl_server_update"]
  *   "C:idle-wait" -> "C:idle-wait"               [label="7:dl_server_update_idle"]
  *   "D:running" -> "A:init"                      [label="4:pick_task_dl"]
@@ -1784,6 +1788,7 @@ void dl_server_start(struct sched_dl_entity *dl_se)
 {
 	struct rq *rq = dl_se->rq;
 
+	dl_se->dl_defer_idle = 0;
 	if (!dl_server(dl_se) || dl_se->dl_server_active)
 		return;