Skip to contentSkip to conversation

Complete EEVDF

PATCH Applied Peter Zijlstra 30 participants 254 messages Jul 27, 2024 – Dec 10, 2024 Page 2 of 2

Subsystems (11)
KVM perf events Tracing Scheduler +7 more subsystems
lore
Share & tools
lore
#144 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

In reply to Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

---
 kernel/sched/core.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
@@ -3794,7 +3794,7 @@ static int ttwu_runnable(struct task_str
 		int queue_flags = DEQUEUE_NOCLOCK;
 
 		if (p->se.sched_delayed)
-			queue_flags |= DEQUEUE_DELAYED;
+			queue_flags |= (DEQUEUE_DELAYED | DEQUEUE_SLEEP);
 
 		/*
 		 * Since we're not current anywhere *AND* hold pi_lock, dequeue
@@ -3802,7 +3802,7 @@ static int ttwu_runnable(struct task_str
 		 * case further along the ttwu() path.
 		 */
 		if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
-			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			dequeue_task(rq, p, queue_flags);
 			return 0;
 		}
 
#145 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.

-Mike

author maintainer
#146 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

#147 lore raw

PATCH 1/1 sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

Quoted 7 lines
Quoted 2 lines

From netperf, I got.. number tabulation practice.  Three runs of each test with and without produced nothing but variance/noise.

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Salut, much obliged for eyeball relief.

---snip---

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

(de-uglified-a-lot-by)

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
 kernel/sched/core.c  |   48 +++++++++++++++++++++++++++++-------------------
 kernel/sched/sched.h |    5 +++++
 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +29 -19
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
+	CLASS(__task_rq_lock, rq_guard)(p);
+	struct rq *rq = rq_guard.rq;
+
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed) {
+		int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+		/*
+		 * Since sched_delayed means we cannot be current anywhere,
+		 * dequeue it here and have it fall through to the
+		 * select_task_rq() case further along the ttwu() path.
+		 */
+		if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
+			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			return 0;
 		}
-		ttwu_do_wakeup(p);
-		ret = 1;
+
+		enqueue_task(rq, p, queue_flags);
+	}
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
 	}
-	__task_rq_unlock(rq, &rf);
+	ttwu_do_wakeup(p);
 
-	return ret;
+	return 1;
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+		    _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+		    __task_rq_unlock(_T->rq, &_T->rf),
+		    struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
 		    _T->rq = task_rq_lock(_T->lock, &_T->rf),
 		    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Reviewed-by Tested-by Jirka Hladky on v1 1/1
#148 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 14 lines · On Fri, Nov 08, 2024 at 01:24:35AM +0100 Mike Galbraith wrote:
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice.  Three runs of each test with and without produced nothing but variance/noise.

Quoted 7 lines
Quoted 4 lines

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Salut, much obliged for eyeball relief.

Thanks Mike (and Peter). We have our full perf tests running on Mike's original verion of this patch. Results probably Monday (there's a long queue). We'll see if this blows up anything else then. I'll queue up a build with this cleaned up version as well but the results will be late next week, probably.

At that point maybe some or all of these:

Cheers,
Phil

Quoted 94 lines

---snip---

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

(de-uglified-a-lot-by)

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
 kernel/sched/core.c  |   48 +++++++++++++++++++++++++++++-------------------
 kernel/sched/sched.h |    5 +++++
 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +29 -19
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
+	CLASS(__task_rq_lock, rq_guard)(p);
+	struct rq *rq = rq_guard.rq;
+
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed) {
+		int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+		/*
+		 * Since sched_delayed means we cannot be current anywhere,
+		 * dequeue it here and have it fall through to the
+		 * select_task_rq() case further along the ttwu() path.
+		 */
+		if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
+			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			return 0;
 		}
-		ttwu_do_wakeup(p);
-		ret = 1;
+
+		enqueue_task(rq, p, queue_flags);
+	}
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
 	}
-	__task_rq_unlock(rq, &rf);
+	ttwu_do_wakeup(p);
 
-	return ret;
+	return 1;
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+		    _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+		    __task_rq_unlock(_T->rq, &_T->rf),
+		    struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
 		    _T->rq = task_rq_lock(_T->lock, &_T->rf),
 		    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
#149 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Mike Galbraith

Hi Mike and Peter,

Quoted 67 lines · Mike Galbraith
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

Quoted 7 lines
Quoted 2 lines

From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Salut, much obliged for eyeball relief.

---snip---

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

(de-uglified-a-lot-by)

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
 kernel/sched/core.c  |   48 +++++++++++++++++++++++++++++-------------------
 kernel/sched/sched.h |    5 +++++
 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +6 -15
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-       struct rq_flags rf;
-       struct rq *rq;
-       int ret = 0;
-
-       rq = __task_rq_lock(p, &rf);
-       if (task_on_rq_queued(p)) {
-               update_rq_clock(rq);
-               if (p->se.sched_delayed)
-                       enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-               if (!task_on_cpu(rq, p)) {
-                       /*
-                        * When on_rq && !on_cpu the task is preempted, see if
-                        * it should preempt the task that is current now.
-                        */
-                       wakeup_preempt(rq, p, wake_flags);
+       CLASS(__task_rq_lock, rq_guard)(p);
+       struct rq *rq = rq_guard.rq;
+
+       if (!task_on_rq_queued(p))
+               return 0;
+

kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3801

Quoted 10 lines
kernel/sched/core.c +10 -0
+       update_rq_clock(rq);
+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+               /*
+                * Since sched_delayed means we cannot be current anywhere,
+                * dequeue it here and have it fall through to the
+                * select_task_rq() case further along the ttwu() path.
+                */
+               if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {

For sched_asym_cpucapacity system, need we consider the task_fits_cpu_capacity there?

Quoted 40 lines
kernel/sched/core.c +13 -4
+                       dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+                       return 0;
                }
-               ttwu_do_wakeup(p);
-               ret = 1;
+
+               enqueue_task(rq, p, queue_flags);
+       }
+       if (!task_on_cpu(rq, p)) {
+               /*
+                * When on_rq && !on_cpu the task is preempted, see if
+                * it should preempt the task that is current now.
+                */
+               wakeup_preempt(rq, p, wake_flags);
        }
-       __task_rq_unlock(rq, &rf);
+       ttwu_do_wakeup(p);
 
-       return ret;
+       return 1;
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas
        raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+                   _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+                   __task_rq_unlock(_T->rq, &_T->rf),
+                   struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
                    _T->rq = task_rq_lock(_T->lock, &_T->rf),
                    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
#150 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 11 lines · Xuewen Yan
Quoted 10 lines

+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; +
+               /*
+                * Since sched_delayed means we cannot be current anywhere, +                * dequeue it here and have it fall through to the +                * select_task_rq() case further along the ttwu() path. +                */
+               if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {

Quoted 2 lines

For sched_asym_cpucapacity system, need we consider the task_fits_cpu_capacity there?

I don't think so. Wakeup placement logic is what we're deflecting the wakee toward, this is not the right spot to add any complexity.

-Mike

#151 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Mike Galbraith
Quoted 14 lines · Mike Galbraith
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice.  Three runs of each test with and without produced nothing but variance/noise.

Quoted 6 lines
Quoted 4 lines

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Salut, much obliged for eyeball relief.

Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.

(adds knob)

tbench 8

NO_MIGRATE_DELAYED    3613.49 MB/sec
MIGRATE_DELAYED       3145.59 MB/sec
NO_DELAY_DEQUEUE      3355.42 MB/sec

First line is DELAY_DEQUEUE restoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.

I thought I might be able to do away with the reservation like side effect of DELAY_DEQUEUE by borrowing h_nr_delayed from...

sched/eevdf: More PELT vs DELAYED_DEQUEUE

...for cgroups free test config, but Q/D poke at idle_cpu() helped not at all.

Quoted 94 lines

---snip---

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

(de-uglified-a-lot-by)

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")

---
 kernel/sched/core.c  |   48 +++++++++++++++++++++++++++++------------------- kernel/sched/sched.h |    5 +++++

 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +28 -18
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t
   */
  static int ttwu_runnable(struct task_struct *p, int wake_flags)
  {
-       struct rq_flags rf;
-       struct rq *rq;
-       int ret = 0;
-
-       rq = __task_rq_lock(p, &rf);
-       if (task_on_rq_queued(p)) {
-               update_rq_clock(rq);
-               if (p->se.sched_delayed)
-                       enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-               if (!task_on_cpu(rq, p)) {
-                       /*
-                        * When on_rq && !on_cpu the task is preempted, see if
-                        * it should preempt the task that is current now.
-                        */
-                       wakeup_preempt(rq, p, wake_flags);
+       CLASS(__task_rq_lock, rq_guard)(p);
+       struct rq *rq = rq_guard.rq;
+
+       if (!task_on_rq_queued(p))
+               return 0;
+
+       update_rq_clock(rq);
+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+               /*
+                * Since sched_delayed means we cannot be current anywhere,
+                * dequeue it here and have it fall through to the
+                * select_task_rq() case further along the ttwu() path.
+                */
+               if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
+                       dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+                       return 0;
                 }
-               ttwu_do_wakeup(p);
-               ret = 1;
+
+               enqueue_task(rq, p, queue_flags);
+       }
+       if (!task_on_cpu(rq, p)) {
+               /*
+                * When on_rq && !on_cpu the task is preempted, see if
+                * it should preempt the task that is current now.
+                */
+               wakeup_preempt(rq, p, wake_flags);
         }
-       __task_rq_unlock(rq, &rf);
+       ttwu_do_wakeup(p);


-       return ret;
+       return 1;
 }

 #ifdef CONFIG_SMP

kernel/sched/sched.h +0 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas
         raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
  }

kernel/sched/sched.h +5 -0
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+                   _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+                   __task_rq_unlock(_T->rq, &_T->rf),
+                   struct rq *rq; struct rq_flags rf)
+

 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct, _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),

#152 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 20 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Quoted 19 lines · Mike Galbraith
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice.  Three runs of each test with and without produced nothing but variance/noise.

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Quoted 8 lines
Quoted 1 line

Salut, much obliged for eyeball relief.

Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.

(adds knob)

Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).

Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.

I'd probably have to take the reads over the writes in such a trade off :)

Quoted 10 lines

tbench 8

NO_MIGRATE_DELAYED    3613.49 MB/sec
MIGRATE_DELAYED       3145.59 MB/sec
NO_DELAY_DEQUEUE      3355.42 MB/sec

First line is DELAY_DEQUEUE restoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.

Yep, that's not pretty.

Quoted 8 lines

I thought I might be able to do away with the reservation like side effect of DELAY_DEQUEUE by borrowing h_nr_delayed from...

sched/eevdf: More PELT vs DELAYED_DEQUEUE

...for cgroups free test config, but Q/D poke at idle_cpu() helped not at all.

I wonder if the last_wakee stuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.

Cheers,
Phil

Quoted 95 lines
Quoted 94 lines

---snip---

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

(de-uglified-a-lot-by)

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")

---
 kernel/sched/core.c  |   48 +++++++++++++++++++++++++++++------------------- kernel/sched/sched.h |    5 +++++

 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +28 -18
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t
   */
  static int ttwu_runnable(struct task_struct *p, int wake_flags)
  {
-       struct rq_flags rf;
-       struct rq *rq;
-       int ret = 0;
-
-       rq = __task_rq_lock(p, &rf);
-       if (task_on_rq_queued(p)) {
-               update_rq_clock(rq);
-               if (p->se.sched_delayed)
-                       enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-               if (!task_on_cpu(rq, p)) {
-                       /*
-                        * When on_rq && !on_cpu the task is preempted, see if
-                        * it should preempt the task that is current now.
-                        */
-                       wakeup_preempt(rq, p, wake_flags);
+       CLASS(__task_rq_lock, rq_guard)(p);
+       struct rq *rq = rq_guard.rq;
+
+       if (!task_on_rq_queued(p))
+               return 0;
+
+       update_rq_clock(rq);
+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+               /*
+                * Since sched_delayed means we cannot be current anywhere,
+                * dequeue it here and have it fall through to the
+                * select_task_rq() case further along the ttwu() path.
+                */
+               if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
+                       dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+                       return 0;
                 }
-               ttwu_do_wakeup(p);
-               ret = 1;
+
+               enqueue_task(rq, p, queue_flags);
+       }
+       if (!task_on_cpu(rq, p)) {
+               /*
+                * When on_rq && !on_cpu the task is preempted, see if
+                * it should preempt the task that is current now.
+                */
+               wakeup_preempt(rq, p, wake_flags);
         }
-       __task_rq_unlock(rq, &rf);
+       ttwu_do_wakeup(p);


-       return ret;
+       return 1;
 }

 #ifdef CONFIG_SMP

kernel/sched/sched.h +0 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas
         raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
  }

kernel/sched/sched.h +5 -0
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+                   _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+                   __task_rq_unlock(_T->rq, &_T->rf),
+                   struct rq *rq; struct rq_flags rf)
+

 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct, _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),

author maintainer
#153 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 30 lines · Phil Auld
Quoted 20 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Quoted 19 lines · Mike Galbraith
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith

I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly.

The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.

From netperf, I got.. number tabulation practice.  Three runs of each test with and without produced nothing but variance/noise.

Make it go away then.

If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.

Quoted 8 lines
Quoted 1 line

Salut, much obliged for eyeball relief.

Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.

(adds knob)

Quoted 8 lines

Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).

Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.

OK, so I'm going to make this commit disappear.

#154 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Phil Auld
Quoted 9 lines · Phil Auld
Quoted 7 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:

Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.

(adds knob)

Quoted 8 lines

Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).

Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.

Ok, that would seem to second "shoot it".

Quoted 8 lines

I'd probably have to take the reads over the writes in such a trade off :)

Quoted 6 lines

tbench 8

NO_MIGRATE_DELAYED    3613.49 MB/sec
MIGRATE_DELAYED       3145.59 MB/sec
NO_DELAY_DEQUEUE      3355.42 MB/sec

Quoted 6 lines
Quoted 4 lines

First line is DELAY_DEQUEUE restoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.

Yep, that's not pretty.

Yeah, not to mention annoying.

I get the "adds bounce cache pain" aspect, but not why pre-EEVDF wouldn't be just as heavily affected, it having nothing blocking high frequency migration (the eternal scheduler boogieman:).

Bottom line would appear to be that these survivors should be left where they ended up, either due to LB or more likely bog standard prev_cpu locality, for they are part and parcel of a progression.

Quoted 7 lines
Quoted 7 lines

I thought I might be able to do away with the reservation like side effect of DELAY_DEQUEUE by borrowing h_nr_delayed from...

     sched/eevdf: More PELT vs DELAYED_DEQUEUE

...for cgroups free test config, but Q/D poke at idle_cpu() helped not at all.

We don't however have to let sched_delayed block SIS though. Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

Quoted 2 lines

I wonder if the last_wakee stuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.

If you mean heuristics, the less of those we have, the better off we are.. they _always_ find a way to embed their teeth in your backside.

-Mike

#155 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 14 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
Quoted 13 lines · Phil Auld
Quoted 7 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:

Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.

(adds knob)

Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).

Quoted 7 lines
Quoted 4 lines

Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.

Ok, that would seem to second "shoot it".

Yes, drop it please, I think. Thanks!

Quoted 28 lines
Quoted 8 lines

I'd probably have to take the reads over the writes in such a trade off :)

Quoted 6 lines

tbench 8

NO_MIGRATE_DELAYED    3613.49 MB/sec
MIGRATE_DELAYED       3145.59 MB/sec
NO_DELAY_DEQUEUE      3355.42 MB/sec

Quoted 6 lines
Quoted 4 lines

First line is DELAY_DEQUEUE restoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.

Yep, that's not pretty.

Yeah, not to mention annoying.

I get the "adds bounce cache pain" aspect, but not why pre-EEVDF wouldn't be just as heavily affected, it having nothing blocking high frequency migration (the eternal scheduler boogieman:).

Bottom line would appear to be that these survivors should be left where they ended up, either due to LB or more likely bog standard prev_cpu locality, for they are part and parcel of a progression.

Quoted 3 lines
Quoted 3 lines

I thought I might be able to do away with the reservation like side effect of DELAY_DEQUEUE by borrowing h_nr_delayed from...

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

     sched/eevdf: More PELT vs DELAYED_DEQUEUE

...for cgroups free test config, but Q/D poke at idle_cpu() helped not at all.

We don't however have to let sched_delayed block SIS though. Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

You mean something like:

if (rq->nr_running > rq->h_nr_delayed)

       return 0;

in idle_cpu() instead of the straight rq->nr_running check? I don't have the h_nr_delayed stuff yet but can look for it. I'm not sure that will help with the delayees being sticky. But I can try to try that if I'm understanding you right.

I'll try to dig into it some more regardless.

Quoted 6 lines
Quoted 2 lines

I wonder if the last_wakee stuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.

If you mean heuristics, the less of those we have, the better off we are.. they _always_ find a way to embed their teeth in your backside.

Sure, I get that. But when you have a trade-off like this being "smarter" about when to do the dequeue might help. But yes, that could go wrong.

I'm not a fan of knobs either but we could do your patch with the feat and default it off.

Cheers,
Phil

Quoted 2 lines

-Mike

#156 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:

We don't however have to let sched_delayed block SIS though.  Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

Quoted 6 lines

You mean something like:

if (rq->nr_running > rq->h_nr_delayed)
       return 0;

in idle_cpu() instead of the straight rq->nr_running check?

Yeah, close enough.

-Mike

#157 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:

We don't however have to let sched_delayed block SIS though.  Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

Quoted 8 lines
Quoted 6 lines

You mean something like:

if (rq->nr_running > rq->h_nr_delayed)
       return 0;

in idle_cpu() instead of the straight rq->nr_running check?

Yeah, close enough.

The below is all you need.

Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?

---
 kernel/sched/fair.c |    4 ++++
 1 file changed, 4 insertions(+)
kernel/sched/fair.c +4 -0
@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct
 
 	/*
 	 * We do not migrate tasks that are:
+	 * 0) not runnable (not useful here/now, but are annoying), or
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
 	 * 3) running (obviously), or
 	 * 4) are cache-hot on their current CPU.
 	 */
+	if (p->se.sched_delayed)
+		return 0;
+
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
 
#158 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 17 lines · On Thu, Nov 14, 2024 at 12:07:03PM +0100 Mike Galbraith wrote:
Quoted 16 lines · Mike Galbraith
Quoted 14 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:

We don't however have to let sched_delayed block SIS though.  Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

You mean something like:

if (rq->nr_running > rq->h_nr_delayed)
       return 0;

in idle_cpu() instead of the straight rq->nr_running check?

Quoted 9 lines
Quoted 1 line

Yeah, close enough.

The below is all you need.

Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?

I'll give this a try on my issue. This'll be simpler than the other way.

Thanks!

Cheers,
Phil

Quoted 23 lines
---
 kernel/sched/fair.c |    4 ++++
 1 file changed, 4 insertions(+)
kernel/sched/fair.c +4 -0
@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct
 
 	/*
 	 * We do not migrate tasks that are:
+	 * 0) not runnable (not useful here/now, but are annoying), or
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
 	 * 3) running (obviously), or
 	 * 4) are cache-hot on their current CPU.
 	 */
+	if (p->se.sched_delayed)
+		return 0;
+
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
 
#159 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 22 lines · On Thu, Nov 14, 2024 at 06:28:54AM -0500 Phil Auld wrote:
Quoted 21 lines · On Thu, Nov 14, 2024 at 12:07:03PM +0100 Mike Galbraith wrote:
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:

We don't however have to let sched_delayed block SIS though.  Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.

Quoted 8 lines
Quoted 6 lines

You mean something like:

if (rq->nr_running > rq->h_nr_delayed)
       return 0;

in idle_cpu() instead of the straight rq->nr_running check?

Yeah, close enough.

The below is all you need.

Quoted 8 lines
Quoted 5 lines

Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?

I'll give this a try on my issue. This'll be simpler than the other way.

This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?

Cheers,
Phil

Quoted 35 lines

Thanks!

Cheers,
Phil

Quoted 23 lines
---
 kernel/sched/fair.c |    4 ++++
 1 file changed, 4 insertions(+)
kernel/sched/fair.c +4 -0
@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct
 
 	/*
 	 * We do not migrate tasks that are:
+	 * 0) not runnable (not useful here/now, but are annoying), or
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
 	 * 3) running (obviously), or
 	 * 4) are cache-hot on their current CPU.
 	 */
+	if (p->se.sched_delayed)
+		return 0;
+
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
 
#160 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess. The numbers said it was quite a reach, no surprise it fell flat.

-Mike

#161 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

-Mike

#162 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 8 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Quoted 8 lines

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

Thanks, I'll try that.

Cheers,
Phil

Quoted 2 lines

-Mike

#163 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 12 lines · On Thu, Nov 21, 2024 at 06:56:28AM -0500 Phil Auld wrote:
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

Quoted 8 lines
Quoted 5 lines

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

Thanks, I'll try that.

Also, fwiw, I think there is another report here

https://lore.kernel.org/lkml/[email protected]/

which seems to be the same thing but mis-bisected. I've asked them to try with NO_DELAY_DEQUEUE just to be sure. But it looks like a duck.

Cheers,
Phil

#164 lore raw

Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 22 lines · On Thu, Nov 21, 2024 at 07:07:04AM -0500 Phil Auld wrote:
Quoted 12 lines · On Thu, Nov 21, 2024 at 06:56:28AM -0500 Phil Auld wrote:
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

Quoted 8 lines
Quoted 5 lines

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

Thanks, I'll try that.

Quoted 7 lines

Also, fwiw, I think there is another report here

https://lore.kernel.org/lkml/[email protected]/

which seems to be the same thing but mis-bisected. I've asked them to try with NO_DELAY_DEQUEUE just to be sure. But it looks like a duck.

But it does not quack like one. Their issue did not go away with NO_DELAY_DEQUEUE so something different is causing that one.

Quoted 7 lines

Cheers,
Phil

#165 lore raw

PATCH v2 1/1 sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Phil Auld
Quoted 12 lines · Phil Auld
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

Quoted 7 lines
Quoted 5 lines

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

Thanks, I'll try that.

You may still want to try that, but my box says probably not. Playing with your write command line, the players I see are pinned kworkers and mobile fio instances.

Maybe try the below instead. The changelog is obsolete BS unless you say otherwise, but while twiddled V2 will still migrate tbench a bit, and per trace_printk() does still let all kinds of stuff wander off to roll the SIS dice, it does not even scratch the paint of the formerly obliterated tbench progression.

Question: did wiping off the evil leave any meaningful goodness behind?

---

sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

V2: do not rip buddies apart, convenient on/off switch

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
 kernel/sched/core.c     |   51 ++++++++++++++++++++++++++++++------------------
 kernel/sched/features.h |    5 ++++
 kernel/sched/sched.h    |    5 ++++
 3 files changed, 42 insertions(+), 19 deletions(-)
kernel/sched/core.c +32 -19
@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
+	CLASS(__task_rq_lock, rq_guard)(p);
+	struct rq *rq = rq_guard.rq;
+
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed) {
+		int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+		int dequeue = sched_feat(DEQUEUE_DELAYED);
+
+		/*
+		 * Since sched_delayed means we cannot be current anywhere,
+		 * dequeue it here and have it fall through to the
+		 * select_task_rq() case further along in ttwu() path.
+		 * Note: Do not rip buddies apart else chaos follows.
+		 */
+		if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&
+		    !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {
+			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			return 0;
 		}
-		ttwu_do_wakeup(p);
-		ret = 1;
+
+		enqueue_task(rq, p, queue_flags);
+	}
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
 	}
-	__task_rq_unlock(rq, &rf);
+	ttwu_do_wakeup(p);
 
-	return ret;
+	return 1;
 }
 
 #ifdef CONFIG_SMP
kernel/sched/features.h +5 -0
@@ -47,6 +47,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
 SCHED_FEAT(DELAY_DEQUEUE, true)
+/*
+ * Free ONLY non-buddy delayed tasks to wakeup-migrate to avoid taking.
+ * an unnecessary latency hit.  Rending buddies asunder inflicts harm.
+ */
+SCHED_FEAT(DEQUEUE_DELAYED, true)
 SCHED_FEAT(DELAY_ZERO, true)
 
 /*
kernel/sched/sched.h +5 -0
@@ -1783,6 +1783,11 @@ task_rq_unlock(struct rq *rq, struct tas
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+		    _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+		    __task_rq_unlock(_T->rq, &_T->rf),
+		    struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
 		    _T->rq = task_rq_lock(_T->lock, &_T->rf),
 		    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
#166 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Hello Mike,

Quoted 52 lines · Mike Galbraith

[..snip..]

Question: did wiping off the evil leave any meaningful goodness behind?

---

sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

V2: do not rip buddies apart, convenient on/off switch

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
  kernel/sched/core.c     |   51 ++++++++++++++++++++++++++++++------------------
  kernel/sched/features.h |    5 ++++
  kernel/sched/sched.h    |    5 ++++
  3 files changed, 42 insertions(+), 19 deletions(-)
kernel/sched/core.c +8 -15
@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t
   */
  static int ttwu_runnable(struct task_struct *p, int wake_flags)
  {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
+	CLASS(__task_rq_lock, rq_guard)(p);
+	struct rq *rq = rq_guard.rq;
+
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed) {

kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3803

Quoted 10 lines
kernel/sched/core.c +10 -0
+		int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+		int dequeue = sched_feat(DEQUEUE_DELAYED);
+
+		/*
+		 * Since sched_delayed means we cannot be current anywhere,
+		 * dequeue it here and have it fall through to the
+		 * select_task_rq() case further along in ttwu() path.
+		 * Note: Do not rip buddies apart else chaos follows.
+		 */
+		if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&

Do we really care if DEQUEUE_DELAYED is enabled / disabled here when we encounter a delayed task?

kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3804

Quoted 1 line
kernel/sched/core.c +1 -0
+		    !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {

Technically, we are still looking at the last wakeup here since record_wakee() is only called later. If we care about 1:1 buddies, should we just see "current == p->last_wakee", otherwise, there is a good chance "p" has a m:n waker-wakee relationship in which case perhaps a "want_affine" like heuristic can help?

For science, I was wondering if the decision to dequeue + migrate or requeue the delayed task can be put off until after the whole select_task_rq() target selection (note: without the h_nr_delayed stuff, some of that wake_affine_idle() logic falls apart). Hackbench (which saw some regression with EEVDF Complete) seem to like it somewhat, but it still falls behind NO_DELAY_DEQUEUE.

==================================================================

    Test          : hackbench
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    NO_DELAY_DEQUEUE	     Mike's v2	  Full ttwu + requeue/migrate
    5.76	           5.72  (  1% )        5.82  ( -1% )
    6.53	           6.56  (  0% )        6.65  ( -2% )
    6.79	           7.04  ( -4% )        7.02  ( -3% )
    6.91	           7.04  ( -2% )        7.03  ( -2% )
    7.63	           8.05  ( -6% )        7.88  ( -3% )

Only subtle changes in IBS profiles; there aren't any obvious shift in hotspots with hackbench at least. Not sure if it is just the act of needing to do a dequeue + enqueue from the wakeup context that adds to the overall regression.

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 24 lines
kernel/sched/core.c +13 -4
+			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			return 0;
  		}
-		ttwu_do_wakeup(p);
-		ret = 1;
+
+		enqueue_task(rq, p, queue_flags);
+	}
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
  	}
-	__task_rq_unlock(rq, &rf);
+	ttwu_do_wakeup(p);
 
-	return ret;
+	return 1;
  }
 
  #ifdef CONFIG_SMP
[..snip..]
#167 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 60 lines · K Prateek Nayak

Hello Mike,

Quoted 57 lines · Mike Galbraith

[..snip..]

Question: did wiping off the evil leave any meaningful goodness behind?

---

sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

V2: do not rip buddies apart, convenient on/off switch

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")

---
  kernel/sched/core.c     |   51 ++++++++++++++++++++++++++++++------------------ kernel/sched/features.h |    5 ++++
  kernel/sched/sched.h    |    5 ++++

  3 files changed, 42 insertions(+), 19 deletions(-)
kernel/sched/core.c +13 -15
@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t
    */
   static int ttwu_runnable(struct task_struct *p, int wake_flags)
   {
-       struct rq_flags rf;
-       struct rq *rq;
-       int ret = 0;
-
-       rq = __task_rq_lock(p, &rf);
-       if (task_on_rq_queued(p)) {
-               update_rq_clock(rq);
-               if (p->se.sched_delayed)
-                       enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-               if (!task_on_cpu(rq, p)) {
-                       /*
-                        * When on_rq && !on_cpu the task is preempted, see if
-                        * it should preempt the task that is current now.
-                        */
-                       wakeup_preempt(rq, p, wake_flags);
+       CLASS(__task_rq_lock, rq_guard)(p);
+       struct rq *rq = rq_guard.rq;
+
+       if (!task_on_rq_queued(p))
+               return 0;
+
+       update_rq_clock(rq);
+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+               int dequeue = sched_feat(DEQUEUE_DELAYED);
+
+               /*
+                * Since sched_delayed means we cannot be current anywhere,
Quoted 8 lines

kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3803

Quoted 5 lines
kernel/sched/core.c +5 -0
+                * dequeue it here and have it fall through to the
+                * select_task_rq() case further along in ttwu() path.
+                * Note: Do not rip buddies apart else chaos follows.
+                */
+               if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&

Do we really care if DEQUEUE_DELAYED is enabled / disabled here when we encounter a delayed task?

The switch is for test convenience.

Quoted 7 lines

kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3804

Quoted 1 line
kernel/sched/core.c +1 -0
+                   !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {

Technically, we are still looking at the last wakeup here since record_wakee() is only called later. If we care about 1:1 buddies, should we just see "current == p->last_wakee", otherwise, there is a good chance "p" has a m:n waker-wakee relationship in which case perhaps a "want_affine" like heuristic can help?

The intent is to blunt the instrument a bit. Paul should have a highly active interrupt source, which will give wakeup credit to whatever is
sitting on that CPU, breaking 1:1 connections.. a little bit. That's
why it still migrates tbench buddies, but NOT at a rate that turns a tbench progression into a new low regression. The hope is that the load shift caused by that active interrupt source is enough to give Paul's regression some of the help it demonstrated wanting, without the collateral damage. It might now be so weak as to not meet the "meaningful" in my question, in which case it lands on the ginormous pile of meh, sorta works, but why would anyone care.

Quoted 6 lines

For science, I was wondering if the decision to dequeue + migrate or requeue the delayed task can be put off until after the whole select_task_rq() target selection (note: without the h_nr_delayed stuff, some of that wake_affine_idle() logic falls apart). Hackbench (which saw some regression with EEVDF Complete) seem to like it somewhat, but it still falls behind NO_DELAY_DEQUEUE.

You can, with a few more fast path cycles and some duplication, none of which looks very desirable.

Quoted 13 lines

    ==================================================================
     Test          : hackbench
     Units         : Normalized time in seconds
     Interpretation: Lower is better
     Statistic     : AMean
     ==================================================================
     NO_DELAY_DEQUEUE        Mike's v2    Full ttwu + requeue/migrate 5.76                  5.72  (  1% )        5.82  ( -1% ) 6.53                  6.56  (  0% )        6.65  ( -2% ) 6.79                  7.04  ( -4% )        7.02  ( -3% ) 6.91                  7.04  ( -2% )        7.03  ( -2% ) 7.63                  8.05  ( -6% )        7.88  ( -3% )

Quoted 4 lines

Only subtle changes in IBS profiles; there aren't any obvious shift in hotspots with hackbench at least. Not sure if it is just the act of needing to do a dequeue + enqueue from the wakeup context that adds to the overall regression.

Those numbers say say to me that hackbench doesn't care deeply. That works for me, because I don't care deeply about nutty fork bombs ;-)

-Mike

#168 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 2 lines · Mike Galbraith

The intent is to blunt the instrument a bit. Paul should have

Yeah I did... ahem, I meant of course Phil.

-Mike

#169 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 6 lines · On Tue, Nov 26, 2024 at 10:42:37AM +0100 Mike Galbraith wrote:
Quoted 2 lines · Mike Galbraith

The intent is to blunt the instrument a bit. Paul should have

Yeah I did... ahem, I meant of course Phil.

Heh, you are not alone, Mike :)

Quoted 2 lines

-Mike

#170 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Mike Galbraith
Quoted 6 lines · Mike Galbraith

The intent is to blunt the instrument a bit. Paul should have a highly active interrupt source, which will give wakeup credit to whatever is sitting on that CPU, breaking 1:1 connections.. a little bit.   That's why it still migrates tbench buddies, but NOT at a rate that turns a tbench progression into a new low regression.

BTW, the reason for the tbench wreckage being so bad is that when the box is near saturation, not only are a portion of the surviving sched_delayed tasks affine wakeups (always the optimal configuration for this fast mover cliebt/server pair in an L3 equipped box), they are exclusively affine wakeups. That is most definitely gonna hurt.

When saturating that becomes the best option for a lot of client/server pairs, even those with a lot of concurrency. Turning them loose to migrate at that time is far more likely than not to hurt a LOT, so V1 was doomed.

Quoted 6 lines

The hope is that the
load shift caused by that active interrupt source is enough to give Paul's regression some of the help it demonstrated wanting, without the collateral damage.  It might now be so weak as to not meet the "meaningful" in my question, in which case it lands on the ginormous pile of meh, sorta works, but why would anyone care.

In my IO challenged box, patch is useless to fio, nothing can help a load where all of the IO action, and wimpy action at that, is nailed to one CPU. I can see it helping other latency sensitive stuff, like say 1:N mother of all work and/or control threads (and ilk), but if Phil's problematic box looks anything like this box.. nah, it's a long reach.

-Mike

#171 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

In reply to Mike Galbraith
Quoted 19 lines · On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Quoted 18 lines · Phil Auld
Quoted 16 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld

This, below, by itself, did not do help and caused a small slowdown on some other tests.  Did this need to be on top of the wakeup change?

No, that made a mess.

Rashly speculating that turning mobile kthread component loose is what helped your write regression...

You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.

Quoted 6 lines
Quoted 1 line

Thanks, I'll try that.

You may still want to try that, but my box says probably not. Playing with your write command line, the players I see are pinned kworkers and mobile fio instances.

Yep. The PF_KTHREAD thing did not help.

Quoted 6 lines

Maybe try the below instead. The changelog is obsolete BS unless you say otherwise, but while twiddled V2 will still migrate tbench a bit, and per trace_printk() does still let all kinds of stuff wander off to roll the SIS dice, it does not even scratch the paint of the formerly obliterated tbench progression.

Will give this one a try when I get caught up from being off all week for US turkey day.

Thanks!

Quoted 1 line

Question: did wiping off the evil leave any meaningful goodness behind?

Is that for this patch?

If you mean for the original patch (which subsequently broke the reads) then no. It was more or less even for all the other tests. It fixed the randwrite issue by moving it to randread. Everything else we run regularly was about the same. So no extra goodness to help decide :)

Cheers,
Phil

Quoted 112 lines

---

sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.

V2: do not rip buddies apart, convenient on/off switch

Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
---
 kernel/sched/core.c     |   51 ++++++++++++++++++++++++++++++------------------
 kernel/sched/features.h |    5 ++++
 kernel/sched/sched.h    |    5 ++++
 3 files changed, 42 insertions(+), 19 deletions(-)
kernel/sched/core.c +32 -19
@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t
  */
 static int ttwu_runnable(struct task_struct *p, int wake_flags)
 {
-	struct rq_flags rf;
-	struct rq *rq;
-	int ret = 0;
-
-	rq = __task_rq_lock(p, &rf);
-	if (task_on_rq_queued(p)) {
-		update_rq_clock(rq);
-		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
-		if (!task_on_cpu(rq, p)) {
-			/*
-			 * When on_rq && !on_cpu the task is preempted, see if
-			 * it should preempt the task that is current now.
-			 */
-			wakeup_preempt(rq, p, wake_flags);
+	CLASS(__task_rq_lock, rq_guard)(p);
+	struct rq *rq = rq_guard.rq;
+
+	if (!task_on_rq_queued(p))
+		return 0;
+
+	update_rq_clock(rq);
+	if (p->se.sched_delayed) {
+		int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+		int dequeue = sched_feat(DEQUEUE_DELAYED);
+
+		/*
+		 * Since sched_delayed means we cannot be current anywhere,
+		 * dequeue it here and have it fall through to the
+		 * select_task_rq() case further along in ttwu() path.
+		 * Note: Do not rip buddies apart else chaos follows.
+		 */
+		if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&
+		    !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {
+			dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+			return 0;
 		}
-		ttwu_do_wakeup(p);
-		ret = 1;
+
+		enqueue_task(rq, p, queue_flags);
+	}
+	if (!task_on_cpu(rq, p)) {
+		/*
+		 * When on_rq && !on_cpu the task is preempted, see if
+		 * it should preempt the task that is current now.
+		 */
+		wakeup_preempt(rq, p, wake_flags);
 	}
-	__task_rq_unlock(rq, &rf);
+	ttwu_do_wakeup(p);
 
-	return ret;
+	return 1;
 }
 
 #ifdef CONFIG_SMP
kernel/sched/features.h +5 -0
@@ -47,6 +47,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
 SCHED_FEAT(DELAY_DEQUEUE, true)
+/*
+ * Free ONLY non-buddy delayed tasks to wakeup-migrate to avoid taking.
+ * an unnecessary latency hit.  Rending buddies asunder inflicts harm.
+ */
+SCHED_FEAT(DEQUEUE_DELAYED, true)
 SCHED_FEAT(DELAY_ZERO, true)
 
 /*
kernel/sched/sched.h +5 -0
@@ -1783,6 +1783,11 @@ task_rq_unlock(struct rq *rq, struct tas
 	raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 }
 
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+		    _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+		    __task_rq_unlock(_T->rq, &_T->rf),
+		    struct rq *rq; struct rq_flags rf)
+
 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct,
 		    _T->rq = task_rq_lock(_T->lock, &_T->rf),
 		    task_rq_unlock(_T->rq, _T->lock, &_T->rf),
#172 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 6 lines · Phil Auld
Quoted 1 line · On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:

Question: did wiping off the evil leave any meaningful goodness behind?

Is that for this patch?

Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.

-Mike

#173 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld

On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

Question: did wiping off the evil leave any meaningful goodness behind?

Is that for this patch?

Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.

I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat. The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.

Cheers,
Phil

Quoted 3 lines

-Mike

#174 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Hi Mike et al.,

Quoted 15 lines · On Mon, Dec 02, 2024 at 02:12:52PM -0500 Phil Auld wrote:
Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld

On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

Question: did wiping off the evil leave any meaningful goodness behind?

Is that for this patch?

Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.

Quoted 4 lines

I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat. The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.

In our testing this has the same effect as the original dequeue-when-delayed fix. It solves the randwrite issue and introduces the ~10-15% randread regression.

Seems to be a real trade-off here. The same guys who benefit from spreading in one case benefit from staying put in the other...

Cheers,
Phil

#175 lore raw

Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU

Quoted 24 lines · Phil Auld

Hi Mike et al.,

Quoted 15 lines · On Mon, Dec 02, 2024 at 02:12:52PM -0500 Phil Auld wrote:
Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld

On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

Question: did wiping off the evil leave any meaningful goodness behind?

Is that for this patch?

Yeah.  Trying it on my box with your write command line didn't improve the confidence level either.  My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances.  Patch was useless for that.

Quoted 4 lines

I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat.  The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.

Quoted 6 lines

In our testing this has the same effect as the original dequeue-when-delayed fix.  It solves the randwrite issue and introduces the ~10-15% randread regression.

Seems to be a real trade-off here. The same guys who benefit from spreading in one case benefit from staying put in the other...

Does as much harm as it does good isn't the mark of a keeper. Oh well.

-Mike

#176 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

In reply to Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

So... I was trying to make that prettier and ended up with something like this:

Passing ENQUEUE_DELAYED to dequeue_task() looks funky until you check the value, but otherwise yeah, when applied that looks better to me.

Quoted 83 lines

---
 kernel/sched/core.c  | 46 ++++++++++++++++++++++++++++------------------ kernel/sched/sched.h |  5 +++++

 2 files changed, 33 insertions(+), 18 deletions(-)
kernel/sched/core.c +2 -3
index 54d82c21fc8e..b083c6385e88 100644
@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
   */
  static int ttwu_runnable(struct task_struct *p, int wake_flags)
  {
-       struct rq_flags rf;
-       struct rq *rq;
-       int ret = 0;
+       CLASS(__task_rq_lock, rq_guard)(p);
+       struct rq *rq = rq_guard.rq;


-       rq = __task_rq_lock(p, &rf);
-       if (task_on_rq_queued(p)) {
-               update_rq_clock(rq);
-               if (p->se.sched_delayed)
-                       enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); -               if (!task_on_cpu(rq, p)) {
-                       /*
-                        * When on_rq && !on_cpu the task is preempted, see if -                        * it should preempt the task that is current now. -                        */
-                       wakeup_preempt(rq, p, wake_flags);
+       if (!task_on_rq_queued(p))
+               return 0;

kernel/sched/core.c +13 -0
+
+       update_rq_clock(rq);
+       if (p->se.sched_delayed) {
+               int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK;
+
+               /*
+                * Since sched_delayed means we cannot be current anywhere,
+                * dequeue it here and have it fall through to the
+                * select_task_rq() case further along the ttwu() path.
+                */
+               if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) {
+                       dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags);
+                       return 0;

                }
-               ttwu_do_wakeup(p);
-               ret = 1;
+
+               enqueue_task(rq, p, queue_flags);
        }
-       __task_rq_unlock(rq, &rf);
+       if (!task_on_cpu(rq, p)) {
+               /*
+                * When on_rq && !on_cpu the task is preempted, see if +                * it should preempt the task that is current now. +                */
+               wakeup_preempt(rq, p, wake_flags);
+       }
+       ttwu_do_wakeup(p);

-       return ret;
+       return 1;
 }

 #ifdef CONFIG_SMP

kernel/sched/sched.h +0 -0
index 21b1780c6695..1714ac38500f 100644
@@ -1787,6 +1787,11 @@ task_rq_unlock(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
         raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
  }

kernel/sched/sched.h +5 -0
+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct,
+                   _T->rq = __task_rq_lock(_T->lock, &_T->rf),
+                   __task_rq_unlock(_T->rq, &_T->rf),
+                   struct rq *rq; struct rq_flags rf)
+

 DEFINE_LOCK_GUARD_1(task_rq_lock, struct task_struct, _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),

author maintainer
#177 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Peter Zijlstra

So... I was trying to make that prettier and ended up with something like this:

Passing ENQUEUE_DELAYED to dequeue_task() looks funky until you check the value, but otherwise yeah, when applied that looks better to me.

Yeah, it does look funneh, but we've been doing that for a long long while.

Still, perhaps I should rename the shared ones to QUEUE_foo and only have the specific ones be {EN,DE}QUEUE_foo.

#178 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

In reply to Phil Auld

Hi Phil,

Quoted 25 lines · Phil Auld

Hi Peterm

Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:

Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.

OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.

Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.

Quoted 6 lines

Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.

Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.

This one is causing a 10-20% performance hit on our filesystem tests.

Quoted 8 lines

On 6.12-rc5 (so with the latest follow ons) we get:

with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s

The test is fio, something like this:

taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Quoted 8 lines

In this case it's ext4, but I'm not sure it will be FS specific.

I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.

It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.

[...]

#179 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

+cc Christian Loehle <[email protected]>

Quoted 35 lines · Dietmar Eggemann

Hi Phil,

Quoted 32 lines · Phil Auld

Hi Peterm

Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:

Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.

OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.

Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.

Quoted 6 lines

Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.

Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.

This one is causing a 10-20% performance hit on our filesystem tests.

On 6.12-rc5 (so with the latest follow ons) we get:

with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s

The test is fio, something like this:

Quoted 7 lines
Quoted 1 line

taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

                 ^^^^^^^
Quoted 3 lines

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

Christian Loehle just told me that my cpumask looks odd. Should be 0xaaaa instead.

Retested:

vanilla features: 954MB/s (mean out of 5 runs, σ: 30.83)
NO_DELAY_DEQUEUE: 932MB/s (mean out of 5 runs, σ: 28.10)

Now there are only 8 CPUs (instead of 10) for the 8 (+2) fio tasks. σ went up probably because of more wakeup/preemption latency.

Quoted 32 lines
# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Quoted 8 lines

In this case it's ext4, but I'm not sure it will be FS specific.

I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.

It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.

[...]

#180 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

In reply to Dietmar Eggemann

Hi Dietmar,

Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

Quoted 25 lines · Phil Auld

Hi Peterm

Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:

Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.

OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.

Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.

Quoted 6 lines

Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.

Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.

This one is causing a 10-20% performance hit on our filesystem tests.

Quoted 8 lines

On 6.12-rc5 (so with the latest follow ons) we get:

with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s

The test is fio, something like this:

taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff
Quoted 4 lines
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Most of the reports are on various NVME drives (samsung mostly I think).

One thing I should add is that it's all on LVM:

vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs

With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.

It's always the randwrite numbers. The rest look fine.

Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.

Thanks for taking a look.

Cheers,
Phil

Quoted 12 lines
Quoted 9 lines

In this case it's ext4, but I'm not sure it will be FS specific.

I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.

It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.

[...]

#181 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 85 lines · Phil Auld

Hi Dietmar,

Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

Quoted 25 lines · Phil Auld

Hi Peterm

Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:

Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.

OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.

Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.

Quoted 6 lines

Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.

Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.

This one is causing a 10-20% performance hit on our filesystem tests.

Quoted 8 lines

On 6.12-rc5 (so with the latest follow ons) we get:

with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s

The test is fio, something like this:

taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff 
Quoted 4 lines
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Most of the reports are on various NVME drives (samsung mostly I think).

One thing I should add is that it's all on LVM:

vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs

Quoted 6 lines

With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.

It's always the randwrite numbers. The rest look fine.

Hi Phil,

Thanks for the detailed instructions. Unfortunately even with your LVM setup on the platforms I've tried I don't see a regression so far, all the numbers are about equal for DELAY_DEQUEUE and NO_DELAY_DEQUEUE.

Anyway I have some follow-ups, first let me trim the fio command for readability:
fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

dropping defaults nr_files, loops, fsync, randrepeat
fio --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs

Adding the CPU affinities directly:
fio --cpus_allowed 1-8 --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs

Now I was wondering about the following:
Is it actually the kworker (not another fio) being preempted? (I'm pretty sure it is) To test: --cpus_allowed_policy split (each fio process gets it's own CPU).

Quoted 2 lines · You

I was thinking maybe the preemption was preventing some batching of IO completions or initiations. But that was wrong it seems.

So while it doesn't reproduce for me, the only thing being preempted regularly is the kworker (running iomap_dio_complete_work). I don't quite follow the "that was wrong it seems" part then. Could you elaborate?

Could you also post the other benchmark numbers? Does any of them score higher in IOPS? Is --rw write the same issue if you set --bs 4k (assuming you set a larger bs for seqwrite).

Can you set the kworkers handling completions to SCHED_BATCH too? Just to confirm.

Regards,
Christian

#182 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Hi Christian,

Quoted 91 lines · On Tue, Nov 05, 2024 at 09:53:49AM +0000 Christian Loehle wrote:
Quoted 90 lines · Phil Auld

Hi Dietmar,

Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

Quoted 25 lines · Phil Auld

Hi Peterm

Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:

Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.

OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.

Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.

Quoted 6 lines

Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.

Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.

This one is causing a 10-20% performance hit on our filesystem tests.

Quoted 8 lines

On 6.12-rc5 (so with the latest follow ons) we get:

with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s

The test is fio, something like this:

taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff 
Quoted 4 lines
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Most of the reports are on various NVME drives (samsung mostly I think).

One thing I should add is that it's all on LVM:

vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs

With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.

Quoted 8 lines
Quoted 1 line

It's always the randwrite numbers. The rest look fine.

Hi Phil,

Thanks for the detailed instructions. Unfortunately even with your LVM setup on the platforms I've tried I don't see a regression so far, all the numbers are about equal for DELAY_DEQUEUE and NO_DELAY_DEQUEUE.

Yeah, that's odd.

Fwiw:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   32
  On-line CPU(s) list:    0-31
Vendor ID:                AuthenticAMD
  BIOS Vendor ID:         Advanced Micro Devices, Inc.
  Model name:             AMD EPYC 7313P 16-Core Processor
    BIOS Model name:      AMD EPYC 7313P 16-Core Processor                Unknown CPU @ 3.0GHz
    BIOS CPU family:      107
    CPU family:           25
    ...

16 SMT2 cores (siblings are 16-31)

#lsblk -N

NAME    TYPE MODEL                      SERIAL              REV TRAN   RQ-SIZE  MQ
nvme3n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605178 GDC5602Q nvme      1023  32
nvme2n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605128 GDC5602Q nvme      1023  32
nvme0n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605125 GDC5602Q nvme      1023  32
nvme1n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605127 GDC5602Q nvme      1023  32

Where nvme0n1 is the one I'm actually using.

I'm on 6.12.0-0.rc5.44.eln143.x86_64 which is v6.12-rc5 with RHEL .config. This should have little to no franken-kernel bits but now that I have the machine I'll build from upstream (with the RHEL .config still) to make sure.

We did see it on all the RCs so far.

Quoted 9 lines

Anyway I have some follow-ups, first let me trim the fio command for readability:
fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs

dropping defaults nr_files, loops, fsync, randrepeat
fio --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs

Adding the CPU affinities directly:
fio --cpus_allowed 1-8 --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs

Fair enough. It should work the same with taskset I suppose except the below bit. I've been given this from our perforance team. They have a framework that produces nice html pages with red and green results and graphs and whatnot. Right now it's in the form of a script that pulls the KB/s number of out the json output which is nice and keeps me from going crosseyed looking at the full fio run output.

Quoted 3 lines

Now I was wondering about the following:
Is it actually the kworker (not another fio) being preempted? (I'm pretty sure it is) To test: --cpus_allowed_policy split (each fio process gets it's own CPU).

with --cpus-allowed and --cpus_allowed_policy split the results with DELAY_DEQUEUE are better (540MB/s) but with NO_DELAY_DEQUEUE they are also better (640 MB/s). It was 510MB/s and 590MB/s before.

Quoted 9 lines
Quoted 2 lines · You

I was thinking maybe the preemption was preventing some batching of IO completions or initiations. But that was wrong it seems.

So while it doesn't reproduce for me, the only thing being preempted regularly is the kworker (running iomap_dio_complete_work). I don't quite follow the "that was wrong it seems" part then. Could you elaborate?

I was thinking that the fio batch test along with the disabling WAKEUP_PREEMPTION was telling me that it wasn't the over preemption issue, but that also I could be wrong about...

Quoted 3 lines

Could you also post the other benchmark numbers? Does any of them score higher in IOPS? Is --rw write the same issue if you set --bs 4k (assuming you set a larger bs for seqwrite).

I don't have numbers for all of the other flavors but I ran --rw write --bs 4k:

DELAY_DEQUEUE ~590MB/s
NO_DELAY_DEQUEUE ~840MB/s

Those results are not good for DELAY_DEQUEUE either.

Quoted 1 line

Can you set the kworkers handling completions to SCHED_BATCH too? Just to confirm.

I think I did the wrong kworkes the first time. So I'll try again to figure out which kworkers to twiddle (or I'll just do all 227 of them...).

Thanks,
Phil

Quoted 4 lines

Regards,
Christian

#183 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

In reply to Phil Auld
Quoted 9 lines · Phil Auld

Hi Dietmar,

Quoted 5 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

Quoted 2 lines · Phil Auld

Hi Peterm

[...]

Quoted 49 lines
Quoted 23 lines

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff
Quoted 4 lines
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Most of the reports are on various NVME drives (samsung mostly I think).

One thing I should add is that it's all on LVM:

vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs

With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.

Quoted 4 lines

It's always the randwrite numbers. The rest look fine.

Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.

One reason I don't see the difference between DELAY_DEQUEUE and NO_DELAY_DEQUEUE could be because of the affinity of the related nvme interrupts:

$ cat /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...

132:   0    0  1523653    0   0        0   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1
133:   0    0        0    0   0  1338451   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2
134:   0    0        0    0   0        0   0    0  2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3
$ cat /proc/irq/132/smp_affinity_list 
0-2
cat /proc/irq/133/smp_affinity_list 
3-5
cat /proc/irq/134/smp_affinity_list 
6-8

So the 8 fio tasks from:

# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k
  --runtime 8s --iodepth 32 --direct 1 --ioengine libaio
  --numjobs 8 --size 30g --name default --time_based
  --group_reporting --cpus_allowed_policy shared
  --directory /testfs

don't have to fight with per-CPU kworkers on each CPU.

e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
run iomap_dio_complete_work() in kworker/8:x'

In case I trace the 'task_on_rq_queued(p) && p->se.sched_delayed && rq->nr_running > 1) condition in ttwu_runnable() condition i only see the per-CPU kworker in there, so p->nr_cpus_allowed == 1.

So the patch shouldn't make a difference for this scenario?

But maybe your VDO or thinpool setup creates waker/wakee pairs with wakee->nr_cpus_allowed > 1?

Does your machine has single CPU smp_affinity masks for these nvme interrupts?

[...]

#184 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 95 lines · On Fri, Nov 08, 2024 at 03:53:26PM +0100 Dietmar Eggemann wrote:
Quoted 9 lines · Phil Auld

Hi Dietmar,

Quoted 5 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

Quoted 2 lines · Phil Auld

Hi Peterm

[...]

Quoted 49 lines
Quoted 23 lines

I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)

Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.

vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)

# sudo lshw -class disk -class storage
  *-nvme                    
       description: NVMe device
       product: GIGABYTE GP-ASM2NE6500GTTD
       vendor: Phison Electronics Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: /dev/nvme0
       version: EGFM13.2
       ...
       capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
       configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
       resources: irq:16 memory:70800000-70803fff
Quoted 4 lines
# mount | grep ^/dev/nvme0
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

Which disk device you're using?

Most of the reports are on various NVME drives (samsung mostly I think).

One thing I should add is that it's all on LVM:

vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs

With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.

Quoted 4 lines

It's always the randwrite numbers. The rest look fine.

Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.

One reason I don't see the difference between DELAY_DEQUEUE and NO_DELAY_DEQUEUE could be because of the affinity of the related nvme interrupts:

$ cat /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...

132:   0    0  1523653    0   0        0   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1
133:   0    0        0    0   0  1338451   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2
134:   0    0        0    0   0        0   0    0  2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3
$ cat /proc/irq/132/smp_affinity_list 
0-2
cat /proc/irq/133/smp_affinity_list 
3-5
cat /proc/irq/134/smp_affinity_list 
6-8

So the 8 fio tasks from:

# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k
  --runtime 8s --iodepth 32 --direct 1 --ioengine libaio
  --numjobs 8 --size 30g --name default --time_based
  --group_reporting --cpus_allowed_policy shared
  --directory /testfs

don't have to fight with per-CPU kworkers on each CPU.

Quoted 9 lines

e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
run iomap_dio_complete_work() in kworker/8:x'

In case I trace the 'task_on_rq_queued(p) && p->se.sched_delayed && rq->nr_running > 1) condition in ttwu_runnable() condition i only see the per-CPU kworker in there, so p->nr_cpus_allowed == 1.

So the patch shouldn't make a difference for this scenario?

If the kworker is waking up an fio task it could. I don't think they are bound to a single cpu.

But yes if your trace is only showing the kworker there then it would not help. Are you actually able to reproduce the difference?

Quoted 3 lines

But maybe your VDO or thinpool setup creates waker/wakee pairs with wakee->nr_cpus_allowed > 1?

That's certainly possible but I don't know for sure. There are well more dio kworkers on the box than cpus though if I recall. I don't know if they all have singel cpu affinities.

Quoted 3 lines

Does your machine has single CPU smp_affinity masks for these nvme interrupts?

I don't know. I had to give the machine back.

Cheers,
Phil

Quoted 35 lines

[...]

#185 lore raw

Re: [PATCH 17/24] sched/fair: Implement delayed dequeue

Quoted 9 lines · Phil Auld
Quoted 8 lines · On Fri, Nov 08, 2024 at 03:53:26PM +0100 Dietmar Eggemann wrote:
Quoted 7 lines · Phil Auld

Hi Dietmar,

Quoted 3 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:

Hi Phil,

On 01/11/2024 13:47, Phil Auld wrote:

[...]

Quoted 35 lines
Quoted 35 lines

One reason I don't see the difference between DELAY_DEQUEUE and NO_DELAY_DEQUEUE could be because of the affinity of the related nvme interrupts:

$ cat /proc/interrupts

CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...

132:   0    0  1523653    0   0        0   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1
133:   0    0        0    0   0  1338451   0    0       0  ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2
134:   0    0        0    0   0        0   0    0  2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3
$ cat /proc/irq/132/smp_affinity_list 
0-2
cat /proc/irq/133/smp_affinity_list 
3-5
cat /proc/irq/134/smp_affinity_list 
6-8

So the 8 fio tasks from:

# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k
  --runtime 8s --iodepth 32 --direct 1 --ioengine libaio
  --numjobs 8 --size 30g --name default --time_based
  --group_reporting --cpus_allowed_policy shared
  --directory /testfs

don't have to fight with per-CPU kworkers on each CPU.

e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
run iomap_dio_complete_work() in kworker/8:x'

In case I trace the 'task_on_rq_queued(p) && p->se.sched_delayed && rq->nr_running > 1) condition in ttwu_runnable() condition i only see the per-CPU kworker in there, so p->nr_cpus_allowed == 1.

Quoted 8 lines
Quoted 2 lines

So the patch shouldn't make a difference for this scenario?

If the kworker is waking up an fio task it could. I don't think they are bound to a single cpu.

But yes if your trace is only showing the kworker there then it would not help. Are you actually able to reproduce the difference?

No, with my setup I don't see any difference running your fio test. But the traces also show me that there are no scenarios in which this patch can make a difference in the scores.

Quoted 7 lines
Quoted 3 lines

But maybe your VDO or thinpool setup creates waker/wakee pairs with wakee->nr_cpus_allowed > 1?

That's certainly possible but I don't know for sure. There are well more dio kworkers on the box than cpus though if I recall. I don't know if they all have singel cpu affinities.

Yeah there must be more tasks (inc. kworkers) w/ 'p->nr_cpus_allowed > 1' involved.

Quoted 5 lines
Quoted 3 lines

Does your machine has single CPU smp_affinity masks for these nvme interrupts?

I don't know. I had to give the machine back.

Ah, too late then ;-)

author maintainer
#186 lore raw

PATCH 18/24 sched/fair: Implement DELAY_ZERO

In reply to Peter Zijlstra

'Extend' DELAY_DEQUEUE by noting that since we wanted to dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).

---
 kernel/sched/fair.c     |   16 ++++++++++++++++
 kernel/sched/features.h |    3 +++
 2 files changed, 19 insertions(+)
kernel/sched/fair.c +16 -0
@@ -5529,6 +5529,8 @@ pick_next_entity(struct rq *rq, struct c
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		SCHED_WARN_ON(se->sched_delayed);
 		SCHED_WARN_ON(se->on_rq);
+		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
+			se->vlag = 0;
 
 		return NULL;
 	}
@@ -6827,6 +6829,20 @@ requeue_delayed_entity(struct sched_enti
 	SCHED_WARN_ON(!se->sched_delayed);
 	SCHED_WARN_ON(!se->on_rq);
 
+	if (sched_feat(DELAY_ZERO)) {
+		update_entity_lag(cfs_rq, se);
+		if (se->vlag > 0) {
+			cfs_rq->nr_running--;
+			if (se != cfs_rq->curr)
+				__dequeue_entity(cfs_rq, se);
+			se->vlag = 0;
+			place_entity(cfs_rq, se, 0);
+			if (se != cfs_rq->curr)
+				__enqueue_entity(cfs_rq, se);
+			cfs_rq->nr_running++;
+		}
+	}
+
 	se->sched_delayed = 0;
 }
 
kernel/sched/features.h +3 -0
@@ -34,8 +34,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  * By delaying the dequeue for non-eligible tasks, they remain in the
  * competition and can burn off their negative lag. When they get selected
  * they'll have positive lag by definition.
+ *
+ * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
 SCHED_FEAT(DELAY_DEQUEUE, true)
+SCHED_FEAT(DELAY_ZERO, true)
 
 /*
  * Allow wakeup-time preemption of the current task:
merge bot Applied
#187 lore raw

[tip: sched/core] sched/fair: Implement DELAY_ZERO

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     54a58a78779169f9c92a51facf6de7ce94962328
Gitweb:        https://git.kernel.org/tip/54a58a78779169f9c92a51facf6de7ce94962328
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Thu, 23 May 2024 12:26:06 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:44 +02:00

sched/fair: Implement DELAY_ZERO

'Extend' DELAY_DEQUEUE by noting that since we wanted to dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).

---
 kernel/sched/fair.c     | 20 ++++++++++++++++++--
 kernel/sched/features.h |  3 +++
 2 files changed, 21 insertions(+), 2 deletions(-)
kernel/sched/fair.c +18 -2
index da5065a..1a59339 100644
@@ -5447,8 +5447,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	if ((flags & (DEQUEUE_SAVE | DEQUEUE_MOVE)) != DEQUEUE_SAVE)
 		update_min_vruntime(cfs_rq);
 
-	if (flags & DEQUEUE_DELAYED)
+	if (flags & DEQUEUE_DELAYED) {
 		se->sched_delayed = 0;
+		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
+			se->vlag = 0;
+	}
 
 	if (cfs_rq->nr_running == 0)
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
@@ -5527,7 +5530,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		SCHED_WARN_ON(se->sched_delayed);
 		SCHED_WARN_ON(se->on_rq);
-
 		return NULL;
 	}
 	return se;
@@ -6825,6 +6827,20 @@ requeue_delayed_entity(struct sched_entity *se)
 	SCHED_WARN_ON(!se->sched_delayed);
 	SCHED_WARN_ON(!se->on_rq);
 
+	if (sched_feat(DELAY_ZERO)) {
+		update_entity_lag(cfs_rq, se);
+		if (se->vlag > 0) {
+			cfs_rq->nr_running--;
+			if (se != cfs_rq->curr)
+				__dequeue_entity(cfs_rq, se);
+			se->vlag = 0;
+			place_entity(cfs_rq, se, 0);
+			if (se != cfs_rq->curr)
+				__enqueue_entity(cfs_rq, se);
+			cfs_rq->nr_running++;
+		}
+	}
+
 	se->sched_delayed = 0;
 }
 
kernel/sched/features.h +3 -0
index 1feaa7b..7fdeb55 100644
@@ -34,8 +34,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  * By delaying the dequeue for non-eligible tasks, they remain in the
  * competition and can burn off their negative lag. When they get selected
  * they'll have positive lag by definition.
+ *
+ * DELAY_ZERO clips the lag on dequeue (or wakeup) to 0.
  */
 SCHED_FEAT(DELAY_DEQUEUE, true)
+SCHED_FEAT(DELAY_ZERO, true)
 
 /*
  * Allow wakeup-time preemption of the current task:
author maintainer
#188 lore raw

PATCH 19/24 sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

In reply to Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)
kernel/sched/fair.c +2 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
 			se->sched_delayed = 1;
+			update_load_avg(cfs_rq, se, 0);
 			return false;
 		}
 	}
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
 	}
 
 	se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);
 }
 
 /*
kernel/sched/sched.h +6 -0
@@ -816,6 +816,9 @@ static inline void se_update_runnable(st
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+
 	if (entity_is_task(se))
 		return !!se->on_rq;
 	else
@@ -830,6 +833,9 @@ static inline void se_update_runnable(st
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+
 	return !!se->on_rq;
 }
 
#189 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 15 lines · Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                      if (cfs_rq->next == se)
                              cfs_rq->next = NULL;
                      se->sched_delayed = 1;
+			update_load_avg(cfs_rq, se, 0);

Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.

kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818

Quoted 8 lines
kernel/sched/fair.c +1 -0
                      return false;
              }
      }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
      se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

kernel/sched/sched.h static inline void se_update_runnable(st · line 821

Quoted 12 lines
kernel/sched/fair.c +0 -0
 }
 
 /*
kernel/sched/sched.h +3 -0
@@ -816,6 +816,9 @@ static inline void se_update_runnable(st
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+

Per __update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of ___update_load_sum()?

Quoted 13 lines
kernel/sched/sched.h +3 -0
      if (entity_is_task(se))
              return !!se->on_rq;
      else
@@ -830,6 +833,9 @@ static inline void se_update_runnable(st
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+
      return !!se->on_rq;
 }
 
author maintainer
#190 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                      if (cfs_rq->next == se)
                              cfs_rq->next = NULL;
                      se->sched_delayed = 1;
+			update_load_avg(cfs_rq, se, 0);

Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.

Quoted 6 lines
kernel/sched/fair.c +0 -0
                      return false;
              }
      }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
Quoted 4 lines

kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818

Quoted 2 lines
kernel/sched/fair.c +1 -0
      se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/

Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +0 -0
 }
 
 /*
kernel/sched/sched.h +0 -0
@@ -816,6 +816,9 @@ static inline void se_update_runnable(st
 
Quoted 9 lines

kernel/sched/sched.h static inline void se_update_runnable(st · line 821

Quoted 5 lines
kernel/sched/sched.h +3 -0
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+

Per __update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of ___update_load_sum()?

So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.

author maintainer
#191 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                      if (cfs_rq->next == se)
                              cfs_rq->next = NULL;
                      se->sched_delayed = 1;
+			update_load_avg(cfs_rq, se, 0);

Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.

Quoted 6 lines
kernel/sched/fair.c +0 -0
                      return false;
              }
      }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
Quoted 8 lines
Quoted 4 lines

kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818

Quoted 2 lines
kernel/sched/fair.c +1 -0
      se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/

OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...

---

kernel/sched/fair.c +11 -15
index 1b15dbfb1ce5..fa8907f2c716 100644
@@ -5461,14 +5461,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	bool sleep = flags & DEQUEUE_SLEEP;
 
+	update_curr(cfs_rq);
+
 	if (flags & DEQUEUE_DELAYED) {
-		/*
-		 * DEQUEUE_DELAYED is typically called from pick_next_entity()
-		 * at which point we've already done update_curr() and do not
-		 * want to do so again.
-		 */
 		SCHED_WARN_ON(!se->sched_delayed);
-		se->sched_delayed = 0;
 	} else {
 		bool delay = sleep;
 		/*
@@ -5479,14 +5475,13 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 			delay = false;
 
 		SCHED_WARN_ON(delay && se->sched_delayed);
-		update_curr(cfs_rq);
 
 		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
-			se->sched_delayed = 1;
 			update_load_avg(cfs_rq, se, 0);
+			se->sched_delayed = 1;
 			return false;
 		}
 	}
@@ -5536,6 +5531,12 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	if ((flags & (DEQUEUE_SAVE | DEQUEUE_MOVE)) != DEQUEUE_SAVE)
 		update_min_vruntime(cfs_rq);
 
+	if (flags & DEQUEUE_DELAYED) {
+		se->sched_delayed = 0;
+		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
+			se->vlag = 0;
+	}
+
 	if (cfs_rq->nr_running == 0)
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 
@@ -5611,11 +5612,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)
 	struct sched_entity *se = pick_eevdf(cfs_rq);
 	if (se->sched_delayed) {
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-		SCHED_WARN_ON(se->sched_delayed);
-		SCHED_WARN_ON(se->on_rq);
-		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
-			se->vlag = 0;
-
 		return NULL;
 	}
 	return se;
@@ -6906,7 +6902,7 @@ requeue_delayed_entity(struct sched_entity *se)
 	struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
 	/*
-	 * se->sched_delayed should imply both: se->on_rq == 1.
+	 * se->sched_delayed should imply: se->on_rq == 1.
 	 * Because a delayed entity is one that is still on
 	 * the runqueue competing until elegibility.
 	 */
@@ -6927,8 +6923,8 @@ requeue_delayed_entity(struct sched_entity *se)
 		}
 	}
 
-	se->sched_delayed = 0;
 	update_load_avg(cfs_rq, se, 0);
+	se->sched_delayed = 0;
 }
 
 /*
author maintainer
#192 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 57 lines · Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                      if (cfs_rq->next == se)
                              cfs_rq->next = NULL;
                      se->sched_delayed = 1;
+			update_load_avg(cfs_rq, se, 0);

Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.

Quoted 6 lines
kernel/sched/fair.c +0 -0
                      return false;
              }
      }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
Quoted 8 lines
Quoted 4 lines

kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818

Quoted 2 lines
kernel/sched/fair.c +1 -0
      se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/

OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...

---

kernel/sched/fair.c +2 -0
index 1b15dbfb1ce5..fa8907f2c716 100644
@@ -5461,14 +5461,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
 	bool sleep = flags & DEQUEUE_SLEEP;
 
+	update_curr(cfs_rq);
+
 	if (flags & DEQUEUE_DELAYED) {

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5470

Quoted 10 lines
kernel/sched/fair.c +0 -6
-		/*
-		 * DEQUEUE_DELAYED is typically called from pick_next_entity()
-		 * at which point we've already done update_curr() and do not
-		 * want to do so again.
-		 */
 		SCHED_WARN_ON(!se->sched_delayed);
-		se->sched_delayed = 0;
 	} else {
 		bool delay = sleep;
 		/*

Because repeated update_curr() is harmless (I think I was thinking it would move the clock, but it doesn't do that), but a missed update_curr() makes a mess.

Quoted 6 lines
kernel/sched/fair.c +0 -1
@@ -5479,14 +5475,13 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 			delay = false;
 
 		SCHED_WARN_ON(delay && se->sched_delayed);
-		update_curr(cfs_rq);
 

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5487

Quoted 10 lines
kernel/sched/fair.c +1 -1
 		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
-			se->sched_delayed = 1;
 			update_load_avg(cfs_rq, se, 0);
+			se->sched_delayed = 1;
 			return false;
 		}
 	}

As you said, update to now, then mark delayed.

Quoted 15 lines
kernel/sched/fair.c +6 -0
@@ -5536,6 +5531,12 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	if ((flags & (DEQUEUE_SAVE | DEQUEUE_MOVE)) != DEQUEUE_SAVE)
 		update_min_vruntime(cfs_rq);
 
+	if (flags & DEQUEUE_DELAYED) {
+		se->sched_delayed = 0;
+		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
+			se->vlag = 0;
+	}
+
 	if (cfs_rq->nr_running == 0)
 		update_idle_cfs_rq_clock_pelt(cfs_rq);
 
@@ -5611,11 +5612,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)
 	struct sched_entity *se = pick_eevdf(cfs_rq);

kernel/sched/fair.c pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq) · line 5617

Quoted 10 lines
kernel/sched/fair.c +0 -5
 	if (se->sched_delayed) {
 		dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-		SCHED_WARN_ON(se->sched_delayed);
-		SCHED_WARN_ON(se->on_rq);
-		if (sched_feat(DELAY_ZERO) && se->vlag > 0)
-			se->vlag = 0;
-
 		return NULL;
 	}
 	return se;

Because if we have to clear delayed at the end up dequeue, we might as well do all of the fixups at that point.

kernel/sched/fair.c requeue_delayed_entity(struct sched_entity *se) · line 6908

Quoted 9 lines
kernel/sched/fair.c +1 -1
@@ -6906,7 +6902,7 @@ requeue_delayed_entity(struct sched_entity *se)
 	struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
 	/*
-	 * se->sched_delayed should imply both: se->on_rq == 1.
+	 * se->sched_delayed should imply: se->on_rq == 1.
 	 * Because a delayed entity is one that is still on
 	 * the runqueue competing until elegibility.
 	 */

edit fail, somewhere along history.

kernel/sched/fair.c requeue_delayed_entity(struct sched_entity *se) · line 6928

Quoted 8 lines
kernel/sched/fair.c +1 -1
@@ -6927,8 +6923,8 @@ requeue_delayed_entity(struct sched_entity *se)
 		}
 	}
 
-	se->sched_delayed = 0;
 	update_load_avg(cfs_rq, se, 0);
+	se->sched_delayed = 0;
 }

What you said..

#193 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Valentin Schneider

Review requeue_delayed_entity(struct sched_enti · line 6818

Quoted 5 lines · Peter Zijlstra
(quoted diff) +1 -0
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
      se->sched_delayed = 0;
+	update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

Quoted 8 lines
Quoted 3 lines

Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/

OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...

Looks right to me, thanks! I'll go test the newer stack of patches.

maintainer
#194 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

In reply to Peter Zijlstra
Quoted 58 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

---
 kernel/sched/fair.c  |    2 ++
 kernel/sched/sched.h |    6 ++++++
 2 files changed, 8 insertions(+)

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                      if (cfs_rq->next == se)
                              cfs_rq->next = NULL;
                      se->sched_delayed = 1;
+                   update_load_avg(cfs_rq, se, 0);

Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.

Quoted 6 lines
kernel/sched/fair.c +0 -0
                      return false;
              }
      }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti
      }
 
Quoted 4 lines

kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818

Quoted 2 lines
kernel/sched/fair.c +1 -0
      se->sched_delayed = 0;
+   update_load_avg(cfs_rq, se, 0);

Ditto on the ordering

Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/

Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +0 -0
 }
 
 /*
kernel/sched/sched.h +0 -0
@@ -816,6 +816,9 @@ static inline void se_update_runnable(st
 
Quoted 9 lines

kernel/sched/sched.h static inline void se_update_runnable(st · line 821

Quoted 5 lines
kernel/sched/sched.h +3 -0
 static inline long se_runnable(struct sched_entity *se)
 {
+   if (se->sched_delayed)
+           return false;
+

Per __update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of ___update_load_sum()?

Quoted 3 lines

So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.

We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the load_avg on this rq

Shouldn't we track separately the sum of the weight of delayed dequeue to apply it only on vruntime update ?

Quoted 1 line
author maintainer
#195 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 7 lines · Vincent Guittot
Quoted 3 lines

So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.

We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the load_avg on this rq

So far load has been a direct sum of all weight. Additionally, we delay until a task gets picked again, migrating tasks to other CPUs will expedite this condition.

Anyway, at the moment I don't have strong evidence either which way, and the above argument seem to suggest not changing things for now.

We can always re-evaluate.

maintainer
#196 lore raw

Re: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Vincent Guittot
Quoted 3 lines

So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.

Quoted 5 lines
Quoted 3 lines

We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the load_avg on this rq

So far load has been a direct sum of all weight. Additionally, we delay

it has been the sum of all runnable tasks but delayed tasks are not runnable anymore. The task stays "enqueued" only to help clearing its lag

Quoted 7 lines

until a task gets picked again, migrating tasks to other CPUs will expedite this condition.

Anyway, at the moment I don't have strong evidence either which way, and the above argument seem to suggest not changing things for now.

We can always re-evaluate.

merge bot Applied
#197 lore raw

[tip: sched/core] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     fc1892becd5672f52329a75c73117b60ac7841b7
Gitweb:        https://git.kernel.org/tip/fc1892becd5672f52329a75c73117b60ac7841b7
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Fri, 26 Apr 2024 13:00:50 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00

sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE

Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.

As such, don't count this time towards runnable.

Thanks to Valentin for spotting I had this backwards initially.

---
 kernel/sched/fair.c  | 2 ++
 kernel/sched/sched.h | 6 ++++++
 2 files changed, 8 insertions(+)
kernel/sched/fair.c +2 -0
index 1a59339..0eb1bbf 100644
@@ -5402,6 +5402,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
+			update_load_avg(cfs_rq, se, 0);
 			se->sched_delayed = 1;
 			return false;
 		}
@@ -6841,6 +6842,7 @@ requeue_delayed_entity(struct sched_entity *se)
 		}
 	}
 
+	update_load_avg(cfs_rq, se, 0);
 	se->sched_delayed = 0;
 }
 
kernel/sched/sched.h +6 -0
index 263b4de..2f5d658 100644
@@ -820,6 +820,9 @@ static inline void se_update_runnable(struct sched_entity *se)
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+
 	if (entity_is_task(se))
 		return !!se->on_rq;
 	else
@@ -834,6 +837,9 @@ static inline void se_update_runnable(struct sched_entity *se) { }
 
 static inline long se_runnable(struct sched_entity *se)
 {
+	if (se->sched_delayed)
+		return false;
+
 	return !!se->on_rq;
 }
 
author maintainer
#198 lore raw

PATCH 20/24 sched/fair: Avoid re-setting virtual deadline on migrations

In reply to Peter Zijlstra

During OSPM24 Youssef noted that migrations are re-setting the virtual deadline. Notably everything that does a dequeue-enqueue, like setting nice, changing preferred numa-node, and a myriad of other random crap, will cause this to happen.

This shouldn't be. Preserve the relative virtual deadline across such dequeue/enqueue cycles.

---
 include/linux/sched.h   |    6 ++++--
 kernel/sched/fair.c     |   23 ++++++++++++++++++-----
 kernel/sched/features.h |    4 ++++
 3 files changed, 26 insertions(+), 7 deletions(-)
include/linux/sched.h +4 -2
@@ -542,8 +542,10 @@ struct sched_entity {
 	u64				min_vruntime;
 
 	struct list_head		group_node;
-	unsigned int			on_rq;
-	unsigned int			sched_delayed;
+	unsigned char			on_rq;
+	unsigned char			sched_delayed;
+	unsigned char			rel_deadline;
+					/* hole */
 
 	u64				exec_start;
 	u64				sum_exec_runtime;
kernel/sched/fair.c +18 -5
@@ -5270,6 +5270,12 @@ place_entity(struct cfs_rq *cfs_rq, stru
 
 	se->vruntime = vruntime - lag;
 
+	if (sched_feat(PLACE_REL_DEADLINE) && se->rel_deadline) {
+		se->deadline += se->vruntime;
+		se->rel_deadline = 0;
+		return;
+	}
+
 	/*
 	 * When joining the competition; the existing tasks will be,
 	 * on average, halfway through their slice, as such start tasks
@@ -5382,6 +5388,8 @@ static __always_inline void return_cfs_r
 static bool
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
+	bool sleep = flags & DEQUEUE_SLEEP;
+
 	if (flags & DEQUEUE_DELAYED) {
 		/*
 		 * DEQUEUE_DELAYED is typically called from pick_next_entity()
@@ -5391,19 +5399,18 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 		SCHED_WARN_ON(!se->sched_delayed);
 		se->sched_delayed = 0;
 	} else {
-		bool sleep = flags & DEQUEUE_SLEEP;
-
+		bool delay = sleep;
 		/*
 		 * DELAY_DEQUEUE relies on spurious wakeups, special task
 		 * states must not suffer spurious wakeups, excempt them.
 		 */
 		if (flags & DEQUEUE_SPECIAL)
-			sleep = false;
+			delay = false;
 
-		SCHED_WARN_ON(sleep && se->sched_delayed);
+		SCHED_WARN_ON(delay && se->sched_delayed);
 		update_curr(cfs_rq);
 
-		if (sched_feat(DELAY_DEQUEUE) && sleep &&
+		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
@@ -5434,6 +5441,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 	clear_buddies(cfs_rq, se);
 
 	update_entity_lag(cfs_rq, se);
+	if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
+		se->deadline -= se->vruntime;
+		se->rel_deadline = 1;
+	}
+
 	if (se != cfs_rq->curr)
 		__dequeue_entity(cfs_rq, se);
 	se->on_rq = 0;
@@ -13024,6 +13036,7 @@ static void switched_from_fair(struct rq
 	 * over the excursion into the new class.
 	 */
 	p->se.vlag = 0;
+	p->se.rel_deadline = 0;
 }
 
 static void switched_to_fair(struct rq *rq, struct task_struct *p)
kernel/sched/features.h +4 -0
@@ -10,6 +10,10 @@ SCHED_FEAT(PLACE_LAG, true)
  */
 SCHED_FEAT(PLACE_DEADLINE_INITIAL, true)
 /*
+ * Preserve relative virtual deadline on 'migration'.
+ */
+SCHED_FEAT(PLACE_REL_DEADLINE, true)
+/*
  * Inhibit (wakeup) preemption until the current task has either matched the
  * 0-lag point or until is has exhausted it's slice.
  */
merge bot Applied
#199 lore raw

[tip: sched/core] sched/fair: Avoid re-setting virtual deadline on 'migrations'

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     82e9d0456e06cebe2c89f3c73cdbc9e3805e9437
Gitweb:        https://git.kernel.org/tip/82e9d0456e06cebe2c89f3c73cdbc9e3805e9437
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Fri, 31 May 2024 15:49:40 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00

sched/fair: Avoid re-setting virtual deadline on 'migrations'

During OSPM24 Youssef noted that migrations are re-setting the virtual deadline. Notably everything that does a dequeue-enqueue, like setting nice, changing preferred numa-node, and a myriad of other random crap, will cause this to happen.

This shouldn't be. Preserve the relative virtual deadline across such dequeue/enqueue cycles.

---
 include/linux/sched.h   |  6 ++++--
 kernel/sched/fair.c     | 23 ++++++++++++++++++-----
 kernel/sched/features.h |  4 ++++
 3 files changed, 26 insertions(+), 7 deletions(-)
include/linux/sched.h +4 -2
index 8a3a389..d25e1cf 100644
@@ -544,8 +544,10 @@ struct sched_entity {
 	u64				min_vruntime;
 
 	struct list_head		group_node;
-	unsigned int			on_rq;
-	unsigned int			sched_delayed;
+	unsigned char			on_rq;
+	unsigned char			sched_delayed;
+	unsigned char			rel_deadline;
+					/* hole */
 
 	u64				exec_start;
 	u64				sum_exec_runtime;
kernel/sched/fair.c +18 -5
index 0eb1bbf..fef0e1f 100644
@@ -5270,6 +5270,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	se->vruntime = vruntime - lag;
 
+	if (sched_feat(PLACE_REL_DEADLINE) && se->rel_deadline) {
+		se->deadline += se->vruntime;
+		se->rel_deadline = 0;
+		return;
+	}
+
 	/*
 	 * When joining the competition; the existing tasks will be,
 	 * on average, halfway through their slice, as such start tasks
@@ -5382,23 +5388,24 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
 static bool
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
+	bool sleep = flags & DEQUEUE_SLEEP;
+
 	update_curr(cfs_rq);
 
 	if (flags & DEQUEUE_DELAYED) {
 		SCHED_WARN_ON(!se->sched_delayed);
 	} else {
-		bool sleep = flags & DEQUEUE_SLEEP;
-
+		bool delay = sleep;
 		/*
 		 * DELAY_DEQUEUE relies on spurious wakeups, special task
 		 * states must not suffer spurious wakeups, excempt them.
 		 */
 		if (flags & DEQUEUE_SPECIAL)
-			sleep = false;
+			delay = false;
 
-		SCHED_WARN_ON(sleep && se->sched_delayed);
+		SCHED_WARN_ON(delay && se->sched_delayed);
 
-		if (sched_feat(DELAY_DEQUEUE) && sleep &&
+		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			if (cfs_rq->next == se)
 				cfs_rq->next = NULL;
@@ -5429,6 +5436,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	clear_buddies(cfs_rq, se);
 
 	update_entity_lag(cfs_rq, se);
+	if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
+		se->deadline -= se->vruntime;
+		se->rel_deadline = 1;
+	}
+
 	if (se != cfs_rq->curr)
 		__dequeue_entity(cfs_rq, se);
 	se->on_rq = 0;
@@ -12992,6 +13004,7 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p)
 	if (p->se.sched_delayed) {
 		dequeue_task(rq, p, DEQUEUE_NOCLOCK | DEQUEUE_SLEEP);
 		p->se.sched_delayed = 0;
+		p->se.rel_deadline = 0;
 		if (sched_feat(DELAY_ZERO) && p->se.vlag > 0)
 			p->se.vlag = 0;
 	}
kernel/sched/features.h +4 -0
index 7fdeb55..caa4d72 100644
@@ -10,6 +10,10 @@ SCHED_FEAT(PLACE_LAG, true)
  */
 SCHED_FEAT(PLACE_DEADLINE_INITIAL, true)
 /*
+ * Preserve relative virtual deadline on 'migration'.
+ */
+SCHED_FEAT(PLACE_REL_DEADLINE, true)
+/*
  * Inhibit (wakeup) preemption until the current task has either matched the
  * 0-lag point or until is has exhausted it's slice.
  */
author maintainer
#200 lore raw

PATCH 21/24 sched/eevdf: Allow shorter slices to wakeup-preempt

In reply to Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

---
 kernel/sched/fair.c     |   64 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/features.h |    5 +++
 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq
  * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
  * this is probably good enough.
  */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if ((s64)(se->vruntime - se->deadline) < 0)
-		return;
+		return false;
 
 	/*
 	 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r
 	/*
 	 * The task has consumed its request, reschedule.
 	 */
-	if (cfs_rq->nr_running > 1) {
-		resched_curr(rq_of(cfs_rq));
-		clear_buddies(cfs_rq, se);
-	}
+	return true;
 }
 
 #include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(stru
 		dl_server_update(p->dl_server, delta_exec);
 }
 
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (curr->vlag == curr->deadline)
+		return false;
+
+	return !entity_eligible(cfs_rq, curr);
+}
+
+static inline bool do_preempt_short(struct cfs_rq *cfs_rq,
+				    struct sched_entity *pse, struct sched_entity *se)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (pse->slice >= se->slice)
+		return false;
+
+	if (!entity_eligible(cfs_rq, pse))
+		return false;
+
+	if (entity_before(pse, se))
+		return true;
+
+	if (!entity_eligible(cfs_rq, se))
+		return true;
+
+	return false;
+}
+
 /*
  * Used by other classes to account runtime.
  */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *c
 	struct sched_entity *curr = cfs_rq->curr;
 	struct rq *rq = rq_of(cfs_rq);
 	s64 delta_exec;
+	bool resched;
 
 	if (unlikely(!curr))
 		return;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *c
 		return;
 
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
-	update_deadline(cfs_rq, curr);
+	resched = update_deadline(cfs_rq, curr);
 	update_min_vruntime(cfs_rq);
 
 	if (entity_is_task(curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c
 	}
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	if (rq->nr_running == 1)
+		return;
+
+	if (resched || did_preempt_short(cfs_rq, curr)) {
+		resched_curr(rq);
+		clear_buddies(cfs_rq, curr);
+	}
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st
 	cfs_rq = cfs_rq_of(se);
 	update_curr(cfs_rq);
 	/*
-	 * XXX pick_eevdf(cfs_rq) != se ?
+	 * If @p has a shorter slice than current and @p is eligible, override
+	 * current's slice protection in order to allow preemption.
+	 *
+	 * Note that even if @p does not turn out to be the most eligible
+	 * task at this moment, current's slice protection will be lost.
+	 */
+	if (do_preempt_short(cfs_rq, pse, se) && se->vlag == se->deadline)
+		se->vlag = se->deadline + 1;
+
+	/*
+	 * If @p has become the most eligible task, force preemption.
 	 */
 	if (pick_eevdf(cfs_rq) == pse)
 		goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)
  * 0-lag point or until is has exhausted it's slice.
  */
 SCHED_FEAT(RUN_TO_PARITY, true)
+/*
+ * Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for
+ * current.
+ */
+SCHED_FEAT(PREEMPT_SHORT, true)
 
 /*
  * Prefer to schedule the task we woke last (assuming it failed
#201 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 78 lines · Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

---
kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++++------
kernel/sched/features.h | 5 +++

2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +3 -6
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq
 * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
 * this is probably good enough.
 */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 if ((s64)(se->vruntime - se->deadline) < 0)
- return;
+ return false;
 
 /*
 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r
 /*
 * The task has consumed its request, reschedule.
 */
- if (cfs_rq->nr_running > 1) {
- resched_curr(rq_of(cfs_rq));
- clear_buddies(cfs_rq, se);
- }
+ return true;
 }
 
 #include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(stru
 dl_server_update(p->dl_server, delta_exec);
 }
 

kernel/sched/fair.c static inline void update_curr_task(stru · line 1143

Quoted 10 lines
kernel/sched/fair.c +10 -0
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

Hi perter

Can this be made more aggressive here? Something like , in the PREEMPT_SHORT
+ NO_RUN_TO_PARITY combination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.

+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr) +{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (sched_feat(RUN_TO_PARITY) && curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.
https://lore.kernel.org/lkml/[email protected]/

thanks
Chunxin

Quoted 93 lines
kernel/sched/fair.c +43 -2
+
+static inline bool do_preempt_short(struct cfs_rq *cfs_rq,
+    struct sched_entity *pse, struct sched_entity *se)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (pse->slice >= se->slice)
+ return false;
+
+ if (!entity_eligible(cfs_rq, pse))
+ return false;
+
+ if (entity_before(pse, se))
+ return true;
+
+ if (!entity_eligible(cfs_rq, se))
+ return true;
+
+ return false;
+}
+
 /*
 * Used by other classes to account runtime.
 */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *c
 struct sched_entity *curr = cfs_rq->curr;
 struct rq *rq = rq_of(cfs_rq);
 s64 delta_exec;
+ bool resched;
 
 if (unlikely(!curr))
 return;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *c
 return;
 
 curr->vruntime += calc_delta_fair(delta_exec, curr);
- update_deadline(cfs_rq, curr);
+ resched = update_deadline(cfs_rq, curr);
 update_min_vruntime(cfs_rq);
 
 if (entity_is_task(curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c
 }
 
 account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+ if (rq->nr_running == 1)
+ return;
+
+ if (resched || did_preempt_short(cfs_rq, curr)) {
+ resched_curr(rq);
+ clear_buddies(cfs_rq, curr);
+ }
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st
 cfs_rq = cfs_rq_of(se);
 update_curr(cfs_rq);
 /*
- * XXX pick_eevdf(cfs_rq) != se ?
+ * If @p has a shorter slice than current and @p is eligible, override
+ * current's slice protection in order to allow preemption.
+ *
+ * Note that even if @p does not turn out to be the most eligible
+ * task at this moment, current's slice protection will be lost.
+ */
+ if (do_preempt_short(cfs_rq, pse, se) && se->vlag == se->deadline)
+ se->vlag = se->deadline + 1;
+
+ /*
+ * If @p has become the most eligible task, force preemption.
 */
 if (pick_eevdf(cfs_rq) == pse)
 goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)
 * 0-lag point or until is has exhausted it's slice.
 */
 SCHED_FEAT(RUN_TO_PARITY, true)
+/*
+ * Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for
+ * current.
+ */
+SCHED_FEAT(PREEMPT_SHORT, true)
 
 /*
 * Prefer to schedule the task we woke last (assuming it failed
author maintainer Tested-by Mike Galbraith
#202 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 17 lines · Chunxin Zang

Review

Quoted 12 lines · Peter Zijlstra
(quoted diff) +10 -0
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

Can this be made more aggressive here? Something like , in the PREEMPT_SHORT
+ NO_RUN_TO_PARITY combination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.

You are aware we have SCHED_DEADLINE for those, right?

Why can't you use that? and what can we do to fix that.

Review

Quoted 10 lines
(quoted diff) +10 -0
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (sched_feat(RUN_TO_PARITY) && curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

No, this will destroy the steady state schedule (where no tasks join/leave) and make it so that all tasks hug the lag=0 state arbitrarily close -- as allowed by the scheduling quanta.

Yes, it will get you better latency, because nobody gets to actually run it's requested slice.

The goal really is for tasks to get their request -- and yes that means you get to wait. PREEMPT_SHORT is already an exception to this rule, and it is very specifically limited to wake-ups so as to retain as much of the intended behaviour as possible.

Quoted 4 lines

Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.

I've made it the below. Does that work for you?

---
Subject: sched/eevdf: Allow shorter slices to wakeup-preempt
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 26 14:32:32 CEST 2023

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Chunxin contributed did_preempt_short() where a task that lost slice protection from PREEMPT_SHORT gets rescheduled once it becomes in-eligible.

---
 kernel/sched/fair.c     |   64 ++++++++++++++++++++++++++++++++++++++++++------
 kernel/sched/features.h |    5 +++
 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq
  * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
  * this is probably good enough.
  */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if ((s64)(se->vruntime - se->deadline) < 0)
-		return;
+		return false;
 
 	/*
 	 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r
 	/*
 	 * The task has consumed its request, reschedule.
 	 */
-	if (cfs_rq->nr_running > 1) {
-		resched_curr(rq_of(cfs_rq));
-		clear_buddies(cfs_rq, se);
-	}
+	return true;
 }
 
 #include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(stru
 		dl_server_update(p->dl_server, delta_exec);
 }
 
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (curr->vlag == curr->deadline)
+		return false;
+
+	return !entity_eligible(cfs_rq, curr);
+}
+
+static inline bool do_preempt_short(struct cfs_rq *cfs_rq,
+				    struct sched_entity *pse, struct sched_entity *se)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (pse->slice >= se->slice)
+		return false;
+
+	if (!entity_eligible(cfs_rq, pse))
+		return false;
+
+	if (entity_before(pse, se))
+		return true;
+
+	if (!entity_eligible(cfs_rq, se))
+		return true;
+
+	return false;
+}
+
 /*
  * Used by other classes to account runtime.
  */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *c
 	struct sched_entity *curr = cfs_rq->curr;
 	struct rq *rq = rq_of(cfs_rq);
 	s64 delta_exec;
+	bool resched;
 
 	if (unlikely(!curr))
 		return;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *c
 		return;
 
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
-	update_deadline(cfs_rq, curr);
+	resched = update_deadline(cfs_rq, curr);
 	update_min_vruntime(cfs_rq);
 
 	if (entity_is_task(curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c
 	}
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	if (rq->nr_running == 1)
+		return;
+
+	if (resched || did_preempt_short(cfs_rq, curr)) {
+		resched_curr(rq);
+		clear_buddies(cfs_rq, curr);
+	}
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st
 	cfs_rq = cfs_rq_of(se);
 	update_curr(cfs_rq);
 	/*
-	 * XXX pick_eevdf(cfs_rq) != se ?
+	 * If @p has a shorter slice than current and @p is eligible, override
+	 * current's slice protection in order to allow preemption.
+	 *
+	 * Note that even if @p does not turn out to be the most eligible
+	 * task at this moment, current's slice protection will be lost.
+	 */
+	if (do_preempt_short(cfs_rq, pse, se) && se->vlag == se->deadline)
+		se->vlag = se->deadline + 1;
+
+	/*
+	 * If @p has become the most eligible task, force preemption.
 	 */
 	if (pick_eevdf(cfs_rq) == pse)
 		goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)
  * 0-lag point or until is has exhausted it's slice.
  */
 SCHED_FEAT(RUN_TO_PARITY, true)
+/*
+ * Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for
+ * current.
+ */
+SCHED_FEAT(PREEMPT_SHORT, true)
 
 /*
  * Prefer to schedule the task we woke last (assuming it failed
#203 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 42 lines · Peter Zijlstra
Quoted 17 lines · Chunxin Zang

kernel/sched/fair.c static inline void update_curr_task(stru · line 1143

Quoted 12 lines · Peter Zijlstra
kernel/sched/fair.c +10 -0
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

Can this be made more aggressive here? Something like , in the PREEMPT_SHORT
+ NO_RUN_TO_PARITY combination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.

You are aware we have SCHED_DEADLINE for those, right?

Why can't you use that? and what can we do to fix that.

kernel/sched/fair.c static inline void update_curr_task(stru · line 1143

Quoted 10 lines
kernel/sched/fair.c +10 -0
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (sched_feat(RUN_TO_PARITY) && curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}

No, this will destroy the steady state schedule (where no tasks join/leave) and make it so that all tasks hug the lag=0 state arbitrarily close -- as allowed by the scheduling quanta.

Quoted 7 lines

Yes, it will get you better latency, because nobody gets to actually run it's requested slice.

The goal really is for tasks to get their request -- and yes that means you get to wait. PREEMPT_SHORT is already an exception to this rule, and it is very specifically limited to wake-ups so as to retain as much of the intended behaviour as possible.

I think I understand your point now. That approach does indeed seem more reasonable. Next, I will try using 'request slice' to conduct more tests in my scenario.

Quoted 7 lines
Quoted 5 lines

Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.

I've made it the below. Does that work for you?

Heartfelt thanks :)

Chunxin

Quoted 187 lines

---
Subject: sched/eevdf: Allow shorter slices to wakeup-preempt
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 26 14:32:32 CEST 2023

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

 1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
 2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
 3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
 1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
 2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
 3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
 1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
 2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
 3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

 1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
 2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
 3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
 1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
 2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
 3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
 1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
 2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
 3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Chunxin contributed did_preempt_short() where a task that lost slice protection from PREEMPT_SHORT gets rescheduled once it becomes in-eligible.

---
kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++++------
kernel/sched/features.h | 5 +++

2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq
 * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
 * this is probably good enough.
 */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 if ((s64)(se->vruntime - se->deadline) < 0)
- return;
+ return false;
 
 /*
 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r
 /*
 * The task has consumed its request, reschedule.
 */
- if (cfs_rq->nr_running > 1) {
- resched_curr(rq_of(cfs_rq));
- clear_buddies(cfs_rq, se);
- }
+ return true;
 }
 
 #include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(stru
 dl_server_update(p->dl_server, delta_exec);
 }
 
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}
+
+static inline bool do_preempt_short(struct cfs_rq *cfs_rq,
+     struct sched_entity *pse, struct sched_entity *se)
+{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (pse->slice >= se->slice)
+ return false;
+
+ if (!entity_eligible(cfs_rq, pse))
+ return false;
+
+ if (entity_before(pse, se))
+ return true;
+
+ if (!entity_eligible(cfs_rq, se))
+ return true;
+
+ return false;
+}
+
 /*
 * Used by other classes to account runtime.
 */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *c
 struct sched_entity *curr = cfs_rq->curr;
 struct rq *rq = rq_of(cfs_rq);
 s64 delta_exec;
+ bool resched;
 
 if (unlikely(!curr))
 return;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *c
 return;
 
 curr->vruntime += calc_delta_fair(delta_exec, curr);
- update_deadline(cfs_rq, curr);
+ resched = update_deadline(cfs_rq, curr);
 update_min_vruntime(cfs_rq);
 
 if (entity_is_task(curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c
 }
 
 account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+ if (rq->nr_running == 1)
+ return;
+
+ if (resched || did_preempt_short(cfs_rq, curr)) {
+ resched_curr(rq);
+ clear_buddies(cfs_rq, curr);
+ }
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st
 cfs_rq = cfs_rq_of(se);
 update_curr(cfs_rq);
 /*
-  * XXX pick_eevdf(cfs_rq) != se ?
+  * If @p has a shorter slice than current and @p is eligible, override
+  * current's slice protection in order to allow preemption.
+  *
+  * Note that even if @p does not turn out to be the most eligible
+  * task at this moment, current's slice protection will be lost.
+  */
+ if (do_preempt_short(cfs_rq, pse, se) && se->vlag == se->deadline)
+ se->vlag = se->deadline + 1;
+
+ /*
+  * If @p has become the most eligible task, force preemption.
 */
 if (pick_eevdf(cfs_rq) == pse)
 goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)
 * 0-lag point or until is has exhausted it's slice.
 */
 SCHED_FEAT(RUN_TO_PARITY, true)
+/*
+ * Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for
+ * current.
+ */
+SCHED_FEAT(PREEMPT_SHORT, true)
 
 /*
 * Prefer to schedule the task we woke last (assuming it failed
#204 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

In reply to Peter Zijlstra

Hi Peter,

Quoted 30 lines · Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |
Quoted 8 lines

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/

The RESPECT_SLICE is introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.

echo RESPECT_SLICE > /sys/kernel/debug/sched/features

echo RUN_TO_PARITY > /sys/kernel/debug/sched/features @task_duration_usecs_before_preempted:

[2, 4)              8732 |@@@                                                 |
[4, 8)            109400 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          |
[8, 16)            95815 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@               |
[16, 32)          110647 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |
[32, 64)          131298 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[64, 128)         132566 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[128, 256)         82095 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                    |
[256, 512)         33771 |@@@@@@@@@@@@@                                       |
[512, 1K)          24180 |@@@@@@@@@                                           |
[1K, 2K)           31056 |@@@@@@@@@@@@                                        |
[2K, 4K)          117533 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
[4K, 8K)            4472 |@                                                   |
[8K, 16K)           1149 |                                                    |
[16K, 32K)           289 |                                                    |
[32K, 64K)           110 |                                                    |
[64K, 128K)           20 |                                                    |
[128K, 256K)           4 |                                                    |

echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features @task_duration_usecs_before_preempted:

[4, 8)                 1 |                                                    |
[8, 16)               12 |                                                    |
[16, 32)              20 |                                                    |
[32, 64)              38 |                                                    |
[64, 128)             64 |                                                    |
[128, 256)            98 |                                                    |
[256, 512)           248 |                                                    |
[512, 1K)           1196 |                                                    |
[1K, 2K)            3456 |                                                    |
[2K, 4K)          417269 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4K, 8K)           22893 |@@                                                  |
[8K, 16K)           7818 |                                                    |
[16K, 32K)          1471 |                                                    |
[32K, 64K)           373 |                                                    |
[64K, 128K)           96 |                                                    |
[128K, 256K)           3 |                                                    |

We can see that without the fix, the task will be preempted and can not reach its time slice budget(we enlarge its slice).

May I know if we can put that patch into this series please?

thanks,
Chenyu

author maintainer
#205 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 42 lines · Chen Yu

Hi Peter,

Quoted 30 lines · Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |
Quoted 8 lines

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Quoted 8 lines

Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/

The RESPECT_SLICE is introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.

So I *think* that running as SCHED_BATCH gets you exactly that behaviour, no?

#206 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Chen Yu

Hi Peter,

Quoted 30 lines · Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |
Quoted 8 lines

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines
Quoted 3 lines

The RESPECT_SLICE is introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.

So I *think* that running as SCHED_BATCH gets you exactly that behaviour, no?

SCHED_BATCH should work as it avoids the wakeup preemption as much as possible. Except that RESPECT_SLICE considers the cgroup hierarchical to check if the current sched_entity has used up its slice, which seems to be less aggressive.

thanks,
Chenyu

author maintainer
#207 lore raw

Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt

Quoted 53 lines · Chen Yu
Quoted 52 lines · Peter Zijlstra
Quoted 50 lines · Chen Yu

Hi Peter,

Quoted 30 lines · Peter Zijlstra

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |
Quoted 8 lines

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/

The RESPECT_SLICE is introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.

Quoted 6 lines
Quoted 2 lines

So I *think* that running as SCHED_BATCH gets you exactly that behaviour, no?

SCHED_BATCH should work as it avoids the wakeup preemption as much as possible. Except that RESPECT_SLICE considers the cgroup hierarchical to check if the current sched_entity has used up its slice, which seems to be less aggressive.

Note that update_deadline() will trigger a resched at the end up a slice regardless -- this is driven from update_curr() and also invoked from any preemption.

merge bot Applied
#208 lore raw

[tip: sched/core] sched/eevdf: Allow shorter slices to wakeup-preempt

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     85e511df3cec46021024176672a748008ed135bf
Gitweb:        https://git.kernel.org/tip/85e511df3cec46021024176672a748008ed135bf
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 Sep 2023 14:32:32 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00

sched/eevdf: Allow shorter slices to wakeup-preempt

Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.

Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |

100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT

  1 massive_intr:(5)      | 846018.956 ms |   779188 | avg:   0.273 ms | max:  58.337 ms | sum:212545.245 ms |
  2 massive_intr:(5)      | 853450.693 ms |   792269 | avg:   0.275 ms | max:  71.193 ms | sum:218263.588 ms |
  3 massive_intr:(5)      | 843888.920 ms |   771456 | avg:   0.277 ms | max:  92.405 ms | sum:213353.221 ms |
  1 chromium-browse:(8)   |  53015.889 ms |   131766 | avg:   0.463 ms | max:  36.341 ms | sum:60959.230  ms |
  2 chromium-browse:(8)   |  53864.088 ms |   136962 | avg:   0.480 ms | max:  27.091 ms | sum:65687.681  ms |
  3 chromium-browse:(9)   |  53637.904 ms |   132637 | avg:   0.481 ms | max:  24.756 ms | sum:63781.673  ms |
  1 cyclictest:(5)        |  12615.604 ms |   639689 | avg:   0.471 ms | max:  32.272 ms | sum:301351.094 ms |
  2 cyclictest:(5)        |  12511.583 ms |   642578 | avg:   0.448 ms | max:  44.243 ms | sum:287632.830 ms |
  3 cyclictest:(5)        |  12545.867 ms |   635953 | avg:   0.475 ms | max:  25.530 ms | sum:302374.658 ms |

100ms massive_intr 500us cyclictest PREEMPT_SHORT

  1 massive_intr:(5)      | 839843.919 ms |   837384 | avg:   0.264 ms | max:  74.366 ms | sum:221476.885 ms |
  2 massive_intr:(5)      | 852449.913 ms |   845086 | avg:   0.252 ms | max:  68.162 ms | sum:212595.968 ms |
  3 massive_intr:(5)      | 839180.725 ms |   836883 | avg:   0.266 ms | max:  69.742 ms | sum:222812.038 ms |
  1 chromium-browse:(11)  |  54591.481 ms |   138388 | avg:   0.458 ms | max:  35.427 ms | sum:63401.508  ms |
  2 chromium-browse:(8)   |  52034.541 ms |   132276 | avg:   0.436 ms | max:  31.826 ms | sum:57732.958  ms |
  3 chromium-browse:(8)   |  55231.771 ms |   141892 | avg:   0.469 ms | max:  27.607 ms | sum:66538.697  ms |
  1 cyclictest:(5)        |  13156.391 ms |   667412 | avg:   0.373 ms | max:  38.247 ms | sum:249174.502 ms |
  2 cyclictest:(5)        |  12688.939 ms |   665144 | avg:   0.374 ms | max:  33.548 ms | sum:248509.392 ms |
  3 cyclictest:(5)        |  13475.623 ms |   669110 | avg:   0.370 ms | max:  37.819 ms | sum:247673.390 ms |

As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.

Chunxin contributed did_preempt_short() where a task that lost slice protection from PREEMPT_SHORT gets rescheduled once it becomes in-eligible.

---
 kernel/sched/fair.c     | 64 +++++++++++++++++++++++++++++++++++-----
 kernel/sched/features.h |  5 +++-
 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
index fef0e1f..cc30ea3 100644
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se);
  * XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i
  * this is probably good enough.
  */
-static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
+static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if ((s64)(se->vruntime - se->deadline) < 0)
-		return;
+		return false;
 
 	/*
 	 * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	/*
 	 * The task has consumed its request, reschedule.
 	 */
-	if (cfs_rq->nr_running > 1) {
-		resched_curr(rq_of(cfs_rq));
-		clear_buddies(cfs_rq, se);
-	}
+	return true;
 }
 
 #include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(struct task_struct *p, s64 delta_exec)
 		dl_server_update(p->dl_server, delta_exec);
 }
 
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (curr->vlag == curr->deadline)
+		return false;
+
+	return !entity_eligible(cfs_rq, curr);
+}
+
+static inline bool do_preempt_short(struct cfs_rq *cfs_rq,
+				    struct sched_entity *pse, struct sched_entity *se)
+{
+	if (!sched_feat(PREEMPT_SHORT))
+		return false;
+
+	if (pse->slice >= se->slice)
+		return false;
+
+	if (!entity_eligible(cfs_rq, pse))
+		return false;
+
+	if (entity_before(pse, se))
+		return true;
+
+	if (!entity_eligible(cfs_rq, se))
+		return true;
+
+	return false;
+}
+
 /*
  * Used by other classes to account runtime.
  */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	struct sched_entity *curr = cfs_rq->curr;
 	struct rq *rq = rq_of(cfs_rq);
 	s64 delta_exec;
+	bool resched;
 
 	if (unlikely(!curr))
 		return;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *cfs_rq)
 		return;
 
 	curr->vruntime += calc_delta_fair(delta_exec, curr);
-	update_deadline(cfs_rq, curr);
+	resched = update_deadline(cfs_rq, curr);
 	update_min_vruntime(cfs_rq);
 
 	if (entity_is_task(curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *cfs_rq)
 	}
 
 	account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+	if (rq->nr_running == 1)
+		return;
+
+	if (resched || did_preempt_short(cfs_rq, curr)) {
+		resched_curr(rq);
+		clear_buddies(cfs_rq, curr);
+	}
 }
 
 static void update_curr_fair(struct rq *rq)
@@ -8605,7 +8643,17 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int 
 	cfs_rq = cfs_rq_of(se);
 	update_curr(cfs_rq);
 	/*
-	 * XXX pick_eevdf(cfs_rq) != se ?
+	 * If @p has a shorter slice than current and @p is eligible, override
+	 * current's slice protection in order to allow preemption.
+	 *
+	 * Note that even if @p does not turn out to be the most eligible
+	 * task at this moment, current's slice protection will be lost.
+	 */
+	if (do_preempt_short(cfs_rq, pse, se) && se->vlag == se->deadline)
+		se->vlag = se->deadline + 1;
+
+	/*
+	 * If @p has become the most eligible task, force preemption.
 	 */
 	if (pick_eevdf(cfs_rq) == pse)
 		goto preempt;
kernel/sched/features.h +5 -0
index caa4d72..2908740 100644
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)
  * 0-lag point or until is has exhausted it's slice.
  */
 SCHED_FEAT(RUN_TO_PARITY, true)
+/*
+ * Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for
+ * current.
+ */
+SCHED_FEAT(PREEMPT_SHORT, true)
 
 /*
  * Prefer to schedule the task we woke last (assuming it failed
author maintainer
#209 lore raw

PATCH 22/24 sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion

In reply to Peter Zijlstra

Allow applications to directly set a suggested request/slice length using sched_attr::sched_runtime.

The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.

Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.

For all the following examples assume a scheduling quantum of 8, and for consistency all examples have W=4:

{A,B,C,D}(w=1,r=8):

ABCD...
+---+---+---+---

t=0, V=1.5 t=1, V=3.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+*------+-------+--- ---+--*----+-------+---

t=2, V=5.5 t=3, V=7.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+----*--+-------+--- ---+------*+-------+---

Note: 4 identical tasks in FIFO order

~~~

{A,B}(w=1,r=16) C(w=2,r=16)

AACCBBCC...
+---+---+---+---

t=0, V=1.25 t=2, V=5.25

  A  |--------------<                   A                  |--------------<
  B   |--------------<                  B   |--------------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+----*--+-------+---

t=4, V=8.25 t=6, V=12.25

  A                  |--------------<   A                  |--------------<
  B   |--------------<                  B                   |--------------<
  C            |------<                 C            |------<
  ---+-------*-------+---               ---+-------+---*---+---

Note: 1 heavy task -- because q=8, double r such that the deadline of the w=2
task doesn't go below q.

Note: observe the full schedule becomes: W*max(r_i/w_i) = 4*2q = 8q in length.

Note: the period of the heavy task is half the full period at:

      W*(r_i/w_i) = 4*(2q/2) = 4q

~~~

{A,C,D}(w=1,r=16) B(w=1,r=8):

BAACCBDD...
+---+---+---+---

t=0, V=1.5 t=1, V=3.5
A |--------------< A |---------------<
B |------< B |------<
C |--------------< C |--------------<
D |--------------< D |--------------<
---+*------+-------+--- ---+--*----+-------+---

t=3, V=7.5 t=5, V=11.5

  A                  |---------------<  A                  |---------------<
  B           |------<                  B           |------<
  C    |--------------<                 C                    |--------------<
  D     |--------------<                D     |--------------<
  ---+------*+-------+---               ---+-------+--*----+---

t=6, V=13.5
A |---------------<
B |------<
C |--------------<
D |--------------<
---+-------+----*--+---

Note: 1 short task -- again double r so that the deadline of the short task
won't be below q. Made B short because its not the leftmost task, but is eligible with the 0,1,2,3 spread.

Note: like with the heavy task, the period of the short task observes:

      W*(r_i/w_i) = 4*(1q/1) = 4q

~~~

A(w=1,r=16) B(w=1,r=8) C(w=2,r=16)

BCCAABCC...
+---+---+---+---

t=0, V=1.25 t=1, V=3.25

  A  |--------------<                   A  |--------------<
  B   |------<                          B           |------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+--*----+-------+---

t=3, V=7.25 t=5, V=11.25

  A  |--------------<                   A                  |--------------<
  B           |------<                  B           |------<
  C            |------<                 C            |------<
  ---+------*+-------+---               ---+-------+--*----+---

t=6, V=13.25
A |--------------<
B |------<
C |------<
---+-------+----*--+---

Note: 1 heavy and 1 short task -- combine them all.

Note: both the short and heavy task end up with a period of 4q

~~~

A(w=1,r=16) B(w=2,r=16) C(w=1,r=8)

BBCAABBC...
+---+---+---+---

t=0, V=1 t=2, V=5

  A  |--------------<                   A  |--------------<
  B   |------<                          B           |------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+----*--+-------+---

t=3, V=7 t=5, V=11

  A  |--------------<                   A                  |--------------<
  B           |------<                  B           |------<
  C            |------<                 C            |------<
  ---+------*+-------+---               ---+-------+--*----+---

t=7, V=15
A |--------------<
B |------<
C |------<
---+-------+------*+---

Note: as before but permuted

~~~

Quoted 1 line

From all this it can be deduced that, for the steady state:

- the total period (P) of a schedule is: W*max(r_i/w_i)
- the average period of a task is: W*(r_i/w_i)
- each task obtains the fair share: w_i/W of each full period P

---
 include/linux/sched.h   |    1 +
 kernel/sched/core.c     |    4 +++-
 kernel/sched/debug.c    |    3 ++-
 kernel/sched/fair.c     |    6 ++++--
 kernel/sched/syscalls.c |   29 +++++++++++++++++++++++------
 5 files changed, 33 insertions(+), 10 deletions(-)
include/linux/sched.h +1 -0
@@ -544,6 +544,7 @@ struct sched_entity {
 	unsigned char			on_rq;
 	unsigned char			sched_delayed;
 	unsigned char			rel_deadline;
+	unsigned char			custom_slice;
 					/* hole */
 
 	u64				exec_start;
kernel/sched/core.c +3 -1
@@ -4347,7 +4347,6 @@ static void __sched_fork(unsigned long c
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	p->se.vlag			= 0;
-	p->se.slice			= sysctl_sched_base_slice;
 	INIT_LIST_HEAD(&p->se.group_node);
 
 	/* A delayed task cannot be in clone(). */
@@ -4600,6 +4599,8 @@ int sched_fork(unsigned long clone_flags
 
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
+		p->se.custom_slice = 0;
+		p->se.slice = sysctl_sched_base_slice;
 
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
@@ -8328,6 +8329,7 @@ void __init sched_init(void)
 	}
 
 	set_load_weight(&init_task, false);
+	init_task.se.slice = sysctl_sched_base_slice,
 
 	/*
 	 * The boot idle thread does lazy MMU switching as well:
kernel/sched/debug.c +2 -1
@@ -580,11 +580,12 @@ print_task(struct seq_file *m, struct rq
 	else
 		SEQ_printf(m, " %c", task_state_to_char(p));
 
-	SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
+	SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
 		p->comm, task_pid_nr(p),
 		SPLIT_NS(p->se.vruntime),
 		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
 		SPLIT_NS(p->se.deadline),
+		p->se.custom_slice ? 'S' : ' ',
 		SPLIT_NS(p->se.slice),
 		SPLIT_NS(p->se.sum_exec_runtime),
 		(long long)(p->nvcsw + p->nivcsw),
kernel/sched/fair.c +4 -2
@@ -995,7 +995,8 @@ static void update_deadline(struct cfs_r
 	 * nice) while the request time r_i is determined by
 	 * sysctl_sched_base_slice.
 	 */
-	se->slice = sysctl_sched_base_slice;
+	if (!se->custom_slice)
+		se->slice = sysctl_sched_base_slice;
 
 	/*
 	 * EEVDF: vd_i = ve_i + r_i / w_i
@@ -5190,7 +5191,8 @@ place_entity(struct cfs_rq *cfs_rq, stru
 	u64 vslice, vruntime = avg_vruntime(cfs_rq);
 	s64 lag = 0;
 
-	se->slice = sysctl_sched_base_slice;
+	if (!se->custom_slice)
+		se->slice = sysctl_sched_base_slice;
 	vslice = calc_delta_fair(se->slice, se);
 
 	/*
kernel/sched/syscalls.c +23 -6
@@ -401,10 +401,20 @@ static void __setscheduler_params(struct
 
 	p->policy = policy;
 
-	if (dl_policy(policy))
+	if (dl_policy(policy)) {
 		__setparam_dl(p, attr);
-	else if (fair_policy(policy))
+	} else if (fair_policy(policy)) {
 		p->static_prio = NICE_TO_PRIO(attr->sched_nice);
+		if (attr->sched_runtime) {
+			p->se.custom_slice = 1;
+			p->se.slice = clamp_t(u64, attr->sched_runtime,
+					      NSEC_PER_MSEC/10,   /* HZ=1000 * 10 */
+					      NSEC_PER_MSEC*100); /* HZ=100  / 10 */
+		} else {
+			p->se.custom_slice = 0;
+			p->se.slice = sysctl_sched_base_slice;
+		}
+	}
 
 	/*
 	 * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -700,7 +710,9 @@ int __sched_setscheduler(struct task_str
 	 * but store a possible modification of reset_on_fork.
 	 */
 	if (unlikely(policy == p->policy)) {
-		if (fair_policy(policy) && attr->sched_nice != task_nice(p))
+		if (fair_policy(policy) &&
+		    (attr->sched_nice != task_nice(p) ||
+		     (attr->sched_runtime != p->se.slice)))
 			goto change;
 		if (rt_policy(policy) && attr->sched_priority != p->rt_priority)
 			goto change;
@@ -846,6 +858,9 @@ static int _sched_setscheduler(struct ta
 		.sched_nice	= PRIO_TO_NICE(p->static_prio),
 	};
 
+	if (p->se.custom_slice)
+		attr.sched_runtime = p->se.slice;
+
 	/* Fixup the legacy SCHED_RESET_ON_FORK hack. */
 	if ((policy != SETPARAM_POLICY) && (policy & SCHED_RESET_ON_FORK)) {
 		attr.sched_flags |= SCHED_FLAG_RESET_ON_FORK;
@@ -1012,12 +1027,14 @@ static int sched_copy_attr(struct sched_
 
 static void get_params(struct task_struct *p, struct sched_attr *attr)
 {
-	if (task_has_dl_policy(p))
+	if (task_has_dl_policy(p)) {
 		__getparam_dl(p, attr);
-	else if (task_has_rt_policy(p))
+	} else if (task_has_rt_policy(p)) {
 		attr->sched_priority = p->rt_priority;
-	else
+	} else {
 		attr->sched_nice = task_nice(p);
+		attr->sched_runtime = p->se.slice;
+	}
 }
 
 /**
merge bot Applied
#210 lore raw

[tip: sched/core] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     857b158dc5e81c6de795ef6be006eed146098fc6
Gitweb:        https://git.kernel.org/tip/857b158dc5e81c6de795ef6be006eed146098fc6
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 22 May 2023 13:46:30 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00

sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion

Allow applications to directly set a suggested request/slice length using sched_attr::sched_runtime.

The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.

Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.

For all the following examples assume a scheduling quantum of 8, and for consistency all examples have W=4:

{A,B,C,D}(w=1,r=8):

ABCD...
+---+---+---+---

t=0, V=1.5 t=1, V=3.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+*------+-------+--- ---+--*----+-------+---

t=2, V=5.5 t=3, V=7.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+----*--+-------+--- ---+------*+-------+---

Note: 4 identical tasks in FIFO order

~~~

{A,B}(w=1,r=16) C(w=2,r=16)

AACCBBCC...
+---+---+---+---

t=0, V=1.25 t=2, V=5.25

  A  |--------------<                   A                  |--------------<
  B   |--------------<                  B   |--------------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+----*--+-------+---

t=4, V=8.25 t=6, V=12.25

  A                  |--------------<   A                  |--------------<
  B   |--------------<                  B                   |--------------<
  C            |------<                 C            |------<
  ---+-------*-------+---               ---+-------+---*---+---

Note: 1 heavy task -- because q=8, double r such that the deadline of the w=2
task doesn't go below q.

Note: observe the full schedule becomes: W*max(r_i/w_i) = 4*2q = 8q in length.

Note: the period of the heavy task is half the full period at:

      W*(r_i/w_i) = 4*(2q/2) = 4q

~~~

{A,C,D}(w=1,r=16) B(w=1,r=8):

BAACCBDD...
+---+---+---+---

t=0, V=1.5 t=1, V=3.5
A |--------------< A |---------------<
B |------< B |------<
C |--------------< C |--------------<
D |--------------< D |--------------<
---+*------+-------+--- ---+--*----+-------+---

t=3, V=7.5 t=5, V=11.5

  A                  |---------------<  A                  |---------------<
  B           |------<                  B           |------<
  C    |--------------<                 C                    |--------------<
  D     |--------------<                D     |--------------<
  ---+------*+-------+---               ---+-------+--*----+---

t=6, V=13.5
A |---------------<
B |------<
C |--------------<
D |--------------<
---+-------+----*--+---

Note: 1 short task -- again double r so that the deadline of the short task
won't be below q. Made B short because its not the leftmost task, but is eligible with the 0,1,2,3 spread.

Note: like with the heavy task, the period of the short task observes:

      W*(r_i/w_i) = 4*(1q/1) = 4q

~~~

A(w=1,r=16) B(w=1,r=8) C(w=2,r=16)

BCCAABCC...
+---+---+---+---

t=0, V=1.25 t=1, V=3.25

  A  |--------------<                   A  |--------------<
  B   |------<                          B           |------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+--*----+-------+---

t=3, V=7.25 t=5, V=11.25

  A  |--------------<                   A                  |--------------<
  B           |------<                  B           |------<
  C            |------<                 C            |------<
  ---+------*+-------+---               ---+-------+--*----+---

t=6, V=13.25
A |--------------<
B |------<
C |------<
---+-------+----*--+---

Note: 1 heavy and 1 short task -- combine them all.

Note: both the short and heavy task end up with a period of 4q

~~~

A(w=1,r=16) B(w=2,r=16) C(w=1,r=8)

BBCAABBC...
+---+---+---+---

t=0, V=1 t=2, V=5

  A  |--------------<                   A  |--------------<
  B   |------<                          B           |------<
  C    |------<                         C    |------<
  ---+*------+-------+---               ---+----*--+-------+---

t=3, V=7 t=5, V=11

  A  |--------------<                   A                  |--------------<
  B           |------<                  B           |------<
  C            |------<                 C            |------<
  ---+------*+-------+---               ---+-------+--*----+---

t=7, V=15
A |--------------<
B |------<
C |------<
---+-------+------*+---

Note: as before but permuted

~~~

Quoted 1 line

From all this it can be deduced that, for the steady state:

- the total period (P) of a schedule is: W*max(r_i/w_i)
- the average period of a task is: W*(r_i/w_i)
- each task obtains the fair share: w_i/W of each full period P

---
 include/linux/sched.h   |  1 +
 kernel/sched/core.c     |  4 +++-
 kernel/sched/debug.c    |  3 ++-
 kernel/sched/fair.c     |  6 ++++--
 kernel/sched/syscalls.c | 29 +++++++++++++++++++++++------
 5 files changed, 33 insertions(+), 10 deletions(-)
include/linux/sched.h +1 -0
index d25e1cf..89a3d8d 100644
@@ -547,6 +547,7 @@ struct sched_entity {
 	unsigned char			on_rq;
 	unsigned char			sched_delayed;
 	unsigned char			rel_deadline;
+	unsigned char			custom_slice;
 					/* hole */
 
 	u64				exec_start;
kernel/sched/core.c +3 -1
index 868b71b..0165811 100644
@@ -4390,7 +4390,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	p->se.vlag			= 0;
-	p->se.slice			= sysctl_sched_base_slice;
 	INIT_LIST_HEAD(&p->se.group_node);
 
 	/* A delayed task cannot be in clone(). */
@@ -4643,6 +4642,8 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 
 		p->prio = p->normal_prio = p->static_prio;
 		set_load_weight(p, false);
+		p->se.custom_slice = 0;
+		p->se.slice = sysctl_sched_base_slice;
 
 		/*
 		 * We don't need the reset flag anymore after the fork. It has
@@ -8412,6 +8413,7 @@ void __init sched_init(void)
 	}
 
 	set_load_weight(&init_task, false);
+	init_task.se.slice = sysctl_sched_base_slice,
 
 	/*
 	 * The boot idle thread does lazy MMU switching as well:
kernel/sched/debug.c +2 -1
index 831a77a..01ce9a7 100644
@@ -739,11 +739,12 @@ print_task(struct seq_file *m, struct rq *rq, struct task_struct *p)
 	else
 		SEQ_printf(m, " %c", task_state_to_char(p));
 
-	SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
+	SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
 		p->comm, task_pid_nr(p),
 		SPLIT_NS(p->se.vruntime),
 		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
 		SPLIT_NS(p->se.deadline),
+		p->se.custom_slice ? 'S' : ' ',
 		SPLIT_NS(p->se.slice),
 		SPLIT_NS(p->se.sum_exec_runtime),
 		(long long)(p->nvcsw + p->nivcsw),
kernel/sched/fair.c +4 -2
index cc30ea3..3284d3c 100644
@@ -983,7 +983,8 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	 * nice) while the request time r_i is determined by
 	 * sysctl_sched_base_slice.
 	 */
-	se->slice = sysctl_sched_base_slice;
+	if (!se->custom_slice)
+		se->slice = sysctl_sched_base_slice;
 
 	/*
 	 * EEVDF: vd_i = ve_i + r_i / w_i
@@ -5227,7 +5228,8 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	u64 vslice, vruntime = avg_vruntime(cfs_rq);
 	s64 lag = 0;
 
-	se->slice = sysctl_sched_base_slice;
+	if (!se->custom_slice)
+		se->slice = sysctl_sched_base_slice;
 	vslice = calc_delta_fair(se->slice, se);
 
 	/*
kernel/sched/syscalls.c +23 -6
index 60e70c8..4fae3cf 100644
@@ -401,10 +401,20 @@ static void __setscheduler_params(struct task_struct *p,
 
 	p->policy = policy;
 
-	if (dl_policy(policy))
+	if (dl_policy(policy)) {
 		__setparam_dl(p, attr);
-	else if (fair_policy(policy))
+	} else if (fair_policy(policy)) {
 		p->static_prio = NICE_TO_PRIO(attr->sched_nice);
+		if (attr->sched_runtime) {
+			p->se.custom_slice = 1;
+			p->se.slice = clamp_t(u64, attr->sched_runtime,
+					      NSEC_PER_MSEC/10,   /* HZ=1000 * 10 */
+					      NSEC_PER_MSEC*100); /* HZ=100  / 10 */
+		} else {
+			p->se.custom_slice = 0;
+			p->se.slice = sysctl_sched_base_slice;
+		}
+	}
 
 	/*
 	 * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -700,7 +710,9 @@ recheck:
 	 * but store a possible modification of reset_on_fork.
 	 */
 	if (unlikely(policy == p->policy)) {
-		if (fair_policy(policy) && attr->sched_nice != task_nice(p))
+		if (fair_policy(policy) &&
+		    (attr->sched_nice != task_nice(p) ||
+		     (attr->sched_runtime != p->se.slice)))
 			goto change;
 		if (rt_policy(policy) && attr->sched_priority != p->rt_priority)
 			goto change;
@@ -846,6 +858,9 @@ static int _sched_setscheduler(struct task_struct *p, int policy,
 		.sched_nice	= PRIO_TO_NICE(p->static_prio),
 	};
 
+	if (p->se.custom_slice)
+		attr.sched_runtime = p->se.slice;
+
 	/* Fixup the legacy SCHED_RESET_ON_FORK hack. */
 	if ((policy != SETPARAM_POLICY) && (policy & SCHED_RESET_ON_FORK)) {
 		attr.sched_flags |= SCHED_FLAG_RESET_ON_FORK;
@@ -1012,12 +1027,14 @@ err_size:
 
 static void get_params(struct task_struct *p, struct sched_attr *attr)
 {
-	if (task_has_dl_policy(p))
+	if (task_has_dl_policy(p)) {
 		__getparam_dl(p, attr);
-	else if (task_has_rt_policy(p))
+	} else if (task_has_rt_policy(p)) {
 		attr->sched_priority = p->rt_priority;
-	else
+	} else {
 		attr->sched_nice = task_nice(p);
+		attr->sched_runtime = p->se.slice;
+	}
 }
 
 /**
author maintainer
#211 lore raw

PATCH 23/24 sched/eevdf: Propagate min_slice up the cgroup hierarchy

In reply to Peter Zijlstra

In the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the min_slice up the hierarchy.

This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.

---
 include/linux/sched.h |    1 
 kernel/sched/fair.c   |   57 +++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 57 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {
 	struct rb_node			run_node;
 	u64				deadline;
 	u64				min_vruntime;
+	u64				min_slice;
 
 	struct list_head		group_node;
 	unsigned char			on_rq;
kernel/sched/fair.c +56 -1

129 lines not on this page · raw message

merge bot Applied
#212 lore raw

[tip: sched/core] sched/eevdf: Propagate min_slice up the cgroup hierarchy

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     aef6987d89544d63a47753cf3741cabff0b5574c
Gitweb:        https://git.kernel.org/tip/aef6987d89544d63a47753cf3741cabff0b5574c
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Thu, 20 Jun 2024 13:16:49 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 17 Aug 2024 11:06:46 +02:00

sched/eevdf: Propagate min_slice up the cgroup hierarchy

In the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the min_slice up the hierarchy.

This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.

---
 include/linux/sched.h |  1 +-
 kernel/sched/fair.c   | 57 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 57 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
index 89a3d8d..3709ded 100644
@@ -542,6 +542,7 @@ struct sched_entity {
 	struct rb_node			run_node;
 	u64				deadline;
 	u64				min_vruntime;
+	u64				min_slice;
 
 	struct list_head		group_node;
 	unsigned char			on_rq;
kernel/sched/fair.c +56 -1

129 lines not on this page · raw message

#213 lore raw

Re: [PATCH 23/24] sched/eevdf: Propagate min_slice up the cgroup hierarchy

In reply to Peter Zijlstra

Hi Peter,

Quoted 3 lines · Peter Zijlstra

In the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the min_slice up the hierarchy.

Will it be acceptable to introduce a cgroup interface (e.g., sth. like cpu.fair_runtime or cpu.fair_slice) to overwrite the caculated min_slice? This could be useful in container scenarios.

Quoted 27 lines

This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.

---
  include/linux/sched.h |    1
  kernel/sched/fair.c   |   57 +++++++++++++++++++++++++++++++++++++++++++++++++-
  2 files changed, 57 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {
  	struct rb_node			run_node;
  	u64				deadline;
  	u64				min_vruntime;
+	u64				min_slice;
 
  	struct list_head		group_node;
  	unsigned char			on_rq;
kernel/sched/fair.c +1 -0
@@ -782,6 +782,21 @@ static void update_min_vruntime(struct c
  	cfs_rq->min_vruntime = __update_min_vruntime(cfs_rq, vruntime);
  }
 
+static inline u64 cfs_rq_min_slice(struct cfs_rq *cfs_rq)

kernel/sched/fair.c static void update_min_vruntime(struct c · line 795

Quoted 10 lines
kernel/sched/fair.c +10 -0
+{
+	struct sched_entity *root = __pick_root_entity(cfs_rq);
+	struct sched_entity *curr = cfs_rq->curr;
+	u64 min_slice = ~0ULL;
+
+	if (curr && curr->on_rq)
+		min_slice = curr->slice;
+
+	if (root)
+		min_slice = min(min_slice, root->min_slice);

If a sched_delayed se keeps the min_slice, then the parent se will receive a shorter slice (from the sched_delayed se) than it should be. Is it a problem?

Thanks.

author maintainer
#214 lore raw

RFC PATCH 24/24 sched/time: Introduce CLOCK_THREAD_DVFS_ID

In reply to Peter Zijlstra

In order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.

The clock does *NOT* support setting timers.

Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.

---
 include/linux/posix-timers_types.h |    5 ++--
 include/linux/sched.h              |    1 
 include/linux/sched/cputime.h      |    3 ++
 include/uapi/linux/time.h          |    1 
 kernel/sched/core.c                |   40 +++++++++++++++++++++++++++++++++++++
 kernel/sched/fair.c                |    8 +++++--
 kernel/time/posix-cpu-timers.c     |   16 +++++++++++++-
 kernel/time/posix-timers.c         |    1 
 kernel/time/posix-timers.h         |    1 
 9 files changed, 71 insertions(+), 5 deletions(-)
include/linux/posix-timers_types.h +3 -2
@@ -13,9 +13,9 @@
  *
  * Bit 2 indicates whether a cpu clock refers to a thread or a process.
  *
- * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3.
+ * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3
  *
- * A clockid is invalid if bits 2, 1, and 0 are all set.
+ * (DVFS is PERTHREAD only)
  */
 #define CPUCLOCK_PID(clock)		((pid_t) ~((clock) >> 3))
 #define CPUCLOCK_PERTHREAD(clock) \
@@ -27,6 +27,7 @@
 #define CPUCLOCK_PROF		0
 #define CPUCLOCK_VIRT		1
 #define CPUCLOCK_SCHED		2
+#define CPUCLOCK_DVFS		3
 #define CPUCLOCK_MAX		3
 #define CLOCKFD			CPUCLOCK_MAX
 #define CLOCKFD_MASK		(CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)
include/linux/sched.h +1 -0
@@ -550,6 +550,7 @@ struct sched_entity {
 	u64				exec_start;
 	u64				sum_exec_runtime;
 	u64				prev_sum_exec_runtime;
+	u64				sum_dvfs_runtime;
 	u64				vruntime;
 	s64				vlag;
 	u64				slice;
include/linux/sched/cputime.h +3 -0
@@ -180,4 +180,7 @@ static inline void prev_cputime_init(str
 extern unsigned long long
 task_sched_runtime(struct task_struct *task);
 
+extern unsigned long long
+task_sched_dvfs_runtime(struct task_struct *task);
+
 #endif /* _LINUX_SCHED_CPUTIME_H */
include/uapi/linux/time.h +1 -0
@@ -62,6 +62,7 @@ struct timezone {
  */
 #define CLOCK_SGI_CYCLE			10
 #define CLOCK_TAI			11
+#define CLOCK_THREAD_DVFS_ID		12
 
 #define MAX_CLOCKS			16
 #define CLOCKS_MASK			(CLOCK_REALTIME | CLOCK_MONOTONIC)
kernel/sched/core.c +40 -0
@@ -4551,6 +4551,7 @@ static void __sched_fork(unsigned long c
 	p->se.exec_start		= 0;
 	p->se.sum_exec_runtime		= 0;
 	p->se.prev_sum_exec_runtime	= 0;
+	p->se.sum_dvfs_runtime		= 0;
 	p->se.nr_migrations		= 0;
 	p->se.vruntime			= 0;
 	p->se.vlag			= 0;
@@ -5632,6 +5633,45 @@ unsigned long long task_sched_runtime(st
 	task_rq_unlock(rq, p, &rf);
 
 	return ns;
+}
+
+unsigned long long task_sched_dvfs_runtime(struct task_struct *p)
+{
+	struct rq_flags rf;
+	struct rq *rq;
+	u64 ns;
+
+#if defined(CONFIG_64BIT) && defined(CONFIG_SMP)
+	/*
+	 * 64-bit doesn't need locks to atomically read a 64-bit value.
+	 * So we have a optimization chance when the task's delta_exec is 0.
+	 * Reading ->on_cpu is racy, but this is ok.
+	 *
+	 * If we race with it leaving CPU, we'll take a lock. So we're correct.
+	 * If we race with it entering CPU, unaccounted time is 0. This is
+	 * indistinguishable from the read occurring a few cycles earlier.
+	 * If we see ->on_cpu without ->on_rq, the task is leaving, and has
+	 * been accounted, so we're correct here as well.
+	 */
+	if (!p->on_cpu || !task_on_rq_queued(p))
+		return p->se.sum_dvfs_runtime;
+#endif
+
+	rq = task_rq_lock(p, &rf);
+	/*
+	 * Must be ->curr _and_ ->on_rq.  If dequeued, we would
+	 * project cycles that may never be accounted to this
+	 * thread, breaking clock_gettime().
+	 */
+	if (task_current(rq, p) && task_on_rq_queued(p)) {
+		prefetch_curr_exec_start(p);
+		update_rq_clock(rq);
+		p->sched_class->update_curr(rq);
+	}
+	ns = p->se.sum_dvfs_runtime;
+	task_rq_unlock(rq, p, &rf);
+
+	return ns;
 }
 
 #ifdef CONFIG_SCHED_DEBUG
kernel/sched/fair.c +6 -2
@@ -1118,15 +1118,19 @@ static void update_tg_load_avg(struct cf
 static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)
 {
 	u64 now = rq_clock_task(rq);
-	s64 delta_exec;
+	s64 delta_exec, delta_dvfs;
 
-	delta_exec = now - curr->exec_start;
+	delta_dvfs = delta_exec = now - curr->exec_start;
 	if (unlikely(delta_exec <= 0))
 		return delta_exec;
 
 	curr->exec_start = now;
 	curr->sum_exec_runtime += delta_exec;
 
+	delta_dvfs = cap_scale(delta_dvfs, arch_scale_freq_capacity(cpu_of(rq)));
+	delta_dvfs = cap_scale(delta_dvfs, arch_scale_cpu_capacity(cpu_of(rq)));
+	curr->sum_dvfs_runtime += delta_dvfs;
+
 	if (schedstat_enabled()) {
 		struct sched_statistics *stats;
 
kernel/time/posix-cpu-timers.c +15 -1
@@ -164,7 +164,7 @@ posix_cpu_clock_getres(const clockid_t w
 	if (!error) {
 		tp->tv_sec = 0;
 		tp->tv_nsec = ((NSEC_PER_SEC + HZ - 1) / HZ);
-		if (CPUCLOCK_WHICH(which_clock) == CPUCLOCK_SCHED) {
+		if (CPUCLOCK_WHICH(which_clock) >= CPUCLOCK_SCHED) {
 			/*
 			 * If sched_clock is using a cycle counter, we
 			 * don't have any idea of its true resolution
@@ -198,6 +198,9 @@ static u64 cpu_clock_sample(const clocki
 	if (clkid == CPUCLOCK_SCHED)
 		return task_sched_runtime(p);
 
+	if (clkid == CPUCLOCK_DVFS)
+		return task_sched_dvfs_runtime(p);
+
 	task_cputime(p, &utime, &stime);
 
 	switch (clkid) {
@@ -1628,6 +1631,7 @@ static long posix_cpu_nsleep_restart(str
 
 #define PROCESS_CLOCK	make_process_cpuclock(0, CPUCLOCK_SCHED)
 #define THREAD_CLOCK	make_thread_cpuclock(0, CPUCLOCK_SCHED)
+#define THREAD_DVFS_CLOCK make_thread_cpuclock(0, CPUCLOCK_DVFS)
 
 static int process_cpu_clock_getres(const clockid_t which_clock,
 				    struct timespec64 *tp)
@@ -1664,6 +1668,11 @@ static int thread_cpu_timer_create(struc
 	timer->it_clock = THREAD_CLOCK;
 	return posix_cpu_timer_create(timer);
 }
+static int thread_dvfs_cpu_clock_get(const clockid_t which_clock,
+				struct timespec64 *tp)
+{
+	return posix_cpu_clock_get(THREAD_DVFS_CLOCK, tp);
+}
 
 const struct k_clock clock_posix_cpu = {
 	.clock_getres		= posix_cpu_clock_getres,
@@ -1690,3 +1699,8 @@ const struct k_clock clock_thread = {
 	.clock_get_timespec	= thread_cpu_clock_get,
 	.timer_create		= thread_cpu_timer_create,
 };
+
+const struct k_clock clock_thread_dvfs = {
+	.clock_getres		= thread_cpu_clock_getres,
+	.clock_get_timespec	= thread_dvfs_cpu_clock_get,
+};
kernel/time/posix-timers.c +1 -0
@@ -1516,6 +1516,7 @@ static const struct k_clock * const posi
 	[CLOCK_MONOTONIC]		= &clock_monotonic,
 	[CLOCK_PROCESS_CPUTIME_ID]	= &clock_process,
 	[CLOCK_THREAD_CPUTIME_ID]	= &clock_thread,
+	[CLOCK_THREAD_DVFS_ID]		= &clock_thread_dvfs,
 	[CLOCK_MONOTONIC_RAW]		= &clock_monotonic_raw,
 	[CLOCK_REALTIME_COARSE]		= &clock_realtime_coarse,
 	[CLOCK_MONOTONIC_COARSE]	= &clock_monotonic_coarse,
kernel/time/posix-timers.h +1 -0
@@ -34,6 +34,7 @@ extern const struct k_clock clock_posix_
 extern const struct k_clock clock_posix_dynamic;
 extern const struct k_clock clock_process;
 extern const struct k_clock clock_thread;
+extern const struct k_clock clock_thread_dvfs;
 extern const struct k_clock alarm_clock;
 
 int posix_timer_event(struct k_itimer *timr, int si_private);
#215 lore raw

Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID

Quoted 5 lines · Peter Zijlstra

In order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.

The clock does *NOT* support setting timers.

That's not the only limitation. See below.

Quoted 2 lines

Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.

Can this please have an explanation about the usage of the previously reserved value of 0x7 in the lower 3 bits?

Quoted 8 lines

*
* Bit 2 indicates whether a cpu clock refers to a thread or a process. *
- * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3.
+ * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3 *
- * A clockid is invalid if bits 2, 1, and 0 are all set.
+ * (DVFS is PERTHREAD only)

This drops the information about the FD usage. Something like:

/*
* Bit fields within a clockid:
*
* Bit 31:3 hold either a pid or a file descriptor. *
* Bit 2 Bit 1 Bit 0

 *   0      0      0     Per process	CPUCLOCK_PROF
 *   0      0      1     Per process	CPUCLOCK_VIRT
 *   0      1      0     Per process	CPUCLOCK_SCHED
 *   0      1      1     Posixclock FD	CLOCKFD
 *   1      0      0     Per thread	CPUCLOCK_PROF
 *   1      0      1     Per thread	CPUCLOCK_VIRT
 *   1      1      0     Per thread	CPUCLOCK_SCHED
 *   1      1      1     Per thread	CPUCLOCK_DVSF

*
* CPUCLOCK_DVSF is per thread only and shares the type code in Bit 1:0
* with CLOCKFD. CLOCKFD uses a file descriptor to access dynamically
* registered POSIX clocks (e.g. PTP hardware clocks).
*/

should be clear enough, no?

But, all of this is wishful thinking because the provided implementation only works for:

      sys_clock_getres(CLOCK_THREAD_DVFS_ID, ...)

which falls back to thread_cpu_clock_getres().

The variant which has the TID encoded in bit 31:3 and the type in bit 2:0 fails the test in pid_for_clock():

if (CPUCLOCK_WHICH(clock) >= CPUCLOCK_MAX)
return NULL;

Worse for sys_clock_gettime(). That fails in both cases for the very same reason.

See the uncompiled delta patch below for a cure of that and the rest of my comments.

Quoted 7 lines

#define CPUCLOCK_PROF 0
#define CPUCLOCK_VIRT 1
#define CPUCLOCK_SCHED 2
+#define CPUCLOCK_DVFS 3
#define CPUCLOCK_MAX 3
#define CLOCKFD CPUCLOCK_MAX
#define CLOCKFD_MASK (CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)

With that DVFS addition CPUCLOCK_MAX is misleading at best. See delta patch.

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+	rq = task_rq_lock(p, &rf);
+	/*
+	 * Must be ->curr _and_ ->on_rq.  If dequeued, we would
+	 * project cycles that may never be accounted to this
+	 * thread, breaking clock_gettime().

Must be? For what? I assume you want to say:

Update the runtime if the task is the current task and on the runqueue. The latter is important because if current is dequeued,
....

Quoted 9 lines
(quoted diff) +8 -0
+	 */
+	if (task_current(rq, p) && task_on_rq_queued(p)) {
+		prefetch_curr_exec_start(p);
+		update_rq_clock(rq);
+		p->sched_class->update_curr(rq);
+	}
+	ns = p->se.sum_dvfs_runtime;
+	task_rq_unlock(rq, p, &rf);

Review static int thread_cpu_timer_create(struc · line 1672

Quoted 5 lines
(quoted diff) +2 -0
@@ -1664,6 +1668,11 @@ static int thread_cpu_timer_create(struc
 	timer->it_clock = THREAD_CLOCK;
 	return posix_cpu_timer_create(timer);
 }
+static int thread_dvfs_cpu_clock_get(const clockid_t which_clock,
+				struct timespec64 *tp)

Please align the second line properly with the argument in the first line.

Thanks,

tglx
---

include/linux/posix-timers_types.h +29 -14
@@ -9,27 +9,42 @@
 /*
  * Bit fields within a clockid:
  *
- * The most significant 29 bits hold either a pid or a file descriptor.
+ * Bit 31:3 hold either a PID/TID or a file descriptor.
  *
- * Bit 2 indicates whether a cpu clock refers to a thread or a process.
+ * Bit 2  Bit 1  Bit 0
+ *   0      0      0     Per process	CPUCLOCK_PROF
+ *   0      0      1     Per process	CPUCLOCK_VIRT
+ *   0      1      0     Per process	CPUCLOCK_SCHED
+ *   0      1      1     Posixclock FD	CLOCKFD
+ *   1      0      0     Per thread	CPUCLOCK_PROF
+ *   1      0      1     Per thread	CPUCLOCK_VIRT
+ *   1      1      0     Per thread	CPUCLOCK_SCHED
+ *   1      1      1     Per thread	CPUCLOCK_DVSF
  *
- * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3
- *
- * (DVFS is PERTHREAD only)
+ * CPUCLOCK_DVSF is per thread only and shares the type code in Bit 1:0
+ * with CLOCKFD. CLOCKFD uses a file descriptor to access dynamically
+ * registered POSIX clocks (e.g. PTP hardware clocks).
  */
+
 #define CPUCLOCK_PID(clock)		((pid_t) ~((clock) >> 3))
-#define CPUCLOCK_PERTHREAD(clock) \
-	(((clock) & (clockid_t) CPUCLOCK_PERTHREAD_MASK) != 0)
+#define CPUCLOCK_PERTHREAD(clock)	(((clock) & (clockid_t) CPUCLOCK_PERTHREAD_MASK) != 0)
 
-#define CPUCLOCK_PERTHREAD_MASK	4
-#define CPUCLOCK_WHICH(clock)	((clock) & (clockid_t) CPUCLOCK_CLOCK_MASK)
-#define CPUCLOCK_CLOCK_MASK	3
 #define CPUCLOCK_PROF		0
 #define CPUCLOCK_VIRT		1
 #define CPUCLOCK_SCHED		2
-#define CPUCLOCK_DVFS		3
-#define CPUCLOCK_MAX		3
-#define CLOCKFD			CPUCLOCK_MAX
+#define CPUCLOCK_SAMPLE_MAX	(CPUCLOCK_SCHED + 1)
+
+#define CPUCLOCK_CLOCK_MASK	3
+#define CPUCLOCK_PERTHREAD_MASK	4
+#define CPUCLOCK_WHICH(clock)	((clock) & (clockid_t) CPUCLOCK_CLOCK_MASK)
+
+/*
+ * CPUCLOCK_DVFS and CLOCKFD share the type code in bit 1:0. CPUCLOCK_DVFS
+ * does not belong to the sampling clocks and does not allow timers to be
+ * armed on it.
+ */
+#define CPUCLOCK_DVFS		CPUCLOCK_SAMPLE_MAX
+#define CLOCKFD			CPUCLOCK_DVFS
 #define CLOCKFD_MASK		(CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)
 
 #ifdef CONFIG_POSIX_TIMERS
@@ -55,7 +70,7 @@ struct posix_cputimer_base {
  * Used in task_struct and signal_struct
  */
 struct posix_cputimers {
-	struct posix_cputimer_base	bases[CPUCLOCK_MAX];
+	struct posix_cputimer_base	bases[CPUCLOCK_SAMPLE_MAX];
 	unsigned int			timers_active;
 	unsigned int			expiry_active;
 };
kernel/sched/core.c +4 -3
@@ -5413,9 +5413,10 @@ unsigned long long task_sched_dvfs_runti
 
 	rq = task_rq_lock(p, &rf);
 	/*
-	 * Must be ->curr _and_ ->on_rq.  If dequeued, we would
-	 * project cycles that may never be accounted to this
-	 * thread, breaking clock_gettime().
+	 * Update the runtime if the task is the current task and on the
+	 * runqueue. The latter is important because if current is
+	 * dequeued, we would project cycles that may never be accounted to
+	 * this thread, breaking clock_gettime().
 	 */
 	if (task_current(rq, p) && task_on_rq_queued(p)) {
 		prefetch_curr_exec_start(p);
kernel/time/posix-cpu-timers.c +17 -12
@@ -54,13 +54,13 @@ int update_rlimit_cpu(struct task_struct
 /*
  * Functions for validating access to tasks.
  */
-static struct pid *pid_for_clock(const clockid_t clock, bool gettime)
+static struct pid *__pid_for_clock(const clockid_t clock, const clockid_t maxclock, bool gettime)
 {
 	const bool thread = !!CPUCLOCK_PERTHREAD(clock);
 	const pid_t upid = CPUCLOCK_PID(clock);
 	struct pid *pid;
 
-	if (CPUCLOCK_WHICH(clock) >= CPUCLOCK_MAX)
+	if (CPUCLOCK_WHICH(clock) > maxclock)
 		return NULL;
 
 	/*
@@ -94,12 +94,17 @@ static struct pid *pid_for_clock(const c
 	return pid_has_task(pid, PIDTYPE_TGID) ? pid : NULL;
 }
 
+static inline struct pid *pid_for_clock(const clockid_t clock, bool gettime)
+{
+	return __pid_for_clock(clock, CPUCLOCK_SCHED, gettime);
+}
+
 static inline int validate_clock_permissions(const clockid_t clock)
 {
 	int ret;
 
 	rcu_read_lock();
-	ret = pid_for_clock(clock, false) ? 0 : -EINVAL;
+	ret = __pid_for_clock(clock, CPUCLOCK_DVFS, false) ? 0 : -EINVAL;
 	rcu_read_unlock();
 
 	return ret;
@@ -344,7 +349,7 @@ static u64 cpu_clock_sample_group(const
 {
 	struct thread_group_cputimer *cputimer = &p->signal->cputimer;
 	struct posix_cputimers *pct = &p->signal->posix_cputimers;
-	u64 samples[CPUCLOCK_MAX];
+	u64 samples[CPUCLOCK_SAMPLE_MAX];
 
 	if (!READ_ONCE(pct->timers_active)) {
 		if (start)
@@ -365,7 +370,7 @@ static int posix_cpu_clock_get(const clo
 	u64 t;
 
 	rcu_read_lock();
-	tsk = pid_task(pid_for_clock(clock, true), clock_pid_type(clock));
+	tsk = pid_task(__pid_for_clock(clock, CPUCLOCK_DVFS, true), clock_pid_type(clock));
 	if (!tsk) {
 		rcu_read_unlock();
 		return -EINVAL;
@@ -864,7 +869,7 @@ static void collect_posix_cputimers(stru
 	struct posix_cputimer_base *base = pct->bases;
 	int i;
 
-	for (i = 0; i < CPUCLOCK_MAX; i++, base++) {
+	for (i = 0; i < CPUCLOCK_SAMPLE_MAX; i++, base++) {
 		base->nextevt = collect_timerqueue(&base->tqhead, firing,
 						    samples[i]);
 	}
@@ -901,7 +906,7 @@ static void check_thread_timers(struct t
 				struct list_head *firing)
 {
 	struct posix_cputimers *pct = &tsk->posix_cputimers;
-	u64 samples[CPUCLOCK_MAX];
+	u64 samples[CPUCLOCK_SAMPLE_MAX];
 	unsigned long soft;
 
 	if (dl_task(tsk))
@@ -979,7 +984,7 @@ static void check_process_timers(struct
 {
 	struct signal_struct *const sig = tsk->signal;
 	struct posix_cputimers *pct = &sig->posix_cputimers;
-	u64 samples[CPUCLOCK_MAX];
+	u64 samples[CPUCLOCK_SAMPLE_MAX];
 	unsigned long soft;
 
 	/*
@@ -1098,7 +1103,7 @@ task_cputimers_expired(const u64 *sample
 {
 	int i;
 
-	for (i = 0; i < CPUCLOCK_MAX; i++) {
+	for (i = 0; i < CPUCLOCK_SAMPLE_MAX; i++) {
 		if (samples[i] >= pct->bases[i].nextevt)
 			return true;
 	}
@@ -1121,7 +1126,7 @@ static inline bool fastpath_timer_check(
 	struct signal_struct *sig;
 
 	if (!expiry_cache_is_inactive(pct)) {
-		u64 samples[CPUCLOCK_MAX];
+		u64 samples[CPUCLOCK_SAMPLE_MAX];
 
 		task_sample_cputime(tsk, samples);
 		if (task_cputimers_expired(samples, pct))
@@ -1146,7 +1151,7 @@ static inline bool fastpath_timer_check(
 	 * delays with signals actually getting sent are expected.
 	 */
 	if (READ_ONCE(pct->timers_active) && !READ_ONCE(pct->expiry_active)) {
-		u64 samples[CPUCLOCK_MAX];
+		u64 samples[CPUCLOCK_SAMPLE_MAX];
 
 		proc_sample_cputime_atomic(&sig->cputimer.cputime_atomic,
 					   samples);
@@ -1669,7 +1674,7 @@ static int thread_cpu_timer_create(struc
 	return posix_cpu_timer_create(timer);
 }
 static int thread_dvfs_cpu_clock_get(const clockid_t which_clock,
-				struct timespec64 *tp)
+				     struct timespec64 *tp)
 {
 	return posix_cpu_clock_get(THREAD_DVFS_CLOCK, tp);
 }
maintainer
#216 lore raw

Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID

In reply to Peter Zijlstra

Hi Peter,

Quoted 8 lines · Peter Zijlstra

In order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.

The clock does *NOT* support setting timers.

Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.

Just so I'm sure I understand, this would be useful for estimating the runtime needs of a (also DEADLINE) task when DVFS is enabled, right?

Thanks,
Juri

author maintainer
#217 lore raw

Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID

Quoted 7 lines · Juri Lelli

Hi Peter,

Quoted 4 lines · Peter Zijlstra

In order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.

Quoted 7 lines
Quoted 4 lines

The clock does *NOT* support setting timers.

Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.

Just so I'm sure I understand, this would be useful for estimating the runtime needs of a (also DEADLINE) task when DVFS is enabled, right?

Correct, DVFS or biggie-smalls CPUs with mixed capacities.

#218 lore raw

Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID

In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra

In order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.

The clock does *NOT* support setting timers.

Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.

How will this look like in practice then?
Is it up to userspace to adjust sched_runtime to capacity/dvfs accordingly every time it changes? I guess not.
Will sched_attr::sched_runtime be for CPUCLOCK_DVFS by default? I assume that would be a uapi change?
Do we need an additional flag in sched_attr to specify the clock to be measured against?

Quoted 16 lines
---
 include/linux/posix-timers_types.h |    5 ++--
 include/linux/sched.h              |    1 
 include/linux/sched/cputime.h      |    3 ++
 include/uapi/linux/time.h          |    1 
 kernel/sched/core.c                |   40 +++++++++++++++++++++++++++++++++++++
 kernel/sched/fair.c                |    8 +++++--
 kernel/time/posix-cpu-timers.c     |   16 +++++++++++++-
 kernel/time/posix-timers.c         |    1 
 kernel/time/posix-timers.h         |    1 
 9 files changed, 71 insertions(+), 5 deletions(-)

include/linux/posix-timers_types.h line 16

Quoted 5 lines
include/linux/posix-timers_types.h +1 -1
@@ -13,9 +13,9 @@
  *
  * Bit 2 indicates whether a cpu clock refers to a thread or a process.
  *
- * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3.
+ * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3

s/DVSF/DVFS

Quoted 1 line

[snip]

#219 lore raw
In reply to Peter Zijlstra

Hi Peter,

Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 11 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

Thanks for the updated series.

FTR, I ran this through our Pixel 6 and did not see significant differences between the baseline (6.8 kernel with EEVDF + fixes) and the baseline with the updated eevdf-complete series applied on top.

The power use and frame metrics seem mostly the same between the two.

I'll run a few more tests just in case.

maintainer
#220 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

Quoted 7 lines

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

I'm using your sched/core branch. Is it the correct one ?

Quoted 12 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

#221 lore raw
Quoted 9 lines · Vincent Guittot

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

I'm using your sched/core branch. Is it the correct one ?

Hm, building that branch, I see the expected tick granularity (4ms).

-Mike

maintainer
#222 lore raw
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

Quoted 3 lines
Quoted 1 line

I'm using your sched/core branch. Is it the correct one ?

Hm, building that branch, I see the expected tick granularity (4ms).

On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.

Quoted 3 lines

-Mike

#223 lore raw
Quoted 11 lines · Vincent Guittot
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'm using your sched/core branch. Is it the correct one ?

Hm, building that branch, I see the expected tick granularity (4ms).

On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.

FWIW, I checked my local master-rt tree as well, which has Peter's latest eevdf series wedged in (plus 4cc290c20a98 now).. it also worked as expected.

-Mike

maintainer
#224 lore raw
Quoted 17 lines · Mike Galbraith
Quoted 15 lines · Vincent Guittot
Quoted 13 lines · Mike Galbraith
Quoted 9 lines · Vincent Guittot

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

I'm using your sched/core branch. Is it the correct one ?

Hm, building that branch, I see the expected tick granularity (4ms).

Quoted 7 lines
Quoted 3 lines

On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.

FWIW, I checked my local master-rt tree as well, which has Peter's latest eevdf series wedged in (plus 4cc290c20a98 now).. it also worked as expected.

After looking at deadline and slice, the issue is on my tool was trying to change the slice (an old trial for previous version) which got clamp to 100ms.
we can forgot this, sorry for the noise

Quoted 3 lines

-Mike

#225 lore raw
In reply to Vincent Guittot

Hello Vincent, Mike,

Quoted 11 lines · Vincent Guittot
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot

While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'm using your sched/core branch. Is it the correct one ?

Hm, building that branch, I see the expected tick granularity (4ms).

On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.

I could not observe this behavior when running two busy loops pinned to one CPU on my end. I'm running with base_slice_ns of 3ms and the sched_feats related to EEVDF complete looks as follows:

    PLACE_LAG
    PLACE_DEADLINE_INITIAL
    PLACE_REL_DEADLINE
    RUN_TO_PARITY
    PREEMPT_SHORT
    NO_NEXT_BUDDY
    CACHE_HOT_BUDDY
    DELAY_DEQUEUE
    DELAY_ZERO
    WAKEUP_PREEMPTION
    ...

Also I'm running with CONFIG_HZ=250 (4ms tick granularity)

    CONFIG_HZ_250=y
    CONFIG_HZ=250

Enabling sched_switch tracepeoint, I see the following:

            ...
            loop-4061    109.710379: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.714377: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.718375: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.722374: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.726379: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.730377: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.734367: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.738365: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.742364: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.746361: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.750359: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.754357: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.758355: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.762353: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.766351: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.770349: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.774347: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.778345: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.782343: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.786346: sched_switch: ... prev_pid=4060 ... prev_state=R ==> next_comm=kworker/1:1 next_pid=1616 next_prio=120
     kworker/1:1-1616    109.786412: sched_switch: prev_comm=kworker/1:1 prev_pid=1616 ... prev_state=I ==> ... next_pid=4061 ...
            loop-4061    109.794337: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.798335: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.802335: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.806331: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.810329: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.814327: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.818325: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.822323: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.826321: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            loop-4060    109.830321: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
            loop-4061    109.834317: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
            ...

(Trimmed traces are for busy loops with pids 4060 and 4061)

I see the expected tick granularity similar to Mike. Since Peter's tree is prone to force-updates, I'm on

git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/core

at commit 4cc290c20a98 "sched/eevdf: Dequeue in switched_from_fair()" which was committed at "2024-08-14 08:15:39 +0200".

Quoted 4 lines
Quoted 4 lines

-Mike

Signature · 2 lines
Thanks and Regards,
Prateek
reviewer Tested-by Reviewed-by on the v1 series
#226 lore raw
In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

So what I've been testing at your queue/sched/core, HEAD at:
4cc290c20a98 ("sched/eevdf: Dequeue in switched_from_fair()")
survives my (simplistic) CFS bandwidth testing, so FWIW:

And for patches 01-20:

with one caveat that I agree with Vincent on maybe doing something about the load PELT signal of delayed dequeued entities.

#227 lore raw
In reply to Peter Zijlstra

Hi Peter,

Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 9 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

The latest tip/sched/core at commit

aef6987d89544d63a47753cf3741cabff0b5574c

crashes very early on on my Juno r2 board (arm64). The trace is here:

[    0.049599] ------------[ cut here ]------------
[    0.054279] kernel BUG at kernel/sched/deadline.c:63!
[    0.059401] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
[    0.066285] Modules linked in:
[    0.069382] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.11.0-rc1-g55404cef33db #1070
[    0.077855] Hardware name: ARM Juno development board (r2) (DT)
[    0.083856] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.090919] pc : enqueue_dl_entity+0x53c/0x540
[    0.095434] lr : dl_server_start+0xb8/0x10c
[    0.099679] sp : ffffffc081ca3c30
[    0.103034] x29: ffffffc081ca3c40 x28: 0000000000000001 x27: 0000000000000002
[    0.110281] x26: 00000000000b71b0 x25: 0000000000000000 x24: 0000000000000001
[    0.117525] x23: ffffff897ef21140 x22: 0000000000000000 x21: 0000000000000000
[    0.124770] x20: ffffff897ef21040 x19: ffffff897ef219a8 x18: ffffffc080d0ad00
[    0.132015] x17: 000000000000002f x16: 0000000000000000 x15: ffffffc081ca8000
[    0.139260] x14: 00000000016ef200 x13: 00000000000e6667 x12: 0000000000000001
[    0.146505] x11: 000000003b9aca00 x10: 0000000002faf080 x9 : 0000000000000030
[    0.153749] x8 : 0000000000000071 x7 : 000000002cf93d25 x6 : 000000002cf93d25
[    0.160994] x5 : ffffffc081e04938 x4 : ffffffc081ca3d40 x3 : 0000000000000001
[    0.168238] x2 : 000000003b9aca00 x1 : 0000000000000001 x0 : ffffff897ef21040
[    0.175483] Call trace:
[    0.177958]  enqueue_dl_entity+0x53c/0x540
[    0.182117]  dl_server_start+0xb8/0x10c
[    0.186010]  enqueue_task_fair+0x5c8/0x6ac
[    0.190165]  enqueue_task+0x54/0x1e8
[    0.193793]  wake_up_new_task+0x250/0x39c
[    0.197862]  kernel_clone+0x140/0x2f0
[    0.201578]  user_mode_thread+0x4c/0x58
[    0.205468]  rest_init+0x24/0xd8
[    0.208743]  start_kernel+0x2bc/0x2fc
[    0.212460]  __primary_switched+0x80/0x88
[    0.216535] Code: b85fc3a8 7100051f 54fff8e9 17ffffce (d4210000)
[    0.222711] ---[ end trace 0000000000000000 ]---
[    0.227391] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.234187] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

I'm not an expert in DL server so I have no idea where the problem could be. If you know where to look off the top of your head then much better. If not, I'll do some bi-section later.

Hongyan

#228 lore raw
Quoted 70 lines · Hongyan Xia

Hi Peter,

Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 9 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

  - split up the huge delay-dequeue patch
  - tested/fixed cfs-bandwidth
  - PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
  - SCHED_BATCH is equivalent to RESPECT_SLICE
  - propagate min_slice up cgroups
  - CLOCK_THREAD_DVFS_ID

The latest tip/sched/core at commit

aef6987d89544d63a47753cf3741cabff0b5574c

crashes very early on on my Juno r2 board (arm64). The trace is here:

[    0.049599] ------------[ cut here ]------------
[    0.054279] kernel BUG at kernel/sched/deadline.c:63!
[    0.059401] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
[    0.066285] Modules linked in:
[    0.069382] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.11.0-rc1-g55404cef33db #1070 [    0.077855] Hardware name: ARM Juno development board (r2) (DT)
[    0.083856] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [    0.090919] pc : enqueue_dl_entity+0x53c/0x540
[    0.095434] lr : dl_server_start+0xb8/0x10c
[    0.099679] sp : ffffffc081ca3c30
[    0.103034] x29: ffffffc081ca3c40 x28: 0000000000000001 x27: 0000000000000002 [    0.110281] x26: 00000000000b71b0 x25: 0000000000000000 x24: 0000000000000001 [    0.117525] x23: ffffff897ef21140 x22: 0000000000000000 x21: 0000000000000000 [    0.124770] x20: ffffff897ef21040 x19: ffffff897ef219a8 x18: ffffffc080d0ad00 [    0.132015] x17: 000000000000002f x16: 0000000000000000 x15: ffffffc081ca8000 [    0.139260] x14: 00000000016ef200 x13: 00000000000e6667 x12: 0000000000000001 [    0.146505] x11: 000000003b9aca00 x10: 0000000002faf080 x9 : 0000000000000030 [    0.153749] x8 : 0000000000000071 x7 : 000000002cf93d25 x6 : 000000002cf93d25 [    0.160994] x5 : ffffffc081e04938 x4 : ffffffc081ca3d40 x3 : 0000000000000001 [    0.168238] x2 : 000000003b9aca00 x1 : 0000000000000001 x0 : ffffff897ef21040 [    0.175483] Call trace:
[    0.177958]  enqueue_dl_entity+0x53c/0x540
[    0.182117]  dl_server_start+0xb8/0x10c
[    0.186010]  enqueue_task_fair+0x5c8/0x6ac
[    0.190165]  enqueue_task+0x54/0x1e8
[    0.193793]  wake_up_new_task+0x250/0x39c
[    0.197862]  kernel_clone+0x140/0x2f0
[    0.201578]  user_mode_thread+0x4c/0x58
[    0.205468]  rest_init+0x24/0xd8
[    0.208743]  start_kernel+0x2bc/0x2fc
[    0.212460]  __primary_switched+0x80/0x88
[    0.216535] Code: b85fc3a8 7100051f 54fff8e9 17ffffce (d4210000)
[    0.222711] ---[ end trace 0000000000000000 ]---
[    0.227391] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.234187] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

Quoted 2 lines

I'm not an expert in DL server so I have no idea where the problem could be. If you know where to look off the top of your head then much better. If not, I'll do some bi-section later.

Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:

dl_server_start()

	enqueue_dl_entity()
		update_stats_enqueue_dl()
			update_stats_enqueue_sleeper_dl()
				__schedstats_from_dl_se()
					dl_task_of() <---------- crash

If I undefine CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.

#229 lore raw
Quoted 4 lines · Hongyan Xia

If I undefine CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.

FWIW, tip.today boots/runs gripe free on dinky rpi4b (SCHEDSTATS=y).

-Mike

author maintainer
#230 lore raw
In reply to Hongyan Xia
Quoted 10 lines · Hongyan Xia

Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:

dl_server_start()

	enqueue_dl_entity()
		update_stats_enqueue_dl()
			update_stats_enqueue_sleeper_dl()
				__schedstats_from_dl_se()
					dl_task_of() <---------- crash
Quoted 3 lines

If I undefine CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.

Thanks for the report -- I'll see if I can spot something. Since you initially fingered these eevdf patches, could you confirm or deny that changing:

kernel/sched/features.h:SCHED_FEAT(DELAY_DEQUEUE, true)

to false, makes any difference in the previously failing case?

#231 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 10 lines · Hongyan Xia

Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:

dl_server_start()

	enqueue_dl_entity()
		update_stats_enqueue_dl()
			update_stats_enqueue_sleeper_dl()
				__schedstats_from_dl_se()
					dl_task_of() <---------- crash
Quoted 3 lines

If I undefine CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.

Quoted 7 lines

Thanks for the report -- I'll see if I can spot something. Since you initially fingered these eevdf patches, could you confirm or deny that changing:

kernel/sched/features.h:SCHED_FEAT(DELAY_DEQUEUE, true)

to false, makes any difference in the previously failing case?

Sadly the issue persists. I'm seeing exactly the same backtrace on my Juno board.

#232 lore raw
In reply to Peter Zijlstra

This series has caused an explosion of different kernel crashes on our syzbot instance that fuzzes linux-next. I guess, such kernel behavior indicates some massive underlying memory corruption (?)

Some of the crash titles we've seen (we didn't release them -- there were too many, 70+):

KASAN: stack-out-of-bounds Write in insn_decode 
kernel panic: stack is corrupted in vprintk_store
kernel panic: stack is corrupted in _printk
BUG: spinlock recursion in __schedule
WARNING in __put_task_struct
BUG: unable to handle kernel NULL pointer dereference in asm_exc_page_fault
WARNING in rng_dev_read
BUG: scheduling while atomic in prb_final_commit
kernel BUG in dequeue_rt_stack
BUG: scheduling while atomic in rcu_is_watching
BUG: spinlock recursion in copy_process
KASAN: slab-use-after-free Read in sched_core_enqueue
kernel panic: stack is corrupted in refill_stock
kernel panic: stack is corrupted in prb_reserve
WARNING: bad unlock balance in timekeeping_get_ns
KASAN: slab-use-after-free Read in set_next_task_fair

I wonder if the actual problem is already known and possibly there are even some fix patches?

If not and if it may be of any help, we can try to come up with some contained instruction to reproduce these issues with syzkaller.

--
Aleksandr

#233 lore raw
In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

I'm seeing crashes/warnings like the following on s390 with linux-next 20240909:

Sometimes the system doesn't manage to print a oops, this one is the best i got:

[  596.146142] ------------[ cut here ]------------
[  596.146161] se->sched_delayed
[  596.146166] WARNING: CPU: 1 PID: 0 at kernel/sched/fair.c:13131 __set_next_task_fair.part.0+0x350/0x400
[  596.146179] Modules linked in: [..]
[  596.146288] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.11.0-rc7-next-20240909 #18
[  596.146294] Hardware name: IBM 3931 A01 704 (LPAR)
[  596.146298] Krnl PSW : 0404e00180000000 001a9c2b5eea4ea4 (__set_next_task_fair.part.0+0x354/0x400)
[  596.146307]            R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
[  596.146314] Krnl GPRS: 001c000300000027 001c000300000023 0000000000000011 0000000000000004
[  596.146319]            0000000000000001 001a9c2b5f1fb118 000000036ef94dd5 0000001b77ca6ea8
[  596.146323]            001c000000000000 001a9c2b5eec6fc0 0000001b77ca6000 00000000b7334800
[  596.146328]            0000000000000000 001a9c2b5eefad70 001a9c2b5eea4ea0 001a9bab5ee8f9f8
[  596.146340] Krnl Code: 001a9c2b5eea4e94: c0200121bbe6        larl    %r2,001a9c2b612dc660
[  596.146340]            001a9c2b5eea4e9a: c0e5fff9e9d3        brasl   %r14,001a9c2b5ede2240
[  596.146340]           #001a9c2b5eea4ea0: af000000            mc      0,0
[  596.146340]           >001a9c2b5eea4ea4: a7f4fe83            brc     15,001a9c2b5eea4baa
[  596.146340]            001a9c2b5eea4ea8: c0e50038ba2c        brasl   %r14,001a9c2b5f5bc300
[  596.146558] CPU: 1 UID: 0 PID: 18582 Comm: prctl-sched-cor Tainted: G        W          6.11.0-rc7-next-20240909 #18
[  596.146564] Tainted: [W]=WARN
[  596.146567] Hardware name: IBM 3931 A01 704 (LPAR)
[  596.146570] Krnl PSW : 0404e00180000000 001a9c2b5eec2de4 (dequeue_entity+0xe64/0x11f0)
[  596.146578]            R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
[  596.146584] Krnl GPRS: 001c000300000027 001c000300000023 000000000000001a 0000000000000004
[  596.146589]            0000000000000001 001a9c2b5f1fb118 001a9c2b61be7144 0000000016e6692a
[  596.146593]            0000000000000001 00000000b7334951 0000000158494800 00000000b7334900
[  596.146597]            000000000000489e 0000000000000009 001a9c2b5eec2de0 001a9bab75dff760
[  596.146607] Krnl Code: 001a9c2b5eec2dd4: c0200120cdf6        larl    %r2,001a9c2b612dc9c0
[  596.146607]            001a9c2b5eec2dda: c0e5fff8fa33        brasl   %r14,001a9c2b5ede2240
[  596.146607]           #001a9c2b5eec2de0: af000000            mc      0,0
[  596.146607]           >001a9c2b5eec2de4: c004fffff90a        brcl    0,001a9c2b5eec1ff8
[  596.146607]            001a9c2b5eec2dea: a7f4fbbe            brc     15,001a9c2b5eec2566
[  596.146607]            001a9c2b5eec2dee: a7d10001            tmll    %r13,1
[  596.146607]            001a9c2b5eec2df2: a774fb1c            brc     7,001a9c2b5eec242a
[  596.146607]            001a9c2b5eec2df6: a7f4f95f            brc     15,001a9c2b5eec20b4
[  596.146637] Call Trace:
[  596.146640]  [<001a9c2b5eec2de4>] dequeue_entity+0xe64/0x11f0 
[  596.146645] ([<001a9c2b5eec2de0>] dequeue_entity+0xe60/0x11f0)
[  596.146650]  [<001a9c2b5eec34b0>] dequeue_entities+0x340/0xe10 
[  596.146655]  [<001a9c2b5eec4208>] dequeue_task_fair+0xb8/0x5a0 
[  596.146660]  [<001a9c2b6115ab68>] __schedule+0xb58/0x14f0 
[  596.146666]  [<001a9c2b6115b59c>] schedule+0x9c/0x240 
[  596.146670]  [<001a9c2b5edf5190>] do_wait+0x160/0x440 
[  596.146676]  [<001a9c2b5edf5936>] kernel_waitid+0xd6/0x110 
[  596.146680]  [<001a9c2b5edf5b4e>] __do_sys_waitid+0x1de/0x1f0 
[  596.146685]  [<001a9c2b5edf5c36>] __s390x_sys_waitid+0xd6/0x120 
[  596.146690]  [<001a9c2b5ed0cbd6>] do_syscall+0x2f6/0x430 
[  596.146695]  [<001a9c2b611543a4>] __do_syscall+0xa4/0x170 
[  596.146700]  [<001a9c2b6117046c>] system_call+0x74/0x98 
[  596.146705] Last Breaking-Event-Address:
[  596.146707]  [<001a9c2b5ede2418>] __warn_printk+0x1d8/0x1e0

This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.

Regards
Sven

#234 lore raw
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

I'm seeing crashes/warnings like the following on s390 with linux-next 20240909:

Quoted 7 lines

Sometimes the system doesn't manage to print a oops, this one is the best i got:

[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.

I failed to add the log from git bisect. Unfortunately i had to skip some commit because the kernel didn't compile:

git bisect start
# status: waiting for both good and bad commits
# bad: [100cc857359b5d731407d1038f7e76cd0e871d94] Add linux-next specific files for 20240909
git bisect bad 100cc857359b5d731407d1038f7e76cd0e871d94
# status: waiting for good commit(s), bad commit known
# good: [da3ea35007d0af457a0afc87e84fddaebc4e0b63] Linux 6.11-rc7
git bisect good da3ea35007d0af457a0afc87e84fddaebc4e0b63
# good: [df20078b9706977cc3308740b56993cf27665f90] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good df20078b9706977cc3308740b56993cf27665f90
# good: [609f9e1b6242e7158ce96f9124372601997ce56c] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git
git bisect good 609f9e1b6242e7158ce96f9124372601997ce56c
# skip: [664c3413e9a6c345a6c926841358314be9da8309] Merge branch 'usb-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git
git bisect skip 664c3413e9a6c345a6c926841358314be9da8309
# good: [16531118ba63dd9bcd65203d04a9c9d6f6800547] iio: bmi323: peripheral in lowest power state on suspend
git bisect good 16531118ba63dd9bcd65203d04a9c9d6f6800547
# bad: [d9c7ac7f8bfb16f431daa7c77bdfe2b163361ead] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86.git
git bisect bad d9c7ac7f8bfb16f431daa7c77bdfe2b163361ead
# bad: [05536babd768b38d84ad168450f48634a013603d] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect bad 05536babd768b38d84ad168450f48634a013603d
# good: [dabef94a179957db117db344b924e5d5c4074e5f] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git
git bisect good dabef94a179957db117db344b924e5d5c4074e5f
# bad: [d4886a325947ecae6867fc858657062211aae3b9] Merge branch into tip/master: 'locking/core'
git bisect bad d4886a325947ecae6867fc858657062211aae3b9
# bad: [51c095bee5c77590d43519f03179342e910d333c] Merge branch into tip/master: 'core/core'
git bisect bad 51c095bee5c77590d43519f03179342e910d333c
# bad: [fc1892becd5672f52329a75c73117b60ac7841b7] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
git bisect bad fc1892becd5672f52329a75c73117b60ac7841b7
# good: [ae04f69de0bef93c7086cf2983dbc8e8fd624ebe] sched/rt: Rename realtime_{prio, task}() to rt_or_dl_{prio, task}()
git bisect good ae04f69de0bef93c7086cf2983dbc8e8fd624ebe
# good: [abc158c82ae555078aa5dd2d8407c3df0f868904] sched: Prepare generic code for delayed dequeue
git bisect good abc158c82ae555078aa5dd2d8407c3df0f868904
# skip: [781773e3b68031bd001c0c18aa72e8470c225ebd] sched/fair: Implement ENQUEUE_DELAYED
git bisect skip 781773e3b68031bd001c0c18aa72e8470c225ebd
# skip: [e1459a50ba31831efdfc35278023d959e4ba775b] sched: Teach dequeue_task() about special task states
git bisect skip e1459a50ba31831efdfc35278023d959e4ba775b
# skip: [a1c446611e31ca5363d4db51e398271da1dce0af] sched,freezer: Mark TASK_FROZEN special
git bisect skip a1c446611e31ca5363d4db51e398271da1dce0af
# good: [e28b5f8bda01720b5ce8456b48cf4b963f9a80a1] sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
git bisect good e28b5f8bda01720b5ce8456b48cf4b963f9a80a1
# skip: [f12e148892ede8d9ee82bcd3e469e6d01fc077ac] sched/fair: Prepare pick_next_task() for delayed dequeue
git bisect skip f12e148892ede8d9ee82bcd3e469e6d01fc077ac
# skip: [152e11f6df293e816a6a37c69757033cdc72667d] sched/fair: Implement delayed dequeue
git bisect skip 152e11f6df293e816a6a37c69757033cdc72667d
# skip: [2e0199df252a536a03f4cb0810324dff523d1e79] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
git bisect skip 2e0199df252a536a03f4cb0810324dff523d1e79
# bad: [54a58a78779169f9c92a51facf6de7ce94962328] sched/fair: Implement DELAY_ZERO
git bisect bad 54a58a78779169f9c92a51facf6de7ce94962328

# only skipped commits left to test
# possible first bad commit: [54a58a78779169f9c92a51facf6de7ce94962328] sched/fair: Implement DELAY_ZERO
# possible first bad commit: [152e11f6df293e816a6a37c69757033cdc72667d] sched/fair: Implement delayed dequeue
# possible first bad commit: [e1459a50ba31831efdfc35278023d959e4ba775b] sched: Teach dequeue_task() about special task states
# possible first bad commit: [a1c446611e31ca5363d4db51e398271da1dce0af] sched,freezer: Mark TASK_FROZEN special
# possible first bad commit: [781773e3b68031bd001c0c18aa72e8470c225ebd] sched/fair: Implement ENQUEUE_DELAYED
# possible first bad commit: [f12e148892ede8d9ee82bcd3e469e6d01fc077ac] sched/fair: Prepare pick_next_task() for delayed dequeue
# possible first bad commit: [2e0199df252a536a03f4cb0810324dff523d1e79] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
author maintainer
#235 lore raw
Quoted 30 lines · Sven Schnelle
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

I'm seeing crashes/warnings like the following on s390 with linux-next 20240909:

Quoted 7 lines
Quoted 7 lines

Sometimes the system doesn't manage to print a oops, this one is the best i got:

[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.

I noted: Comm: prctl-sched-cor, which is testing core scheduling, right?

Only today I;ve merged a fix for that:

c662e2b1e8cf ("sched: Fix sched_delayed vs sched_core")

Could you double check if merging tip/sched/core into your next tree helps anything at all?

#236 lore raw
Quoted 39 lines · Peter Zijlstra
Quoted 30 lines · Sven Schnelle
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

I'm seeing crashes/warnings like the following on s390 with linux-next 20240909:

Quoted 7 lines
Quoted 7 lines

Sometimes the system doesn't manage to print a oops, this one is the best i got:

[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.

Quoted 8 lines

I noted: Comm: prctl-sched-cor, which is testing core scheduling, right?

Only today I;ve merged a fix for that:

c662e2b1e8cf ("sched: Fix sched_delayed vs sched_core")

Could you double check if merging tip/sched/core into your next tree helps anything at all?

Yes, that fixes the issue. Thanks!

#237 lore raw
In reply to Peter Zijlstra
Quoted 16 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 9 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

Hi Peter,

TL;DR:
We run some basic sched/cpufreq behavior tests on a Pixel 6 for every change we accept. Some of these changes are merges from Linus's tree. We can see a very clear change in behavior with this patch series. Based on what we are seeing, we'd expect this change in behavior to cause pretty serious power regression (7-25%) depending on what the actual bug is and the use case.

Intro:
We run these tests 20 times for every build (a bunch of changes). All the data below is from the 20+ builds before this series and 20 builds after this series (inclusive). So, all the "before numbers" are from (20 x 20) 400+ runs and all the after numbers are from another 400+ runs.

Test:
We create a synthetic "tiny" thread that runs for 3ms and sleeps for 10ms at Fmin. We let it run like this for several seconds to make sure the util is low and all the "new thread" boost stuff isn't kicking in. So, at this point, the CPU frequency is at Fmin. Then we let this thread run continuously without sleeping and measure (using ftrace) the time it takes for the CPU to get to Fmax.

We do this separately (fresh run) on the Pixel 6 with the cpu affinity set to each cluster and once without any cpu affinity (thread starts at little).

Data:
All the values below are in milliseconds.

When the thread is not affined to any CPU: So thread starts on little, ramps up to fmax, migrates to middle, ramps up to fmax, migrates to big, ramps up to fmax.

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 169    | 151   |
|-----------------------+----------|
| Median          | 221    | 177   |
|-----------------------+----------|
| Mean            | 221    | 177   |
|-----------------------+----------|
| 95th percentile | 249    | 200   |
+----------------------------------+

When thread is affined to the little cluster:
The average time to reach Fmax is 104 ms without this series and 66 ms after this series. We didn't collect the individual per run data. We can if you really need it. We also noticed that the little cluster wouldn't go to Fmin (300 MHz) after this series even when the CPUs are mostly idle. It was instead hovering at 738 MHz (the Fmax is ~1800 MHz).

When thread is affined to the middle cluster:

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 99     | 84    |
|-----------------------+----------|
| Median          | 111    | 104   |
|-----------------------+----------|
| Mean            | 111    | 104   |
|-----------------------+----------|
| 95th percentile | 120    | 119   |
+----------------------------------+

When thread is affined to the big cluster:

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 138    | 96    |
|-----------------------+----------|
| Median          | 147    | 144   |
|-----------------------+----------|
| Mean            | 145    | 134   |
|-----------------------+----------|
| 95th percentile | 151    | 150   |
+----------------------------------+

As you can see, the ramp up time has decreased noticeably. Also, as you can tell from the 5th percentile numbers, the standard deviation has also increased a lot too, causing a wider spread of the ramp up time (more noticeable in the middle and big clusters). So in general this looks like it's going to increase the usage of the middle and big CPU clusters and also going to shift the CPU frequency residency to frequencies that are 5 to 25% higher.

We already checked the rate_limit_us value and it is the same for both the before/after cases and it's set to 7.5 ms (jiffies is 4ms in our case). Also, based on my limited understanding the DELAYED_DEQUEUE stuff is only relevant if there are multiple contending threads in a CPU. In this case it's just 1 continuously running thread with a kworker that runs sporadically less than 1% of the time.

So, without a deeper understanding of this patch series, it's behaving as if the PELT signal is accumulating faster than expected. Which is a bit surprising to me because AFAIK (which is not much) the EEVDF series isn't supposed to change the PELT behavior.

If you want to get a visual idea of what the system is doing, here are some perfetto links that visualize the traces. Hopefully you have access permissions to these. You can use the W, S, A, D keys to pan and zoom around the timeline.

Big Before:
https://ui.perfetto.dev/#!/?s=01aa3ad3a5ddd78f2948c86db4265ce2249da8aa
Big After:
https://ui.perfetto.dev/#!/?s=7729ee012f238e96cfa026459eac3f8c3e88d9a9

Thanks,
Saravana, Sam and David

#238 lore raw

(+ Mike, Luis)

Hello Saravana, Sam, David,

Quoted 29 lines · Saravana Kannan
Quoted 16 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 9 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID

Hi Peter,

Quoted 7 lines

TL;DR:
We run some basic sched/cpufreq behavior tests on a Pixel 6 for every change we accept. Some of these changes are merges from Linus's tree. We can see a very clear change in behavior with this patch series. Based on what we are seeing, we'd expect this change in behavior to cause pretty serious power regression (7-25%) depending on what the actual bug is and the use case.

Do the regressions persist with NO_DELAY_DEQUEUE? You can disable the DELAY_DEQUEUE feature added in EEVDF Complete via debugfs by doing a:

    # echo NO_DELAY_DEQUEUE > /sys/kernel/debug/sched/features

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

There is also the fact that delayed entities don't update the tg loadavg since the delayed path calls update_load_avg() without UPDATE_TG flag set which can again cause some changes in PELT calculation since the averages are used to estimate the entity shares when running with cgroups.

Quoted 80 lines

Intro:
We run these tests 20 times for every build (a bunch of changes). All the data below is from the 20+ builds before this series and 20 builds after this series (inclusive). So, all the "before numbers" are from (20 x 20) 400+ runs and all the after numbers are from another 400+ runs.

Test:
We create a synthetic "tiny" thread that runs for 3ms and sleeps for 10ms at Fmin. We let it run like this for several seconds to make sure the util is low and all the "new thread" boost stuff isn't kicking in. So, at this point, the CPU frequency is at Fmin. Then we let this thread run continuously without sleeping and measure (using ftrace) the time it takes for the CPU to get to Fmax.

We do this separately (fresh run) on the Pixel 6 with the cpu affinity set to each cluster and once without any cpu affinity (thread starts at little).

Data:
All the values below are in milliseconds.

When the thread is not affined to any CPU: So thread starts on little, ramps up to fmax, migrates to middle, ramps up to fmax, migrates to big, ramps up to fmax.

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 169    | 151   |
|-----------------------+----------|
| Median          | 221    | 177   |
|-----------------------+----------|
| Mean            | 221    | 177   |
|-----------------------+----------|
| 95th percentile | 249    | 200   |
+----------------------------------+

When thread is affined to the little cluster:
The average time to reach Fmax is 104 ms without this series and 66 ms after this series. We didn't collect the individual per run data. We can if you really need it. We also noticed that the little cluster wouldn't go to Fmin (300 MHz) after this series even when the CPUs are mostly idle. It was instead hovering at 738 MHz (the Fmax is ~1800 MHz).

When thread is affined to the middle cluster:

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 99     | 84    |
|-----------------------+----------|
| Median          | 111    | 104   |
|-----------------------+----------|
| Mean            | 111    | 104   |
|-----------------------+----------|
| 95th percentile | 120    | 119   |
+----------------------------------+

When thread is affined to the big cluster:

+----------------------------------+
| Data            | Before | After |
|-----------------------+----------|
| 5th percentile  | 138    | 96    |
|-----------------------+----------|
| Median          | 147    | 144   |
|-----------------------+----------|
| Mean            | 145    | 134   |
|-----------------------+----------|
| 95th percentile | 151    | 150   |
+----------------------------------+

As you can see, the ramp up time has decreased noticeably. Also, as you can tell from the 5th percentile numbers, the standard deviation has also increased a lot too, causing a wider spread of the ramp up time (more noticeable in the middle and big clusters). So in general this looks like it's going to increase the usage of the middle and big CPU clusters and also going to shift the CPU frequency residency to frequencies that are 5 to 25% higher.

Quoted 6 lines

We already checked the rate_limit_us value and it is the same for both the before/after cases and it's set to 7.5 ms (jiffies is 4ms in our case). Also, based on my limited understanding the DELAYED_DEQUEUE stuff is only relevant if there are multiple contending threads in a CPU. In this case it's just 1 continuously running thread with a kworker that runs sporadically less than 1% of the time.

There is an ongoing investigation on delayed entities possibly not migrating if they are woken up before they are fully dequeued. Since you mention there is only one task, this should not matter but could you also try out Mike's suggestion from
https://lore.kernel.org/lkml/[email protected]/
and see if it makes a difference on your test suite?

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 11 lines

So, without a deeper understanding of this patch series, it's behaving as if the PELT signal is accumulating faster than expected. Which is a bit surprising to me because AFAIK (which is not much) the EEVDF series isn't supposed to change the PELT behavior.

If you want to get a visual idea of what the system is doing, here are some perfetto links that visualize the traces. Hopefully you have access permissions to these. You can use the W, S, A, D keys to pan and zoom around the timeline.

Quoted 4 lines

Big Before:
https://ui.perfetto.dev/#!/?s=01aa3ad3a5ddd78f2948c86db4265ce2249da8aa
Big After:
https://ui.perfetto.dev/#!/?s=7729ee012f238e96cfa026459eac3f8c3e88d9a9

P.S. I only gave a quick glance but I do see the frequency ramp up with larger deltas and reach Fmax much quickly in case of "Big After"

Quoted 3 lines

Thanks,
Saravana, Sam and David

author maintainer
#239 lore raw
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

#240 lore raw

Hi,

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Quoted 5 lines

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

Sorry for the late reply, I got sidetracked on something else.

There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.

The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]

The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.

So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.

Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]

Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.

If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

#241 lore raw
Quoted 40 lines · Luis Machado

Hi,

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Quoted 5 lines

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

Sorry for the late reply, I got sidetracked on something else.

There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.

The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]

The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.

So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.

Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]

Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.

Quoted 8 lines

If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Thanks for the replies. We are trying to disable DELAY_DEQUEUE and recollect the data to see if that's the cause. We'll get back to this thread once we have some data.

-Saravana

#242 lore raw
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado

Hi,

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Quoted 5 lines

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

Sorry for the late reply, I got sidetracked on something else.

There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.

The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]

The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.

So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.

Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]

Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.

Quoted 8 lines

If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Quoted 5 lines

Thanks for the replies. We are trying to disable DELAY_DEQUEUE and recollect the data to see if that's the cause. We'll get back to this thread once we have some data.

-Saravana

The test data is back to pre-EEVDF state with DELAY_DEQUEUE disabled.

Same test example from before, when thread is affined to the big cluster:

+----------------------------------+
| Data            | Enabled | Disabled |
|-----------------------+----------|
| 5th percentile  | 96     | 143    |
|-----------------------+----------|
| Median          | 144    | 147   |
|-----------------------+----------|
| Mean            | 134    | 147   |
|-----------------------+----------|
| 95th percentile | 150    | 150   |
+----------------------------------+

What are the next steps to bring this behavior back? Will DELAY_DEQUEUE always be enabled by default and/or is there a fix coming for 6.12?

Thanks,
Sam

#243 lore raw

Hello Sam,

Quoted 73 lines · Samuel Wu
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado

Hi,

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Quoted 5 lines

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

Sorry for the late reply, I got sidetracked on something else.

There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.

The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]

The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.

So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.

Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]

Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.

Quoted 8 lines

If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Quoted 5 lines

Thanks for the replies. We are trying to disable DELAY_DEQUEUE and recollect the data to see if that's the cause. We'll get back to this thread once we have some data.

-Saravana

The test data is back to pre-EEVDF state with DELAY_DEQUEUE disabled.

Same test example from before, when thread is affined to the big cluster:

+----------------------------------+
| Data            | Enabled | Disabled |
|-----------------------+----------|
| 5th percentile  | 96     | 143    |
|-----------------------+----------|
| Median          | 144    | 147   |
|-----------------------+----------|
| Mean            | 134    | 147   |
|-----------------------+----------|
| 95th percentile | 150    | 150   |
+----------------------------------+
Quoted 2 lines

What are the next steps to bring this behavior back? Will DELAY_DEQUEUE always be enabled by default and/or is there a fix coming for 6.12?

DELAY_DEQUEUE should be enabled by default from v6.12 but there are a few fixes for the same in-flight. Could try running with the changes from [1] and [2] and see if you could reproduce the behavior and if you can, is it equally bad?

Both changes apply cleanly for me on top of current

git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core

at commit fe9beaaa802d ("sched: No PREEMPT_RT=y for all{yes,mod}config") when applied in that order.

[1] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[2] https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines

Thanks,
Sam

Signature · 2 lines
Thanks and Regards,
Prateek
#244 lore raw
Quoted 87 lines · K Prateek Nayak

Hello Sam,

Quoted 73 lines · Samuel Wu
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado

Hi,

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak

Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.

Quoted 5 lines

Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.

But I might be misremembering things. I'll get to it eventually :/

Sorry for the late reply, I got sidetracked on something else.

There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.

The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]

The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.

So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.

Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]

Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.

Quoted 8 lines

If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Quoted 5 lines

Thanks for the replies. We are trying to disable DELAY_DEQUEUE and recollect the data to see if that's the cause. We'll get back to this thread once we have some data.

-Saravana

The test data is back to pre-EEVDF state with DELAY_DEQUEUE disabled.

Same test example from before, when thread is affined to the big cluster:

+----------------------------------+
| Data            | Enabled | Disabled |
|-----------------------+----------|
| 5th percentile  | 96     | 143    |
|-----------------------+----------|
| Median          | 144    | 147   |
|-----------------------+----------|
| Mean            | 134    | 147   |
|-----------------------+----------|
| 95th percentile | 150    | 150   |
+----------------------------------+
Quoted 2 lines

What are the next steps to bring this behavior back? Will DELAY_DEQUEUE always be enabled by default and/or is there a fix coming for 6.12?

DELAY_DEQUEUE should be enabled by default from v6.12 but there are a few fixes for the same in-flight. Could try running with the changes from [1] and [2] and see if you could reproduce the behavior and if you can, is it equally bad?

Both changes apply cleanly for me on top of current

Quoted 7 lines

git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core

at commit fe9beaaa802d ("sched: No PREEMPT_RT=y for all{yes,mod}config") when applied in that order.

[1] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[2] https://lore.kernel.org/lkml/[email protected]/

Have these changes landed in 6.12? Or will these in 6.13?

We tested 6.12 and the issue we reported is still present. What should we do for any products we want to ship on 6.12? Disable Delayed Dequeue or backport any fixes to 6.12 LTS?

Peter/Vincent, do you plan on backporting the future fixes to the 6.12 LTS kernel? Anything else we can do to help with making sure this is fixed on the LTS kernel?

-Saravana

#245 lore raw
In reply to Peter Zijlstra

Hi all,

Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

 - split up the huge delay-dequeue patch
 - tested/fixed cfs-bandwidth
 - PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
 - SCHED_BATCH is equivalent to RESPECT_SLICE
 - propagate min_slice up cgroups
 - CLOCK_THREAD_DVFS_ID

We found the following 7 commits from this patch set to crash in enqueue_dl_entity():

54a58a787791 sched/fair: Implement DELAY_ZERO
152e11f6df29 sched/fair: Implement delayed dequeue e1459a50ba31 sched: Teach dequeue_task() about special task states a1c446611e31 sched,freezer: Mark TASK_FROZEN special 781773e3b680 sched/fair: Implement ENQUEUE_DELAYED f12e148892ed sched/fair: Prepare pick_next_task() for delayed dequeue 2e0199df252a sched/fair: Prepare exit/cleanup paths for delayed_dequeue

Resulting in the following crash dump (this is running v6.12.1):

[   14.652856] sched: DL replenish lagged too much
[   16.572706] ------------[ cut here ]------------
[   16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c
[   16.573900] Modules linked in: overlay crct10dif_ce rk805_pwrkey snd_soc_es8316 pwm_fan

phy_rockchip_naneng_combphy rockchip_saradc rtc_hym8563 industrialio_trigg
ered_buffer kfifo_buf rockchip_thermal phy_rockchip_usbdp typec spi_rockchip_sfc snd_soc_rockchip_i2s_tdm hantro_vpu v4l2_vp9 v4l2_h264 v4l2_jpeg panthor v4l2_mem2me
m rockchipdrm drm_gpuvm drm_exec drm_shmem_helper analogix_dp gpu_sched dw_mipi_dsi dw_hdmi cec drm_display_helper snd_soc_audio_graph_card snd_soc_simple_card_utils
drm_dma_helper drm_kms_helper cfg80211 rfkill pci_endpoint_test drm backlight dm_mod dax

[   16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15
[   16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024
[   16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   16.580273] pc : enqueue_dl_entity+0x46c/0x55c
[   16.580661] lr : dl_server_start+0x44/0x12c
[   16.581028] sp : ffff80008002bc00
[   16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000
[   16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009
[   16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0
[   16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000
[   16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698
[   16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000
[   16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009
[   16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440
[   16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0
[   16.587540] Call trace:
[   16.587754]  enqueue_dl_entity+0x46c/0x55c
[   16.588113]  dl_server_start+0x44/0x12c
[   16.588449]  enqueue_task_fair+0x124/0x49c
[   16.588807]  enqueue_task+0x3c/0xe0
[   16.589113]  ttwu_do_activate.isra.0+0x6c/0x208
[   16.589511]  try_to_wake_up+0x1d0/0x61c
[   16.589847]  wake_up_process+0x18/0x24
[   16.590175]  kick_pool+0x84/0x150
[   16.590467]  __queue_work+0x2f4/0x544
[   16.590788]  delayed_work_timer_fn+0x1c/0x28
[   16.591161]  call_timer_fn+0x34/0x1ac
[   16.591481]  __run_timer_base+0x20c/0x314
[   16.591832]  run_timer_softirq+0x3c/0x78
[   16.592176]  handle_softirqs+0x124/0x35c
[   16.592520]  __do_softirq+0x14/0x20
[   16.592827]  ____do_softirq+0x10/0x1c
[   16.593148]  call_on_irq_stack+0x24/0x4c
[   16.593490]  do_softirq_own_stack+0x1c/0x2c
[   16.593857]  irq_exit_rcu+0x8c/0xc0
[   16.594163]  el0_interrupt+0x48/0xbc
[   16.594477]  __el0_irq_handler_common+0x18/0x24
[   16.594874]  el0t_64_irq_handler+0x10/0x1c
[   16.595232]  el0t_64_irq+0x190/0x194
[   16.595545] ---[ end trace 0000000000000000 ]---
[   16.595950] ------------[ cut here ]------------

It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].

We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.

We haven't had time to do a non-proprietary reproduction case as of yet but I wanted to report our current findings asking for any feedback/suggestions from the community.

Thanks!

Cheers

Marcel

[1] https://hastebin.skyra.pw/hoqesigaye.yaml

author maintainer
#246 lore raw
Quoted 49 lines · Marcel Ziswiler

Resulting in the following crash dump (this is running v6.12.1):

[   14.652856] sched: DL replenish lagged too much
[   16.572706] ------------[ cut here ]------------
[   16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c
[   16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15
[   16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024
[   16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   16.580273] pc : enqueue_dl_entity+0x46c/0x55c
[   16.580661] lr : dl_server_start+0x44/0x12c
[   16.581028] sp : ffff80008002bc00
[   16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000
[   16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009
[   16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0
[   16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000
[   16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698
[   16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000
[   16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009
[   16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440
[   16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0
[   16.587540] Call trace:
[   16.587754]  enqueue_dl_entity+0x46c/0x55c
[   16.588113]  dl_server_start+0x44/0x12c
[   16.588449]  enqueue_task_fair+0x124/0x49c
[   16.588807]  enqueue_task+0x3c/0xe0
[   16.589113]  ttwu_do_activate.isra.0+0x6c/0x208
[   16.589511]  try_to_wake_up+0x1d0/0x61c
[   16.589847]  wake_up_process+0x18/0x24
[   16.590175]  kick_pool+0x84/0x150
[   16.590467]  __queue_work+0x2f4/0x544
[   16.590788]  delayed_work_timer_fn+0x1c/0x28
[   16.591161]  call_timer_fn+0x34/0x1ac
[   16.591481]  __run_timer_base+0x20c/0x314
[   16.591832]  run_timer_softirq+0x3c/0x78
[   16.592176]  handle_softirqs+0x124/0x35c
[   16.592520]  __do_softirq+0x14/0x20
[   16.592827]  ____do_softirq+0x10/0x1c
[   16.593148]  call_on_irq_stack+0x24/0x4c
[   16.593490]  do_softirq_own_stack+0x1c/0x2c
[   16.593857]  irq_exit_rcu+0x8c/0xc0
[   16.594163]  el0_interrupt+0x48/0xbc
[   16.594477]  __el0_irq_handler_common+0x18/0x24
[   16.594874]  el0t_64_irq_handler+0x10/0x1c
[   16.595232]  el0t_64_irq+0x190/0x194
[   16.595545] ---[ end trace 0000000000000000 ]---
[   16.595950] ------------[ cut here ]------------
Quoted 1 line

It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].

Right, I've had a number of these reports, but so far we've not yet managed to figure out how it's all happening.

Quoted 3 lines

We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.

Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?

A very long shot:

https://lkml.kernel.org/r/[email protected]

doesn't help, does it?

#247 lore raw
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Marcel Ziswiler

Resulting in the following crash dump (this is running v6.12.1):

[snip]

Quoted 6 lines
Quoted 2 lines

It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].

Right, I've had a number of these reports, but so far we've not yet managed to figure out how it's all happening.

Quoted 7 lines
Quoted 3 lines

We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms.  While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.

Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?

As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.

Quoted 5 lines

A very long shot:

 https://lkml.kernel.org/r/[email protected]

doesn't help, does it?

No, still the same with this on top of today's -next.

Thanks!

Cheers

Marcel

author maintainer
#248 lore raw
Quoted 6 lines · Marcel Ziswiler
Quoted 3 lines

Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?

As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.

I was hoping you could perhaps share a binary with Juri privately or with RHT (same difference etc), such that he can poke at it too.

Anyway, if you don't mind a bit of back and forth, would you mind adding the below patch to your kernel and doing:

(all assuming your kernel has ftrace enabled)

echo 1 > /sys/kernel/debug/tracing/options/stacktrace echo 1 > /proc/sys/kernel/traceoff_on_warning

running your test to failure and then dumping the trace into a file like:

cat /sys/kernel/debug/tracing/trace > ~/trace

Then compress the file (bzip2 or whatever is popular these days) and send it my way along with a dmesg dump (private is fine -- these things tend to be large-ish).

Hopefully, this will give us a little clue as to where the double enqueue happens.

---

kernel/sched/deadline.c +20 -0
index d9d5a702f1a6..b9cd9b40a19f 100644
@@ -1203,6 +1203,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 	scoped_guard (rq_lock, rq) {
 		struct rq_flags *rf = &scope.rf;
 
+		if (dl_se == &rq->fair_server) {
+			trace_printk("timer fair server %d throttled %d\n",
+				     cpu_of(rq), dl_se->dl_throttled);
+		}
+
 		if (!dl_se->dl_throttled || !dl_se->dl_runtime)
 			return HRTIMER_NORESTART;
 
@@ -1772,6 +1777,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		rq_lock(rq, &rf);
 	}
 
+	if (dl_se == &rq->fair_server)
+		trace_printk("inactive fair server %d\n", cpu_of(rq));
+
 	sched_clock_tick();
 	update_rq_clock(rq);
 
@@ -1967,6 +1975,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,
 static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
 {
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("enqueue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}
 
 	WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));
 
@@ -1978,6 +1992,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
 static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
 {
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("dequeue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}
 
 	if (RB_EMPTY_NODE(&dl_se->rb_node))
 		return;
#249 lore raw

Sorry for my late reply, I was traveling back from Manchester to Switzerland but I am all settled down again.

Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Marcel Ziswiler
Quoted 3 lines

Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?

As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.

I was hoping you could perhaps share a binary with Juri privately or with RHT (same difference etc), such that he can poke at it too.

Sure, there is nothing secret about it, it is just that we have not gotten around open sourcing all parts of it just yet.

The UEFI aarch64 embedded Linux image I am using may be found here [1]. Plus matching bmap file should you fancy using that [2]. And the SSH key may help when interacting with the system (e.g. that is how I trigger the failure as the console is quite busy with tracing) [3]. However, that was built by CI and does not contain a kernel with below patch applied yet. I manually dumped the kernel config and compiled v6.12.1 with your patch applied and deployed it (to /lib/modules, /usr/lib/kernel et. al.) in the below case where I provide the dump.

Quoted 1 line

Anyway, if you don't mind a bit of back and forth,

Sure.

Quoted 5 lines

would you mind adding
the below patch to your kernel and doing:

(all assuming your kernel has ftrace enabled)

Quoted 7 lines

  echo 1 > /sys/kernel/debug/tracing/options/stacktrace echo 1 > /proc/sys/kernel/traceoff_on_warning

running your test to failure and then dumping the trace into a file like:

  cat /sys/kernel/debug/tracing/trace > ~/trace

Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:

tail -f /sys/kernel/debug/tracing/trace

Not sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.

Quoted 1 line

Then compress the file (bzip2 or whatever is popular these days)

xz or zstd (;-p)

Quoted 3 lines

and
send it my way along with a dmesg dump (private is fine -- these things tend to be large-ish).

As mentioned before, there is nothing secret about it. Please find it here [4].

Quoted 2 lines

Hopefully, this will give us a little clue as to where the double enqueue happens.

Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!

Quoted 50 lines

---

kernel/sched/deadline.c +0 -0
index d9d5a702f1a6..b9cd9b40a19f 100644
@@ -1203,6 +1203,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
   scoped_guard (rq_lock, rq) {
   struct rq_flags *rf = &scope.rf;


+ if (dl_se == &rq->fair_server) {
+ trace_printk("timer fair server %d throttled %d\n",
+      cpu_of(rq), dl_se->dl_throttled);
+ }
+
  if (!dl_se->dl_throttled || !dl_se->dl_runtime) return HRTIMER_NORESTART;

kernel/sched/deadline.c +0 -0
@@ -1772,6 +1777,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
   rq_lock(rq, &rf);
   }


+ if (dl_se == &rq->fair_server)
+ trace_printk("inactive fair server %d\n", cpu_of(rq)); +
  sched_clock_tick();
  update_rq_clock(rq);

kernel/sched/deadline.c +6 -0
@@ -1967,6 +1975,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,
  static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
  {
   struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+ struct rq *rq = rq_of_dl_se(dl_se);
+
+ if (dl_se == &rq->fair_server) {
+ trace_printk("enqueue fair server %d h_nr_running %d\n",
+      cpu_of(rq), rq->cfs.h_nr_running);
+ }


  WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));

kernel/sched/deadline.c +6 -0
@@ -1978,6 +1992,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
  static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
  {
   struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+ struct rq *rq = rq_of_dl_se(dl_se);
+
+ if (dl_se == &rq->fair_server) {
+ trace_printk("dequeue fair server %d h_nr_running %d\n",
+      cpu_of(rq), rq->cfs.h_nr_running);
+ }
Quoted 3 lines


  if (RB_EMPTY_NODE(&dl_se->rb_node))
  return;

[1] https://drive.codethink.co.uk/s/N8CQipaNNN45gYM

[2] https://drive.codethink.co.uk/s/mpcPawXpCjPL8D3

[3] https://drive.codethink.co.uk/s/8RjHNTQQRpYgaLc

[4] https://drive.codethink.co.uk/s/MWtzWjLDtdD3E5i

Cheers

Marcel

author maintainer
#250 lore raw

Sorry for the delay, I got laid low by snot monsters :/

Quoted 7 lines · Marcel Ziswiler

Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:

tail -f /sys/kernel/debug/tracing/trace

Not sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.

Ah, that is unfortunate. There is a ftrace_dump_on_oops option that might be of help. And yes, dumping trace buffers over 1m5 serial lines is tedious -- been there done that, got a t-shirt and all that.

Still, let me see if perhaps making that WARN in enqueue_dl_entity() return makes the whole thing less fatal.

I've included the traceoff_on_warning and ftrace_dump in the code, so all you really need to still do is enable the stacktrace option.

  echo 1 > /sys/kernel/debug/tracing/options/stacktrace

Quoted 1 line

Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!

Could I bother you to try again with the below patch?

There are two new hunks vs the previous one, the hunk in enqueue_dl_entity() (the very last bit) will stop tracing and dump the buffers when that condition is hit in addition to then aborting the double enqueue, hopefully leaving the system is a slightly better state.

The other new hunk is the one for dl_server_stop() (second hunk), while going over the code last week, I found that this might be a possible hole leading to the observed double enqueue, so fingers crossed.

---

kernel/sched/deadline.c +31 -1
index 33b4646f8b24..bd1df7612482 100644
@@ -1223,6 +1223,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 	scoped_guard (rq_lock, rq) {
 		struct rq_flags *rf = &scope.rf;
 
+		if (dl_se == &rq->fair_server) {
+			trace_printk("timer fair server %d throttled %d\n",
+				     cpu_of(rq), dl_se->dl_throttled);
+		}
+
 		if (!dl_se->dl_throttled || !dl_se->dl_runtime)
 			return HRTIMER_NORESTART;
 
@@ -1674,6 +1679,12 @@ void dl_server_start(struct sched_dl_entity *dl_se)
 
 void dl_server_stop(struct sched_dl_entity *dl_se)
 {
+	if (current->dl_server == dl_se) {
+		struct rq *rq = rq_of_dl_se(dl_se);
+		trace_printk("stop fair server %d\n", cpu_of(rq));
+		current->dl_server = NULL;
+	}
+
 	if (!dl_se->dl_runtime)
 		return;
 
@@ -1792,6 +1803,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		rq_lock(rq, &rf);
 	}
 
+	if (dl_se == &rq->fair_server)
+		trace_printk("inactive fair server %d\n", cpu_of(rq));
+
 	sched_clock_tick();
 	update_rq_clock(rq);
 
@@ -1987,6 +2001,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,
 static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
 {
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("enqueue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}
 
 	WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));
 
@@ -1998,6 +2018,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
 static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
 {
 	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("dequeue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}
 
 	if (RB_EMPTY_NODE(&dl_se->rb_node))
 		return;
@@ -2012,7 +2038,11 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
 static void
 enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
 {
-	WARN_ON_ONCE(on_dl_rq(dl_se));
+	if (WARN_ON_ONCE(on_dl_rq(dl_se))) {
+		tracing_off();
+		ftrace_dump(DUMP_ALL);
+		return;
+	}
 
 	update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
 
#251 lore raw
Quoted 21 lines · Peter Zijlstra

Sorry for the delay, I got laid low by snot monsters :/

Quoted 8 lines · Marcel Ziswiler

Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace
buffer any longer. So I did the following instead on the serial console terminal:

tail -f /sys/kernel/debug/tracing/trace

Not sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.

Ah, that is unfortunate. There is a ftrace_dump_on_oops option that might be of help. And yes, dumping trace buffers over 1m5 serial lines is tedious -- been there done that, got a t-shirt and all that.

Still, let me see if perhaps making that WARN in enqueue_dl_entity() return makes the whole thing less fatal.

Quoted 8 lines

I've included the traceoff_on_warning and ftrace_dump in the code, so all you really need to still do is enable the stacktrace option.

   echo 1 > /sys/kernel/debug/tracing/options/stacktrace

Quoted 1 line

Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!

Could I bother you to try again with the below patch?

Sure, here you go.

https://drive.codethink.co.uk/s/HniZCtccDBMHpAK

Quoted 88 lines

There are two new hunks vs the previous one, the hunk in enqueue_dl_entity() (the very last bit) will stop tracing and dump the buffers when that condition is hit in addition to then aborting the double enqueue, hopefully leaving the system is a slightly better state.

The other new hunk is the one for dl_server_stop() (second hunk), while going over the code last week, I found that this might be a possible hole leading to the observed double enqueue, so fingers crossed.

---

kernel/sched/deadline.c +0 -0
index 33b4646f8b24..bd1df7612482 100644
@@ -1223,6 +1223,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
  	scoped_guard (rq_lock, rq) {
  		struct rq_flags *rf = &scope.rf;

kernel/sched/deadline.c +5 -0
+		if (dl_se == &rq->fair_server) {
+			trace_printk("timer fair server %d throttled %d\n",
+				     cpu_of(rq), dl_se->dl_throttled);
+		}
+

  if (!dl_se->dl_throttled || !dl_se->dl_runtime) return HRTIMER_NORESTART;

kernel/sched/deadline.c +0 -0
@@ -1674,6 +1679,12 @@ void dl_server_start(struct sched_dl_entity *dl_se)


 void dl_server_stop(struct sched_dl_entity *dl_se) {

kernel/sched/deadline.c +6 -0
+	if (current->dl_server == dl_se) {
+		struct rq *rq = rq_of_dl_se(dl_se);
+		trace_printk("stop fair server %d\n", cpu_of(rq));
+		current->dl_server = NULL;
+	}
+

  if (!dl_se->dl_runtime)
  return;

kernel/sched/deadline.c +0 -0
@@ -1792,6 +1803,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
  		rq_lock(rq, &rf);
  	}

kernel/sched/deadline.c +3 -0
+	if (dl_se == &rq->fair_server)
+		trace_printk("inactive fair server %d\n", cpu_of(rq));
+

  sched_clock_tick();
  update_rq_clock(rq);

kernel/sched/deadline.c +6 -0
@@ -1987,6 +2001,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,
  static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
  {
  	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("enqueue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}

WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));

kernel/sched/deadline.c +6 -0
@@ -1998,6 +2018,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)
  static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)
  {
  	struct dl_rq *dl_rq = dl_rq_of_se(dl_se);
+	struct rq *rq = rq_of_dl_se(dl_se);
+
+	if (dl_se == &rq->fair_server) {
+		trace_printk("dequeue fair server %d h_nr_running %d\n",
+			     cpu_of(rq), rq->cfs.h_nr_running);
+	}


  if (RB_EMPTY_NODE(&dl_se->rb_node))
  return;

kernel/sched/deadline.c +0 -0
@@ -2012,7 +2038,11 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)

 static void
 enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags) {

kernel/sched/deadline.c +5 -1
-	WARN_ON_ONCE(on_dl_rq(dl_se));
+	if (WARN_ON_ONCE(on_dl_rq(dl_se))) {
+		tracing_off();
+		ftrace_dump(DUMP_ALL);
+		return;
+	}


  update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);

maintainer
#252 lore raw
In reply to Marcel Ziswiler
Quoted 7 lines · On Mon, 02 Dec 2024 19:46:21 +0100 Marcel Ziswiler…

Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:

tail -f /sys/kernel/debug/tracing/trace

Not sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.

If the memory of the machine is persistent (it is on several of my machines) you can use the persistent ring buffer.

Add to the kernel command line (or enable bootconfig that attaches a command line to the kernel if you can't change the parameters):

reserve_mem=20M:12M:trace trace_instance=boot_map^traceoff^traceprintk@trace

The above will create a "boot_map" instance with tracing off on boot and trace_printk() going to it. Start tracing:

trace-cmd start -B boot_map -p nop

If or replace "-p nop" with any events or tracers you want, including function tracing", then after a crash.

trace-cmd show -B boot_map

If the memory is persistent and you don't use KASLR (may want to also add nokaslr if arm64 supports KASLR it and you use it), you should get the trace right up to the crash.

See Documentation/trace/debugging.rst for more details.

-- Steve

#253 lore raw
In reply to Marcel Ziswiler
Quoted 83 lines · Marcel Ziswiler

Hi all,

Quoted 15 lines · Peter Zijlstra

Hi all,

So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.

I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.

These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.

I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.

Quoted 8 lines

Aside from a ton of bug fixes -- thanks all! -- new in this version is:

 - split up the huge delay-dequeue patch
 - tested/fixed cfs-bandwidth
 - PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
 - SCHED_BATCH is equivalent to RESPECT_SLICE
 - propagate min_slice up cgroups
 - CLOCK_THREAD_DVFS_ID

We found the following 7 commits from this patch set to crash in enqueue_dl_entity():

54a58a787791 sched/fair: Implement DELAY_ZERO
152e11f6df29 sched/fair: Implement delayed dequeue e1459a50ba31 sched: Teach dequeue_task() about special task states a1c446611e31 sched,freezer: Mark TASK_FROZEN special 781773e3b680 sched/fair: Implement ENQUEUE_DELAYED f12e148892ed sched/fair: Prepare pick_next_task() for delayed dequeue 2e0199df252a sched/fair: Prepare exit/cleanup paths for delayed_dequeue

Resulting in the following crash dump (this is running v6.12.1):

[   14.652856] sched: DL replenish lagged too much
[   16.572706] ------------[ cut here ]------------
[   16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c
[   16.573900] Modules linked in: overlay crct10dif_ce rk805_pwrkey snd_soc_es8316 pwm_fan

phy_rockchip_naneng_combphy rockchip_saradc rtc_hym8563 industrialio_trigg
ered_buffer kfifo_buf rockchip_thermal phy_rockchip_usbdp typec spi_rockchip_sfc snd_soc_rockchip_i2s_tdm hantro_vpu v4l2_vp9 v4l2_h264 v4l2_jpeg panthor v4l2_mem2me
m rockchipdrm drm_gpuvm drm_exec drm_shmem_helper analogix_dp gpu_sched dw_mipi_dsi dw_hdmi cec drm_display_helper snd_soc_audio_graph_card snd_soc_simple_card_utils
drm_dma_helper drm_kms_helper cfg80211 rfkill pci_endpoint_test drm backlight dm_mod dax

[   16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15
[   16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024
[   16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   16.580273] pc : enqueue_dl_entity+0x46c/0x55c
[   16.580661] lr : dl_server_start+0x44/0x12c
[   16.581028] sp : ffff80008002bc00
[   16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000
[   16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009
[   16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0
[   16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000
[   16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698
[   16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000
[   16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009
[   16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440
[   16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[   16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0
[   16.587540] Call trace:
[   16.587754]  enqueue_dl_entity+0x46c/0x55c
[   16.588113]  dl_server_start+0x44/0x12c
[   16.588449]  enqueue_task_fair+0x124/0x49c
[   16.588807]  enqueue_task+0x3c/0xe0
[   16.589113]  ttwu_do_activate.isra.0+0x6c/0x208
[   16.589511]  try_to_wake_up+0x1d0/0x61c
[   16.589847]  wake_up_process+0x18/0x24
[   16.590175]  kick_pool+0x84/0x150
[   16.590467]  __queue_work+0x2f4/0x544
[   16.590788]  delayed_work_timer_fn+0x1c/0x28
[   16.591161]  call_timer_fn+0x34/0x1ac
[   16.591481]  __run_timer_base+0x20c/0x314
[   16.591832]  run_timer_softirq+0x3c/0x78
[   16.592176]  handle_softirqs+0x124/0x35c
[   16.592520]  __do_softirq+0x14/0x20
[   16.592827]  ____do_softirq+0x10/0x1c
[   16.593148]  call_on_irq_stack+0x24/0x4c
Quoted 8 lines
[   16.593490]  do_softirq_own_stack+0x1c/0x2c
[   16.593857]  irq_exit_rcu+0x8c/0xc0
[   16.594163]  el0_interrupt+0x48/0xbc
[   16.594477]  __el0_irq_handler_common+0x18/0x24
[   16.594874]  el0t_64_irq_handler+0x10/0x1c
[   16.595232]  el0t_64_irq+0x190/0x194
[   16.595545] ---[ end trace 0000000000000000 ]---
[   16.595950] ------------[ cut here ]------------

Random piece of data, but I also had some difficulty making things boot on Android when trying Vincent's nr_running accounting series due to a very similar crash/stack trace. Though what I saw went a bit further and actually crashed within task_contending, called from enqueue_dl_entity. Possibly crashed in one of the inlined functions.

Even though the kernel was 6.8 and it was backports, it seems awfully similar to the above.

Quoted 17 lines

It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].

We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.

We haven't had time to do a non-proprietary reproduction case as of yet but I wanted to report our current findings asking for any feedback/suggestions from the community.

Thanks!

Cheers

Marcel

[1] https://hastebin.skyra.pw/hoqesigaye.yaml