Complete EEVDF
Subsystems (11)
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Mike GalbraithQuoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.
The below improved uptime, and trace_printk() says it's doing the intended, so I suppose I'll add a feature and see what falls out.
--- kernel/sched/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/core.c +2 -2
@@ -3794,7 +3794,7 @@ static int ttwu_runnable(struct task_strint queue_flags =DEQUEUE_NOCLOCK ;if (p->se.sched_delayed) - queue_flags |=DEQUEUE_DELAYED ; + queue_flags |= (DEQUEUE_DELAYED |DEQUEUE_SLEEP );/* * Since we're not current anywhere *AND* hold pi_lock, dequeue
@@ -3802,7 +3802,7 @@ static int ttwu_runnable(struct task_str* case further along the ttwu() path. */ if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { -dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +dequeue_task (rq, p, queue_flags);return 0 ; }
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.
From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
-Mike
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
PATCH 1/1 sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.
Quoted 7 lines
Quoted 2 lines
From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Salut, much obliged for eyeball relief.
---snip---
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.
(de-uglified-a-lot-by)
--- kernel/sched/core.c | 48 +++++++++++++++++++++++++++++------------------- kernel/sched/sched.h | 5 +++++ 2 files changed, 34 insertions(+), 19 deletions(-)
kernel/sched/core.c +29 -19
@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tasraw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 14 lines · On Fri, Nov 08, 2024 at 01:24:35AM +0100 Mike Galbraith wrote:
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Quoted 7 lines
Quoted 4 lines
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Salut, much obliged for eyeball relief.
Thanks Mike (and Peter). We have our full perf tests running on Mike's original verion of this patch. Results probably Monday (there's a long queue). We'll see if this blows up anything else then. I'll queue up a build with this cleaned up version as well but the results will be late next week, probably.
At that point maybe some or all of these:
Cheers,
Phil
Quoted 94 lines
---snip---
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.(de-uglified-a-lot-by)
Reported-by Phil Auld <[email protected]>Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>--- kernel/sched/core.c | 48 +++++++++++++++++++++++++++++------------------- kernel/sched/sched.h | 5 +++++ 2 files changed, 34 insertions(+), 19 deletions(-)kernel/sched/core.c +29
-19@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP kernel/sched/sched.h +5
-0@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tasraw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Mike GalbraithHi Mike and Peter,
Quoted 67 lines · Mike Galbraith
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.Quoted 7 lines
Quoted 2 lines
From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Salut, much obliged for eyeball relief.
---snip---
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.(de-uglified-a-lot-by)
Reported-by Phil Auld <[email protected]>Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>--- kernel/sched/core.c | 48 +++++++++++++++++++++++++++++------------------- kernel/sched/sched.h | 5 +++++ 2 files changed, 34 insertions(+), 19 deletions(-)kernel/sched/core.c +6
-15@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; +
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3801
Quoted 10 lines
kernel/sched/core.c +10
-0+update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) {
For sched_asym_cpucapacity system, need we consider the task_fits_cpu_capacity there?
Quoted 40 lines
kernel/sched/core.c +13
-4+dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP kernel/sched/sched.h +5
-0@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tasraw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 11 lines · Xuewen Yan
Quoted 10 lines
+ if (
p->se.sched_delayed) {
+ intqueue_flags=ENQUEUE_DELAYED|ENQUEUE_NOCLOCK; +
+ /*
+ * Sincesched_delayedmeans we cannot be current anywhere, + * dequeue it here and have it fall through to the + *select_task_rq()case further along thettwu()path. + */
+ if (rq->nr_running> 1 &&p->nr_cpus_allowed> 1) {
Quoted 2 lines
For
sched_asym_cpucapacitysystem, need we consider thetask_fits_cpu_capacitythere?
I don't think so. Wakeup placement logic is what we're deflecting the wakee toward, this is not the right spot to add any complexity.
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Mike GalbraithQuoted 14 lines · Mike Galbraith
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Quoted 6 lines
Quoted 4 lines
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Salut, much obliged for eyeball relief.
Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.
(adds knob)
tbench 8
NO_MIGRATE_DELAYED 3613.49 MB/sec MIGRATE_DELAYED 3145.59 MB/sec NO_DELAY_DEQUEUE 3355.42 MB/sec
First line is DELAY_DEQUEUE restoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.
I thought I might be able to do away with the reservation like side effect of DELAY_DEQUEUE by borrowing h_nr_delayed from...
sched/eevdf: More PELT vs DELAYED_DEQUEUE
...for cgroups free test config, but Q/D poke at idle_cpu() helped not at all.
Quoted 94 lines
---snip---
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.(de-uglified-a-lot-by)
Reported-by Phil Auld <[email protected]>Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>---
kernel/sched/core.c | 48 +++++++++++++++++++++++++++++-------------------kernel/sched/sched.h| 5 +++++2 files changed, 34 insertions(+), 19 deletions(-)kernel/sched/core.c +28
-18@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable(struct task_struct *p, int wake_flags) { - struct rq_flags rf; - struct rq *rq; - int ret = 0; - - rq = __task_rq_lock(p, &rf); - if (task_on_rq_queued(p)) { - update_rq_clock(rq); - if (p->se.sched_delayed) - enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); - if (!task_on_cpu(rq, p)) { - /* - * When on_rq && !on_cpu the task is preempted, see if - * it should preempt the task that is current now. - */ - wakeup_preempt(rq, p, wake_flags); + CLASS(__task_rq_lock, rq_guard)(p); + struct rq *rq = rq_guard.rq; + + if (!task_on_rq_queued(p)) + return 0; + + update_rq_clock(rq); + if (p->se.sched_delayed) { + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; + + /* + * Since sched_delayed means we cannot be current anywhere, + * dequeue it here and have it fall through to the + * select_task_rq() case further along the ttwu() path. + */ + if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) { + dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags); + return 0; } - ttwu_do_wakeup(p); - ret = 1; + + enqueue_task(rq, p, queue_flags); + } + if (!task_on_cpu(rq, p)) { + /* + * When on_rq && !on_cpu the task is preempted, see if + * it should preempt the task that is current now. + */ + wakeup_preempt(rq, p, wake_flags); } - __task_rq_unlock(rq, &rf); + ttwu_do_wakeup(p);
- return ret;
+ return 1;
}
#ifdefCONFIG_SMPkernel/sched/sched.h +0
-0@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags); }kernel/sched/sched.h +5
-0+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq = __task_rq_lock(_T->lock, &_T->rf), + __task_rq_unlock(_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1(task_rq_lock, struct
task_struct,_T->rq=task_rq_lock(_T->lock, &_T->rf),task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 20 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Quoted 19 lines · Mike Galbraith
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Quoted 8 lines
Quoted 1 line
Salut, much obliged for eyeball relief.
Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.
(adds knob)
Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).
Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.
I'd probably have to take the reads over the writes in such a trade off :)
Quoted 10 lines
tbench 8
NO_MIGRATE_DELAYED 3613.49 MB/sec MIGRATE_DELAYED 3145.59 MB/sec NO_DELAY_DEQUEUE 3355.42 MB/secFirst line is
DELAY_DEQUEUErestoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.
Yep, that's not pretty.
Quoted 8 lines
I thought I might be able to do away with the reservation like side effect of
DELAY_DEQUEUEby borrowingh_nr_delayedfrom...sched/eevdf: More PELT vs
DELAYED_DEQUEUE...for cgroups free test config, but Q/D poke at
idle_cpu()helped not at all.
I wonder if the last_wakee stuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.
Cheers,
Phil
Quoted 95 lines
Quoted 94 lines
---snip---
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.(de-uglified-a-lot-by)
Reported-by Phil Auld <[email protected]>Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>---
kernel/sched/core.c | 48 +++++++++++++++++++++++++++++-------------------kernel/sched/sched.h| 5 +++++2 files changed, 34 insertions(+), 19 deletions(-)kernel/sched/core.c +28
-18@@ -3783,28 +3783,38 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable(struct task_struct *p, int wake_flags) { - struct rq_flags rf; - struct rq *rq; - int ret = 0; - - rq = __task_rq_lock(p, &rf); - if (task_on_rq_queued(p)) { - update_rq_clock(rq); - if (p->se.sched_delayed) - enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); - if (!task_on_cpu(rq, p)) { - /* - * When on_rq && !on_cpu the task is preempted, see if - * it should preempt the task that is current now. - */ - wakeup_preempt(rq, p, wake_flags); + CLASS(__task_rq_lock, rq_guard)(p); + struct rq *rq = rq_guard.rq; + + if (!task_on_rq_queued(p)) + return 0; + + update_rq_clock(rq); + if (p->se.sched_delayed) { + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; + + /* + * Since sched_delayed means we cannot be current anywhere, + * dequeue it here and have it fall through to the + * select_task_rq() case further along the ttwu() path. + */ + if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) { + dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags); + return 0; } - ttwu_do_wakeup(p); - ret = 1; + + enqueue_task(rq, p, queue_flags); + } + if (!task_on_cpu(rq, p)) { + /* + * When on_rq && !on_cpu the task is preempted, see if + * it should preempt the task that is current now. + */ + wakeup_preempt(rq, p, wake_flags); } - __task_rq_unlock(rq, &rf); + ttwu_do_wakeup(p);
- return ret;
+ return 1;
}
#ifdefCONFIG_SMPkernel/sched/sched.h +0
-0@@ -1779,6 +1779,11 @@ task_rq_unlock(struct rq *rq, struct tas raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags); }kernel/sched/sched.h +5
-0+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq = __task_rq_lock(_T->lock, &_T->rf), + __task_rq_unlock(_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1(task_rq_lock, struct
task_struct,_T->rq=task_rq_lock(_T->lock, &_T->rf),task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Peter Zijlstra
maintainerRe: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 30 lines · Phil Auld
Quoted 20 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Quoted 19 lines · Mike Galbraith
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
I built that patch out of curiosity, and yeah,
set_next_task_fair()finding acfs_rq->currends play time pretty quickly.The below improved uptime, and
trace_printk()says it's doing the intended, so I suppose I'll add a feature and see what falls out.From netperf, I got.. number tabulation practice. Three runs of each test with and without produced nothing but variance/noise.
Make it go away then.
If you could write a Changelog for you inspired bit and stick my cleaned up version under it, I'd be much obliged.
Quoted 8 lines
Quoted 1 line
Salut, much obliged for eyeball relief.
Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.
(adds knob)
Quoted 8 lines
Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).
Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.
OK, so I'm going to make this commit disappear.
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Phil AuldQuoted 9 lines · Phil Auld
Quoted 7 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.
(adds knob)
Quoted 8 lines
Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).
Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.
Ok, that would seem to second "shoot it".
Quoted 8 lines
I'd probably have to take the reads over the writes in such a trade off :)
Quoted 6 lines
tbench 8
NO_MIGRATE_DELAYED 3613.49 MB/sec
MIGRATE_DELAYED 3145.59 MB/sec
NO_DELAY_DEQUEUE 3355.42 MB/sec
Quoted 6 lines
Quoted 4 lines
First line is
DELAY_DEQUEUErestoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.Yep, that's not pretty.
Yeah, not to mention annoying.
I get the "adds bounce cache pain" aspect, but not why pre-EEVDF wouldn't be just as heavily affected, it having nothing blocking high frequency migration (the eternal scheduler boogieman:).
Bottom line would appear to be that these survivors should be left where they ended up, either due to LB or more likely bog standard prev_cpu locality, for they are part and parcel of a progression.
Quoted 7 lines
Quoted 7 lines
I thought I might be able to do away with the reservation like side effect of
DELAY_DEQUEUEby borrowingh_nr_delayedfrom...sched/eevdf: More PELT vs
DELAYED_DEQUEUE...for cgroups free test config, but Q/D poke at
idle_cpu()helped not at all.
We don't however have to let sched_delayed block SIS though. Rendering them transparent in idle_cpu() did NOT wreck the progression, so maaaybe could help your regression.
Quoted 2 lines
I wonder if the
last_wakeestuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.
If you mean heuristics, the less of those we have, the better off we are.. they _always_ find a way to embed their teeth in your backside.
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 14 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
Quoted 13 lines · Phil Auld
Quoted 7 lines · On Tue, Nov 12, 2024 at 08:05:04AM +0100 Mike Galbraith wrote:
Unfortunate change log place holder below aside, I think this patch may need to be yanked as trading one not readily repeatable regression for at least one that definitely is, and likely multiple others.
(adds knob)
Yes, I ws just coming here to reply. I have the results from the first version of the patch (I don't think the later one fundemtally changed enough that it will matter but those results are still pending).
Quoted 7 lines
Quoted 4 lines
Not entirely surprisingly we've traded a ~10% rand write regression for 5-10% rand read regression. This makes sense to me since the reads are more likely to be synchronous and thus be more buddy-like and benefit from flipping back and forth on the same cpu.
Ok, that would seem to second "shoot it".
Yes, drop it please, I think. Thanks!
Quoted 28 lines
Quoted 8 lines
I'd probably have to take the reads over the writes in such a trade off :)
Quoted 6 lines
tbench 8
NO_MIGRATE_DELAYED 3613.49 MB/sec
MIGRATE_DELAYED 3145.59 MB/sec
NO_DELAY_DEQUEUE 3355.42 MB/secQuoted 6 lines
Quoted 4 lines
First line is
DELAY_DEQUEUErestoring pre-EEVDF tbench throughput as I've mentioned it doing, but $subject promptly did away with that and then some.Yep, that's not pretty.
Yeah, not to mention annoying.
I get the "adds bounce cache pain" aspect, but not why pre-EEVDF wouldn't be just as heavily affected, it having nothing blocking high frequency migration (the eternal scheduler boogieman:).
Bottom line would appear to be that these survivors should be left where they ended up, either due to LB or more likely bog standard
prev_cpulocality, for they are part and parcel of a progression.Quoted 3 lines
Quoted 3 lines
I thought I might be able to do away with the reservation like side effect of
DELAY_DEQUEUEby borrowingh_nr_delayedfrom...
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
sched/eevdf: More PELT vs
DELAYED_DEQUEUE...for cgroups free test config, but Q/D poke at
idle_cpu()helped not at all.We don't however have to let
sched_delayedblock SIS though. Rendering them transparent inidle_cpu()did NOT wreck the progression, so maaaybe could help your regression.
You mean something like:
if (rq->nr_running > rq->h_nr_delayed)
return 0;
in idle_cpu() instead of the straight rq->nr_running check? I don't have the h_nr_delayed stuff yet but can look for it. I'm not sure that will help with the delayees being sticky. But I can try to try that if I'm understanding you right.
I'll try to dig into it some more regardless.
Quoted 6 lines
Quoted 2 lines
I wonder if the
last_wakeestuff could be leveraged here (an idle thought, so to speak). Haven't looked closely enough.If you mean heuristics, the less of those we have, the better off we are.. they _always_ find a way to embed their teeth in your backside.
Sure, I get that. But when you have a trade-off like this being "smarter" about when to do the dequeue might help. But yes, that could go wrong.
I'm not a fan of knobs either but we could do your patch with the feat and default it off.
Cheers,
Phil
Quoted 2 lines
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
We don't however have to let
sched_delayedblock SIS though. Rendering them transparent inidle_cpu()did NOT wreck the progression, so maaaybe could help your regression.
Quoted 6 lines
You mean something like:
if (
rq->nr_running>rq->h_nr_delayed)
return 0;in
idle_cpu()instead of the straightrq->nr_runningcheck?
Yeah, close enough.
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
We don't however have to let
sched_delayedblock SIS though. Rendering them transparent inidle_cpu()did NOT wreck the progression, so maaaybe could help your regression.
Quoted 8 lines
Quoted 6 lines
You mean something like:
if (
rq->nr_running>rq->h_nr_delayed)
return 0;in
idle_cpu()instead of the straightrq->nr_runningcheck?Yeah, close enough.
The below is all you need.
Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?
--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/sched/fair.c +4 -0
@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct/* * We do not migrate tasks that are: +* 0) not runnable (not useful here/now, but are annoying), or * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or * 3) running (obviously), or * 4) are cache-hot on their current CPU. */ +if (p->se.sched_delayed) +return 0 ; +if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 17 lines · On Thu, Nov 14, 2024 at 12:07:03PM +0100 Mike Galbraith wrote:
Quoted 16 lines · Mike Galbraith
Quoted 14 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
We don't however have to let
sched_delayedblock SIS though. Rendering them transparent inidle_cpu()did NOT wreck the progression, so maaaybe could help your regression.You mean something like:
if (
rq->nr_running>rq->h_nr_delayed)
return 0;in
idle_cpu()instead of the straightrq->nr_runningcheck?
Quoted 9 lines
Quoted 1 line
Yeah, close enough.
The below is all you need.
Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?
I'll give this a try on my issue. This'll be simpler than the other way.
Thanks!
Cheers,
Phil
Quoted 23 lines
--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +4
-0@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct/* * We do not migrate tasks that are: +* 0) not runnable (not useful here/now, but are annoying), or * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or * 3) running (obviously), or * 4) are cache-hot on their current CPU. */ +if (p->se.sched_delayed) +return 0 ; +if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 22 lines · On Thu, Nov 14, 2024 at 06:28:54AM -0500 Phil Auld wrote:
Quoted 21 lines · On Thu, Nov 14, 2024 at 12:07:03PM +0100 Mike Galbraith wrote:
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Phil Auld
Quoted 5 lines · On Tue, Nov 12, 2024 at 03:23:38PM +0100 Mike Galbraith wrote:
We don't however have to let
sched_delayedblock SIS though. Rendering them transparent inidle_cpu()did NOT wreck the progression, so maaaybe could help your regression.Quoted 8 lines
Quoted 6 lines
You mean something like:
if (
rq->nr_running>rq->h_nr_delayed)
return 0;in
idle_cpu()instead of the straightrq->nr_runningcheck?Yeah, close enough.
The below is all you need.
Quoted 8 lines
Quoted 5 lines
Watching blockage rate during part of a netperf scaling run without, a bit over 2/sec was the highest it got, but with, that drops to the same zero as turning off the feature, so... relevance highly unlikely but not quite impossible?
I'll give this a try on my issue. This'll be simpler than the other way.
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
Cheers,
Phil
Quoted 35 lines
Thanks!
Cheers,
PhilQuoted 23 lines
--- kernel/sched/fair.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/sched/fair.c +4
-0@@ -9454,11 +9454,15 @@ int can_migrate_task(struct task_struct/* * We do not migrate tasks that are: +* 0) not runnable (not useful here/now, but are annoying), or * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or * 3) running (obviously), or * 4) are cache-hot on their current CPU. */ +if (p->se.sched_delayed) +return 0 ; +if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess. The numbers said it was quite a reach, no surprise it fell flat.
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
You could try adding (p->flags & PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 8 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Quoted 8 lines
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
You could try adding (
p->flags&PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.
Thanks, I'll try that.
Cheers,
Phil
Quoted 2 lines
-Mike
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 12 lines · On Thu, Nov 21, 2024 at 06:56:28AM -0500 Phil Auld wrote:
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
Quoted 8 lines
Quoted 5 lines
You could try adding (
p->flags&PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.Thanks, I'll try that.
Also, fwiw, I think there is another report here
https://lore.kernel.org/lkml/[email protected]/
which seems to be the same thing but mis-bisected. I've asked them to try with NO_DELAY_DEQUEUE just to be sure. But it looks like a duck.
Cheers,
Phil
Re: [PATCH] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 22 lines · On Thu, Nov 21, 2024 at 07:07:04AM -0500 Phil Auld wrote:
Quoted 12 lines · On Thu, Nov 21, 2024 at 06:56:28AM -0500 Phil Auld wrote:
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
Quoted 8 lines
Quoted 5 lines
You could try adding (
p->flags&PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.Thanks, I'll try that.
Quoted 7 lines
Also, fwiw, I think there is another report here
https://lore.kernel.org/lkml/[email protected]/
which seems to be the same thing but mis-bisected. I've asked them to try with
NO_DELAY_DEQUEUEjust to be sure. But it looks like a duck.
But it does not quack like one. Their issue did not go away with NO_DELAY_DEQUEUE so something different is causing that one.
Quoted 7 lines
Cheers,
Phil
PATCH v2 1/1 sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Phil AuldQuoted 12 lines · Phil Auld
Quoted 11 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
Quoted 7 lines
Quoted 5 lines
You could try adding (
p->flags&PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.Thanks, I'll try that.
You may still want to try that, but my box says probably not. Playing with your write command line, the players I see are pinned kworkers and mobile fio instances.
Maybe try the below instead. The changelog is obsolete BS unless you say otherwise, but while twiddled V2 will still migrate tbench a bit, and per trace_printk() does still let all kinds of stuff wander off to roll the SIS dice, it does not even scratch the paint of the formerly obliterated tbench progression.
Question: did wiping off the evil leave any meaningful goodness behind?
---
sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Phil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the DELAY_DEQUEUE feature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.
V2: do not rip buddies apart, convenient on/off switch
--- kernel/sched/core.c | 51 ++++++++++++++++++++++++++++++------------------ kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 5 ++++ 3 files changed, 42 insertions(+), 19 deletions(-)
kernel/sched/core.c +32 -19
@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; +int dequeue =sched_feat (DEQUEUE_DELAYED ); + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along in ttwu() path. +* Note: Do not rip buddies apart else chaos follows. +*/ +if (dequeue && rq->nr_running >1 && p->nr_cpus_allowed >1 && + !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP
kernel/sched/features.h +5 -0
@@ -47,6 +47,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. */ SCHED_FEAT (DELAY_DEQUEUE , true) +/* +* Free ONLY non-buddy delayed tasks to wakeup-migrate to avoid taking. +* an unnecessary latency hit. Rending buddies asunder inflicts harm. + */ +SCHED_FEAT(DEQUEUE_DELAYED, true)SCHED_FEAT (DELAY_ZERO , true)/*
kernel/sched/sched.h +5 -0
@@ -1783,6 +1783,11 @@ task_rq_unlock(struct rq *rq, struct tasraw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
K Prateek Nayak
reviewerRe: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Hello Mike,
Quoted 52 lines · Mike Galbraith
[..snip..]
Question: did wiping off the evil leave any meaningful goodness behind?
---
sched/fair: Dequeue
sched_delayedtasks when waking to a busy CPUPhil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.V2: do not rip buddies apart, convenient on/off switch
Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>--- kernel/sched/core.c | 51 ++++++++++++++++++++++++++++++------------------ kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 5 ++++ 3 files changed, 42 insertions(+), 19 deletions(-)kernel/sched/core.c +8
-15@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) {
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3803
Quoted 10 lines
kernel/sched/core.c +10
-0+int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; +int dequeue =sched_feat (DEQUEUE_DELAYED ); + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along in ttwu() path. +* Note: Do not rip buddies apart else chaos follows. +*/ + if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&
Do we really care if DEQUEUE_DELAYED is enabled / disabled here when we encounter a delayed task?
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3804
Quoted 1 line
kernel/sched/core.c +1
-0+ !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {
Technically, we are still looking at the last wakeup here since record_wakee() is only called later. If we care about 1:1 buddies, should we just see "current == p->last_wakee", otherwise, there is a good chance "p" has a m:n waker-wakee relationship in which case perhaps a "want_affine" like heuristic can help?
For science, I was wondering if the decision to dequeue + migrate or requeue the delayed task can be put off until after the whole select_task_rq() target selection (note: without the h_nr_delayed stuff, some of that wake_affine_idle() logic falls apart). Hackbench (which saw some regression with EEVDF Complete) seem to like it somewhat, but it still falls behind NO_DELAY_DEQUEUE.
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
NO_DELAY_DEQUEUE Mike's v2 Full ttwu + requeue/migrate
5.76 5.72 ( 1% ) 5.82 ( -1% )
6.53 6.56 ( 0% ) 6.65 ( -2% )
6.79 7.04 ( -4% ) 7.02 ( -3% )
6.91 7.04 ( -2% ) 7.03 ( -2% )
7.63 8.05 ( -6% ) 7.88 ( -3% )Only subtle changes in IBS profiles; there aren't any obvious shift in hotspots with hackbench at least. Not sure if it is just the act of needing to do a dequeue + enqueue from the wakeup context that adds to the overall regression.
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 24 lines
kernel/sched/core.c +13
-4+dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 60 lines · K Prateek Nayak
Hello Mike,
Quoted 57 lines · Mike Galbraith
[..snip..]
Question: did wiping off the evil leave any meaningful goodness behind?
---
sched/fair: Dequeue
sched_delayedtasks when waking to a busy CPUPhil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.V2: do not rip buddies apart, convenient on/off switch
Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>---
kernel/sched/core.c | 51 ++++++++++++++++++++++++++++++------------------kernel/sched/features.h| 5 ++++
kernel/sched/sched.h | 5 ++++3 files changed, 42 insertions(+), 19 deletions(-)kernel/sched/core.c +13
-15@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable(struct task_struct *p, int wake_flags) { - struct rq_flags rf; - struct rq *rq; - int ret = 0; - - rq = __task_rq_lock(p, &rf); - if (task_on_rq_queued(p)) { - update_rq_clock(rq); - if (p->se.sched_delayed) - enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); - if (!task_on_cpu(rq, p)) { - /* - * When on_rq && !on_cpu the task is preempted, see if - * it should preempt the task that is current now. - */ - wakeup_preempt(rq, p, wake_flags); + CLASS(__task_rq_lock, rq_guard)(p); + struct rq *rq = rq_guard.rq; + + if (!task_on_rq_queued(p)) + return 0; + + update_rq_clock(rq); + if (p->se.sched_delayed) { + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; + int dequeue = sched_feat(DEQUEUE_DELAYED); + + /* +* Since sched_delayed means we cannot be current anywhere,
Quoted 8 lines
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3803
Quoted 5 lines
kernel/sched/core.c +5
-0+* dequeue it here and have it fall through to the +* select_task_rq() case further along in ttwu() path. +* Note: Do not rip buddies apart else chaos follows. +*/ + if (dequeue && rq->nr_running > 1 && p->nr_cpus_allowed > 1 &&Do we really care if
DEQUEUE_DELAYEDis enabled / disabled here when we encounter a delayed task?
The switch is for test convenience.
Quoted 7 lines
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct t · line 3804
Quoted 1 line
kernel/sched/core.c +1
-0+ !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) {Technically, we are still looking at the last wakeup here since
record_wakee()is only called later. If we care about 1:1 buddies, should we just see "current ==p->last_wakee", otherwise, there is a good chance "p" has a m:n waker-wakee relationship in which case perhaps a "want_affine" like heuristic can help?
The intent is to blunt the instrument a bit. Paul should have a highly active interrupt source, which will give wakeup credit to whatever is
sitting on that CPU, breaking 1:1 connections.. a little bit. That's
why it still migrates tbench buddies, but NOT at a rate that turns a tbench progression into a new low regression. The hope is that the load shift caused by that active interrupt source is enough to give Paul's regression some of the help it demonstrated wanting, without the collateral damage. It might now be so weak as to not meet the "meaningful" in my question, in which case it lands on the ginormous pile of meh, sorta works, but why would anyone care.
Quoted 6 lines
For science, I was wondering if the decision to dequeue + migrate or requeue the delayed task can be put off until after the whole
select_task_rq()target selection (note: without theh_nr_delayedstuff, some of thatwake_affine_idle()logic falls apart). Hackbench (which saw some regression with EEVDF Complete) seem to like it somewhat, but it still falls behindNO_DELAY_DEQUEUE.
You can, with a few more fast path cycles and some duplication, none of which looks very desirable.
Quoted 13 lines
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
NO_DELAY_DEQUEUE Mike's v2 Full ttwu + requeue/migrate 5.76 5.72 ( 1% ) 5.82 ( -1% ) 6.53 6.56 ( 0% ) 6.65 ( -2% ) 6.79 7.04 ( -4% ) 7.02 ( -3% ) 6.91 7.04 ( -2% ) 7.03 ( -2% ) 7.63 8.05 ( -6% ) 7.88 ( -3% )
Quoted 4 lines
Only subtle changes in IBS profiles; there aren't any obvious shift in hotspots with hackbench at least. Not sure if it is just the act of needing to do a dequeue + enqueue from the wakeup context that adds to the overall regression.
Those numbers say say to me that hackbench doesn't care deeply. That works for me, because I don't care deeply about nutty fork bombs ;-)
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 2 lines · Mike Galbraith
The intent is to blunt the instrument a bit. Paul should have
Yeah I did... ahem, I meant of course Phil.
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 6 lines · On Tue, Nov 26, 2024 at 10:42:37AM +0100 Mike Galbraith wrote:
Quoted 2 lines · Mike Galbraith
The intent is to blunt the instrument a bit. Paul should have
Yeah I did... ahem, I meant of course Phil.
Heh, you are not alone, Mike :)
Quoted 2 lines
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Mike GalbraithQuoted 6 lines · Mike Galbraith
The intent is to blunt the instrument a bit. Paul should have a highly active interrupt source, which will give wakeup credit to whatever is sitting on that CPU, breaking 1:1 connections.. a little bit. That's why it still migrates tbench buddies, but NOT at a rate that turns a tbench progression into a new low regression.
BTW, the reason for the tbench wreckage being so bad is that when the box is near saturation, not only are a portion of the surviving sched_delayed tasks affine wakeups (always the optimal configuration for this fast mover cliebt/server pair in an L3 equipped box), they are exclusively affine wakeups. That is most definitely gonna hurt.
When saturating that becomes the best option for a lot of client/server pairs, even those with a lot of concurrency. Turning them loose to migrate at that time is far more likely than not to hurt a LOT, so V1 was doomed.
Quoted 6 lines
The hope is that the
load shift caused by that active interrupt source is enough to give Paul's regression some of the help it demonstrated wanting, without the collateral damage. It might now be so weak as to not meet the "meaningful" in my question, in which case it lands on the ginormous pile of meh, sorta works, but why would anyone care.
In my IO challenged box, patch is useless to fio, nothing can help a load where all of the IO action, and wimpy action at that, is nailed to one CPU. I can see it helping other latency sensitive stuff, like say 1:N mother of all work and/or control threads (and ilk), but if Phil's problematic box looks anything like this box.. nah, it's a long reach.
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
In reply to Mike GalbraithQuoted 19 lines · On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Quoted 18 lines · Phil Auld
Quoted 16 lines · On Wed, Nov 20, 2024 at 07:37:39PM +0100 Mike Galbraith wrote:
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Phil Auld
This, below, by itself, did not do help and caused a small slowdown on some other tests. Did this need to be on top of the wakeup change?
No, that made a mess.
Rashly speculating that turning mobile kthread component loose is what helped your write regression...
You could try adding (
p->flags&PF_KTHREAD) to the wakeup patch to only turn hard working kthreads loose to try to dodge service latency. Seems unlikely wakeup frequency * instances would combine to shred fio the way turning tbench loose did.
Quoted 6 lines
Quoted 1 line
Thanks, I'll try that.
You may still want to try that, but my box says probably not. Playing with your write command line, the players I see are pinned kworkers and mobile fio instances.
Yep. The PF_KTHREAD thing did not help.
Quoted 6 lines
Maybe try the below instead. The changelog is obsolete BS unless you say otherwise, but while twiddled V2 will still migrate tbench a bit, and per
trace_printk()does still let all kinds of stuff wander off to roll the SIS dice, it does not even scratch the paint of the formerly obliterated tbench progression.
Will give this one a try when I get caught up from being off all week for US turkey day.
Thanks!
Quoted 1 line
Question: did wiping off the evil leave any meaningful goodness behind?
Is that for this patch?
If you mean for the original patch (which subsequently broke the reads) then no. It was more or less even for all the other tests. It fixed the randwrite issue by moving it to randread. Everything else we run regularly was about the same. So no extra goodness to help decide :)
Cheers,
Phil
Quoted 112 lines
---
sched/fair: Dequeue
sched_delayedtasks when waking to a busy CPUPhil Auld (Redhat) reported an fio benchmark regression having been found to have been caused by addition of the
DELAY_DEQUEUEfeature, suggested it may be related to wakees losing the ability to migrate, and confirmed that restoration of same indeed did restore previous performance.V2: do not rip buddies apart, convenient on/off switch
Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Signed-off-by Mike Galbraith <[email protected]>--- kernel/sched/core.c | 51 ++++++++++++++++++++++++++++++------------------ kernel/sched/features.h | 5 ++++ kernel/sched/sched.h | 5 ++++ 3 files changed, 42 insertions(+), 19 deletions(-)kernel/sched/core.c +32
-19@@ -3783,28 +3783,41 @@ ttwu_do_activate(struct rq *rq, struct t*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; - - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; + +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; +int dequeue =sched_feat (DEQUEUE_DELAYED ); + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along in ttwu() path. +* Note: Do not rip buddies apart else chaos follows. +*/ +if (dequeue && rq->nr_running >1 && p->nr_cpus_allowed >1 && + !(rq->curr->last_wakee == p || p->last_wakee == rq->curr)) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); + } +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); } -__task_rq_unlock (rq, &rf); +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP kernel/sched/features.h +5
-0@@ -47,6 +47,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. */ SCHED_FEAT (DELAY_DEQUEUE , true) +/* +* Free ONLY non-buddy delayed tasks to wakeup-migrate to avoid taking. +* an unnecessary latency hit. Rending buddies asunder inflicts harm. + */ +SCHED_FEAT(DEQUEUE_DELAYED, true)SCHED_FEAT (DELAY_ZERO , true)/* kernel/sched/sched.h +5
-0@@ -1783,6 +1783,11 @@ task_rq_unlock(struct rq *rq, struct tasraw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 6 lines · Phil Auld
Quoted 1 line · On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Question: did wiping off the evil leave any meaningful goodness behind?
Is that for this patch?
Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld
On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Question: did wiping off the evil leave any meaningful goodness behind?
Is that for this patch?
Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.
I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat. The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.
Cheers,
Phil
Quoted 3 lines
-Mike
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Hi Mike et al.,
Quoted 15 lines · On Mon, Dec 02, 2024 at 02:12:52PM -0500 Phil Auld wrote:
Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld
On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Question: did wiping off the evil leave any meaningful goodness behind?
Is that for this patch?
Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.
Quoted 4 lines
I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat. The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.
In our testing this has the same effect as the original dequeue-when-delayed fix. It solves the randwrite issue and introduces the ~10-15% randread regression.
Seems to be a real trade-off here. The same guys who benefit from spreading in one case benefit from staying put in the other...
Cheers,
Phil
Re: [PATCH V2] sched/fair: Dequeue sched_delayed tasks when waking to a busy CPU
Quoted 24 lines · Phil Auld
Hi Mike et al.,
Quoted 15 lines · On Mon, Dec 02, 2024 at 02:12:52PM -0500 Phil Auld wrote:
Quoted 4 lines · On Mon, Dec 02, 2024 at 05:55:28PM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld
On Sat, Nov 23, 2024 at 09:44:40AM +0100 Mike Galbraith wrote:
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Question: did wiping off the evil leave any meaningful goodness behind?
Is that for this patch?
Yeah. Trying it on my box with your write command line didn't improve the confidence level either. My box has one CPU handling IRQs and waking pinned workers to service 8 fio instances. Patch was useless for that.
Quoted 4 lines
I'll give it a try. Our "box" is multiple different boxes but the results vary somewhat. The one I sent info about earlier in this thread is just one of the more egregious and is the one the perf team lent me for a while.
Quoted 6 lines
In our testing this has the same effect as the original dequeue-when-delayed fix. It solves the randwrite issue and introduces the ~10-15% randread regression.
Seems to be a real trade-off here. The same guys who benefit from spreading in one case benefit from staying put in the other...
Does as much harm as it does good isn't the mark of a keeper. Oh well.
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 3 lines · Peter Zijlstra
So... I was trying to make that prettier and ended up with something like this:
Passing ENQUEUE_DELAYED to dequeue_task() looks funky until you check the value, but otherwise yeah, when applied that looks better to me.
Quoted 83 lines
---
kernel/sched/core.c | 46 ++++++++++++++++++++++++++++------------------kernel/sched/sched.h| 5 +++++2 files changed, 33 insertions(+), 18 deletions(-)kernel/sched/core.c +2
-3@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,*/ static int ttwu_runnable(struct task_struct *p, int wake_flags) { - struct rq_flags rf; - struct rq *rq; - int ret = 0; + CLASS(__task_rq_lock, rq_guard)(p); + struct rq *rq = rq_guard.rq;
- rq =__task_rq_lock(p, &rf);
- if (task_on_rq_queued(p)) {
- update_rq_clock(rq);
- if (p->se.sched_delayed)
- enqueue_task(rq, p,ENQUEUE_NOCLOCK|ENQUEUE_DELAYED); - if (!task_on_cpu(rq, p)) {
- /*
- * Whenon_rq&& !on_cpu the task is preempted, see if - * it should preempt the task that is current now. - */
- wakeup_preempt(rq, p,wake_flags);
+ if (!task_on_rq_queued(p))
+ return 0;kernel/sched/core.c +13
-0+ + update_rq_clock(rq); + if (p->se.sched_delayed) { + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; + + /* + * Since sched_delayed means we cannot be current anywhere, + * dequeue it here and have it fall through to the + * select_task_rq() case further along the ttwu() path. + */ + if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) { + dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags); + return 0;}
- ttwu_do_wakeup(p);
- ret = 1;
+
+ enqueue_task(rq, p,queue_flags);
}
- __task_rq_unlock(rq, &rf);
+ if (!task_on_cpu(rq, p)) {
+ /*
+ * Whenon_rq&& !on_cpu the task is preempted, see if + * it should preempt the task that is current now. + */
+ wakeup_preempt(rq, p,wake_flags);
+ }
+ ttwu_do_wakeup(p);
- return ret;
+ return 1;
}
#ifdefCONFIG_SMPkernel/sched/sched.h +0
-0@@ -1787,6 +1787,11 @@ task_rq_unlock(struct rq *rq, struct task_struct *p, struct rq_flags *rf) raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags); }kernel/sched/sched.h +5
-0+DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq = __task_rq_lock(_T->lock, &_T->rf), + __task_rq_unlock(_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1(task_rq_lock, struct
task_struct,_T->rq=task_rq_lock(_T->lock, &_T->rf),task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Peter Zijlstra
So... I was trying to make that prettier and ended up with something like this:
Passing
ENQUEUE_DELAYEDtodequeue_task()looks funky until you check the value, but otherwise yeah, when applied that looks better to me.
Yeah, it does look funneh, but we've been doing that for a long long while.
Still, perhaps I should rename the shared ones to QUEUE_foo and only have the specific ones be {EN,DE}QUEUE_foo.
Dietmar Eggemann
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Phil AuldHi Phil,
Quoted 25 lines · Phil Auld
Hi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
Quoted 8 lines
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
I'm not seeing this on my i7-13700K running tip sched/core (1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)
Using 'taskset 0xaaaaa' avoiding SMT and running only on P-cores.
vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)
# sudo lshw -class disk -class storage
*-nvme
description: NVMe device
product: GIGABYTE GP-ASM2NE6500GTTD
vendor: Phison Electronics Corporation
physical id: 0
bus info: pci@0000:01:00.0
logical name: /dev/nvme0
version: EGFM13.2
...
capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live
resources: irq:16 memory:70800000-70803fff# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
Which disk device you're using?
Quoted 8 lines
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
[...]
Dietmar Eggemann
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
+cc Christian Loehle <[email protected]>
Quoted 35 lines · Dietmar Eggemann
Hi Phil,
Quoted 32 lines · Phil Auld
Hi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
Quoted 7 lines
Quoted 1 line
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.
^^^^^^^
Quoted 3 lines
vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)
Christian Loehle just told me that my cpumask looks odd. Should be 0xaaaa instead.
Retested:
vanilla features: 954MB/s (mean out of 5 runs, σ: 30.83)NO_DELAY_DEQUEUE: 932MB/s (mean out of 5 runs, σ: 28.10)
Now there are only 8 CPUs (instead of 10) for the 8 (+2) fio tasks. σ went up probably because of more wakeup/preemption latency.
Quoted 32 lines
# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fff# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Quoted 8 lines
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
[...]
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Dietmar EggemannHi Dietmar,
Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
Quoted 25 lines · Phil Auld
Hi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
Quoted 8 lines
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fff
Quoted 4 lines
# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Most of the reports are on various NVME drives (samsung mostly I think).
One thing I should add is that it's all on LVM:
vgcreate vg /dev/nvme0n1 -y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool /dev/mapper/vg-thinPool --poolmetadata /dev/mapper/vg-thinMeta -Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a /dev/mapper/vg-testLV
mkfs.ext4 /dev/mapper/vg-testLV -E lazy_itable_init=0,lazy_journal_init=0 -F mount /dev/mapper/vg-testLV /testfs
With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.
It's always the randwrite numbers. The rest look fine.
Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.
Thanks for taking a look.
Cheers,
Phil
Quoted 12 lines
Quoted 9 lines
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
[...]
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 85 lines · Phil Auld
Hi Dietmar,
Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
Quoted 25 lines · Phil Auld
Hi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
Quoted 8 lines
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fffQuoted 4 lines
# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Most of the reports are on various NVME drives (samsung mostly I think).
One thing I should add is that it's all on LVM:
vgcreate vg
/dev/nvme0n1-y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool/dev/mapper/vg-thinPool--poolmetadata/dev/mapper/vg-thinMeta-Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a/dev/mapper/vg-testLV
mkfs.ext4/dev/mapper/vg-testLV-E lazy_itable_init=0,lazy_journal_init=0 -F mount/dev/mapper/vg-testLV/testfs
Quoted 6 lines
With VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.
It's always the randwrite numbers. The rest look fine.
Hi Phil,
Thanks for the detailed instructions. Unfortunately even with your LVM setup on the platforms I've tried I don't see a regression so far, all the numbers are about equal for DELAY_DEQUEUE and NO_DELAY_DEQUEUE.
Anyway I have some follow-ups, first let me trim the fio command for readability:
fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
dropping defaults nr_files, loops, fsync, randrepeat
fio --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs
Adding the CPU affinities directly:
fio --cpus_allowed 1-8 --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs
Now I was wondering about the following:
Is it actually the kworker (not another fio) being preempted? (I'm pretty sure it is) To test: --cpus_allowed_policy split (each fio process gets it's own CPU).
Quoted 2 lines · You
I was thinking maybe the preemption was preventing some batching of IO completions or initiations. But that was wrong it seems.
So while it doesn't reproduce for me, the only thing being preempted regularly is the kworker (running iomap_dio_complete_work). I don't quite follow the "that was wrong it seems" part then. Could you elaborate?
Could you also post the other benchmark numbers? Does any of them score higher in IOPS? Is --rw write the same issue if you set --bs 4k (assuming you set a larger bs for seqwrite).
Can you set the kworkers handling completions to SCHED_BATCH too? Just to confirm.
Regards,
Christian
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hi Christian,
Quoted 91 lines · On Tue, Nov 05, 2024 at 09:53:49AM +0000 Christian Loehle wrote:
Quoted 90 lines · Phil Auld
Hi Dietmar,
Quoted 60 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
Quoted 25 lines · Phil Auld
Hi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
Quoted 8 lines
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fffQuoted 4 lines
# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Most of the reports are on various NVME drives (samsung mostly I think).
One thing I should add is that it's all on LVM:
vgcreate vg
/dev/nvme0n1-y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool/dev/mapper/vg-thinPool--poolmetadata/dev/mapper/vg-thinMeta-Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a/dev/mapper/vg-testLV
mkfs.ext4/dev/mapper/vg-testLV-E lazy_itable_init=0,lazy_journal_init=0 -F mount/dev/mapper/vg-testLV/testfsWith VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.
Quoted 8 lines
Quoted 1 line
It's always the randwrite numbers. The rest look fine.
Hi Phil,
Thanks for the detailed instructions. Unfortunately even with your LVM setup on the platforms I've tried I don't see a regression so far, all the numbers are about equal for
DELAY_DEQUEUEandNO_DELAY_DEQUEUE.
Yeah, that's odd.
Fwiw:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
Model name: AMD EPYC 7313P 16-Core Processor
BIOS Model name: AMD EPYC 7313P 16-Core Processor Unknown CPU @ 3.0GHz
BIOS CPU family: 107
CPU family: 25
...16 SMT2 cores (siblings are 16-31)
#lsblk -N
NAME TYPE MODEL SERIAL REV TRAN RQ-SIZE MQ nvme3n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605178 GDC5602Q nvme 1023 32 nvme2n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605128 GDC5602Q nvme 1023 32 nvme0n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605125 GDC5602Q nvme 1023 32 nvme1n1 disk SAMSUNG MZQL21T9HCJR-00A07 S64GNJ0T605127 GDC5602Q nvme 1023 32
Where nvme0n1 is the one I'm actually using.
I'm on 6.12.0-0.rc5.44.eln143.x86_64 which is v6.12-rc5 with RHEL .config. This should have little to no franken-kernel bits but now that I have the machine I'll build from upstream (with the RHEL .config still) to make sure.
We did see it on all the RCs so far.
Quoted 9 lines
Anyway I have some follow-ups, first let me trim the fio command for readability:
fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfsdropping defaults
nr_files, loops, fsync, randrepeat
fio --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfsAdding the CPU affinities directly:
fio --cpus_allowed 1-8 --rw randwrite --bs 4k --runtime 1m --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --directory /testfs
Fair enough. It should work the same with taskset I suppose except the below bit. I've been given this from our perforance team. They have a framework that produces nice html pages with red and green results and graphs and whatnot. Right now it's in the form of a script that pulls the KB/s number of out the json output which is nice and keeps me from going crosseyed looking at the full fio run output.
Quoted 3 lines
Now I was wondering about the following:
Is it actually the kworker (not another fio) being preempted? (I'm pretty sure it is) To test: --cpus_allowed_policy split (each fio process gets it's own CPU).
with --cpus-allowed and --cpus_allowed_policy split the results with DELAY_DEQUEUE are better (540MB/s) but with NO_DELAY_DEQUEUE they are also better (640 MB/s). It was 510MB/s and 590MB/s before.
Quoted 9 lines
Quoted 2 lines · You
I was thinking maybe the preemption was preventing some batching of IO completions or initiations. But that was wrong it seems.
So while it doesn't reproduce for me, the only thing being preempted regularly is the kworker (running
iomap_dio_complete_work). I don't quite follow the "that was wrong it seems" part then. Could you elaborate?
I was thinking that the fio batch test along with the disabling WAKEUP_PREEMPTION was telling me that it wasn't the over preemption issue, but that also I could be wrong about...
Quoted 3 lines
Could you also post the other benchmark numbers? Does any of them score higher in IOPS? Is --rw write the same issue if you set --bs 4k (assuming you set a larger bs for seqwrite).
I don't have numbers for all of the other flavors but I ran --rw write --bs 4k:
DELAY_DEQUEUE ~590MB/sNO_DELAY_DEQUEUE ~840MB/s
Those results are not good for DELAY_DEQUEUE either.
Quoted 1 line
Can you set the kworkers handling completions to
SCHED_BATCHtoo? Just to confirm.
I think I did the wrong kworkes the first time. So I'll try again to figure out which kworkers to twiddle (or I'll just do all 227 of them...).
Thanks,
Phil
Quoted 4 lines
Regards,
Christian
Dietmar Eggemann
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Phil AuldQuoted 9 lines · Phil Auld
Hi Dietmar,
Quoted 5 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
Quoted 2 lines · Phil Auld
Hi Peterm
[...]
Quoted 49 lines
Quoted 23 lines
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fffQuoted 4 lines
# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Most of the reports are on various NVME drives (samsung mostly I think).
One thing I should add is that it's all on LVM:
vgcreate vg
/dev/nvme0n1-y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool/dev/mapper/vg-thinPool--poolmetadata/dev/mapper/vg-thinMeta-Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a/dev/mapper/vg-testLV
mkfs.ext4/dev/mapper/vg-testLV-E lazy_itable_init=0,lazy_journal_init=0 -F mount/dev/mapper/vg-testLV/testfsWith VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.
Quoted 4 lines
It's always the randwrite numbers. The rest look fine.
Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.
One reason I don't see the difference between DELAY_DEQUEUE and NO_DELAY_DEQUEUE could be because of the affinity of the related nvme interrupts:
$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...
132: 0 0 1523653 0 0 0 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1 133: 0 0 0 0 0 1338451 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2 134: 0 0 0 0 0 0 0 0 2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3
$ cat /proc/irq/132/smp_affinity_list 0-2 cat /proc/irq/133/smp_affinity_list 3-5 cat /proc/irq/134/smp_affinity_list 6-8
So the 8 fio tasks from:
# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k --runtime 8s --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --cpus_allowed_policy shared --directory /testfs
don't have to fight with per-CPU kworkers on each CPU.
e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
run iomap_dio_complete_work() in kworker/8:x'
In case I trace the 'task_on_rq_queued(p) && p->se.sched_delayed && rq->nr_running > 1) condition in ttwu_runnable() condition i only see the per-CPU kworker in there, so p->nr_cpus_allowed == 1.
So the patch shouldn't make a difference for this scenario?
But maybe your VDO or thinpool setup creates waker/wakee pairs with wakee->nr_cpus_allowed > 1?
Does your machine has single CPU smp_affinity masks for these nvme interrupts?
[...]
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 95 lines · On Fri, Nov 08, 2024 at 03:53:26PM +0100 Dietmar Eggemann wrote:
Quoted 9 lines · Phil Auld
Hi Dietmar,
Quoted 5 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
Quoted 2 lines · Phil Auld
Hi Peterm
[...]
Quoted 49 lines
Quoted 23 lines
I'm not seeing this on my i7-13700K running tip sched/core (
1a6151017ee5
- sched: psi: pass enqueue/dequeue flags to psi callbacks directly (2024-10-26 Johannes Weiner)) (6.12.0-rc4 - based)Using 'taskset
0xaaaaa' avoiding SMT and running only on P-cores.vanilla features: 990MB/s (mean out of 5 runs, σ: 9.38)
NO_DELAY_DEQUEUE: 992MB/s (mean out of 5 runs, σ: 10.61)# sudo lshw -class disk -class storage *-nvme description: NVMe device product: GIGABYTE GP-ASM2NE6500GTTD vendor: Phison Electronics Corporation physical id: 0 bus info: pci@0000:01:00.0 logical name: /dev/nvme0 version: EGFM13.2 ... capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:19871987SN215108954872 GIGABYTE GP-ASM2NE6500GTTD state=live resources: irq:16 memory:70800000-70803fffQuoted 4 lines
# mount | grep ^/dev/nvme0 /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)Which disk device you're using?
Most of the reports are on various NVME drives (samsung mostly I think).
One thing I should add is that it's all on LVM:
vgcreate vg
/dev/nvme0n1-y
lvcreate -n thinMeta -L 3GB vg -y
lvcreate -n thinPool -l 99%FREE vg -y
lvconvert --thinpool/dev/mapper/vg-thinPool--poolmetadata/dev/mapper/vg-thinMeta-Zn -y lvcreate -n testLV -V 1300G --thinpool thinPool vg
wipefs -a/dev/mapper/vg-testLV
mkfs.ext4/dev/mapper/vg-testLV-E lazy_itable_init=0,lazy_journal_init=0 -F mount/dev/mapper/vg-testLV/testfsWith VDO or thinpool (as above) it shows on both ext4 and xfs. With fs on drive directly it's a little more variable. Some it shows on xfs, some it show on ext4 and not vice-versa, seems to depend on the drive or hw raid. But when it shows it's 100% reproducible on that setup.
Quoted 4 lines
It's always the randwrite numbers. The rest look fine.
Also, as yet I'm not personally doing this testing, just looking into it and passing on the information I have.
One reason I don't see the difference between
DELAY_DEQUEUEandNO_DELAY_DEQUEUEcould be because of the affinity of the related nvme interrupts:$ cat /proc/interruptsCPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...
132: 0 0 1523653 0 0 0 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1 133: 0 0 0 0 0 1338451 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2 134: 0 0 0 0 0 0 0 0 2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3$ cat /proc/irq/132/smp_affinity_list 0-2 cat /proc/irq/133/smp_affinity_list 3-5 cat /proc/irq/134/smp_affinity_list 6-8So the 8 fio tasks from:
# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k --runtime 8s --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --cpus_allowed_policy shared --directory /testfsdon't have to fight with per-CPU kworkers on each CPU.
Quoted 9 lines
e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
runiomap_dio_complete_work()in kworker/8:x'In case I trace the '
task_on_rq_queued(p)&&p->se.sched_delayed&&rq->nr_running> 1) condition inttwu_runnable()condition i only see the per-CPU kworker in there, sop->nr_cpus_allowed== 1.So the patch shouldn't make a difference for this scenario?
If the kworker is waking up an fio task it could. I don't think they are bound to a single cpu.
But yes if your trace is only showing the kworker there then it would not help. Are you actually able to reproduce the difference?
Quoted 3 lines
But maybe your VDO or thinpool setup creates waker/wakee pairs with
wakee->nr_cpus_allowed> 1?
That's certainly possible but I don't know for sure. There are well more dio kworkers on the box than cpus though if I recall. I don't know if they all have singel cpu affinities.
Quoted 3 lines
Does your machine has single CPU
smp_affinitymasks for these nvme interrupts?
I don't know. I had to give the machine back.
Cheers,
Phil
Quoted 35 lines
[...]
Dietmar Eggemann
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 9 lines · Phil Auld
Quoted 8 lines · On Fri, Nov 08, 2024 at 03:53:26PM +0100 Dietmar Eggemann wrote:
Quoted 7 lines · Phil Auld
Hi Dietmar,
Quoted 3 lines · On Mon, Nov 04, 2024 at 10:28:37AM +0100 Dietmar Eggemann wrote:
Hi Phil,
On 01/11/2024 13:47, Phil Auld wrote:
[...]
Quoted 35 lines
Quoted 35 lines
One reason I don't see the difference between
DELAY_DEQUEUEandNO_DELAY_DEQUEUEcould be because of the affinity of the related nvme interrupts:$ cat /proc/interruptsCPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 ...
132: 0 0 1523653 0 0 0 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 1-edge nvme0q1 133: 0 0 0 0 0 1338451 0 0 0 ... IR-PCI-MSIX-0000:01:00.0 2-edge nvme0q2 134: 0 0 0 0 0 0 0 0 2252297 ... IR-PCI-MSIX-0000:01:00.0 3-edge nvme0q3$ cat /proc/irq/132/smp_affinity_list 0-2 cat /proc/irq/133/smp_affinity_list 3-5 cat /proc/irq/134/smp_affinity_list 6-8So the 8 fio tasks from:
# fio --cpus_allowed 1,2,3,4,5,6,7,8 --rw randwrite --bs 4k --runtime 8s --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --name default --time_based --group_reporting --cpus_allowed_policy shared --directory /testfsdon't have to fight with per-CPU kworkers on each CPU.
e.g. 'nvme0q3 interrupt -> queue on workqueue dio/nvme0n1p2 ->
runiomap_dio_complete_work()in kworker/8:x'In case I trace the '
task_on_rq_queued(p)&&p->se.sched_delayed&&rq->nr_running> 1) condition inttwu_runnable()condition i only see the per-CPU kworker in there, sop->nr_cpus_allowed== 1.
Quoted 8 lines
Quoted 2 lines
So the patch shouldn't make a difference for this scenario?
If the kworker is waking up an fio task it could. I don't think they are bound to a single cpu.
But yes if your trace is only showing the kworker there then it would not help. Are you actually able to reproduce the difference?
No, with my setup I don't see any difference running your fio test. But the traces also show me that there are no scenarios in which this patch can make a difference in the scores.
Quoted 7 lines
Quoted 3 lines
But maybe your VDO or thinpool setup creates waker/wakee pairs with
wakee->nr_cpus_allowed> 1?That's certainly possible but I don't know for sure. There are well more dio kworkers on the box than cpus though if I recall. I don't know if they all have singel cpu affinities.
Yeah there must be more tasks (inc. kworkers) w/ 'p->nr_cpus_allowed > 1' involved.
Quoted 5 lines
Quoted 3 lines
Does your machine has single CPU
smp_affinitymasks for these nvme interrupts?I don't know. I had to give the machine back.
Ah, too late then ;-)
Peter Zijlstra
maintainerPATCH 18/24 sched/fair: Implement DELAY_ZERO
In reply to Peter Zijlstra'Extend' DELAY_DEQUEUE by noting that since we wanted to dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
--- kernel/sched/fair.c | 16 ++++++++++++++++ kernel/sched/features.h | 3 +++ 2 files changed, 19 insertions(+)
kernel/sched/fair.c +16 -0
@@ -5529,6 +5529,8 @@ pick_next_entity(struct rq *rq, struct cdequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED );SCHED_WARN_ON (se->sched_delayed);SCHED_WARN_ON (se->on_rq); +if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) + se->vlag =0 ;return NULL ; }
@@ -6827,6 +6829,20 @@ requeue_delayed_entity(struct sched_entiSCHED_WARN_ON (!se->sched_delayed);SCHED_WARN_ON (!se->on_rq); +if (sched_feat (DELAY_ZERO )) { +update_entity_lag (cfs_rq, se); +if (se->vlag >0 ) { + cfs_rq->nr_running--; +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); + se->vlag =0 ; +place_entity (cfs_rq, se,0 ); +if (se != cfs_rq->curr) +__enqueue_entity (cfs_rq, se); + cfs_rq->nr_running++; + } + } + se->sched_delayed =0 ; }
kernel/sched/features.h +3 -0
@@ -34,8 +34,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)* By delaying the dequeue for non-eligible tasks, they remain in the * competition and can burn off their negative lag. When they get selected * they'll have positive lag by definition. +* +* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. */ SCHED_FEAT(DELAY_DEQUEUE, true) +SCHED_FEAT (DELAY_ZERO , true)/* * Allow wakeup-time preemption of the current task:
[tip: sched/core] sched/fair: Implement DELAY_ZERO
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 54a58a78779169f9c92a51facf6de7ce94962328 Gitweb: https://git.kernel.org/tip/54a58a78779169f9c92a51facf6de7ce94962328 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 12:26:06 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:44 +02:00
sched/fair: Implement DELAY_ZERO
'Extend' DELAY_DEQUEUE by noting that since we wanted to dequeued them at the 0-lag point, truncate lag (eg. don't let them earn positive lag).
--- kernel/sched/fair.c | 20 ++++++++++++++++++-- kernel/sched/features.h | 3 +++ 2 files changed, 21 insertions(+), 2 deletions(-)
kernel/sched/fair.c +18 -2
@@ -5447,8 +5447,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if ((flags & (DEQUEUE_SAVE |DEQUEUE_MOVE )) !=DEQUEUE_SAVE )update_min_vruntime (cfs_rq); -if (flags &DEQUEUE_DELAYED ) +if (flags &DEQUEUE_DELAYED ) { se->sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) + se->vlag =0 ; + }if (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq);
@@ -5527,7 +5530,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED );SCHED_WARN_ON (se->sched_delayed);SCHED_WARN_ON (se->on_rq); -return NULL ; }return se;
@@ -6825,6 +6827,20 @@ requeue_delayed_entity(struct sched_entity *se)SCHED_WARN_ON (!se->sched_delayed);SCHED_WARN_ON (!se->on_rq); +if (sched_feat (DELAY_ZERO )) { +update_entity_lag (cfs_rq, se); +if (se->vlag >0 ) { + cfs_rq->nr_running--; +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); + se->vlag =0 ; +place_entity (cfs_rq, se,0 ); +if (se != cfs_rq->curr) +__enqueue_entity (cfs_rq, se); + cfs_rq->nr_running++; + } + } + se->sched_delayed =0 ; }
kernel/sched/features.h +3 -0
@@ -34,8 +34,11 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)* By delaying the dequeue for non-eligible tasks, they remain in the * competition and can burn off their negative lag. When they get selected * they'll have positive lag by definition. +* +* DELAY_ZERO clips the lag on dequeue (or wakeup) to 0. */ SCHED_FEAT(DELAY_DEQUEUE, true) +SCHED_FEAT (DELAY_ZERO , true)/* * Allow wakeup-time preemption of the current task:
Peter Zijlstra
maintainerPATCH 19/24 sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
In reply to Peter ZijlstraNote that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)
kernel/sched/fair.c +2 -0
@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );return false; } }
@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti } se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 ); }/*
kernel/sched/sched.h +6 -0
@@ -816,6 +816,9 @@ static inline void se_update_runnable(ststatic inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +if (entity_is_task (se))return !!se->on_rq;else
@@ -830,6 +833,9 @@ static inline void se_update_runnable(ststatic inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }
Valentin Schneider
reviewerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 15 lines · Peter Zijlstra
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );
Shouldn't this be before setting ->sched_delayed? accumulate_sum() should see the time delta as spent being runnable.
kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818
Quoted 8 lines
kernel/sched/fair.c +1
-0return false; } }@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti } se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 );
Ditto on the ordering
kernel/sched/sched.h static inline void se_update_runnable(st · line 821
Quoted 12 lines
kernel/sched/fair.c +0
-0}/* kernel/sched/sched.h +3
-0@@ -816,6 +816,9 @@ static inline void se_update_runnable(ststatic inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +
Per __update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of ___update_load_sum()?
Quoted 13 lines
kernel/sched/sched.h +3
-0if (entity_is_task (se))return !!se->on_rq;else @@ -830,6 +833,9 @@ static inline void se_update_runnable(ststatic inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }
Peter Zijlstra
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );Shouldn't this be before setting ->sched_delayed?
accumulate_sum()should see the time delta as spent being runnable.Quoted 6 lines
kernel/sched/fair.c +0
-0return false; } }@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti }
Quoted 4 lines
kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818
Quoted 2 lines
kernel/sched/fair.c +1
-0se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 );Ditto on the ordering
Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/
Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +0
-0}/* kernel/sched/sched.h +0
-0@@ -816,6 +816,9 @@ static inline void se_update_runnable(st
Quoted 9 lines
kernel/sched/sched.h static inline void se_update_runnable(st · line 821
Quoted 5 lines
kernel/sched/sched.h +3
-0static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +Per
__update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of___update_load_sum()?
So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.
Peter Zijlstra
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );Shouldn't this be before setting ->sched_delayed?
accumulate_sum()should see the time delta as spent being runnable.Quoted 6 lines
kernel/sched/fair.c +0
-0return false; } }@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti }
Quoted 8 lines
Quoted 4 lines
kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818
Quoted 2 lines
kernel/sched/fair.c +1
-0se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 );Ditto on the ordering
Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/
OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...
---
kernel/sched/fair.c +11 -15
@@ -5461,14 +5461,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) {bool sleep = flags &DEQUEUE_SLEEP ; +update_curr (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { -/* -* DEQUEUE_DELAYED is typically called from pick_next_entity() -* at which point we've already done update_curr() and do not -* want to do so again. -*/ SCHED_WARN_ON (!se->sched_delayed); - se->sched_delayed =0 ; }else {bool delay = sleep;/*
@@ -5479,14 +5475,13 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) delay = false;SCHED_WARN_ON (delay && se->sched_delayed); -update_curr (cfs_rq);if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ; - se->sched_delayed =1 ;update_load_avg (cfs_rq, se,0 ); + se->sched_delayed =1 ;return false; } }
@@ -5536,6 +5531,12 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if ((flags & (DEQUEUE_SAVE |DEQUEUE_MOVE )) !=DEQUEUE_SAVE )update_min_vruntime (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { + se->sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) + se->vlag =0 ; + } +if (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq);
@@ -5611,11 +5612,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)struct sched_entity *se =pick_eevdf (cfs_rq);if (se->sched_delayed) {dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); -SCHED_WARN_ON (se->sched_delayed); -SCHED_WARN_ON (se->on_rq); -if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) - se->vlag =0 ; -return NULL ; }return se;
@@ -6906,7 +6902,7 @@ requeue_delayed_entity(struct sched_entity *se)struct cfs_rq *cfs_rq =cfs_rq_of (se);/* -* se->sched_delayed should imply both: se->on_rq == 1. +* se->sched_delayed should imply: se->on_rq == 1. * Because a delayed entity is one that is still on * the runqueue competing until elegibility. */
@@ -6927,8 +6923,8 @@ requeue_delayed_entity(struct sched_entity *se) } } - se->sched_delayed =0 ;update_load_avg (cfs_rq, se,0 ); + se->sched_delayed =0 ; }/*
Peter Zijlstra
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 57 lines · Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );Shouldn't this be before setting ->sched_delayed?
accumulate_sum()should see the time delta as spent being runnable.Quoted 6 lines
kernel/sched/fair.c +0
-0return false; } }@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti }Quoted 8 lines
Quoted 4 lines
kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818
Quoted 2 lines
kernel/sched/fair.c +1
-0se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 );Ditto on the ordering
Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/
OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...
---
kernel/sched/fair.c +2
-0@@ -5461,14 +5461,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) {bool sleep = flags &DEQUEUE_SLEEP ; +update_curr (cfs_rq); +if (flags &DEQUEUE_DELAYED ) {
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5470
Quoted 10 lines
kernel/sched/fair.c +0
-6-/* -* DEQUEUE_DELAYED is typically called from pick_next_entity() -* at which point we've already done update_curr() and do not -* want to do so again. -*/ SCHED_WARN_ON (!se->sched_delayed); - se->sched_delayed =0 ; }else {bool delay = sleep;/*
Because repeated update_curr() is harmless (I think I was thinking it would move the clock, but it doesn't do that), but a missed update_curr() makes a mess.
Quoted 6 lines
kernel/sched/fair.c +0
-1@@ -5479,14 +5475,13 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) delay = false;SCHED_WARN_ON (delay && se->sched_delayed); -update_curr (cfs_rq);
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5487
Quoted 10 lines
kernel/sched/fair.c +1
-1if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ; - se->sched_delayed =1 ;update_load_avg (cfs_rq, se,0 ); + se->sched_delayed =1 ;return false; } }
As you said, update to now, then mark delayed.
Quoted 15 lines
kernel/sched/fair.c +6
-0@@ -5536,6 +5531,12 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if ((flags & (DEQUEUE_SAVE |DEQUEUE_MOVE )) !=DEQUEUE_SAVE )update_min_vruntime (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { + se->sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) + se->vlag =0 ; + } +if (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq);@@ -5611,11 +5612,6 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq)struct sched_entity *se =pick_eevdf (cfs_rq);
kernel/sched/fair.c pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq) · line 5617
Quoted 10 lines
kernel/sched/fair.c +0
-5if (se->sched_delayed) {dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); -SCHED_WARN_ON (se->sched_delayed); -SCHED_WARN_ON (se->on_rq); -if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) - se->vlag =0 ; -return NULL ; }return se;
Because if we have to clear delayed at the end up dequeue, we might as well do all of the fixups at that point.
kernel/sched/fair.c requeue_delayed_entity(struct sched_entity *se) · line 6908
Quoted 9 lines
kernel/sched/fair.c +1
-1@@ -6906,7 +6902,7 @@ requeue_delayed_entity(struct sched_entity *se)struct cfs_rq *cfs_rq =cfs_rq_of (se);/* -* se->sched_delayed should imply both: se->on_rq == 1. +* se->sched_delayed should imply: se->on_rq == 1. * Because a delayed entity is one that is still on * the runqueue competing until elegibility. */
edit fail, somewhere along history.
kernel/sched/fair.c requeue_delayed_entity(struct sched_entity *se) · line 6928
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -6927,8 +6923,8 @@ requeue_delayed_entity(struct sched_entity *se) } } - se->sched_delayed =0 ;update_load_avg (cfs_rq, se,0 ); + se->sched_delayed =0 ; }
What you said..
Valentin Schneider
reviewerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Valentin Schneider
Review requeue_delayed_entity(struct sched_enti · line 6818
Quoted 5 lines · Peter Zijlstra
(quoted diff) +1
-0@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti } se->sched_delayed = 0; + update_load_avg(cfs_rq, se, 0);Ditto on the ordering
Quoted 8 lines
Quoted 3 lines
Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/
OK, so I went over it again and I ended up with the below diff -- which assuming I didn't make a giant mess of things *again*, I should go fold back into various other patches ...
Looks right to me, thanks! I'll go test the newer stack of patches.
Vincent Guittot
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
In reply to Peter ZijlstraQuoted 58 lines · Peter Zijlstra
Quoted 30 lines · Valentin Schneider
Quoted 15 lines · Peter Zijlstra
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, st · line 5391
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -5388,6 +5388,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->next == se) cfs_rq->next =NULL ; se->sched_delayed =1 ; +update_load_avg (cfs_rq, se,0 );Shouldn't this be before setting ->sched_delayed?
accumulate_sum()should see the time delta as spent being runnable.Quoted 6 lines
kernel/sched/fair.c +0
-0return false; } }@@ -6814,6 +6815,7 @@ requeue_delayed_entity(struct sched_enti }Quoted 4 lines
kernel/sched/fair.c requeue_delayed_entity(struct sched_enti · line 6818
Quoted 2 lines
kernel/sched/fair.c +1
-0se->sched_delayed =0 ; +update_load_avg (cfs_rq, se,0 );Ditto on the ordering
Bah, so I remember thinking about it and then I obviously go and do it the exact wrong way around eh? Let me double check this tomorrow morning with the brain slightly more awake :/
Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +0
-0}/* kernel/sched/sched.h +0
-0@@ -816,6 +816,9 @@ static inline void se_update_runnable(stQuoted 9 lines
kernel/sched/sched.h static inline void se_update_runnable(st · line 821
Quoted 5 lines
kernel/sched/sched.h +3
-0static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +Per
__update_load_avg_se(), delayed-dequeue entities are still ->on_rq, so their load signal will increase. Do we want a similar helper for the @load input of___update_load_sum()?
Quoted 3 lines
So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.
We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the load_avg on this rq
Shouldn't we track separately the sum of the weight of delayed dequeue to apply it only on vruntime update ?
Quoted 1 line
Peter Zijlstra
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 7 lines · Vincent Guittot
Quoted 3 lines
So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.
We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the
load_avgon this rq
So far load has been a direct sum of all weight. Additionally, we delay until a task gets picked again, migrating tasks to other CPUs will expedite this condition.
Anyway, at the moment I don't have strong evidence either which way, and the above argument seem to suggest not changing things for now.
We can always re-evaluate.
Vincent Guittot
maintainerRe: [PATCH 19/24] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Vincent Guittot
Quoted 3 lines
So the whole reason to keep then enqueued is so that they can continue to compete for vruntime, and vruntime is load based. So it would be very weird to remove them from load.
Quoted 5 lines
Quoted 3 lines
We only use the weight to update vruntime, not the load. The load is used to balance tasks between cpus and if we keep a "delayed" dequeued task in the load, we will artificially inflate the
load_avgon this rqSo far load has been a direct sum of all weight. Additionally, we delay
it has been the sum of all runnable tasks but delayed tasks are not runnable anymore. The task stays "enqueued" only to help clearing its lag
Quoted 7 lines
until a task gets picked again, migrating tasks to other CPUs will expedite this condition.
Anyway, at the moment I don't have strong evidence either which way, and the above argument seem to suggest not changing things for now.
We can always re-evaluate.
[tip: sched/core] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: fc1892becd5672f52329a75c73117b60ac7841b7 Gitweb: https://git.kernel.org/tip/fc1892becd5672f52329a75c73117b60ac7841b7 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 26 Apr 2024 13:00:50 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00
sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
Note that tasks that are kept on the runqueue to burn off negative lag, are not in fact runnable anymore, they'll get dequeued the moment they get picked.
As such, don't count this time towards runnable.
Thanks to Valentin for spotting I had this backwards initially.
--- kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)
kernel/sched/fair.c +2 -0
@@ -5402,6 +5402,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) !entity_eligible(cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ; +update_load_avg (cfs_rq, se,0 ); se->sched_delayed =1 ;return false; }
@@ -6841,6 +6842,7 @@ requeue_delayed_entity(struct sched_entity *se) } } +update_load_avg (cfs_rq, se,0 ); se->sched_delayed =0 ; }
kernel/sched/sched.h +6 -0
@@ -820,6 +820,9 @@ static inline void se_update_runnable(struct sched_entity *se)static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +if (entity_is_task (se))return !!se->on_rq;else
@@ -834,6 +837,9 @@ static inline void se_update_runnable(struct sched_entity *se) { }static inline long se_runnable (struct sched_entity *se) { +if (se->sched_delayed) +return false; +return !!se->on_rq; }
Peter Zijlstra
maintainerPATCH 20/24 sched/fair: Avoid re-setting virtual deadline on migrations
In reply to Peter ZijlstraDuring OSPM24 Youssef noted that migrations are re-setting the virtual deadline. Notably everything that does a dequeue-enqueue, like setting nice, changing preferred numa-node, and a myriad of other random crap, will cause this to happen.
This shouldn't be. Preserve the relative virtual deadline across such dequeue/enqueue cycles.
--- include/linux/sched.h | 6 ++++-- kernel/sched/fair.c | 23 ++++++++++++++++++----- kernel/sched/features.h | 4 ++++ 3 files changed, 26 insertions(+), 7 deletions(-)
include/linux/sched.h +4 -2
@@ -542,8 +542,10 @@ struct sched_entity {u64 min_vruntime;struct list_head group_node; -unsigned int on_rq; -unsigned int sched_delayed; +unsigned char on_rq; +unsigned char sched_delayed; +unsigned char rel_deadline; +/* hole */ u64 exec_start;u64 sum_exec_runtime;
kernel/sched/fair.c +18 -5
@@ -5270,6 +5270,12 @@ place_entity(struct cfs_rq *cfs_rq, stru se->vruntime = vruntime - lag; +if (sched_feat (PLACE_REL_DEADLINE ) && se->rel_deadline) { + se->deadline += se->vruntime; + se->rel_deadline =0 ; +return ; + } +/* * When joining the competition; the existing tasks will be, * on average, halfway through their slice, as such start tasks
@@ -5382,6 +5388,8 @@ static __always_inline void return_cfs_rstatic bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { +bool sleep = flags &DEQUEUE_SLEEP ; +if (flags &DEQUEUE_DELAYED ) {/* * DEQUEUE_DELAYED is typically called from pick_next_entity()
@@ -5391,19 +5399,18 @@ dequeue_entity(struct cfs_rq *cfs_rq, stSCHED_WARN_ON (!se->sched_delayed); se->sched_delayed =0 ; }else { -bool sleep = flags &DEQUEUE_SLEEP ; - +bool delay = sleep;/* * DELAY_DEQUEUE relies on spurious wakeups, special task * states must not suffer spurious wakeups, excempt them. */ if (flags &DEQUEUE_SPECIAL ) - sleep = false; + delay = false; -SCHED_WARN_ON (sleep && se->sched_delayed); +SCHED_WARN_ON (delay && se->sched_delayed);update_curr (cfs_rq); -if (sched_feat (DELAY_DEQUEUE ) && sleep && +if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;
@@ -5434,6 +5441,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, stclear_buddies (cfs_rq, se);update_entity_lag (cfs_rq, se); +if (sched_feat (PLACE_REL_DEADLINE ) && !sleep) { + se->deadline -= se->vruntime; + se->rel_deadline =1 ; + } +if (se != cfs_rq->curr)__dequeue_entity (cfs_rq, se); se->on_rq =0 ;
@@ -13024,6 +13036,7 @@ static void switched_from_fair(struct rq* over the excursion into the new class. */ p->se.vlag =0 ; + p->se.rel_deadline =0 ; }static void switched_to_fair (struct rq *rq,struct task_struct *p)
kernel/sched/features.h +4 -0
@@ -10,6 +10,10 @@ SCHED_FEAT(PLACE_LAG, true)*/ SCHED_FEAT (PLACE_DEADLINE_INITIAL , true)/* +* Preserve relative virtual deadline on 'migration'. + */ +SCHED_FEAT (PLACE_REL_DEADLINE , true) +/* * Inhibit (wakeup) preemption until the current task has either matched the * 0-lag point or until is has exhausted it's slice. */
[tip: sched/core] sched/fair: Avoid re-setting virtual deadline on 'migrations'
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 82e9d0456e06cebe2c89f3c73cdbc9e3805e9437 Gitweb: https://git.kernel.org/tip/82e9d0456e06cebe2c89f3c73cdbc9e3805e9437 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 31 May 2024 15:49:40 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00
sched/fair: Avoid re-setting virtual deadline on 'migrations'
During OSPM24 Youssef noted that migrations are re-setting the virtual deadline. Notably everything that does a dequeue-enqueue, like setting nice, changing preferred numa-node, and a myriad of other random crap, will cause this to happen.
This shouldn't be. Preserve the relative virtual deadline across such dequeue/enqueue cycles.
--- include/linux/sched.h | 6 ++++-- kernel/sched/fair.c | 23 ++++++++++++++++++----- kernel/sched/features.h | 4 ++++ 3 files changed, 26 insertions(+), 7 deletions(-)
include/linux/sched.h +4 -2
@@ -544,8 +544,10 @@ struct sched_entity {u64 min_vruntime;struct list_head group_node; -unsigned int on_rq; -unsigned int sched_delayed; +unsigned char on_rq; +unsigned char sched_delayed; +unsigned char rel_deadline; +/* hole */ u64 exec_start;u64 sum_exec_runtime;
kernel/sched/fair.c +18 -5
@@ -5270,6 +5270,12 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) se->vruntime = vruntime - lag; +if (sched_feat (PLACE_REL_DEADLINE ) && se->rel_deadline) { + se->deadline += se->vruntime; + se->rel_deadline =0 ; +return ; + } +/* * When joining the competition; the existing tasks will be, * on average, halfway through their slice, as such start tasks
@@ -5382,23 +5388,24 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { +bool sleep = flags &DEQUEUE_SLEEP ; +update_curr (cfs_rq);if (flags &DEQUEUE_DELAYED ) {SCHED_WARN_ON (!se->sched_delayed); }else { -bool sleep = flags &DEQUEUE_SLEEP ; - +bool delay = sleep;/* * DELAY_DEQUEUE relies on spurious wakeups, special task * states must not suffer spurious wakeups, excempt them. */ if (flags &DEQUEUE_SPECIAL ) - sleep = false; + delay = false; -SCHED_WARN_ON (sleep && se->sched_delayed); +SCHED_WARN_ON (delay && se->sched_delayed); -if (sched_feat (DELAY_DEQUEUE ) && sleep && +if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {if (cfs_rq->next == se) cfs_rq->next =NULL ;
@@ -5429,6 +5436,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)clear_buddies (cfs_rq, se);update_entity_lag (cfs_rq, se); +if (sched_feat (PLACE_REL_DEADLINE ) && !sleep) { + se->deadline -= se->vruntime; + se->rel_deadline =1 ; + } +if (se != cfs_rq->curr)__dequeue_entity (cfs_rq, se); se->on_rq =0 ;
@@ -12992,6 +13004,7 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p)if (p->se.sched_delayed) {dequeue_task (rq, p,DEQUEUE_NOCLOCK |DEQUEUE_SLEEP ); p->se.sched_delayed =0 ; + p->se.rel_deadline =0 ;if (sched_feat (DELAY_ZERO ) && p->se.vlag >0 ) p->se.vlag =0 ; }
kernel/sched/features.h +4 -0
@@ -10,6 +10,10 @@ SCHED_FEAT(PLACE_LAG, true)*/ SCHED_FEAT (PLACE_DEADLINE_INITIAL , true)/* +* Preserve relative virtual deadline on 'migration'. + */ +SCHED_FEAT (PLACE_REL_DEADLINE , true) +/* * Inhibit (wakeup) preemption until the current task has either matched the * 0-lag point or until is has exhausted it's slice. */
Peter Zijlstra
maintainerPATCH 21/24 sched/eevdf: Allow shorter slices to wakeup-preempt
In reply to Peter ZijlstraPart of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT
1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |
100ms massive_intr 500us cyclictest PREEMPT_SHORT
1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.
--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 5 +++ 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq* XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i * this is probably good enough. */ -static void update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) +static bool update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) {if ((s64)(se->vruntime - se->deadline) <0 ) -return ; +return false;/* * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r/* * The task has consumed its request, reschedule. */ -if (cfs_rq->nr_running >1 ) { -resched_curr (rq_of (cfs_rq)); -clear_buddies (cfs_rq, se); - } +return true; }#include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(strudl_server_update (p->dl_server, delta_exec); } +static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +} + +static inline bool do_preempt_short (struct cfs_rq *cfs_rq, +struct sched_entity *pse,struct sched_entity *se) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (pse->slice >= se->slice) +return false; + +if (!entity_eligible (cfs_rq, pse)) +return false; + +if (entity_before (pse, se)) +return true; + +if (!entity_eligible (cfs_rq, se)) +return true; + +return false; +} +/* * Used by other classes to account runtime. */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cstruct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec; +bool resched;if (unlikely (!curr))return ;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *creturn ; curr->vruntime +=calc_delta_fair (delta_exec, curr); -update_deadline (cfs_rq, curr); + resched =update_deadline (cfs_rq, curr);update_min_vruntime (cfs_rq);if (entity_is_task (curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c }account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (rq->nr_running ==1 ) +return ; + +if (resched ||did_preempt_short (cfs_rq, curr)) { +resched_curr (rq); +clear_buddies (cfs_rq, curr); + } }static void update_curr_fair (struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st cfs_rq =cfs_rq_of (se);update_curr (cfs_rq);/* -* XXX pick_eevdf(cfs_rq) != se ? +* If @p has a shorter slice than current and @p is eligible, override +* current's slice protection in order to allow preemption. +* +* Note that even if @p does not turn out to be the most eligible +* task at this moment, current's slice protection will be lost. +*/ +if (do_preempt_short (cfs_rq, pse, se) && se->vlag == se->deadline) + se->vlag = se->deadline +1 ; + +/* +* If @p has become the most eligible task, force preemption. */ if (pick_eevdf (cfs_rq) == pse) goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for +* current. + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 78 lines · Peter Zijlstra
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.[mike: numbers]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>---
kernel/sched/fair.c| 64 ++++++++++++++++++++++++++++++++++++++++++------kernel/sched/features.h| 5 +++2 files changed, 61 insertions(+), 8 deletions(-)kernel/sched/fair.c +3
-6@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq* XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i * this is probably good enough. */ -static void update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) +static bool update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) {if ((s64)(se->vruntime - se->deadline) <0 ) -return ; +return false;/* * For EEVDF the virtual time slope is determined by w_i (iow. @@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r/* * The task has consumed its request, reschedule. */ -if (cfs_rq->nr_running >1 ) { -resched_curr (rq_of (cfs_rq)); -clear_buddies (cfs_rq, se); - } +return true; }#include "pelt.h" @@ -1134,6 +1131,38 @@ static inline void update_curr_task(strudl_server_update (p->dl_server, delta_exec); }
kernel/sched/fair.c static inline void update_curr_task(stru · line 1143
Quoted 10 lines
kernel/sched/fair.c +10
-0+static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +}
Hi perter
Can this be made more aggressive here? Something like , in the PREEMPT_SHORT
+ NO_RUN_TO_PARITY combination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.
+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr) +{
+ if (!sched_feat(PREEMPT_SHORT))
+ return false;
+
+ if (sched_feat(RUN_TO_PARITY) && curr->vlag == curr->deadline)
+ return false;
+
+ return !entity_eligible(cfs_rq, curr);
+}
Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.
https://lore.kernel.org/lkml/[email protected]/
thanks
Chunxin
Quoted 93 lines
kernel/sched/fair.c +43
-2+ +static inline bool do_preempt_short (struct cfs_rq *cfs_rq, +struct sched_entity *pse,struct sched_entity *se) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (pse->slice >= se->slice) +return false; + +if (!entity_eligible (cfs_rq, pse)) +return false; + +if (entity_before (pse, se)) +return true; + +if (!entity_eligible (cfs_rq, se)) +return true; + +return false; +} +/* * Used by other classes to account runtime. */ @@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cstruct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec; +bool resched;if (unlikely (!curr))return ;@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *creturn ; curr->vruntime +=calc_delta_fair (delta_exec, curr); -update_deadline (cfs_rq, curr); + resched =update_deadline (cfs_rq, curr);update_min_vruntime (cfs_rq);if (entity_is_task (curr)) {@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c }account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (rq->nr_running ==1 ) +return ; + +if (resched ||did_preempt_short (cfs_rq, curr)) { +resched_curr (rq); +clear_buddies (cfs_rq, curr); + } }static void update_curr_fair (struct rq *rq)@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st cfs_rq =cfs_rq_of (se);update_curr (cfs_rq);/* -* XXX pick_eevdf(cfs_rq) != se ? +* If @p has a shorter slice than current and @p is eligible, override +* current's slice protection in order to allow preemption. +* +* Note that even if @p does not turn out to be the most eligible +* task at this moment, current's slice protection will be lost. +*/ +if (do_preempt_short (cfs_rq, pse, se) && se->vlag == se->deadline) + se->vlag = se->deadline +1 ; + +/* +* If @p has become the most eligible task, force preemption. */ if (pick_eevdf (cfs_rq) == pse) goto preempt;kernel/sched/features.h +5
-0@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for +* current. + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 17 lines · Chunxin Zang
Review
Quoted 12 lines · Peter Zijlstra
(quoted diff) +10
-0+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr) +{ + if (!sched_feat(PREEMPT_SHORT)) + return false; + + if (curr->vlag == curr->deadline) + return false; + + return !entity_eligible(cfs_rq, curr); +}Can this be made more aggressive here? Something like , in the
PREEMPT_SHORT
+NO_RUN_TO_PARITYcombination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.
You are aware we have SCHED_DEADLINE for those, right?
Why can't you use that? and what can we do to fix that.
Review
Quoted 10 lines
(quoted diff) +10
-0+static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_entity *curr) +{ + if (!sched_feat(PREEMPT_SHORT)) + return false; + + if (sched_feat(RUN_TO_PARITY) && curr->vlag == curr->deadline) + return false; + + return !entity_eligible(cfs_rq, curr); +}
No, this will destroy the steady state schedule (where no tasks join/leave) and make it so that all tasks hug the lag=0 state arbitrarily close -- as allowed by the scheduling quanta.
Yes, it will get you better latency, because nobody gets to actually run it's requested slice.
The goal really is for tasks to get their request -- and yes that means you get to wait. PREEMPT_SHORT is already an exception to this rule, and it is very specifically limited to wake-ups so as to retain as much of the intended behaviour as possible.
Quoted 4 lines
Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.
I've made it the below. Does that work for you?
---
Subject: sched/eevdf: Allow shorter slices to wakeup-preempt
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 26 14:32:32 CEST 2023
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT
1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |
100ms massive_intr 500us cyclictest PREEMPT_SHORT
1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.
Chunxin contributed did_preempt_short() where a task that lost slice protection from PREEMPT_SHORT gets rescheduled once it becomes in-eligible.
--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 5 +++ 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq* XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i * this is probably good enough. */ -static void update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) +static bool update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) {if ((s64)(se->vruntime - se->deadline) <0 ) -return ; +return false;/* * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r/* * The task has consumed its request, reschedule. */ -if (cfs_rq->nr_running >1 ) { -resched_curr (rq_of (cfs_rq)); -clear_buddies (cfs_rq, se); - } +return true; }#include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(strudl_server_update (p->dl_server, delta_exec); } +static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +} + +static inline bool do_preempt_short (struct cfs_rq *cfs_rq, +struct sched_entity *pse,struct sched_entity *se) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (pse->slice >= se->slice) +return false; + +if (!entity_eligible (cfs_rq, pse)) +return false; + +if (entity_before (pse, se)) +return true; + +if (!entity_eligible (cfs_rq, se)) +return true; + +return false; +} +/* * Used by other classes to account runtime. */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cstruct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec; +bool resched;if (unlikely (!curr))return ;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *creturn ; curr->vruntime +=calc_delta_fair (delta_exec, curr); -update_deadline (cfs_rq, curr); + resched =update_deadline (cfs_rq, curr);update_min_vruntime (cfs_rq);if (entity_is_task (curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c }account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (rq->nr_running ==1 ) +return ; + +if (resched ||did_preempt_short (cfs_rq, curr)) { +resched_curr (rq); +clear_buddies (cfs_rq, curr); + } }static void update_curr_fair (struct rq *rq)
@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st cfs_rq =cfs_rq_of (se);update_curr (cfs_rq);/* -* XXX pick_eevdf(cfs_rq) != se ? +* If @p has a shorter slice than current and @p is eligible, override +* current's slice protection in order to allow preemption. +* +* Note that even if @p does not turn out to be the most eligible +* task at this moment, current's slice protection will be lost. +*/ +if (do_preempt_short (cfs_rq, pse, se) && se->vlag == se->deadline) + se->vlag = se->deadline +1 ; + +/* +* If @p has become the most eligible task, force preemption. */ if (pick_eevdf (cfs_rq) == pse) goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for +* current. + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 42 lines · Peter Zijlstra
Quoted 17 lines · Chunxin Zang
kernel/sched/fair.c static inline void update_curr_task(stru · line 1143
Quoted 12 lines · Peter Zijlstra
kernel/sched/fair.c +10
-0+static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +}Can this be made more aggressive here? Something like , in the
PREEMPT_SHORT
+NO_RUN_TO_PARITYcombination, it could break the first deadline of the current task. This can achieve better latency benefits in certain embedded scenarios, such as high-priority periodic tasks.You are aware we have
SCHED_DEADLINEfor those, right?Why can't you use that? and what can we do to fix that.
kernel/sched/fair.c static inline void update_curr_task(stru · line 1143
Quoted 10 lines
kernel/sched/fair.c +10
-0+static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (sched_feat (RUN_TO_PARITY ) && curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +}No, this will destroy the steady state schedule (where no tasks join/leave) and make it so that all tasks hug the lag=0 state arbitrarily close -- as allowed by the scheduling quanta.
Quoted 7 lines
Yes, it will get you better latency, because nobody gets to actually run it's requested slice.
The goal really is for tasks to get their request -- and yes that means you get to wait.
PREEMPT_SHORTis already an exception to this rule, and it is very specifically limited to wake-ups so as to retain as much of the intended behaviour as possible.
I think I understand your point now. That approach does indeed seem more reasonable. Next, I will try using 'request slice' to conduct more tests in my scenario.
Quoted 7 lines
Quoted 5 lines
Additionally, if possible, could you please include my name in this patch? I spent over a month finding this solution and conducting the tests, and I hope to leave some trace of my efforts during that time. This is also one of the reasons why I love Linux and am eager to contribute to open source. I would be extremely grateful.
I've made it the below. Does that work for you?
Heartfelt thanks :)
Chunxin
Quoted 187 lines
---
Subject: sched/eevdf: Allow shorter slices to wakeup-preempt
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 26 14:32:32 CEST 2023Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.Chunxin contributed
did_preempt_short()where a task that lost slice protection fromPREEMPT_SHORTgets rescheduled once it becomes in-eligible.[mike: numbers]Co-Developed-by Chunxin Zang <[email protected]>Signed-off-by Chunxin Zang <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>---
kernel/sched/fair.c| 64 ++++++++++++++++++++++++++++++++++++++++++------kernel/sched/features.h| 5 +++2 files changed, 61 insertions(+), 8 deletions(-)kernel/sched/fair.c +56
-8@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq* XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i * this is probably good enough. */ -static void update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) +static bool update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) {if ((s64)(se->vruntime - se->deadline) <0 ) -return ; +return false;/* * For EEVDF the virtual time slope is determined by w_i (iow. @@ -993,10 +993,7 @@ static void update_deadline(struct cfs_r/* * The task has consumed its request, reschedule. */ -if (cfs_rq->nr_running >1 ) { -resched_curr (rq_of (cfs_rq)); -clear_buddies (cfs_rq, se); - } +return true; }#include "pelt.h" @@ -1134,6 +1131,38 @@ static inline void update_curr_task(strudl_server_update (p->dl_server, delta_exec); } +static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +} + +static inline bool do_preempt_short (struct cfs_rq *cfs_rq, +struct sched_entity *pse,struct sched_entity *se) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (pse->slice >= se->slice) +return false; + +if (!entity_eligible (cfs_rq, pse)) +return false; + +if (entity_before (pse, se)) +return true; + +if (!entity_eligible (cfs_rq, se)) +return true; + +return false; +} +/* * Used by other classes to account runtime. */ @@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cstruct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec; +bool resched;if (unlikely (!curr))return ;@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *creturn ; curr->vruntime +=calc_delta_fair (delta_exec, curr); -update_deadline (cfs_rq, curr); + resched =update_deadline (cfs_rq, curr);update_min_vruntime (cfs_rq);if (entity_is_task (curr)) {@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *c }account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (rq->nr_running ==1 ) +return ; + +if (resched ||did_preempt_short (cfs_rq, curr)) { +resched_curr (rq); +clear_buddies (cfs_rq, curr); + } }static void update_curr_fair (struct rq *rq)@@ -8611,7 +8649,17 @@ static void check_preempt_wakeup_fair(st cfs_rq =cfs_rq_of (se);update_curr (cfs_rq);/* -* XXX pick_eevdf(cfs_rq) != se ? +* If @p has a shorter slice than current and @p is eligible, override +* current's slice protection in order to allow preemption. +* +* Note that even if @p does not turn out to be the most eligible +* task at this moment, current's slice protection will be lost. +*/ +if (do_preempt_short (cfs_rq, pse, se) && se->vlag == se->deadline) + se->vlag = se->deadline +1 ; + +/* +* If @p has become the most eligible task, force preemption. */ if (pick_eevdf (cfs_rq) == pse) goto preempt;kernel/sched/features.h +5
-0@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for +* current. + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
In reply to Peter ZijlstraHi Peter,
Quoted 30 lines · Peter Zijlstra
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |
Quoted 8 lines
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.[mike: numbers]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>
Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/
The RESPECT_SLICE is introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.
echo RESPECT_SLICE > /sys/kernel/debug/sched/features
echo RUN_TO_PARITY > /sys/kernel/debug/sched/features @task_duration_usecs_before_preempted:
[2, 4) 8732 |@@@ | [4, 8) 109400 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 95815 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 110647 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 131298 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [64, 128) 132566 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [128, 256) 82095 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [256, 512) 33771 |@@@@@@@@@@@@@ | [512, 1K) 24180 |@@@@@@@@@ | [1K, 2K) 31056 |@@@@@@@@@@@@ | [2K, 4K) 117533 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [4K, 8K) 4472 |@ | [8K, 16K) 1149 | | [16K, 32K) 289 | | [32K, 64K) 110 | | [64K, 128K) 20 | | [128K, 256K) 4 | |
echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features @task_duration_usecs_before_preempted:
[4, 8) 1 | | [8, 16) 12 | | [16, 32) 20 | | [32, 64) 38 | | [64, 128) 64 | | [128, 256) 98 | | [256, 512) 248 | | [512, 1K) 1196 | | [1K, 2K) 3456 | | [2K, 4K) 417269 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4K, 8K) 22893 |@@ | [8K, 16K) 7818 | | [16K, 32K) 1471 | | [32K, 64K) 373 | | [64K, 128K) 96 | | [128K, 256K) 3 | |
We can see that without the fix, the task will be preempted and can not reach its time slice budget(we enlarge its slice).
May I know if we can put that patch into this series please?
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 42 lines · Chen Yu
Hi Peter,
Quoted 30 lines · Peter Zijlstra
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |Quoted 8 lines
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.[mike: numbers]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>
Quoted 8 lines
Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/The
RESPECT_SLICEis introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.
So I *think* that running as SCHED_BATCH gets you exactly that behaviour, no?
Re: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Chen Yu
Hi Peter,
Quoted 30 lines · Peter Zijlstra
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |Quoted 8 lines
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.[mike: numbers]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/
Quoted 6 lines
Quoted 3 lines
The
RESPECT_SLICEis introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.So I *think* that running as
SCHED_BATCHgets you exactly that behaviour, no?
SCHED_BATCH should work as it avoids the wakeup preemption as much as possible. Except that RESPECT_SLICE considers the cgroup hierarchical to check if the current sched_entity has used up its slice, which seems to be less aggressive.
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 21/24] sched/eevdf: Allow shorter slices to wakeup-preempt
Quoted 53 lines · Chen Yu
Quoted 52 lines · Peter Zijlstra
Quoted 50 lines · Chen Yu
Hi Peter,
Quoted 30 lines · Peter Zijlstra
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms
massive_intr500us cyclictestNO_PREEMPT_SHORT1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |100ms
massive_intr500us cyclictestPREEMPT_SHORT1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |Quoted 8 lines
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the
massive_intr(long slice) gets more context switches and a slight increase in sum-delay.[mike: numbers]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Mike Galbraith <[email protected]>Besides this short preemption, it seems that an important patch is missing from this patch set, that was originally from Prateek and you refined it to fix the current task's false negative eligibility:
https://lore.kernel.org/lkml/[email protected]/The
RESPECT_SLICEis introduced to honor the current task's slice during wakeup preemption. Without it we got reported that over-preemption and performance downgrading are observed when running SPECjbb on servers.
Quoted 6 lines
Quoted 2 lines
So I *think* that running as
SCHED_BATCHgets you exactly that behaviour, no?
SCHED_BATCHshould work as it avoids the wakeup preemption as much as possible. Except thatRESPECT_SLICEconsiders the cgroup hierarchical to check if the currentsched_entityhas used up its slice, which seems to be less aggressive.
Note that update_deadline() will trigger a resched at the end up a slice regardless -- this is driven from update_curr() and also invoked from any preemption.
[tip: sched/core] sched/eevdf: Allow shorter slices to wakeup-preempt
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 85e511df3cec46021024176672a748008ed135bf Gitweb: https://git.kernel.org/tip/85e511df3cec46021024176672a748008ed135bf Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 26 Sep 2023 14:32:32 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00
sched/eevdf: Allow shorter slices to wakeup-preempt
Part of the reason to have shorter slices is to improve responsiveness. Allow shorter slices to preempt longer slices on wakeup.
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms |
100ms massive_intr 500us cyclictest NO_PREEMPT_SHORT
1 massive_intr:(5) | 846018.956 ms | 779188 | avg: 0.273 ms | max: 58.337 ms | sum:212545.245 ms | 2 massive_intr:(5) | 853450.693 ms | 792269 | avg: 0.275 ms | max: 71.193 ms | sum:218263.588 ms | 3 massive_intr:(5) | 843888.920 ms | 771456 | avg: 0.277 ms | max: 92.405 ms | sum:213353.221 ms | 1 chromium-browse:(8) | 53015.889 ms | 131766 | avg: 0.463 ms | max: 36.341 ms | sum:60959.230 ms | 2 chromium-browse:(8) | 53864.088 ms | 136962 | avg: 0.480 ms | max: 27.091 ms | sum:65687.681 ms | 3 chromium-browse:(9) | 53637.904 ms | 132637 | avg: 0.481 ms | max: 24.756 ms | sum:63781.673 ms | 1 cyclictest:(5) | 12615.604 ms | 639689 | avg: 0.471 ms | max: 32.272 ms | sum:301351.094 ms | 2 cyclictest:(5) | 12511.583 ms | 642578 | avg: 0.448 ms | max: 44.243 ms | sum:287632.830 ms | 3 cyclictest:(5) | 12545.867 ms | 635953 | avg: 0.475 ms | max: 25.530 ms | sum:302374.658 ms |
100ms massive_intr 500us cyclictest PREEMPT_SHORT
1 massive_intr:(5) | 839843.919 ms | 837384 | avg: 0.264 ms | max: 74.366 ms | sum:221476.885 ms | 2 massive_intr:(5) | 852449.913 ms | 845086 | avg: 0.252 ms | max: 68.162 ms | sum:212595.968 ms | 3 massive_intr:(5) | 839180.725 ms | 836883 | avg: 0.266 ms | max: 69.742 ms | sum:222812.038 ms | 1 chromium-browse:(11) | 54591.481 ms | 138388 | avg: 0.458 ms | max: 35.427 ms | sum:63401.508 ms | 2 chromium-browse:(8) | 52034.541 ms | 132276 | avg: 0.436 ms | max: 31.826 ms | sum:57732.958 ms | 3 chromium-browse:(8) | 55231.771 ms | 141892 | avg: 0.469 ms | max: 27.607 ms | sum:66538.697 ms | 1 cyclictest:(5) | 13156.391 ms | 667412 | avg: 0.373 ms | max: 38.247 ms | sum:249174.502 ms | 2 cyclictest:(5) | 12688.939 ms | 665144 | avg: 0.374 ms | max: 33.548 ms | sum:248509.392 ms | 3 cyclictest:(5) | 13475.623 ms | 669110 | avg: 0.370 ms | max: 37.819 ms | sum:247673.390 ms |
As per the numbers the, this makes cyclictest (short slice) it's max-delay more consistent and consistency drops the sum-delay. The trade-off is that the massive_intr (long slice) gets more context switches and a slight increase in sum-delay.
Chunxin contributed did_preempt_short() where a task that lost slice protection from PREEMPT_SHORT gets rescheduled once it becomes in-eligible.
--- kernel/sched/fair.c | 64 +++++++++++++++++++++++++++++++++++----- kernel/sched/features.h | 5 +++- 2 files changed, 61 insertions(+), 8 deletions(-)
kernel/sched/fair.c +56 -8
@@ -973,10 +973,10 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se);* XXX: strictly: vd_i += N*r_i/w_i such that: vd_i > ve_i * this is probably good enough. */ -static void update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) +static bool update_deadline (struct cfs_rq *cfs_rq,struct sched_entity *se) {if ((s64)(se->vruntime - se->deadline) <0 ) -return ; +return false;/* * For EEVDF the virtual time slope is determined by w_i (iow.
@@ -993,10 +993,7 @@ static void update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)/* * The task has consumed its request, reschedule. */ -if (cfs_rq->nr_running >1 ) { -resched_curr (rq_of (cfs_rq)); -clear_buddies (cfs_rq, se); - } +return true; }#include "pelt.h"
@@ -1134,6 +1131,38 @@ static inline void update_curr_task(struct task_struct *p, s64 delta_exec)dl_server_update (p->dl_server, delta_exec); } +static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (curr->vlag == curr->deadline) +return false; + +return !entity_eligible (cfs_rq, curr); +} + +static inline bool do_preempt_short (struct cfs_rq *cfs_rq, +struct sched_entity *pse,struct sched_entity *se) +{ +if (!sched_feat (PREEMPT_SHORT )) +return false; + +if (pse->slice >= se->slice) +return false; + +if (!entity_eligible (cfs_rq, pse)) +return false; + +if (entity_before (pse, se)) +return true; + +if (!entity_eligible (cfs_rq, se)) +return true; + +return false; +} +/* * Used by other classes to account runtime. */
@@ -1157,6 +1186,7 @@ static void update_curr(struct cfs_rq *cfs_rq)struct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec; +bool resched;if (unlikely (!curr))return ;
@@ -1166,7 +1196,7 @@ static void update_curr(struct cfs_rq *cfs_rq)return ; curr->vruntime +=calc_delta_fair (delta_exec, curr); -update_deadline (cfs_rq, curr); + resched =update_deadline (cfs_rq, curr);update_min_vruntime (cfs_rq);if (entity_is_task (curr)) {
@@ -1184,6 +1214,14 @@ static void update_curr(struct cfs_rq *cfs_rq) }account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (rq->nr_running ==1 ) +return ; + +if (resched ||did_preempt_short (cfs_rq, curr)) { +resched_curr (rq); +clear_buddies (cfs_rq, curr); + } }static void update_curr_fair (struct rq *rq)
@@ -8605,7 +8643,17 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int cfs_rq =cfs_rq_of (se);update_curr (cfs_rq);/* -* XXX pick_eevdf(cfs_rq) != se ? +* If @p has a shorter slice than current and @p is eligible, override +* current's slice protection in order to allow preemption. +* +* Note that even if @p does not turn out to be the most eligible +* task at this moment, current's slice protection will be lost. +*/ +if (do_preempt_short (cfs_rq, pse, se) && se->vlag == se->deadline) + se->vlag = se->deadline +1 ; + +/* +* If @p has become the most eligible task, force preemption. */ if (pick_eevdf (cfs_rq) == pse) goto preempt;
kernel/sched/features.h +5 -0
@@ -18,6 +18,11 @@ SCHED_FEAT(PLACE_REL_DEADLINE, true)* 0-lag point or until is has exhausted it's slice. */ SCHED_FEAT (RUN_TO_PARITY , true) +/* +* Allow wakeup of tasks with a shorter slice to cancel RESPECT_SLICE for +* current. + */ +SCHED_FEAT (PREEMPT_SHORT , true)/* * Prefer to schedule the task we woke last (assuming it failed
Peter Zijlstra
maintainerPATCH 22/24 sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
In reply to Peter ZijlstraAllow applications to directly set a suggested request/slice length using sched_attr::sched_runtime.
The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
For all the following examples assume a scheduling quantum of 8, and for consistency all examples have W=4:
{A,B,C,D}(w=1,r=8):
ABCD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+*------+-------+--- ---+--*----+-------+---
t=2, V=5.5 t=3, V=7.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+----*--+-------+--- ---+------*+-------+---
Note: 4 identical tasks in FIFO order
~~~
{A,B}(w=1,r=16) C(w=2,r=16)
AACCBBCC...
+---+---+---+---
t=0, V=1.25 t=2, V=5.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=4, V=8.25 t=6, V=12.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+-------*-------+--- ---+-------+---*---+---
Note: 1 heavy task -- because q=8, double r such that the deadline of the w=2
task doesn't go below q.
Note: observe the full schedule becomes: W*max(r_i/w_i) = 4*2q = 8q in length.
Note: the period of the heavy task is half the full period at:
W*(r_i/w_i) = 4*(2q/2) = 4q
~~~
{A,C,D}(w=1,r=16) B(w=1,r=8):
BAACCBDD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |--------------< A |---------------<
B |------< B |------<
C |--------------< C |--------------<
D |--------------< D |--------------<
---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.5 t=5, V=11.5
A |---------------< A |---------------< B |------< B |------< C |--------------< C |--------------< D |--------------< D |--------------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.5
A |---------------<
B |------<
C |--------------<
D |--------------<
---+-------+----*--+---
Note: 1 short task -- again double r so that the deadline of the short task
won't be below q. Made B short because its not the leftmost task, but is eligible with the 0,1,2,3 spread.
Note: like with the heavy task, the period of the short task observes:
W*(r_i/w_i) = 4*(1q/1) = 4q
~~~
A(w=1,r=16) B(w=1,r=8) C(w=2,r=16)
BCCAABCC...
+---+---+---+---
t=0, V=1.25 t=1, V=3.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.25 t=5, V=11.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.25
A |--------------<
B |------<
C |------<
---+-------+----*--+---
Note: 1 heavy and 1 short task -- combine them all.
Note: both the short and heavy task end up with a period of 4q
~~~
A(w=1,r=16) B(w=2,r=16) C(w=1,r=8)
BBCAABBC...
+---+---+---+---
t=0, V=1 t=2, V=5
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=3, V=7 t=5, V=11
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=7, V=15
A |--------------<
B |------<
C |------<
---+-------+------*+---
Note: as before but permuted
~~~
Quoted 1 line
From all this it can be deduced that, for the steady state:
- the total period (P) of a schedule is: W*max(r_i/w_i)
- the average period of a task is: W*(r_i/w_i)
- each task obtains the fair share: w_i/W of each full period P
--- include/linux/sched.h | 1 + kernel/sched/core.c | 4 +++- kernel/sched/debug.c | 3 ++- kernel/sched/fair.c | 6 ++++-- kernel/sched/syscalls.c | 29 +++++++++++++++++++++++------ 5 files changed, 33 insertions(+), 10 deletions(-)
include/linux/sched.h +1 -0
@@ -544,6 +544,7 @@ struct sched_entity {unsigned char on_rq;unsigned char sched_delayed;unsigned char rel_deadline; +unsigned char custom_slice;/* hole */ u64 exec_start;
kernel/sched/core.c +3 -1
@@ -4347,7 +4347,6 @@ static void __sched_fork(unsigned long c p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ; - p->se.slice = sysctl_sched_base_slice;INIT_LIST_HEAD (&p->se.group_node);/* A delayed task cannot be in clone(). */
@@ -4600,6 +4599,8 @@ int sched_fork(unsigned long clone_flags p->prio = p->normal_prio = p->static_prio;set_load_weight (p, false); + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice;/* * We don't need the reset flag anymore after the fork. It has
@@ -8328,6 +8329,7 @@ void __init sched_init(void) }set_load_weight (&init_task, false); + init_task.se.slice = sysctl_sched_base_slice,/* * The boot idle thread does lazy MMU switching as well:
kernel/sched/debug.c +2 -1
@@ -580,11 +580,12 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), SPLIT_NS(p->se.vruntime), entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), + p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice), SPLIT_NS(p->se.sum_exec_runtime), (long long)(p->nvcsw + p->nivcsw),
kernel/sched/fair.c +4 -2
@@ -995,7 +995,8 @@ static void update_deadline(struct cfs_r* nice) while the request time r_i is determined by * sysctl_sched_base_slice. */ - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice;/* * EEVDF: vd_i = ve_i + r_i / w_i
@@ -5190,7 +5191,8 @@ place_entity(struct cfs_rq *cfs_rq, struu64 vslice, vruntime =avg_vruntime (cfs_rq);s64 lag =0 ; - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se);/*
kernel/sched/syscalls.c +23 -6
@@ -401,10 +401,20 @@ static void __setscheduler_params(struct p->policy = policy; -if (dl_policy (policy)) +if (dl_policy (policy)) {__setparam_dl (p, attr); -else if (fair_policy (policy)) + }else if (fair_policy (policy)) { p->static_prio =NICE_TO_PRIO (attr->sched_nice); +if (attr->sched_runtime) { + p->se.custom_slice =1 ; + p->se.slice =clamp_t (u64, attr->sched_runtime, +NSEC_PER_MSEC /10 ,/* HZ=1000 * 10 */ +NSEC_PER_MSEC *100 );/* HZ=100 / 10 */ + }else { + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice; + } + }/* * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -700,7 +710,9 @@ int __sched_setscheduler(struct task_str* but store a possible modification of reset_on_fork. */ if (unlikely (policy == p->policy)) { -if (fair_policy (policy) && attr->sched_nice !=task_nice (p)) +if (fair_policy (policy) && + (attr->sched_nice !=task_nice (p) || + (attr->sched_runtime != p->se.slice))) goto change;if (rt_policy (policy) && attr->sched_priority != p->rt_priority) goto change;
@@ -846,6 +858,9 @@ static int _sched_setscheduler(struct ta .sched_nice = PRIO_TO_NICE(p->static_prio), }; +if (p->se.custom_slice) + attr.sched_runtime = p->se.slice; +/* Fixup the legacy SCHED_RESET_ON_FORK hack. */ if ((policy !=SETPARAM_POLICY ) && (policy &SCHED_RESET_ON_FORK )) { attr.sched_flags |=SCHED_FLAG_RESET_ON_FORK ;
@@ -1012,12 +1027,14 @@ static int sched_copy_attr(struct sched_static void get_params (struct task_struct *p,struct sched_attr *attr) { -if (task_has_dl_policy (p)) +if (task_has_dl_policy (p)) {__getparam_dl (p, attr); -else if (task_has_rt_policy (p)) + }else if (task_has_rt_policy (p)) { attr->sched_priority = p->rt_priority; -else + }else { attr->sched_nice =task_nice (p); + attr->sched_runtime = p->se.slice; + } }/**
[tip: sched/core] sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 857b158dc5e81c6de795ef6be006eed146098fc6 Gitweb: https://git.kernel.org/tip/857b158dc5e81c6de795ef6be006eed146098fc6 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 22 May 2023 13:46:30 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:45 +02:00
sched/eevdf: Use sched_attr::sched_runtime to set request/slice suggestion
Allow applications to directly set a suggested request/slice length using sched_attr::sched_runtime.
The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms] which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
Applications should strive to use their periodic runtime at a high confidence interval (95%+) as the target slice. Using a smaller slice will introduce undue preemptions, while using a larger value will increase latency.
For all the following examples assume a scheduling quantum of 8, and for consistency all examples have W=4:
{A,B,C,D}(w=1,r=8):
ABCD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+*------+-------+--- ---+--*----+-------+---
t=2, V=5.5 t=3, V=7.5
A |------< A |------<
B |------< B |------<
C |------< C |------<
D |------< D |------<
---+----*--+-------+--- ---+------*+-------+---
Note: 4 identical tasks in FIFO order
~~~
{A,B}(w=1,r=16) C(w=2,r=16)
AACCBBCC...
+---+---+---+---
t=0, V=1.25 t=2, V=5.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=4, V=8.25 t=6, V=12.25
A |--------------< A |--------------< B |--------------< B |--------------< C |------< C |------< ---+-------*-------+--- ---+-------+---*---+---
Note: 1 heavy task -- because q=8, double r such that the deadline of the w=2
task doesn't go below q.
Note: observe the full schedule becomes: W*max(r_i/w_i) = 4*2q = 8q in length.
Note: the period of the heavy task is half the full period at:
W*(r_i/w_i) = 4*(2q/2) = 4q
~~~
{A,C,D}(w=1,r=16) B(w=1,r=8):
BAACCBDD...
+---+---+---+---
t=0, V=1.5 t=1, V=3.5
A |--------------< A |---------------<
B |------< B |------<
C |--------------< C |--------------<
D |--------------< D |--------------<
---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.5 t=5, V=11.5
A |---------------< A |---------------< B |------< B |------< C |--------------< C |--------------< D |--------------< D |--------------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.5
A |---------------<
B |------<
C |--------------<
D |--------------<
---+-------+----*--+---
Note: 1 short task -- again double r so that the deadline of the short task
won't be below q. Made B short because its not the leftmost task, but is eligible with the 0,1,2,3 spread.
Note: like with the heavy task, the period of the short task observes:
W*(r_i/w_i) = 4*(1q/1) = 4q
~~~
A(w=1,r=16) B(w=1,r=8) C(w=2,r=16)
BCCAABCC...
+---+---+---+---
t=0, V=1.25 t=1, V=3.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+--*----+-------+---
t=3, V=7.25 t=5, V=11.25
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=6, V=13.25
A |--------------<
B |------<
C |------<
---+-------+----*--+---
Note: 1 heavy and 1 short task -- combine them all.
Note: both the short and heavy task end up with a period of 4q
~~~
A(w=1,r=16) B(w=2,r=16) C(w=1,r=8)
BBCAABBC...
+---+---+---+---
t=0, V=1 t=2, V=5
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+*------+-------+--- ---+----*--+-------+---
t=3, V=7 t=5, V=11
A |--------------< A |--------------< B |------< B |------< C |------< C |------< ---+------*+-------+--- ---+-------+--*----+---
t=7, V=15
A |--------------<
B |------<
C |------<
---+-------+------*+---
Note: as before but permuted
~~~
Quoted 1 line
From all this it can be deduced that, for the steady state:
- the total period (P) of a schedule is: W*max(r_i/w_i)
- the average period of a task is: W*(r_i/w_i)
- each task obtains the fair share: w_i/W of each full period P
--- include/linux/sched.h | 1 + kernel/sched/core.c | 4 +++- kernel/sched/debug.c | 3 ++- kernel/sched/fair.c | 6 ++++-- kernel/sched/syscalls.c | 29 +++++++++++++++++++++++------ 5 files changed, 33 insertions(+), 10 deletions(-)
include/linux/sched.h +1 -0
@@ -547,6 +547,7 @@ struct sched_entity {unsigned char on_rq;unsigned char sched_delayed;unsigned char rel_deadline; +unsigned char custom_slice;/* hole */ u64 exec_start;
kernel/sched/core.c +3 -1
@@ -4390,7 +4390,6 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ; - p->se.slice = sysctl_sched_base_slice;INIT_LIST_HEAD (&p->se.group_node);/* A delayed task cannot be in clone(). */
@@ -4643,6 +4642,8 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio;set_load_weight (p, false); + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice;/* * We don't need the reset flag anymore after the fork. It has
@@ -8412,6 +8413,7 @@ void __init sched_init(void) }set_load_weight (&init_task, false); + init_task.se.slice = sysctl_sched_base_slice,/* * The boot idle thread does lazy MMU switching as well:
kernel/sched/debug.c +2 -1
@@ -739,11 +739,12 @@ print_task(struct seq_file *m, struct rq *rq, struct task_struct *p)else SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, "%15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), SPLIT_NS(p->se.vruntime), entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), + p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice), SPLIT_NS(p->se.sum_exec_runtime), (long long)(p->nvcsw + p->nivcsw),
kernel/sched/fair.c +4 -2
@@ -983,7 +983,8 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)* nice) while the request time r_i is determined by * sysctl_sched_base_slice. */ - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice;/* * EEVDF: vd_i = ve_i + r_i / w_i
@@ -5227,7 +5228,8 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)u64 vslice, vruntime =avg_vruntime (cfs_rq);s64 lag =0 ; - se->slice = sysctl_sched_base_slice; +if (!se->custom_slice) + se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se);/*
kernel/sched/syscalls.c +23 -6
@@ -401,10 +401,20 @@ static void __setscheduler_params(struct task_struct *p, p->policy = policy; -if (dl_policy (policy)) +if (dl_policy (policy)) {__setparam_dl (p, attr); -else if (fair_policy (policy)) + }else if (fair_policy (policy)) { p->static_prio =NICE_TO_PRIO (attr->sched_nice); +if (attr->sched_runtime) { + p->se.custom_slice =1 ; + p->se.slice =clamp_t (u64, attr->sched_runtime, +NSEC_PER_MSEC /10 ,/* HZ=1000 * 10 */ +NSEC_PER_MSEC *100 );/* HZ=100 / 10 */ + }else { + p->se.custom_slice =0 ; + p->se.slice = sysctl_sched_base_slice; + } + }/* * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -700,7 +710,9 @@ recheck:* but store a possible modification of reset_on_fork. */ if (unlikely (policy == p->policy)) { -if (fair_policy (policy) && attr->sched_nice !=task_nice (p)) +if (fair_policy (policy) && + (attr->sched_nice !=task_nice (p) || + (attr->sched_runtime != p->se.slice))) goto change;if (rt_policy (policy) && attr->sched_priority != p->rt_priority) goto change;
@@ -846,6 +858,9 @@ static int _sched_setscheduler(struct task_struct *p, int policy, .sched_nice = PRIO_TO_NICE(p->static_prio), }; +if (p->se.custom_slice) + attr.sched_runtime = p->se.slice; +/* Fixup the legacy SCHED_RESET_ON_FORK hack. */ if ((policy !=SETPARAM_POLICY ) && (policy &SCHED_RESET_ON_FORK )) { attr.sched_flags |=SCHED_FLAG_RESET_ON_FORK ;
@@ -1012,12 +1027,14 @@ err_size:static void get_params (struct task_struct *p,struct sched_attr *attr) { -if (task_has_dl_policy (p)) +if (task_has_dl_policy (p)) {__getparam_dl (p, attr); -else if (task_has_rt_policy (p)) + }else if (task_has_rt_policy (p)) { attr->sched_priority = p->rt_priority; -else + }else { attr->sched_nice =task_nice (p); + attr->sched_runtime = p->se.slice; + } }/**
Peter Zijlstra
maintainerPATCH 23/24 sched/eevdf: Propagate min_slice up the cgroup hierarchy
In reply to Peter ZijlstraIn the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the min_slice up the hierarchy.
This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.
--- include/linux/sched.h | 1 kernel/sched/fair.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {struct rb_node run_node;u64 deadline;u64 min_vruntime; +u64 min_slice;struct list_head group_node;unsigned char on_rq;
kernel/sched/fair.c +56 -1
129 lines not on this page · raw message
[tip: sched/core] sched/eevdf: Propagate min_slice up the cgroup hierarchy
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: aef6987d89544d63a47753cf3741cabff0b5574c Gitweb: https://git.kernel.org/tip/aef6987d89544d63a47753cf3741cabff0b5574c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 20 Jun 2024 13:16:49 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:46 +02:00
sched/eevdf: Propagate min_slice up the cgroup hierarchy
In the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the min_slice up the hierarchy.
This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.
--- include/linux/sched.h | 1 +- kernel/sched/fair.c | 57 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {struct rb_node run_node;u64 deadline;u64 min_vruntime; +u64 min_slice;struct list_head group_node;unsigned char on_rq;
kernel/sched/fair.c +56 -1
129 lines not on this page · raw message
Re: [PATCH 23/24] sched/eevdf: Propagate min_slice up the cgroup hierarchy
In reply to Peter ZijlstraHi Peter,
Quoted 3 lines · Peter Zijlstra
In the absence of an explicit cgroup slice configureation, make mixed slice length work with cgroups by propagating the
min_sliceup the hierarchy.
Will it be acceptable to introduce a cgroup interface (e.g., sth. like cpu.fair_runtime or cpu.fair_slice) to overwrite the caculated min_slice? This could be useful in container scenarios.
Quoted 27 lines
This ensures the cgroup entity gets timely service to service its entities that have this timing constraint set on them.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/sched.h | 1 kernel/sched/fair.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 1 deletion(-)include/linux/sched.h +1
-0@@ -542,6 +542,7 @@ struct sched_entity {struct rb_node run_node;u64 deadline;u64 min_vruntime; +u64 min_slice;struct list_head group_node;unsigned char on_rq;kernel/sched/fair.c +1
-0@@ -782,6 +782,21 @@ static void update_min_vruntime(struct c cfs_rq->min_vruntime =__update_min_vruntime (cfs_rq, vruntime); } +static inline u64 cfs_rq_min_slice (struct cfs_rq *cfs_rq)
kernel/sched/fair.c static void update_min_vruntime(struct c · line 795
Quoted 10 lines
kernel/sched/fair.c +10
-0+{ +struct sched_entity *root =__pick_root_entity (cfs_rq); +struct sched_entity *curr = cfs_rq->curr; +u64 min_slice = ~0ULL ; + +if (curr && curr->on_rq) + min_slice = curr->slice; + +if (root) + min_slice =min (min_slice, root->min_slice);
If a sched_delayed se keeps the min_slice, then the parent se will receive a shorter slice (from the sched_delayed se) than it should be. Is it a problem?
Thanks.
Peter Zijlstra
maintainerRFC PATCH 24/24 sched/time: Introduce CLOCK_THREAD_DVFS_ID
In reply to Peter ZijlstraIn order to measure thread time in a DVFS world, introduce CLOCK_THREAD_DVFS_ID -- a copy of CLOCK_THREAD_CPUTIME_ID that slows down with both DVFS scaling and CPU capacity.
The clock does *NOT* support setting timers.
Useful for both SCHED_DEADLINE and the newly introduced sched_attr::sched_runtime usage for SCHED_NORMAL.
--- include/linux/posix-timers_types.h | 5 ++-- include/linux/sched.h | 1 include/linux/sched/cputime.h | 3 ++ include/uapi/linux/time.h | 1 kernel/sched/core.c | 40 +++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 8 +++++-- kernel/time/posix-cpu-timers.c | 16 +++++++++++++- kernel/time/posix-timers.c | 1 kernel/time/posix-timers.h | 1 9 files changed, 71 insertions(+), 5 deletions(-)
include/linux/posix-timers_types.h +3 -2
@@ -13,9 +13,9 @@* * Bit 2 indicates whether a cpu clock refers to a thread or a process. * -* Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3. +* Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3 * -* A clockid is invalid if bits 2, 1, and 0 are all set. +* (DVFS is PERTHREAD only) */ #define CPUCLOCK_PID (clock) ((pid_t) ~((clock) >> 3))#define CPUCLOCK_PERTHREAD (clock) \
@@ -27,6 +27,7 @@#define CPUCLOCK_PROF 0#define CPUCLOCK_VIRT 1#define CPUCLOCK_SCHED 2 +#define CPUCLOCK_DVFS 3#define CPUCLOCK_MAX 3#define CLOCKFD CPUCLOCK_MAX#define CLOCKFD_MASK (CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)
include/linux/sched.h +1 -0
@@ -550,6 +550,7 @@ struct sched_entity {u64 exec_start;u64 sum_exec_runtime;u64 prev_sum_exec_runtime; +u64 sum_dvfs_runtime;u64 vruntime;s64 vlag;u64 slice;
include/linux/sched/cputime.h +3 -0
@@ -180,4 +180,7 @@ static inline void prev_cputime_init(strextern unsigned long long task_sched_runtime (struct task_struct *task); +extern unsigned long long +task_sched_dvfs_runtime (struct task_struct *task); +#endif /* _LINUX_SCHED_CPUTIME_H */
include/uapi/linux/time.h +1 -0
@@ -62,6 +62,7 @@ struct timezone {*/ #define CLOCK_SGI_CYCLE 10#define CLOCK_TAI 11 +#define CLOCK_THREAD_DVFS_ID 12#define MAX_CLOCKS 16#define CLOCKS_MASK (CLOCK_REALTIME | CLOCK_MONOTONIC)
kernel/sched/core.c +40 -0
@@ -4551,6 +4551,7 @@ static void __sched_fork(unsigned long c p->se.exec_start =0 ; p->se.sum_exec_runtime =0 ; p->se.prev_sum_exec_runtime =0 ; + p->se.sum_dvfs_runtime =0 ; p->se.nr_migrations =0 ; p->se.vruntime =0 ; p->se.vlag =0 ;
@@ -5632,6 +5633,45 @@ unsigned long long task_sched_runtime(sttask_rq_unlock (rq, p, &rf);return ns; +} + +unsigned long long task_sched_dvfs_runtime (struct task_struct *p) +{ +struct rq_flags rf; +struct rq *rq; +u64 ns; + +#if defined(CONFIG_64BIT ) && defined(CONFIG_SMP ) +/* +* 64-bit doesn't need locks to atomically read a 64-bit value. +* So we have a optimization chance when the task's delta_exec is 0. +* Reading ->on_cpu is racy, but this is ok. +* +* If we race with it leaving CPU, we'll take a lock. So we're correct. +* If we race with it entering CPU, unaccounted time is 0. This is +* indistinguishable from the read occurring a few cycles earlier. +* If we see ->on_cpu without ->on_rq, the task is leaving, and has +* been accounted, so we're correct here as well. +*/ +if (!p->on_cpu || !task_on_rq_queued (p)) +return p->se.sum_dvfs_runtime; +#endif + + rq =task_rq_lock (p, &rf); +/* +* Must be ->curr _and_ ->on_rq. If dequeued, we would +* project cycles that may never be accounted to this +* thread, breaking clock_gettime(). +*/ +if (task_current (rq, p) &&task_on_rq_queued (p)) { +prefetch_curr_exec_start (p); +update_rq_clock (rq); + p->sched_class->update_curr (rq); + } + ns = p->se.sum_dvfs_runtime; +task_rq_unlock (rq, p, &rf); + +return ns; }#ifdef CONFIG_SCHED_DEBUG
kernel/sched/fair.c +6 -2
@@ -1118,15 +1118,19 @@ static void update_tg_load_avg(struct cfstatic s64 update_curr_se (struct rq *rq,struct sched_entity *curr) {u64 now =rq_clock_task (rq); -s64 delta_exec; +s64 delta_exec, delta_dvfs; - delta_exec = now - curr->exec_start; + delta_dvfs = delta_exec = now - curr->exec_start;if (unlikely (delta_exec <=0 ))return delta_exec; curr->exec_start = now; curr->sum_exec_runtime += delta_exec; + delta_dvfs =cap_scale (delta_dvfs,arch_scale_freq_capacity (cpu_of (rq))); + delta_dvfs =cap_scale (delta_dvfs,arch_scale_cpu_capacity (cpu_of (rq))); + curr->sum_dvfs_runtime += delta_dvfs; +if (schedstat_enabled ()) {struct sched_statistics *stats;
kernel/time/posix-cpu-timers.c +15 -1
@@ -164,7 +164,7 @@ posix_cpu_clock_getres(const clockid_t wif (!error) { tp->tv_sec =0 ; tp->tv_nsec = ((NSEC_PER_SEC +HZ -1 ) /HZ ); -if (CPUCLOCK_WHICH (which_clock) ==CPUCLOCK_SCHED ) { +if (CPUCLOCK_WHICH (which_clock) >=CPUCLOCK_SCHED ) {/* * If sched_clock is using a cycle counter, we * don't have any idea of its true resolution
@@ -198,6 +198,9 @@ static u64 cpu_clock_sample(const clockiif (clkid ==CPUCLOCK_SCHED )return task_sched_runtime (p); +if (clkid ==CPUCLOCK_DVFS ) +return task_sched_dvfs_runtime (p); +task_cputime (p, &utime, &stime);switch (clkid) {
@@ -1628,6 +1631,7 @@ static long posix_cpu_nsleep_restart(str#define PROCESS_CLOCK make_process_cpuclock(0, CPUCLOCK_SCHED)#define THREAD_CLOCK make_thread_cpuclock(0, CPUCLOCK_SCHED) +#define THREAD_DVFS_CLOCK make_thread_cpuclock(0, CPUCLOCK_DVFS)static int process_cpu_clock_getres (const clockid_t which_clock,struct timespec64 *tp)
@@ -1664,6 +1668,11 @@ static int thread_cpu_timer_create(struc timer->it_clock =THREAD_CLOCK ;return posix_cpu_timer_create (timer); } +static int thread_dvfs_cpu_clock_get (const clockid_t which_clock, +struct timespec64 *tp) +{ +return posix_cpu_clock_get (THREAD_DVFS_CLOCK , tp); +}const struct k_clock clock_posix_cpu = { .clock_getres = posix_cpu_clock_getres,
@@ -1690,3 +1699,8 @@ const struct k_clock clock_thread = { .clock_get_timespec = thread_cpu_clock_get, .timer_create = thread_cpu_timer_create, }; + +const struct k_clock clock_thread_dvfs = { + .clock_getres = thread_cpu_clock_getres, + .clock_get_timespec = thread_dvfs_cpu_clock_get, +};
kernel/time/posix-timers.c +1 -0
@@ -1516,6 +1516,7 @@ static const struct k_clock * const posi [CLOCK_MONOTONIC] = &clock_monotonic, [CLOCK_PROCESS_CPUTIME_ID] = &clock_process, [CLOCK_THREAD_CPUTIME_ID] = &clock_thread, + [CLOCK_THREAD_DVFS_ID] = &clock_thread_dvfs, [CLOCK_MONOTONIC_RAW] = &clock_monotonic_raw, [CLOCK_REALTIME_COARSE] = &clock_realtime_coarse, [CLOCK_MONOTONIC_COARSE] = &clock_monotonic_coarse,
kernel/time/posix-timers.h +1 -0
@@ -34,6 +34,7 @@ extern const struct k_clock clock_posix_extern const struct k_clock clock_posix_dynamic;extern const struct k_clock clock_process;extern const struct k_clock clock_thread; +extern const struct k_clock clock_thread_dvfs;extern const struct k_clock alarm_clock;int posix_timer_event (struct k_itimer *timr,int si_private);
Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID
Quoted 5 lines · Peter Zijlstra
In order to measure thread time in a DVFS world, introduce
CLOCK_THREAD_DVFS_ID-- a copy ofCLOCK_THREAD_CPUTIME_IDthat slows down with both DVFS scaling and CPU capacity.The clock does *NOT* support setting timers.
That's not the only limitation. See below.
Quoted 2 lines
Useful for both
SCHED_DEADLINEand the newly introducedsched_attr::sched_runtimeusage forSCHED_NORMAL.
Can this please have an explanation about the usage of the previously reserved value of 0x7 in the lower 3 bits?
Quoted 8 lines
*
* Bit 2 indicates whether a cpu clock refers to a thread or a process. *
- * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3.
+ * Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3 *
- * A clockid is invalid if bits 2, 1, and 0 are all set.
+ * (DVFS is PERTHREAD only)
This drops the information about the FD usage. Something like:
/*
* Bit fields within a clockid:
*
* Bit 31:3 hold either a pid or a file descriptor. *
* Bit 2 Bit 1 Bit 0
* 0 0 0 Per process CPUCLOCK_PROF * 0 0 1 Per process CPUCLOCK_VIRT * 0 1 0 Per process CPUCLOCK_SCHED * 0 1 1 Posixclock FD CLOCKFD * 1 0 0 Per thread CPUCLOCK_PROF * 1 0 1 Per thread CPUCLOCK_VIRT * 1 1 0 Per thread CPUCLOCK_SCHED * 1 1 1 Per thread CPUCLOCK_DVSF
*
* CPUCLOCK_DVSF is per thread only and shares the type code in Bit 1:0
* with CLOCKFD. CLOCKFD uses a file descriptor to access dynamically
* registered POSIX clocks (e.g. PTP hardware clocks).
*/
should be clear enough, no?
But, all of this is wishful thinking because the provided implementation only works for:
sys_clock_getres(CLOCK_THREAD_DVFS_ID, ...)
which falls back to thread_cpu_clock_getres().
The variant which has the TID encoded in bit 31:3 and the type in bit 2:0 fails the test in pid_for_clock():
if (CPUCLOCK_WHICH(clock) >= CPUCLOCK_MAX)
return NULL;
Worse for sys_clock_gettime(). That fails in both cases for the very same reason.
See the uncompiled delta patch below for a cure of that and the rest of my comments.
Quoted 7 lines
#define
CPUCLOCK_PROF0
#defineCPUCLOCK_VIRT1
#defineCPUCLOCK_SCHED2
+#defineCPUCLOCK_DVFS3
#defineCPUCLOCK_MAX3
#define CLOCKFDCPUCLOCK_MAX
#defineCLOCKFD_MASK(CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)
With that DVFS addition CPUCLOCK_MAX is misleading at best. See delta patch.
Review
Quoted 6 lines
(quoted diff) +6
-0+ + rq = task_rq_lock(p, &rf); + /* + * Must be ->curr _and_ ->on_rq. If dequeued, we would + * project cycles that may never be accounted to this + * thread, breaking clock_gettime().
Must be? For what? I assume you want to say:
Update the runtime if the task is the current task and on the runqueue. The latter is important because if current is dequeued,
....
Quoted 9 lines
(quoted diff) +8
-0+ */ + if (task_current(rq, p) && task_on_rq_queued(p)) { + prefetch_curr_exec_start(p); + update_rq_clock(rq); + p->sched_class->update_curr(rq); + } + ns = p->se.sum_dvfs_runtime; + task_rq_unlock(rq, p, &rf);
Review static int thread_cpu_timer_create(struc · line 1672
Quoted 5 lines
(quoted diff) +2
-0@@ -1664,6 +1668,11 @@ static int thread_cpu_timer_create(struc timer->it_clock = THREAD_CLOCK; return posix_cpu_timer_create(timer); } +static int thread_dvfs_cpu_clock_get(const clockid_t which_clock, + struct timespec64 *tp)
Please align the second line properly with the argument in the first line.
Thanks,
tglx
---
include/linux/posix-timers_types.h +29 -14
@@ -9,27 +9,42 @@/* * Bit fields within a clockid: * -* The most significant 29 bits hold either a pid or a file descriptor. +* Bit 31:3 hold either a PID/TID or a file descriptor. * -* Bit 2 indicates whether a cpu clock refers to a thread or a process. +* Bit 2 Bit 1 Bit 0 +* 0 0 0 Per process CPUCLOCK_PROF +* 0 0 1 Per process CPUCLOCK_VIRT +* 0 1 0 Per process CPUCLOCK_SCHED +* 0 1 1 Posixclock FD CLOCKFD +* 1 0 0 Per thread CPUCLOCK_PROF +* 1 0 1 Per thread CPUCLOCK_VIRT +* 1 1 0 Per thread CPUCLOCK_SCHED +* 1 1 1 Per thread CPUCLOCK_DVSF * -* Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3 -* -* (DVFS is PERTHREAD only) +* CPUCLOCK_DVSF is per thread only and shares the type code in Bit 1:0 +* with CLOCKFD. CLOCKFD uses a file descriptor to access dynamically +* registered POSIX clocks (e.g. PTP hardware clocks). */ +#define CPUCLOCK_PID (clock) ((pid_t) ~((clock) >> 3)) -#define CPUCLOCK_PERTHREAD (clock) \ - (((clock) & (clockid_t) CPUCLOCK_PERTHREAD_MASK) != 0) +#define CPUCLOCK_PERTHREAD (clock) (((clock) & (clockid_t) CPUCLOCK_PERTHREAD_MASK) != 0) -#define CPUCLOCK_PERTHREAD_MASK 4 -#define CPUCLOCK_WHICH (clock) ((clock) & (clockid_t) CPUCLOCK_CLOCK_MASK) -#define CPUCLOCK_CLOCK_MASK 3#define CPUCLOCK_PROF 0#define CPUCLOCK_VIRT 1#define CPUCLOCK_SCHED 2 -#define CPUCLOCK_DVFS 3 -#define CPUCLOCK_MAX 3 -#define CLOCKFD CPUCLOCK_MAX +#define CPUCLOCK_SAMPLE_MAX (CPUCLOCK_SCHED + 1) + +#define CPUCLOCK_CLOCK_MASK 3 +#define CPUCLOCK_PERTHREAD_MASK 4 +#define CPUCLOCK_WHICH (clock) ((clock) & (clockid_t) CPUCLOCK_CLOCK_MASK) + +/* +* CPUCLOCK_DVFS and CLOCKFD share the type code in bit 1:0. CPUCLOCK_DVFS +* does not belong to the sampling clocks and does not allow timers to be +* armed on it. +*/ +#define CPUCLOCK_DVFS CPUCLOCK_SAMPLE_MAX +#define CLOCKFD CPUCLOCK_DVFS#define CLOCKFD_MASK (CPUCLOCK_PERTHREAD_MASK|CPUCLOCK_CLOCK_MASK)#ifdef CONFIG_POSIX_TIMERS
@@ -55,7 +70,7 @@ struct posix_cputimer_base {* Used in task_struct and signal_struct */ struct posix_cputimers { -struct posix_cputimer_base bases[CPUCLOCK_MAX ]; +struct posix_cputimer_base bases[CPUCLOCK_SAMPLE_MAX ];unsigned int timers_active;unsigned int expiry_active; };
kernel/sched/core.c +4 -3
@@ -5413,9 +5413,10 @@ unsigned long long task_sched_dvfs_runti rq =task_rq_lock (p, &rf);/* -* Must be ->curr _and_ ->on_rq. If dequeued, we would -* project cycles that may never be accounted to this -* thread, breaking clock_gettime(). +* Update the runtime if the task is the current task and on the +* runqueue. The latter is important because if current is +* dequeued, we would project cycles that may never be accounted to +* this thread, breaking clock_gettime(). */ if (task_current (rq, p) &&task_on_rq_queued (p)) {prefetch_curr_exec_start (p);
kernel/time/posix-cpu-timers.c +17 -12
@@ -54,13 +54,13 @@ int update_rlimit_cpu(struct task_struct/* * Functions for validating access to tasks. */ -static struct pid *pid_for_clock (const clockid_t clock,bool gettime) +static struct pid *__pid_for_clock (const clockid_t clock,const clockid_t maxclock,bool gettime) {const bool thread = !!CPUCLOCK_PERTHREAD (clock);const pid_t upid =CPUCLOCK_PID (clock);struct pid *pid; -if (CPUCLOCK_WHICH (clock) >=CPUCLOCK_MAX ) +if (CPUCLOCK_WHICH (clock) > maxclock)return NULL ;/*
@@ -94,12 +94,17 @@ static struct pid *pid_for_clock(const creturn pid_has_task (pid,PIDTYPE_TGID ) ? pid :NULL ; } +static inline struct pid *pid_for_clock (const clockid_t clock,bool gettime) +{ +return __pid_for_clock (clock,CPUCLOCK_SCHED , gettime); +} +static inline int validate_clock_permissions (const clockid_t clock) {int ret;rcu_read_lock (); - ret =pid_for_clock (clock, false) ?0 : -EINVAL ; + ret =__pid_for_clock (clock,CPUCLOCK_DVFS , false) ?0 : -EINVAL ;rcu_read_unlock ();return ret;
@@ -344,7 +349,7 @@ static u64 cpu_clock_sample_group(const {struct thread_group_cputimer *cputimer = &p->signal->cputimer;struct posix_cputimers *pct = &p->signal->posix_cputimers; -u64 samples[CPUCLOCK_MAX ]; +u64 samples[CPUCLOCK_SAMPLE_MAX ];if (!READ_ONCE (pct->timers_active)) {if (start)
@@ -365,7 +370,7 @@ static int posix_cpu_clock_get(const clou64 t;rcu_read_lock (); - tsk =pid_task (pid_for_clock (clock, true),clock_pid_type (clock)); + tsk =pid_task (__pid_for_clock (clock,CPUCLOCK_DVFS , true),clock_pid_type (clock));if (!tsk) {rcu_read_unlock ();return -EINVAL ;
@@ -864,7 +869,7 @@ static void collect_posix_cputimers(strustruct posix_cputimer_base *base = pct->bases;int i; -for (i =0 ; i <CPUCLOCK_MAX ; i++, base++) { +for (i =0 ; i <CPUCLOCK_SAMPLE_MAX ; i++, base++) { base->nextevt =collect_timerqueue (&base->tqhead, firing, samples[i]); }
@@ -901,7 +906,7 @@ static void check_thread_timers(struct t struct list_head *firing) {struct posix_cputimers *pct = &tsk->posix_cputimers; -u64 samples[CPUCLOCK_MAX ]; +u64 samples[CPUCLOCK_SAMPLE_MAX ];unsigned long soft;if (dl_task (tsk))
@@ -979,7 +984,7 @@ static void check_process_timers(struct {struct signal_struct *const sig = tsk->signal;struct posix_cputimers *pct = &sig->posix_cputimers; -u64 samples[CPUCLOCK_MAX ]; +u64 samples[CPUCLOCK_SAMPLE_MAX ];unsigned long soft;/*
@@ -1098,7 +1103,7 @@ task_cputimers_expired(const u64 *sample {int i; -for (i =0 ; i <CPUCLOCK_MAX ; i++) { +for (i =0 ; i <CPUCLOCK_SAMPLE_MAX ; i++) {if (samples[i] >= pct->bases[i].nextevt)return true; }
@@ -1121,7 +1126,7 @@ static inline bool fastpath_timer_check(struct signal_struct *sig;if (!expiry_cache_is_inactive (pct)) { -u64 samples[CPUCLOCK_MAX ]; +u64 samples[CPUCLOCK_SAMPLE_MAX ];task_sample_cputime (tsk, samples);if (task_cputimers_expired (samples, pct))
@@ -1146,7 +1151,7 @@ static inline bool fastpath_timer_check(* delays with signals actually getting sent are expected. */ if (READ_ONCE (pct->timers_active) && !READ_ONCE (pct->expiry_active)) { -u64 samples[CPUCLOCK_MAX ]; +u64 samples[CPUCLOCK_SAMPLE_MAX ];proc_sample_cputime_atomic (&sig->cputimer.cputime_atomic, samples);
@@ -1669,7 +1674,7 @@ static int thread_cpu_timer_create(strucreturn posix_cpu_timer_create (timer); }static int thread_dvfs_cpu_clock_get (const clockid_t which_clock, -struct timespec64 *tp) +struct timespec64 *tp) {return posix_cpu_clock_get (THREAD_DVFS_CLOCK , tp); }
Juri Lelli
maintainerRe: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID
In reply to Peter ZijlstraHi Peter,
Quoted 8 lines · Peter Zijlstra
In order to measure thread time in a DVFS world, introduce
CLOCK_THREAD_DVFS_ID-- a copy ofCLOCK_THREAD_CPUTIME_IDthat slows down with both DVFS scaling and CPU capacity.The clock does *NOT* support setting timers.
Useful for both
SCHED_DEADLINEand the newly introducedsched_attr::sched_runtimeusage forSCHED_NORMAL.
Just so I'm sure I understand, this would be useful for estimating the runtime needs of a (also DEADLINE) task when DVFS is enabled, right?
Thanks,
Juri
Peter Zijlstra
maintainerRe: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID
Quoted 7 lines · Juri Lelli
Hi Peter,
Quoted 4 lines · Peter Zijlstra
In order to measure thread time in a DVFS world, introduce
CLOCK_THREAD_DVFS_ID-- a copy ofCLOCK_THREAD_CPUTIME_IDthat slows down with both DVFS scaling and CPU capacity.
Quoted 7 lines
Quoted 4 lines
The clock does *NOT* support setting timers.
Useful for both
SCHED_DEADLINEand the newly introducedsched_attr::sched_runtimeusage forSCHED_NORMAL.Just so I'm sure I understand, this would be useful for estimating the runtime needs of a (also DEADLINE) task when DVFS is enabled, right?
Correct, DVFS or biggie-smalls CPUs with mixed capacities.
Re: [RFC PATCH 24/24] sched/time: Introduce CLOCK_THREAD_DVFS_ID
In reply to Peter ZijlstraQuoted 9 lines · Peter Zijlstra
In order to measure thread time in a DVFS world, introduce
CLOCK_THREAD_DVFS_ID-- a copy ofCLOCK_THREAD_CPUTIME_IDthat slows down with both DVFS scaling and CPU capacity.The clock does *NOT* support setting timers.
Useful for both
SCHED_DEADLINEand the newly introducedsched_attr::sched_runtimeusage forSCHED_NORMAL.
How will this look like in practice then?
Is it up to userspace to adjust sched_runtime to capacity/dvfs accordingly every time it changes? I guess not.
Will sched_attr::sched_runtime be for CPUCLOCK_DVFS by default? I assume that would be a uapi change?
Do we need an additional flag in sched_attr to specify the clock to be measured against?
Quoted 16 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/posix-timers_types.h | 5 ++-- include/linux/sched.h | 1 include/linux/sched/cputime.h | 3 ++ include/uapi/linux/time.h | 1 kernel/sched/core.c | 40 +++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 8 +++++-- kernel/time/posix-cpu-timers.c | 16 +++++++++++++- kernel/time/posix-timers.c | 1 kernel/time/posix-timers.h | 1 9 files changed, 71 insertions(+), 5 deletions(-)
include/linux/posix-timers_types.h line 16
Quoted 5 lines
include/linux/posix-timers_types.h +1
-1@@ -13,9 +13,9 @@* * Bit 2 indicates whether a cpu clock refers to a thread or a process. * -* Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or FD=3. +* Bits 1 and 0 give the type: PROF=0, VIRT=1, SCHED=2, or DVSF=3
s/DVSF/DVFS
Quoted 1 line
[snip]
Hi Peter,
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 11 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
Thanks for the updated series.
FTR, I ran this through our Pixel 6 and did not see significant differences between the baseline (6.8 kernel with EEVDF + fixes) and the baseline with the updated eevdf-complete series applied on top.
The power use and frame metrics seem mostly the same between the two.
I'll run a few more tests just in case.
Vincent Guittot
maintainerQuoted 7 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
Quoted 7 lines
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
While trying to test what would be the impact of delayed dequeue on load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (the sysctl_sched_base_slice on my system) between 2 context swicthes
I'm using your sched/core branch. Is it the correct one ?
Quoted 12 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
Quoted 9 lines · Vincent Guittot
While trying to test what would be the impact of delayed dequeue on
load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (thesysctl_sched_base_sliceon my system) between 2 context swicthesI'm using your sched/core branch. Is it the correct one ?
Hm, building that branch, I see the expected tick granularity (4ms).
-Mike
Vincent Guittot
maintainerQuoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot
While trying to test what would be the impact of delayed dequeue on
load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (thesysctl_sched_base_sliceon my system) between 2 context swicthes
Quoted 3 lines
Quoted 1 line
I'm using your sched/core branch. Is it the correct one ?
Hm, building that branch, I see the expected tick granularity (4ms).
On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.
Quoted 3 lines
-Mike
Quoted 11 lines · Vincent Guittot
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot
While trying to test what would be the impact of delayed dequeue on
load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (thesysctl_sched_base_sliceon my system) between 2 context swicthes
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I'm using your sched/core branch. Is it the correct one ?
Hm, building that branch, I see the expected tick granularity (4ms).
On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.
FWIW, I checked my local master-rt tree as well, which has Peter's latest eevdf series wedged in (plus 4cc290c20a98 now).. it also worked as expected.
-Mike
Vincent Guittot
maintainerQuoted 17 lines · Mike Galbraith
Quoted 15 lines · Vincent Guittot
Quoted 13 lines · Mike Galbraith
Quoted 9 lines · Vincent Guittot
While trying to test what would be the impact of delayed dequeue on
load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (thesysctl_sched_base_sliceon my system) between 2 context swicthesI'm using your sched/core branch. Is it the correct one ?
Hm, building that branch, I see the expected tick granularity (4ms).
Quoted 7 lines
Quoted 3 lines
On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.
FWIW, I checked my local master-rt tree as well, which has Peter's latest eevdf series wedged in (plus
4cc290c20a98now).. it also worked as expected.
After looking at deadline and slice, the issue is on my tool was trying to change the slice (an old trial for previous version) which got clamp to 100ms.
we can forgot this, sorry for the noise
Quoted 3 lines
-Mike
K Prateek Nayak
reviewerHello Vincent, Mike,
Quoted 11 lines · Vincent Guittot
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Vincent Guittot
While trying to test what would be the impact of delayed dequeue on
load_avg, I noticed something strange with the running slice. I have a simple test with 2 always running threads on 1 CPU and the each thread runs around 100ms continuously before switching to the other one whereas I was expecting 3ms (thesysctl_sched_base_sliceon my system) between 2 context swicthes
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I'm using your sched/core branch. Is it the correct one ?
Hm, building that branch, I see the expected tick granularity (4ms).
On my side tip/sched/core switches every 4ms but Peter's sched/core, which is delayed queued on top of tip/sched/core if I don't get it wrong, switches every 100ms.
I could not observe this behavior when running two busy loops pinned to one CPU on my end. I'm running with base_slice_ns of 3ms and the sched_feats related to EEVDF complete looks as follows:
PLACE_LAG
PLACE_DEADLINE_INITIAL
PLACE_REL_DEADLINE
RUN_TO_PARITY
PREEMPT_SHORT
NO_NEXT_BUDDY
CACHE_HOT_BUDDY
DELAY_DEQUEUE
DELAY_ZERO
WAKEUP_PREEMPTION
...Also I'm running with CONFIG_HZ=250 (4ms tick granularity)
CONFIG_HZ_250=y
CONFIG_HZ=250Enabling sched_switch tracepeoint, I see the following:
...
loop-4061 109.710379: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.714377: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.718375: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.722374: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.726379: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.730377: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.734367: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.738365: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.742364: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.746361: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.750359: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.754357: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.758355: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.762353: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.766351: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.770349: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.774347: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.778345: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.782343: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.786346: sched_switch: ... prev_pid=4060 ... prev_state=R ==> next_comm=kworker/1:1 next_pid=1616 next_prio=120
kworker/1:1-1616 109.786412: sched_switch: prev_comm=kworker/1:1 prev_pid=1616 ... prev_state=I ==> ... next_pid=4061 ...
loop-4061 109.794337: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.798335: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.802335: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.806331: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.810329: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.814327: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.818325: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.822323: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.826321: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
loop-4060 109.830321: sched_switch: ... prev_pid=4060 ... prev_state=R ==> ... next_pid=4061 ...
loop-4061 109.834317: sched_switch: ... prev_pid=4061 ... prev_state=R ==> ... next_pid=4060 ...
...(Trimmed traces are for busy loops with pids 4060 and 4061)
I see the expected tick granularity similar to Mike. Since Peter's tree is prone to force-updates, I'm on
git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/core
at commit 4cc290c20a98 "sched/eevdf: Dequeue in switched_from_fair()" which was committed at "2024-08-14 08:15:39 +0200".
Quoted 4 lines
Quoted 4 lines
-Mike
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
So what I've been testing at your queue/sched/core, HEAD at:
4cc290c20a98 ("sched/eevdf: Dequeue in switched_from_fair()")
survives my (simplistic) CFS bandwidth testing, so FWIW:
And for patches 01-20:
with one caveat that I agree with Vincent on maybe doing something about the load PELT signal of delayed dequeued entities.
Hi Peter,
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 9 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
The latest tip/sched/core at commit
aef6987d89544d63a47753cf3741cabff0b5574c
crashes very early on on my Juno r2 board (arm64). The trace is here:
[ 0.049599] ------------[ cut here ]------------ [ 0.054279] kernel BUG at kernel/sched/deadline.c:63! [ 0.059401] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP [ 0.066285] Modules linked in: [ 0.069382] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.11.0-rc1-g55404cef33db #1070 [ 0.077855] Hardware name: ARM Juno development board (r2) (DT) [ 0.083856] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.090919] pc : enqueue_dl_entity+0x53c/0x540 [ 0.095434] lr : dl_server_start+0xb8/0x10c [ 0.099679] sp : ffffffc081ca3c30 [ 0.103034] x29: ffffffc081ca3c40 x28: 0000000000000001 x27: 0000000000000002 [ 0.110281] x26: 00000000000b71b0 x25: 0000000000000000 x24: 0000000000000001 [ 0.117525] x23: ffffff897ef21140 x22: 0000000000000000 x21: 0000000000000000 [ 0.124770] x20: ffffff897ef21040 x19: ffffff897ef219a8 x18: ffffffc080d0ad00 [ 0.132015] x17: 000000000000002f x16: 0000000000000000 x15: ffffffc081ca8000 [ 0.139260] x14: 00000000016ef200 x13: 00000000000e6667 x12: 0000000000000001 [ 0.146505] x11: 000000003b9aca00 x10: 0000000002faf080 x9 : 0000000000000030 [ 0.153749] x8 : 0000000000000071 x7 : 000000002cf93d25 x6 : 000000002cf93d25 [ 0.160994] x5 : ffffffc081e04938 x4 : ffffffc081ca3d40 x3 : 0000000000000001 [ 0.168238] x2 : 000000003b9aca00 x1 : 0000000000000001 x0 : ffffff897ef21040 [ 0.175483] Call trace: [ 0.177958] enqueue_dl_entity+0x53c/0x540 [ 0.182117] dl_server_start+0xb8/0x10c [ 0.186010] enqueue_task_fair+0x5c8/0x6ac [ 0.190165] enqueue_task+0x54/0x1e8 [ 0.193793] wake_up_new_task+0x250/0x39c [ 0.197862] kernel_clone+0x140/0x2f0 [ 0.201578] user_mode_thread+0x4c/0x58 [ 0.205468] rest_init+0x24/0xd8 [ 0.208743] start_kernel+0x2bc/0x2fc [ 0.212460] __primary_switched+0x80/0x88 [ 0.216535] Code: b85fc3a8 7100051f 54fff8e9 17ffffce (d4210000) [ 0.222711] ---[ end trace 0000000000000000 ]--- [ 0.227391] Kernel panic - not syncing: Attempted to kill the idle task! [ 0.234187] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
I'm not an expert in DL server so I have no idea where the problem could be. If you know where to look off the top of your head then much better. If not, I'll do some bi-section later.
Hongyan
Quoted 70 lines · Hongyan Xia
Hi Peter,
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 9 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDThe latest tip/sched/core at commit
aef6987d89544d63a47753cf3741cabff0b5574ccrashes very early on on my Juno r2 board (arm64). The trace is here:
[ 0.049599] ------------[ cut here ]------------
[ 0.054279] kernel BUG atkernel/sched/deadline.c:63!
[ 0.059401] Internal error: Oops - BUG:00000000f2000800[#1] PREEMPT SMP
[ 0.066285] Modules linked in:
[ 0.069382] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.11.0-rc1-g55404cef33db #1070 [ 0.077855] Hardware name: ARM Juno development board (r2) (DT)
[ 0.083856] pstate:600000c5(nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.090919] pc :enqueue_dl_entity+0x53c/0x540
[ 0.095434] lr :dl_server_start+0xb8/0x10c
[ 0.099679] sp :ffffffc081ca3c30
[ 0.103034] x29:ffffffc081ca3c40x28: 0000000000000001 x27: 0000000000000002 [ 0.110281] x26:00000000000b71b0x25: 0000000000000000 x24: 0000000000000001 [ 0.117525] x23:ffffff897ef21140x22: 0000000000000000 x21: 0000000000000000 [ 0.124770] x20:ffffff897ef21040x19:ffffff897ef219a8x18:ffffffc080d0ad00[ 0.132015] x17:000000000000002fx16: 0000000000000000 x15:ffffffc081ca8000[ 0.139260] x14:00000000016ef200x13:00000000000e6667x12: 0000000000000001 [ 0.146505] x11:000000003b9aca00x10:0000000002faf080x9 : 0000000000000030 [ 0.153749] x8 : 0000000000000071 x7 :000000002cf93d25x6 :000000002cf93d25[ 0.160994] x5 :ffffffc081e04938x4 :ffffffc081ca3d40x3 : 0000000000000001 [ 0.168238] x2 :000000003b9aca00x1 : 0000000000000001 x0 :ffffff897ef21040[ 0.175483] Call trace:
[ 0.177958]enqueue_dl_entity+0x53c/0x540
[ 0.182117]dl_server_start+0xb8/0x10c
[ 0.186010]enqueue_task_fair+0x5c8/0x6ac
[ 0.190165]enqueue_task+0x54/0x1e8
[ 0.193793]wake_up_new_task+0x250/0x39c
[ 0.197862]kernel_clone+0x140/0x2f0
[ 0.201578]user_mode_thread+0x4c/0x58
[ 0.205468]rest_init+0x24/0xd8
[ 0.208743]start_kernel+0x2bc/0x2fc
[ 0.212460]__primary_switched+0x80/0x88
[ 0.216535] Code:b85fc3a87100051f54fff8e917ffffce(d4210000)
[ 0.222711] ---[ end trace 0000000000000000 ]---
[ 0.227391] Kernel panic - not syncing: Attempted to kill the idle task!
[ 0.234187] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
Quoted 2 lines
I'm not an expert in DL server so I have no idea where the problem could be. If you know where to look off the top of your head then much better. If not, I'll do some bi-section later.
Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:
dl_server_start()
enqueue_dl_entity() update_stats_enqueue_dl() update_stats_enqueue_sleeper_dl() __schedstats_from_dl_se() dl_task_of() <---------- crash
If I undefine CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.
Quoted 4 lines · Hongyan Xia
If I undefine
CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.
FWIW, tip.today boots/runs gripe free on dinky rpi4b (SCHEDSTATS=y).
-Mike
Peter Zijlstra
maintainerQuoted 10 lines · Hongyan Xia
Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:
dl_server_start()enqueue_dl_entity() update_stats_enqueue_dl() update_stats_enqueue_sleeper_dl() __schedstats_from_dl_se() dl_task_of() <---------- crash
Quoted 3 lines
If I undefine
CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.
Thanks for the report -- I'll see if I can spot something. Since you initially fingered these eevdf patches, could you confirm or deny that changing:
kernel/sched/features.h:SCHED_FEAT(DELAY_DEQUEUE, true)
to false, makes any difference in the previously failing case?
Quoted 15 lines · Peter Zijlstra
Quoted 10 lines · Hongyan Xia
Okay, in case the trace I provided isn't clear enough, I traced the crash to a call chain like this:
dl_server_start()enqueue_dl_entity() update_stats_enqueue_dl() update_stats_enqueue_sleeper_dl() __schedstats_from_dl_se() dl_task_of() <---------- crashQuoted 3 lines
If I undefine
CONFIG_SCHEDSTATS, then it boots fine, and I wonder if this is the reason why other people are not seeing this. This is probably not EEVDF but DL refactoring related.
Quoted 7 lines
Thanks for the report -- I'll see if I can spot something. Since you initially fingered these eevdf patches, could you confirm or deny that changing:
kernel/sched/features.h:SCHED_FEAT(DELAY_DEQUEUE, true)
to false, makes any difference in the previously failing case?
Sadly the issue persists. I'm seeing exactly the same backtrace on my Juno board.
This series has caused an explosion of different kernel crashes on our syzbot instance that fuzzes linux-next. I guess, such kernel behavior indicates some massive underlying memory corruption (?)
Some of the crash titles we've seen (we didn't release them -- there were too many, 70+):
KASAN: stack-out-of-bounds Write in insn_decode kernel panic: stack is corrupted in vprintk_store kernel panic: stack is corrupted in _printk BUG: spinlock recursion in __schedule WARNING in __put_task_struct BUG: unable to handle kernel NULL pointer dereference in asm_exc_page_fault WARNING in rng_dev_read BUG: scheduling while atomic in prb_final_commit kernel BUG in dequeue_rt_stack BUG: scheduling while atomic in rcu_is_watching BUG: spinlock recursion in copy_process KASAN: slab-use-after-free Read in sched_core_enqueue kernel panic: stack is corrupted in refill_stock kernel panic: stack is corrupted in prb_reserve WARNING: bad unlock balance in timekeeping_get_ns KASAN: slab-use-after-free Read in set_next_task_fair
I wonder if the actual problem is already known and possibly there are even some fix patches?
If not and if it may be of any help, we can try to come up with some contained instruction to reproduce these issues with syzkaller.
--
Aleksandr
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
I'm seeing crashes/warnings like the following on s390 with linux-next 20240909:
Sometimes the system doesn't manage to print a oops, this one is the best i got:
[ 596.146142] ------------[ cut here ]------------ [ 596.146161] se->sched_delayed [ 596.146166] WARNING: CPU: 1 PID: 0 at kernel/sched/fair.c:13131 __set_next_task_fair.part.0+0x350/0x400 [ 596.146179] Modules linked in: [..] [ 596.146288] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.11.0-rc7-next-20240909 #18 [ 596.146294] Hardware name: IBM 3931 A01 704 (LPAR) [ 596.146298] Krnl PSW : 0404e00180000000 001a9c2b5eea4ea4 (__set_next_task_fair.part.0+0x354/0x400) [ 596.146307] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 [ 596.146314] Krnl GPRS: 001c000300000027 001c000300000023 0000000000000011 0000000000000004 [ 596.146319] 0000000000000001 001a9c2b5f1fb118 000000036ef94dd5 0000001b77ca6ea8 [ 596.146323] 001c000000000000 001a9c2b5eec6fc0 0000001b77ca6000 00000000b7334800 [ 596.146328] 0000000000000000 001a9c2b5eefad70 001a9c2b5eea4ea0 001a9bab5ee8f9f8 [ 596.146340] Krnl Code: 001a9c2b5eea4e94: c0200121bbe6 larl %r2,001a9c2b612dc660 [ 596.146340] 001a9c2b5eea4e9a: c0e5fff9e9d3 brasl %r14,001a9c2b5ede2240 [ 596.146340] #001a9c2b5eea4ea0: af000000 mc 0,0 [ 596.146340] >001a9c2b5eea4ea4: a7f4fe83 brc 15,001a9c2b5eea4baa [ 596.146340] 001a9c2b5eea4ea8: c0e50038ba2c brasl %r14,001a9c2b5f5bc300
[ 596.146558] CPU: 1 UID: 0 PID: 18582 Comm: prctl-sched-cor Tainted: G W 6.11.0-rc7-next-20240909 #18 [ 596.146564] Tainted: [W]=WARN [ 596.146567] Hardware name: IBM 3931 A01 704 (LPAR) [ 596.146570] Krnl PSW : 0404e00180000000 001a9c2b5eec2de4 (dequeue_entity+0xe64/0x11f0) [ 596.146578] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 [ 596.146584] Krnl GPRS: 001c000300000027 001c000300000023 000000000000001a 0000000000000004 [ 596.146589] 0000000000000001 001a9c2b5f1fb118 001a9c2b61be7144 0000000016e6692a [ 596.146593] 0000000000000001 00000000b7334951 0000000158494800 00000000b7334900 [ 596.146597] 000000000000489e 0000000000000009 001a9c2b5eec2de0 001a9bab75dff760 [ 596.146607] Krnl Code: 001a9c2b5eec2dd4: c0200120cdf6 larl %r2,001a9c2b612dc9c0 [ 596.146607] 001a9c2b5eec2dda: c0e5fff8fa33 brasl %r14,001a9c2b5ede2240 [ 596.146607] #001a9c2b5eec2de0: af000000 mc 0,0 [ 596.146607] >001a9c2b5eec2de4: c004fffff90a brcl 0,001a9c2b5eec1ff8 [ 596.146607] 001a9c2b5eec2dea: a7f4fbbe brc 15,001a9c2b5eec2566 [ 596.146607] 001a9c2b5eec2dee: a7d10001 tmll %r13,1 [ 596.146607] 001a9c2b5eec2df2: a774fb1c brc 7,001a9c2b5eec242a [ 596.146607] 001a9c2b5eec2df6: a7f4f95f brc 15,001a9c2b5eec20b4 [ 596.146637] Call Trace: [ 596.146640] [<001a9c2b5eec2de4>] dequeue_entity+0xe64/0x11f0 [ 596.146645] ([<001a9c2b5eec2de0>] dequeue_entity+0xe60/0x11f0) [ 596.146650] [<001a9c2b5eec34b0>] dequeue_entities+0x340/0xe10 [ 596.146655] [<001a9c2b5eec4208>] dequeue_task_fair+0xb8/0x5a0 [ 596.146660] [<001a9c2b6115ab68>] __schedule+0xb58/0x14f0 [ 596.146666] [<001a9c2b6115b59c>] schedule+0x9c/0x240 [ 596.146670] [<001a9c2b5edf5190>] do_wait+0x160/0x440 [ 596.146676] [<001a9c2b5edf5936>] kernel_waitid+0xd6/0x110 [ 596.146680] [<001a9c2b5edf5b4e>] __do_sys_waitid+0x1de/0x1f0 [ 596.146685] [<001a9c2b5edf5c36>] __s390x_sys_waitid+0xd6/0x120 [ 596.146690] [<001a9c2b5ed0cbd6>] do_syscall+0x2f6/0x430 [ 596.146695] [<001a9c2b611543a4>] __do_syscall+0xa4/0x170 [ 596.146700] [<001a9c2b6117046c>] system_call+0x74/0x98 [ 596.146705] Last Breaking-Event-Address: [ 596.146707] [<001a9c2b5ede2418>] __warn_printk+0x1d8/0x1e0
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.
Regards
Sven
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDI'm seeing crashes/warnings like the following on s390 with linux-next 20240909:
Quoted 7 lines
Sometimes the system doesn't manage to print a oops, this one is the best i got:
[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.
I failed to add the log from git bisect. Unfortunately i had to skip some commit because the kernel didn't compile:
git bisect start
# status: waiting for both good and bad commits
# bad: [100cc857359b5d731407d1038f7e76cd0e871d94] Add linux-next specific files for 20240909
git bisect bad 100cc857359b5d731407d1038f7e76cd0e871d94
# status: waiting for good commit(s), bad commit known
# good: [da3ea35007d0af457a0afc87e84fddaebc4e0b63] Linux 6.11-rc7
git bisect good da3ea35007d0af457a0afc87e84fddaebc4e0b63
# good: [df20078b9706977cc3308740b56993cf27665f90] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good df20078b9706977cc3308740b56993cf27665f90
# good: [609f9e1b6242e7158ce96f9124372601997ce56c] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git
git bisect good 609f9e1b6242e7158ce96f9124372601997ce56c
# skip: [664c3413e9a6c345a6c926841358314be9da8309] Merge branch 'usb-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git
git bisect skip 664c3413e9a6c345a6c926841358314be9da8309
# good: [16531118ba63dd9bcd65203d04a9c9d6f6800547] iio: bmi323: peripheral in lowest power state on suspend
git bisect good 16531118ba63dd9bcd65203d04a9c9d6f6800547
# bad: [d9c7ac7f8bfb16f431daa7c77bdfe2b163361ead] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86.git
git bisect bad d9c7ac7f8bfb16f431daa7c77bdfe2b163361ead
# bad: [05536babd768b38d84ad168450f48634a013603d] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect bad 05536babd768b38d84ad168450f48634a013603d
# good: [dabef94a179957db117db344b924e5d5c4074e5f] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git
git bisect good dabef94a179957db117db344b924e5d5c4074e5f
# bad: [d4886a325947ecae6867fc858657062211aae3b9] Merge branch into tip/master: 'locking/core'
git bisect bad d4886a325947ecae6867fc858657062211aae3b9
# bad: [51c095bee5c77590d43519f03179342e910d333c] Merge branch into tip/master: 'core/core'
git bisect bad 51c095bee5c77590d43519f03179342e910d333c
# bad: [fc1892becd5672f52329a75c73117b60ac7841b7] sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE
git bisect bad fc1892becd5672f52329a75c73117b60ac7841b7
# good: [ae04f69de0bef93c7086cf2983dbc8e8fd624ebe] sched/rt: Rename realtime_{prio, task}() to rt_or_dl_{prio, task}()
git bisect good ae04f69de0bef93c7086cf2983dbc8e8fd624ebe
# good: [abc158c82ae555078aa5dd2d8407c3df0f868904] sched: Prepare generic code for delayed dequeue
git bisect good abc158c82ae555078aa5dd2d8407c3df0f868904
# skip: [781773e3b68031bd001c0c18aa72e8470c225ebd] sched/fair: Implement ENQUEUE_DELAYED
git bisect skip 781773e3b68031bd001c0c18aa72e8470c225ebd
# skip: [e1459a50ba31831efdfc35278023d959e4ba775b] sched: Teach dequeue_task() about special task states
git bisect skip e1459a50ba31831efdfc35278023d959e4ba775b
# skip: [a1c446611e31ca5363d4db51e398271da1dce0af] sched,freezer: Mark TASK_FROZEN special
git bisect skip a1c446611e31ca5363d4db51e398271da1dce0af
# good: [e28b5f8bda01720b5ce8456b48cf4b963f9a80a1] sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
git bisect good e28b5f8bda01720b5ce8456b48cf4b963f9a80a1
# skip: [f12e148892ede8d9ee82bcd3e469e6d01fc077ac] sched/fair: Prepare pick_next_task() for delayed dequeue
git bisect skip f12e148892ede8d9ee82bcd3e469e6d01fc077ac
# skip: [152e11f6df293e816a6a37c69757033cdc72667d] sched/fair: Implement delayed dequeue
git bisect skip 152e11f6df293e816a6a37c69757033cdc72667d
# skip: [2e0199df252a536a03f4cb0810324dff523d1e79] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
git bisect skip 2e0199df252a536a03f4cb0810324dff523d1e79
# bad: [54a58a78779169f9c92a51facf6de7ce94962328] sched/fair: Implement DELAY_ZERO
git bisect bad 54a58a78779169f9c92a51facf6de7ce94962328
# only skipped commits left to test # possible first bad commit: [54a58a78779169f9c92a51facf6de7ce94962328] sched/fair: Implement DELAY_ZERO # possible first bad commit: [152e11f6df293e816a6a37c69757033cdc72667d] sched/fair: Implement delayed dequeue # possible first bad commit: [e1459a50ba31831efdfc35278023d959e4ba775b] sched: Teach dequeue_task() about special task states # possible first bad commit: [a1c446611e31ca5363d4db51e398271da1dce0af] sched,freezer: Mark TASK_FROZEN special # possible first bad commit: [781773e3b68031bd001c0c18aa72e8470c225ebd] sched/fair: Implement ENQUEUE_DELAYED # possible first bad commit: [f12e148892ede8d9ee82bcd3e469e6d01fc077ac] sched/fair: Prepare pick_next_task() for delayed dequeue # possible first bad commit: [2e0199df252a536a03f4cb0810324dff523d1e79] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Peter Zijlstra
maintainerQuoted 30 lines · Sven Schnelle
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDI'm seeing crashes/warnings like the following on s390 with linux-next 20240909:
Quoted 7 lines
Quoted 7 lines
Sometimes the system doesn't manage to print a oops, this one is the best i got:
[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.
I noted: Comm: prctl-sched-cor, which is testing core scheduling, right?
Only today I;ve merged a fix for that:
c662e2b1e8cf ("sched: Fix sched_delayed vs sched_core")
Could you double check if merging tip/sched/core into your next tree helps anything at all?
Quoted 39 lines · Peter Zijlstra
Quoted 30 lines · Sven Schnelle
Quoted 28 lines · Sven Schnelle
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDI'm seeing crashes/warnings like the following on s390 with linux-next 20240909:
Quoted 7 lines
Quoted 7 lines
Sometimes the system doesn't manage to print a oops, this one is the best i got:
[..]
This happens when running the strace test suite. The system normaly has 128 CPUs. With this configuration the crash doesn't happen, but when disabling all but four CPUs and running 'make check -j16' in the strace test suite the crash is almost always reproducable.
Quoted 8 lines
I noted: Comm: prctl-sched-cor, which is testing core scheduling, right?
Only today I;ve merged a fix for that:
c662e2b1e8cf("sched: Fixsched_delayedvssched_core")Could you double check if merging tip/sched/core into your next tree helps anything at all?
Yes, that fixes the issue. Thanks!
Quoted 16 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 9 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
Hi Peter,
TL;DR:
We run some basic sched/cpufreq behavior tests on a Pixel 6 for every change we accept. Some of these changes are merges from Linus's tree. We can see a very clear change in behavior with this patch series. Based on what we are seeing, we'd expect this change in behavior to cause pretty serious power regression (7-25%) depending on what the actual bug is and the use case.
Intro:
We run these tests 20 times for every build (a bunch of changes). All the data below is from the 20+ builds before this series and 20 builds after this series (inclusive). So, all the "before numbers" are from (20 x 20) 400+ runs and all the after numbers are from another 400+ runs.
Test:
We create a synthetic "tiny" thread that runs for 3ms and sleeps for 10ms at Fmin. We let it run like this for several seconds to make sure the util is low and all the "new thread" boost stuff isn't kicking in. So, at this point, the CPU frequency is at Fmin. Then we let this thread run continuously without sleeping and measure (using ftrace) the time it takes for the CPU to get to Fmax.
We do this separately (fresh run) on the Pixel 6 with the cpu affinity set to each cluster and once without any cpu affinity (thread starts at little).
Data:
All the values below are in milliseconds.
When the thread is not affined to any CPU: So thread starts on little, ramps up to fmax, migrates to middle, ramps up to fmax, migrates to big, ramps up to fmax.
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 169 | 151 | |-----------------------+----------| | Median | 221 | 177 | |-----------------------+----------| | Mean | 221 | 177 | |-----------------------+----------| | 95th percentile | 249 | 200 | +----------------------------------+
When thread is affined to the little cluster:
The average time to reach Fmax is 104 ms without this series and 66 ms after this series. We didn't collect the individual per run data. We can if you really need it. We also noticed that the little cluster wouldn't go to Fmin (300 MHz) after this series even when the CPUs are mostly idle. It was instead hovering at 738 MHz (the Fmax is ~1800 MHz).
When thread is affined to the middle cluster:
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 99 | 84 | |-----------------------+----------| | Median | 111 | 104 | |-----------------------+----------| | Mean | 111 | 104 | |-----------------------+----------| | 95th percentile | 120 | 119 | +----------------------------------+
When thread is affined to the big cluster:
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 138 | 96 | |-----------------------+----------| | Median | 147 | 144 | |-----------------------+----------| | Mean | 145 | 134 | |-----------------------+----------| | 95th percentile | 151 | 150 | +----------------------------------+
As you can see, the ramp up time has decreased noticeably. Also, as you can tell from the 5th percentile numbers, the standard deviation has also increased a lot too, causing a wider spread of the ramp up time (more noticeable in the middle and big clusters). So in general this looks like it's going to increase the usage of the middle and big CPU clusters and also going to shift the CPU frequency residency to frequencies that are 5 to 25% higher.
We already checked the rate_limit_us value and it is the same for both the before/after cases and it's set to 7.5 ms (jiffies is 4ms in our case). Also, based on my limited understanding the DELAYED_DEQUEUE stuff is only relevant if there are multiple contending threads in a CPU. In this case it's just 1 continuously running thread with a kworker that runs sporadically less than 1% of the time.
So, without a deeper understanding of this patch series, it's behaving as if the PELT signal is accumulating faster than expected. Which is a bit surprising to me because AFAIK (which is not much) the EEVDF series isn't supposed to change the PELT behavior.
If you want to get a visual idea of what the system is doing, here are some perfetto links that visualize the traces. Hopefully you have access permissions to these. You can use the W, S, A, D keys to pan and zoom around the timeline.
Big Before:
https://ui.perfetto.dev/#!/?s=01aa3ad3a5ddd78f2948c86db4265ce2249da8aa
Big After:
https://ui.perfetto.dev/#!/?s=7729ee012f238e96cfa026459eac3f8c3e88d9a9
Thanks,
Saravana, Sam and David
K Prateek Nayak
reviewer(+ Mike, Luis)
Hello Saravana, Sam, David,
Quoted 29 lines · Saravana Kannan
Quoted 16 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 9 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDHi Peter,
Quoted 7 lines
TL;DR:
We run some basic sched/cpufreq behavior tests on a Pixel 6 for every change we accept. Some of these changes are merges from Linus's tree. We can see a very clear change in behavior with this patch series. Based on what we are seeing, we'd expect this change in behavior to cause pretty serious power regression (7-25%) depending on what the actual bug is and the use case.
Do the regressions persist with NO_DELAY_DEQUEUE? You can disable the DELAY_DEQUEUE feature added in EEVDF Complete via debugfs by doing a:
# echo NO_DELAY_DEQUEUE > /sys/kernel/debug/sched/features
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values for h_nr_delayed on his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempt h_nr_delayed tracking.
There is also the fact that delayed entities don't update the tg loadavg since the delayed path calls update_load_avg() without UPDATE_TG flag set which can again cause some changes in PELT calculation since the averages are used to estimate the entity shares when running with cgroups.
Quoted 80 lines
Intro:
We run these tests 20 times for every build (a bunch of changes). All the data below is from the 20+ builds before this series and 20 builds after this series (inclusive). So, all the "before numbers" are from (20 x 20) 400+ runs and all the after numbers are from another 400+ runs.Test:
We create a synthetic "tiny" thread that runs for 3ms and sleeps for 10ms at Fmin. We let it run like this for several seconds to make sure the util is low and all the "new thread" boost stuff isn't kicking in. So, at this point, the CPU frequency is at Fmin. Then we let this thread run continuously without sleeping and measure (using ftrace) the time it takes for the CPU to get to Fmax.We do this separately (fresh run) on the Pixel 6 with the cpu affinity set to each cluster and once without any cpu affinity (thread starts at little).
Data:
All the values below are in milliseconds.When the thread is not affined to any CPU: So thread starts on little, ramps up to fmax, migrates to middle, ramps up to fmax, migrates to big, ramps up to fmax.
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 169 | 151 | |-----------------------+----------| | Median | 221 | 177 | |-----------------------+----------| | Mean | 221 | 177 | |-----------------------+----------| | 95th percentile | 249 | 200 | +----------------------------------+When thread is affined to the little cluster:
The average time to reach Fmax is 104 ms without this series and 66 ms after this series. We didn't collect the individual per run data. We can if you really need it. We also noticed that the little cluster wouldn't go to Fmin (300 MHz) after this series even when the CPUs are mostly idle. It was instead hovering at 738 MHz (the Fmax is ~1800 MHz).When thread is affined to the middle cluster:
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 99 | 84 | |-----------------------+----------| | Median | 111 | 104 | |-----------------------+----------| | Mean | 111 | 104 | |-----------------------+----------| | 95th percentile | 120 | 119 | +----------------------------------+When thread is affined to the big cluster:
+----------------------------------+ | Data | Before | After | |-----------------------+----------| | 5th percentile | 138 | 96 | |-----------------------+----------| | Median | 147 | 144 | |-----------------------+----------| | Mean | 145 | 134 | |-----------------------+----------| | 95th percentile | 151 | 150 | +----------------------------------+As you can see, the ramp up time has decreased noticeably. Also, as you can tell from the 5th percentile numbers, the standard deviation has also increased a lot too, causing a wider spread of the ramp up time (more noticeable in the middle and big clusters). So in general this looks like it's going to increase the usage of the middle and big CPU clusters and also going to shift the CPU frequency residency to frequencies that are 5 to 25% higher.
Quoted 6 lines
We already checked the
rate_limit_usvalue and it is the same for both the before/after cases and it's set to 7.5 ms (jiffies is 4ms in our case). Also, based on my limited understanding theDELAYED_DEQUEUEstuff is only relevant if there are multiple contending threads in a CPU. In this case it's just 1 continuously running thread with a kworker that runs sporadically less than 1% of the time.
There is an ongoing investigation on delayed entities possibly not migrating if they are woken up before they are fully dequeued. Since you mention there is only one task, this should not matter but could you also try out Mike's suggestion from
https://lore.kernel.org/lkml/[email protected]/
and see if it makes a difference on your test suite?
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 11 lines
So, without a deeper understanding of this patch series, it's behaving as if the PELT signal is accumulating faster than expected. Which is a bit surprising to me because AFAIK (which is not much) the EEVDF series isn't supposed to change the PELT behavior.
If you want to get a visual idea of what the system is doing, here are some perfetto links that visualize the traces. Hopefully you have access permissions to these. You can use the W, S, A, D keys to pan and zoom around the timeline.
Quoted 4 lines
Big Before:
https://ui.perfetto.dev/#!/?s=01aa3ad3a5ddd78f2948c86db4265ce2249da8aa
Big After:
https://ui.perfetto.dev/#!/?s=7729ee012f238e96cfa026459eac3f8c3e88d9a9
P.S. I only gave a quick glance but I do see the frequency ramp up with larger deltas and reach Fmax much quickly in case of "Big After"
Quoted 3 lines
Thanks,
Saravana, Sam and David
Peter Zijlstra
maintainerQuoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Hi,
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.
Quoted 5 lines
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Sorry for the late reply, I got sidetracked on something else.
There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.
The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]
The bug also made it so disabling DELAY_DEQUEUE at runtime didn't fix things, because the imbalance/stale state would be perpetuated. Disabling DELAY_DEQUEUE before boot did fix things.
So power use was brought down by the above fix, but some issues still remained, like the accounting issues with h_nr_running and not taking sched_delayed tasks into account.
Dietmar addressed some of it with "kernel/sched: Fix util_est accounting for DELAY_DEQUEUE". [2]
Peter sent another patch to add accounting for sched_delayed tasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in the sched_delayed accounting that slowly drove frequencies up again.
If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
Quoted 40 lines · Luis Machado
Hi,
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.Quoted 5 lines
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Sorry for the late reply, I got sidetracked on something else.
There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.
The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]
The bug also made it so disabling
DELAY_DEQUEUEat runtime didn't fix things, because the imbalance/stale state would be perpetuated. DisablingDELAY_DEQUEUEbefore boot did fix things.So power use was brought down by the above fix, but some issues still remained, like the accounting issues with
h_nr_runningand not takingsched_delayedtasks into account.Dietmar addressed some of it with "kernel/sched: Fix
util_estaccounting forDELAY_DEQUEUE". [2]Peter sent another patch to add accounting for
sched_delayedtasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in thesched_delayedaccounting that slowly drove frequencies up again.
Quoted 8 lines
If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
Thanks for the replies. We are trying to disable DELAY_DEQUEUE and recollect the data to see if that's the cause. We'll get back to this thread once we have some data.
-Saravana
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado
Hi,
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.Quoted 5 lines
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Sorry for the late reply, I got sidetracked on something else.
There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.
The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]
The bug also made it so disabling
DELAY_DEQUEUEat runtime didn't fix things, because the imbalance/stale state would be perpetuated. DisablingDELAY_DEQUEUEbefore boot did fix things.So power use was brought down by the above fix, but some issues still remained, like the accounting issues with
h_nr_runningand not takingsched_delayedtasks into account.Dietmar addressed some of it with "kernel/sched: Fix
util_estaccounting forDELAY_DEQUEUE". [2]Peter sent another patch to add accounting for
sched_delayedtasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in thesched_delayedaccounting that slowly drove frequencies up again.Quoted 8 lines
If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
Quoted 5 lines
Thanks for the replies. We are trying to disable
DELAY_DEQUEUEand recollect the data to see if that's the cause. We'll get back to this thread once we have some data.-Saravana
The test data is back to pre-EEVDF state with DELAY_DEQUEUE disabled.
Same test example from before, when thread is affined to the big cluster:
+----------------------------------+ | Data | Enabled | Disabled | |-----------------------+----------| | 5th percentile | 96 | 143 | |-----------------------+----------| | Median | 144 | 147 | |-----------------------+----------| | Mean | 134 | 147 | |-----------------------+----------| | 95th percentile | 150 | 150 | +----------------------------------+
What are the next steps to bring this behavior back? Will DELAY_DEQUEUE always be enabled by default and/or is there a fix coming for 6.12?
Thanks,
Sam
K Prateek Nayak
reviewerHello Sam,
Quoted 73 lines · Samuel Wu
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado
Hi,
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.Quoted 5 lines
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Sorry for the late reply, I got sidetracked on something else.
There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.
The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]
The bug also made it so disabling
DELAY_DEQUEUEat runtime didn't fix things, because the imbalance/stale state would be perpetuated. DisablingDELAY_DEQUEUEbefore boot did fix things.So power use was brought down by the above fix, but some issues still remained, like the accounting issues with
h_nr_runningand not takingsched_delayedtasks into account.Dietmar addressed some of it with "kernel/sched: Fix
util_estaccounting forDELAY_DEQUEUE". [2]Peter sent another patch to add accounting for
sched_delayedtasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in thesched_delayedaccounting that slowly drove frequencies up again.Quoted 8 lines
If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/Quoted 5 lines
Thanks for the replies. We are trying to disable
DELAY_DEQUEUEand recollect the data to see if that's the cause. We'll get back to this thread once we have some data.-Saravana
The test data is back to pre-EEVDF state with
DELAY_DEQUEUEdisabled.Same test example from before, when thread is affined to the big cluster:
+----------------------------------+ | Data | Enabled | Disabled | |-----------------------+----------| | 5th percentile | 96 | 143 | |-----------------------+----------| | Median | 144 | 147 | |-----------------------+----------| | Mean | 134 | 147 | |-----------------------+----------| | 95th percentile | 150 | 150 | +----------------------------------+
Quoted 2 lines
What are the next steps to bring this behavior back? Will
DELAY_DEQUEUEalways be enabled by default and/or is there a fix coming for 6.12?
DELAY_DEQUEUE should be enabled by default from v6.12 but there are a few fixes for the same in-flight. Could try running with the changes from [1] and [2] and see if you could reproduce the behavior and if you can, is it equally bad?
Both changes apply cleanly for me on top of current
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
at commit fe9beaaa802d ("sched: No PREEMPT_RT=y for all{yes,mod}config") when applied in that order.
[1] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[2] https://lore.kernel.org/lkml/[email protected]/
Quoted 3 lines
Thanks,
Sam
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 87 lines · K Prateek Nayak
Hello Sam,
Quoted 73 lines · Samuel Wu
Quoted 51 lines · Saravana Kannan
Quoted 40 lines · Luis Machado
Hi,
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · K Prateek Nayak
Since delayed entities are still on the runqueue, they can affect PELT calculation. Vincent and Dietmar have both noted this and Peter posted
https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
in response but it was pulled out since Luis reported observing -ve values forh_nr_delayedon his setup. A lot has been fixed around delayed dequeue since and I wonder if now would be the right time to re-attempth_nr_delayedtracking.Quoted 5 lines
Yeah, it's something I meant to get back to. I think the patch as posted was actually right and it didn't work for Luis because of some other, since fixed issue.
But I might be misremembering things. I'll get to it eventually :/
Sorry for the late reply, I got sidetracked on something else.
There have been a few power regressions (based on our Pixel6-based testing) due to the delayed-dequeue series.
The main one drove the frequencies up due to an imbalance in the uclamp inc/dec handling. That has since been fixed by "[PATCH 10/24] sched/uclamg: Handle delayed dequeue". [1]
The bug also made it so disabling
DELAY_DEQUEUEat runtime didn't fix things, because the imbalance/stale state would be perpetuated. DisablingDELAY_DEQUEUEbefore boot did fix things.So power use was brought down by the above fix, but some issues still remained, like the accounting issues with
h_nr_runningand not takingsched_delayedtasks into account.Dietmar addressed some of it with "kernel/sched: Fix
util_estaccounting forDELAY_DEQUEUE". [2]Peter sent another patch to add accounting for
sched_delayedtasks [3]. Though the patch was mostly correct, under some circumstances [4] we spotted imbalances in thesched_delayedaccounting that slowly drove frequencies up again.Quoted 8 lines
If I recall correctly, Peter has pulled that particular patch from the tree, but we should definitely revisit it with a proper fix for the imbalance. Suggestion in [5].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/Quoted 5 lines
Thanks for the replies. We are trying to disable
DELAY_DEQUEUEand recollect the data to see if that's the cause. We'll get back to this thread once we have some data.-Saravana
The test data is back to pre-EEVDF state with
DELAY_DEQUEUEdisabled.Same test example from before, when thread is affined to the big cluster:
+----------------------------------+ | Data | Enabled | Disabled | |-----------------------+----------| | 5th percentile | 96 | 143 | |-----------------------+----------| | Median | 144 | 147 | |-----------------------+----------| | Mean | 134 | 147 | |-----------------------+----------| | 95th percentile | 150 | 150 | +----------------------------------+Quoted 2 lines
What are the next steps to bring this behavior back? Will
DELAY_DEQUEUEalways be enabled by default and/or is there a fix coming for 6.12?
DELAY_DEQUEUEshould be enabled by default from v6.12 but there are a few fixes for the same in-flight. Could try running with the changes from [1] and [2] and see if you could reproduce the behavior and if you can, is it equally bad?Both changes apply cleanly for me on top of current
Quoted 7 lines
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
at commit
fe9beaaa802d("sched: No PREEMPT_RT=y for all{yes,mod}config") when applied in that order.[1] https://lore.kernel.org/lkml/172595576232.2215.18027704125134691219.tip-bot2@tip-bot2/
[2] https://lore.kernel.org/lkml/[email protected]/
Have these changes landed in 6.12? Or will these in 6.13?
We tested 6.12 and the issue we reported is still present. What should we do for any products we want to ship on 6.12? Disable Delayed Dequeue or backport any fixes to 6.12 LTS?
Peter/Vincent, do you plan on backporting the future fixes to the 6.12 LTS kernel? Anything else we can do to help with making sure this is fixed on the LTS kernel?
-Saravana
Hi all,
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_ID
We found the following 7 commits from this patch set to crash in enqueue_dl_entity():
54a58a787791 sched/fair: Implement DELAY_ZERO152e11f6df29 sched/fair: Implement delayed dequeue e1459a50ba31 sched: Teach dequeue_task() about special task states a1c446611e31 sched,freezer: Mark TASK_FROZEN special 781773e3b680 sched/fair: Implement ENQUEUE_DELAYED f12e148892ed sched/fair: Prepare pick_next_task() for delayed dequeue 2e0199df252a sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Resulting in the following crash dump (this is running v6.12.1):
[ 14.652856] sched: DL replenish lagged too much [ 16.572706] ------------[ cut here ]------------ [ 16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c [ 16.573900] Modules linked in: overlay crct10dif_ce rk805_pwrkey snd_soc_es8316 pwm_fan
phy_rockchip_naneng_combphy rockchip_saradc rtc_hym8563 industrialio_triggered_buffer kfifo_buf rockchip_thermal phy_rockchip_usbdp typec spi_rockchip_sfc snd_soc_rockchip_i2s_tdm hantro_vpu v4l2_vp9 v4l2_h264 v4l2_jpeg panthor v4l2_mem2me
m rockchipdrm drm_gpuvm drm_exec drm_shmem_helper analogix_dp gpu_sched dw_mipi_dsi dw_hdmi cec drm_display_helper snd_soc_audio_graph_card snd_soc_simple_card_utils
drm_dma_helper drm_kms_helper cfg80211 rfkill pci_endpoint_test drm backlight dm_mod dax
[ 16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15 [ 16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024 [ 16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 16.580273] pc : enqueue_dl_entity+0x46c/0x55c [ 16.580661] lr : dl_server_start+0x44/0x12c [ 16.581028] sp : ffff80008002bc00 [ 16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000 [ 16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009 [ 16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0 [ 16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000 [ 16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698 [ 16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000 [ 16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009 [ 16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440 [ 16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0 [ 16.587540] Call trace: [ 16.587754] enqueue_dl_entity+0x46c/0x55c [ 16.588113] dl_server_start+0x44/0x12c [ 16.588449] enqueue_task_fair+0x124/0x49c [ 16.588807] enqueue_task+0x3c/0xe0 [ 16.589113] ttwu_do_activate.isra.0+0x6c/0x208 [ 16.589511] try_to_wake_up+0x1d0/0x61c [ 16.589847] wake_up_process+0x18/0x24 [ 16.590175] kick_pool+0x84/0x150 [ 16.590467] __queue_work+0x2f4/0x544 [ 16.590788] delayed_work_timer_fn+0x1c/0x28 [ 16.591161] call_timer_fn+0x34/0x1ac [ 16.591481] __run_timer_base+0x20c/0x314 [ 16.591832] run_timer_softirq+0x3c/0x78 [ 16.592176] handle_softirqs+0x124/0x35c [ 16.592520] __do_softirq+0x14/0x20 [ 16.592827] ____do_softirq+0x10/0x1c [ 16.593148] call_on_irq_stack+0x24/0x4c [ 16.593490] do_softirq_own_stack+0x1c/0x2c [ 16.593857] irq_exit_rcu+0x8c/0xc0 [ 16.594163] el0_interrupt+0x48/0xbc [ 16.594477] __el0_irq_handler_common+0x18/0x24 [ 16.594874] el0t_64_irq_handler+0x10/0x1c [ 16.595232] el0t_64_irq+0x190/0x194 [ 16.595545] ---[ end trace 0000000000000000 ]--- [ 16.595950] ------------[ cut here ]------------
It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].
We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.
We haven't had time to do a non-proprietary reproduction case as of yet but I wanted to report our current findings asking for any feedback/suggestions from the community.
Thanks!
Cheers
Marcel
Peter Zijlstra
maintainerQuoted 49 lines · Marcel Ziswiler
Resulting in the following crash dump (this is running v6.12.1):
[ 14.652856] sched: DL replenish lagged too much [ 16.572706] ------------[ cut here ]------------ [ 16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c[ 16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15 [ 16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024 [ 16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 16.580273] pc : enqueue_dl_entity+0x46c/0x55c [ 16.580661] lr : dl_server_start+0x44/0x12c [ 16.581028] sp : ffff80008002bc00 [ 16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000 [ 16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009 [ 16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0 [ 16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000 [ 16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698 [ 16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000 [ 16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009 [ 16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440 [ 16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0 [ 16.587540] Call trace: [ 16.587754] enqueue_dl_entity+0x46c/0x55c [ 16.588113] dl_server_start+0x44/0x12c [ 16.588449] enqueue_task_fair+0x124/0x49c [ 16.588807] enqueue_task+0x3c/0xe0 [ 16.589113] ttwu_do_activate.isra.0+0x6c/0x208 [ 16.589511] try_to_wake_up+0x1d0/0x61c [ 16.589847] wake_up_process+0x18/0x24 [ 16.590175] kick_pool+0x84/0x150 [ 16.590467] __queue_work+0x2f4/0x544 [ 16.590788] delayed_work_timer_fn+0x1c/0x28 [ 16.591161] call_timer_fn+0x34/0x1ac [ 16.591481] __run_timer_base+0x20c/0x314 [ 16.591832] run_timer_softirq+0x3c/0x78 [ 16.592176] handle_softirqs+0x124/0x35c [ 16.592520] __do_softirq+0x14/0x20 [ 16.592827] ____do_softirq+0x10/0x1c [ 16.593148] call_on_irq_stack+0x24/0x4c [ 16.593490] do_softirq_own_stack+0x1c/0x2c [ 16.593857] irq_exit_rcu+0x8c/0xc0 [ 16.594163] el0_interrupt+0x48/0xbc [ 16.594477] __el0_irq_handler_common+0x18/0x24 [ 16.594874] el0t_64_irq_handler+0x10/0x1c [ 16.595232] el0t_64_irq+0x190/0x194 [ 16.595545] ---[ end trace 0000000000000000 ]--- [ 16.595950] ------------[ cut here ]------------
Quoted 1 line
It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].
Right, I've had a number of these reports, but so far we've not yet managed to figure out how it's all happening.
Quoted 3 lines
We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.
Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?
A very long shot:
https://lkml.kernel.org/r/[email protected]
doesn't help, does it?
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Marcel Ziswiler
Resulting in the following crash dump (this is running v6.12.1):
[snip]
Quoted 6 lines
Quoted 2 lines
It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].
Right, I've had a number of these reports, but so far we've not yet managed to figure out how it's all happening.
Quoted 7 lines
Quoted 3 lines
We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.
Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?
As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.
Quoted 5 lines
A very long shot:
https://lkml.kernel.org/r/[email protected]
doesn't help, does it?
No, still the same with this on top of today's -next.
Thanks!
Cheers
Marcel
Peter Zijlstra
maintainerQuoted 6 lines · Marcel Ziswiler
Quoted 3 lines
Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?
As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.
I was hoping you could perhaps share a binary with Juri privately or with RHT (same difference etc), such that he can poke at it too.
Anyway, if you don't mind a bit of back and forth, would you mind adding the below patch to your kernel and doing:
(all assuming your kernel has ftrace enabled)
echo 1 > /sys/kernel/debug/tracing/options/stacktrace echo 1 > /proc/sys/kernel/traceoff_on_warning
running your test to failure and then dumping the trace into a file like:
cat /sys/kernel/debug/tracing/trace > ~/trace
Then compress the file (bzip2 or whatever is popular these days) and send it my way along with a dmesg dump (private is fine -- these things tend to be large-ish).
Hopefully, this will give us a little clue as to where the double enqueue happens.
---
kernel/sched/deadline.c +20 -0
@@ -1203,6 +1203,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ scoped_guard (rq_lock, rq) {struct rq_flags *rf = &scope.rf; +if (dl_se == &rq->fair_server) { +trace_printk ("timer fair server %d throttled %d\n" , +cpu_of (rq), dl_se->dl_throttled); + } +if (!dl_se->dl_throttled || !dl_se->dl_runtime)return HRTIMER_NORESTART ;
@@ -1772,6 +1777,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)rq_lock (rq, &rf); } +if (dl_se == &rq->fair_server) +trace_printk ("inactive fair server %d\n" ,cpu_of (rq)); +sched_clock_tick ();update_rq_clock (rq);
@@ -1967,6 +1975,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("enqueue fair server %d h_nr_running %d\n" , +cpu_of (rq), rq->cfs.h_nr_running); + }WARN_ON_ONCE (!RB_EMPTY_NODE (&dl_se->rb_node));
@@ -1978,6 +1992,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)static void __dequeue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("dequeue fair server %d h_nr_running %d\n" , +cpu_of (rq), rq->cfs.h_nr_running); + }if (RB_EMPTY_NODE (&dl_se->rb_node))return ;
Sorry for my late reply, I was traveling back from Manchester to Switzerland but I am all settled down again.
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Marcel Ziswiler
Quoted 3 lines
Oooh, that's something. So far the few reports have not been (easily) reproducible. If this is readily reproducible on arm64 that would help a lot. Juri, do you have access to an arm64 test box?
As mentioned above, so far our scheduler stress test is not yet open source but Codethink is eager to share anything which helps in resolving this.
I was hoping you could perhaps share a binary with Juri privately or with RHT (same difference etc), such that he can poke at it too.
Sure, there is nothing secret about it, it is just that we have not gotten around open sourcing all parts of it just yet.
The UEFI aarch64 embedded Linux image I am using may be found here [1]. Plus matching bmap file should you fancy using that [2]. And the SSH key may help when interacting with the system (e.g. that is how I trigger the failure as the console is quite busy with tracing) [3]. However, that was built by CI and does not contain a kernel with below patch applied yet. I manually dumped the kernel config and compiled v6.12.1 with your patch applied and deployed it (to /lib/modules, /usr/lib/kernel et. al.) in the below case where I provide the dump.
Quoted 1 line
Anyway, if you don't mind a bit of back and forth,
Sure.
Quoted 5 lines
would you mind adding
the below patch to your kernel and doing:(all assuming your kernel has ftrace enabled)
Quoted 7 lines
echo 1 >
/sys/kernel/debug/tracing/options/stacktraceecho 1 >/proc/sys/kernel/traceoff_on_warningrunning your test to failure and then dumping the trace into a file like:
cat
/sys/kernel/debug/tracing/trace> ~/trace
Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:
tail -f /sys/kernel/debug/tracing/trace
Not sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.
Quoted 1 line
Then compress the file (bzip2 or whatever is popular these days)
xz or zstd (;-p)
Quoted 3 lines
and
send it my way along with a dmesg dump (private is fine -- these things tend to be large-ish).
As mentioned before, there is nothing secret about it. Please find it here [4].
Quoted 2 lines
Hopefully, this will give us a little clue as to where the double enqueue happens.
Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!
Quoted 50 lines
---
kernel/sched/deadline.c +0
-0@@ -1203,6 +1203,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ scoped_guard (rq_lock, rq) { struct rq_flags *rf = &scope.rf;
+ if (dl_se==&rq->fair_server) {
+ trace_printk("timer fair server %d throttled %d\n",
+cpu_of(rq),dl_se->dl_throttled);
+ }
+
if (!dl_se->dl_throttled || !dl_se->dl_runtime) returnHRTIMER_NORESTART;kernel/sched/deadline.c +0
-0@@ -1772,6 +1777,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) rq_lock(rq, &rf); }
+ if (dl_se==&rq->fair_server)
+trace_printk("inactive fair server %d\n", cpu_of(rq)); +
sched_clock_tick();
update_rq_clock(rq);kernel/sched/deadline.c +6
-0@@ -1967,6 +1975,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, static void __enqueue_dl_entity(struct sched_dl_entity *dl_se) { struct dl_rq *dl_rq = dl_rq_of_se(dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("enqueue fair server %d h_nr_running %d\n" , + cpu_of(rq), rq->cfs.h_nr_running); + }
WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));kernel/sched/deadline.c +6
-0@@ -1978,6 +1992,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se) static void __dequeue_dl_entity(struct sched_dl_entity *dl_se) { struct dl_rq *dl_rq = dl_rq_of_se(dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("dequeue fair server %d h_nr_running %d\n" , + cpu_of(rq), rq->cfs.h_nr_running); + }
Quoted 3 lines
if (RB_EMPTY_NODE(&dl_se->rb_node))
return;
[1] https://drive.codethink.co.uk/s/N8CQipaNNN45gYM
[2] https://drive.codethink.co.uk/s/mpcPawXpCjPL8D3
[3] https://drive.codethink.co.uk/s/8RjHNTQQRpYgaLc
[4] https://drive.codethink.co.uk/s/MWtzWjLDtdD3E5i
Cheers
Marcel
Peter Zijlstra
maintainerSorry for the delay, I got laid low by snot monsters :/
Quoted 7 lines · Marcel Ziswiler
Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:
tail -f
/sys/kernel/debug/tracing/traceNot sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.
Ah, that is unfortunate. There is a ftrace_dump_on_oops option that might be of help. And yes, dumping trace buffers over 1m5 serial lines is tedious -- been there done that, got a t-shirt and all that.
Still, let me see if perhaps making that WARN in enqueue_dl_entity() return makes the whole thing less fatal.
I've included the traceoff_on_warning and ftrace_dump in the code, so all you really need to still do is enable the stacktrace option.
echo 1 > /sys/kernel/debug/tracing/options/stacktrace
Quoted 1 line
Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!
Could I bother you to try again with the below patch?
There are two new hunks vs the previous one, the hunk in enqueue_dl_entity() (the very last bit) will stop tracing and dump the buffers when that condition is hit in addition to then aborting the double enqueue, hopefully leaving the system is a slightly better state.
The other new hunk is the one for dl_server_stop() (second hunk), while going over the code last week, I found that this might be a possible hole leading to the observed double enqueue, so fingers crossed.
---
kernel/sched/deadline.c +31 -1
@@ -1223,6 +1223,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ scoped_guard (rq_lock, rq) {struct rq_flags *rf = &scope.rf; +if (dl_se == &rq->fair_server) { +trace_printk ("timer fair server %d throttled %d\n" , +cpu_of (rq), dl_se->dl_throttled); + } +if (!dl_se->dl_throttled || !dl_se->dl_runtime)return HRTIMER_NORESTART ;
@@ -1674,6 +1679,12 @@ void dl_server_start(struct sched_dl_entity *dl_se)void dl_server_stop (struct sched_dl_entity *dl_se) { +if (current->dl_server == dl_se) { +struct rq *rq =rq_of_dl_se (dl_se); +trace_printk ("stop fair server %d\n" ,cpu_of (rq)); + current->dl_server =NULL ; + } +if (!dl_se->dl_runtime)return ;
@@ -1792,6 +1803,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)rq_lock (rq, &rf); } +if (dl_se == &rq->fair_server) +trace_printk ("inactive fair server %d\n" ,cpu_of (rq)); +sched_clock_tick ();update_rq_clock (rq);
@@ -1987,6 +2001,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se,static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("enqueue fair server %d h_nr_running %d\n" , +cpu_of (rq), rq->cfs.h_nr_running); + }WARN_ON_ONCE (!RB_EMPTY_NODE (&dl_se->rb_node));
@@ -1998,6 +2018,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)static void __dequeue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("dequeue fair server %d h_nr_running %d\n" , +cpu_of (rq), rq->cfs.h_nr_running); + }if (RB_EMPTY_NODE (&dl_se->rb_node))return ;
@@ -2012,7 +2038,11 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)static void enqueue_dl_entity (struct sched_dl_entity *dl_se,int flags) { -WARN_ON_ONCE (on_dl_rq (dl_se)); +if (WARN_ON_ONCE (on_dl_rq (dl_se))) { +tracing_off (); +ftrace_dump (DUMP_ALL ); +return ; + }update_stats_enqueue_dl (dl_rq_of_se (dl_se), dl_se, flags);
Quoted 21 lines · Peter Zijlstra
Sorry for the delay, I got laid low by snot monsters :/
Quoted 8 lines · Marcel Ziswiler
Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace
buffer any longer. So I did the following instead on the serial console terminal:tail -f
/sys/kernel/debug/tracing/traceNot sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.
Ah, that is unfortunate. There is a
ftrace_dump_on_oopsoption that might be of help. And yes, dumping trace buffers over 1m5 serial lines is tedious -- been there done that, got a t-shirt and all that.Still, let me see if perhaps making that WARN in
enqueue_dl_entity()return makes the whole thing less fatal.
Quoted 8 lines
I've included the
traceoff_on_warningandftrace_dumpin the code, so all you really need to still do is enable the stacktrace option.echo 1 >
/sys/kernel/debug/tracing/options/stacktraceQuoted 1 line
Yes, and do not hesitate to ask for any additional information et. al. we are happy to help. Thanks!
Could I bother you to try again with the below patch?
Sure, here you go.
https://drive.codethink.co.uk/s/HniZCtccDBMHpAK
Quoted 88 lines
There are two new hunks vs the previous one, the hunk in
enqueue_dl_entity()(the very last bit) will stop tracing and dump the buffers when that condition is hit in addition to then aborting the double enqueue, hopefully leaving the system is a slightly better state.The other new hunk is the one for
dl_server_stop()(second hunk), while going over the code last week, I found that this might be a possible hole leading to the observed double enqueue, so fingers crossed.---
kernel/sched/deadline.c +0
-0@@ -1223,6 +1223,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ scoped_guard (rq_lock, rq) { struct rq_flags *rf = &scope.rf;kernel/sched/deadline.c +5
-0+if (dl_se == &rq->fair_server) { +trace_printk ("timer fair server %d throttled %d\n" , + cpu_of(rq), dl_se->dl_throttled); + } +if (!dl_se->dl_throttled || !dl_se->dl_runtime) return
HRTIMER_NORESTART;kernel/sched/deadline.c +0
-0@@ -1674,6 +1679,12 @@ void dl_server_start(struct sched_dl_entity *dl_se)
voiddl_server_stop(struct sched_dl_entity *dl_se){kernel/sched/deadline.c +6
-0+if (current->dl_server == dl_se) { +struct rq *rq =rq_of_dl_se (dl_se); +trace_printk ("stop fair server %d\n" ,cpu_of (rq)); + current->dl_server =NULL ; + } +if (!dl_se->dl_runtime)
return;kernel/sched/deadline.c +0
-0@@ -1792,6 +1803,9 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) rq_lock(rq, &rf); }kernel/sched/deadline.c +3
-0+if (dl_se == &rq->fair_server) +trace_printk ("inactive fair server %d\n" ,cpu_of (rq)); +
sched_clock_tick();
update_rq_clock(rq);kernel/sched/deadline.c +6
-0@@ -1987,6 +2001,12 @@ update_stats_dequeue_dl(struct dl_rq *dl_rq, struct sched_dl_entity *dl_se, static void __enqueue_dl_entity(struct sched_dl_entity *dl_se) { struct dl_rq *dl_rq = dl_rq_of_se(dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("enqueue fair server %d h_nr_running %d\n" , + cpu_of(rq), rq->cfs.h_nr_running); + }WARN_ON_ONCE(!RB_EMPTY_NODE(&dl_se->rb_node));
kernel/sched/deadline.c +6
-0@@ -1998,6 +2018,12 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se) static void __dequeue_dl_entity(struct sched_dl_entity *dl_se) { struct dl_rq *dl_rq = dl_rq_of_se(dl_se); +struct rq *rq =rq_of_dl_se (dl_se); + +if (dl_se == &rq->fair_server) { +trace_printk ("dequeue fair server %d h_nr_running %d\n" , + cpu_of(rq), rq->cfs.h_nr_running); + }
if (RB_EMPTY_NODE(&dl_se->rb_node))
return;kernel/sched/deadline.c +0
-0@@ -2012,7 +2038,11 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)static void
enqueue_dl_entity(structsched_dl_entity*dl_se, int flags) {kernel/sched/deadline.c +5
-1-WARN_ON_ONCE (on_dl_rq (dl_se)); +if (WARN_ON_ONCE (on_dl_rq (dl_se))) { +tracing_off (); +ftrace_dump (DUMP_ALL ); +return ; + }
update_stats_enqueue_dl(dl_rq_of_se(dl_se), dl_se, flags);
Steven Rostedt
maintainerQuoted 7 lines · On Mon, 02 Dec 2024 19:46:21 +0100 Marcel Ziswiler…
Unfortunately, once I trigger the failure the system is completely dead and won't allow me to dump the trace buffer any longer. So I did the following instead on the serial console terminal:
tail -f
/sys/kernel/debug/tracing/traceNot sure whether there is any better way to go about this. Plus even so we run the serial console at 1.5 megabaud I am not fully sure whether it was able to keep up logging what you are looking for.
If the memory of the machine is persistent (it is on several of my machines) you can use the persistent ring buffer.
Add to the kernel command line (or enable bootconfig that attaches a command line to the kernel if you can't change the parameters):
reserve_mem=20M:12M:trace trace_instance=boot_map^traceoff^traceprintk@trace
The above will create a "boot_map" instance with tracing off on boot and trace_printk() going to it. Start tracing:
trace-cmd start -B boot_map -p nop
If or replace "-p nop" with any events or tracers you want, including function tracing", then after a crash.
trace-cmd show -B boot_map
If the memory is persistent and you don't use KASLR (may want to also add nokaslr if arm64 supports KASLR it and you use it), you should get the trace right up to the crash.
See Documentation/trace/debugging.rst for more details.
-- Steve
Quoted 83 lines · Marcel Ziswiler
Hi all,
Quoted 15 lines · Peter Zijlstra
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Quoted 8 lines
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
-PLACE_REL_DEADLINE-- preserve the relative deadline when migrating
-SCHED_BATCHis equivalent toRESPECT_SLICE
- propagatemin_sliceup cgroups
-CLOCK_THREAD_DVFS_IDWe found the following 7 commits from this patch set to crash in
enqueue_dl_entity():
54a58a787791sched/fair: ImplementDELAY_ZERO152e11f6df29sched/fair: Implement delayed dequeuee1459a50ba31sched: Teachdequeue_task()about special task statesa1c446611e31sched,freezer: MarkTASK_FROZENspecial781773e3b680sched/fair: ImplementENQUEUE_DELAYEDf12e148892edsched/fair: Preparepick_next_task()for delayed dequeue2e0199df252asched/fair: Prepare exit/cleanup paths fordelayed_dequeueResulting in the following crash dump (this is running v6.12.1):
[ 14.652856] sched: DL replenish lagged too much [ 16.572706] ------------[ cut here ]------------ [ 16.573115] WARNING: CPU: 5 PID: 912 at kernel/sched/deadline.c:1995 enqueue_dl_entity+0x46c/0x55c [ 16.573900] Modules linked in: overlay crct10dif_ce rk805_pwrkey snd_soc_es8316 pwm_fan
phy_rockchip_naneng_combphyrockchip_saradcrtc_hym8563industrialio_triggered_bufferkfifo_bufrockchip_thermalphy_rockchip_usbdptypecspi_rockchip_sfcsnd_soc_rockchip_i2s_tdmhantro_vpuv4l2_vp9v4l2_h264v4l2_jpegpanthorv4l2_mem2me
m rockchipdrmdrm_gpuvmdrm_execdrm_shmem_helperanalogix_dpgpu_scheddw_mipi_dsidw_hdmicecdrm_display_helpersnd_soc_audio_graph_cardsnd_soc_simple_card_utils
drm_dma_helperdrm_kms_helpercfg80211 rfkillpci_endpoint_testdrm backlightdm_moddax[ 16.578350] CPU: 5 UID: 0 PID: 912 Comm: job10 Not tainted 6.12.1-dirty #15 [ 16.578956] Hardware name: radxa Radxa ROCK 5B/Radxa ROCK 5B, BIOS 2024.10-rc3 10/01/2024 [ 16.579667] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 16.580273] pc : enqueue_dl_entity+0x46c/0x55c [ 16.580661] lr : dl_server_start+0x44/0x12c [ 16.581028] sp : ffff80008002bc00 [ 16.581318] x29: ffff80008002bc00 x28: dead000000000122 x27: 0000000000000000 [ 16.581941] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000009 [ 16.582563] x23: ffff33c976db0e40 x22: 0000000000000001 x21: 00000000002dc6c0 [ 16.583186] x20: 0000000000000001 x19: ffff33c976db17a8 x18: 0000000000000000 [ 16.583808] x17: ffff5dd9779ac000 x16: ffff800080028000 x15: 11c3485b851e0698 [ 16.584430] x14: 11b4b257e4156000 x13: 0000000000000255 x12: 0000000000000000 [ 16.585053] x11: ffff33c976db0ec0 x10: 0000000000000000 x9 : 0000000000000009 [ 16.585674] x8 : 0000000000000005 x7 : ffff33c976db19a0 x6 : ffff33c78258b440 [ 16.586296] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 16.586918] x2 : 0000000000000001 x1 : 0000000000000001 x0 : ffff33c798e112f0 [ 16.587540] Call trace: [ 16.587754] enqueue_dl_entity+0x46c/0x55c [ 16.588113] dl_server_start+0x44/0x12c [ 16.588449] enqueue_task_fair+0x124/0x49c [ 16.588807] enqueue_task+0x3c/0xe0 [ 16.589113] ttwu_do_activate.isra.0+0x6c/0x208 [ 16.589511] try_to_wake_up+0x1d0/0x61c [ 16.589847] wake_up_process+0x18/0x24 [ 16.590175] kick_pool+0x84/0x150 [ 16.590467] __queue_work+0x2f4/0x544 [ 16.590788] delayed_work_timer_fn+0x1c/0x28 [ 16.591161] call_timer_fn+0x34/0x1ac [ 16.591481] __run_timer_base+0x20c/0x314 [ 16.591832] run_timer_softirq+0x3c/0x78 [ 16.592176] handle_softirqs+0x124/0x35c [ 16.592520] __do_softirq+0x14/0x20 [ 16.592827] ____do_softirq+0x10/0x1c [ 16.593148] call_on_irq_stack+0x24/0x4c
Quoted 8 lines
[ 16.593490] do_softirq_own_stack+0x1c/0x2c [ 16.593857] irq_exit_rcu+0x8c/0xc0 [ 16.594163] el0_interrupt+0x48/0xbc [ 16.594477] __el0_irq_handler_common+0x18/0x24 [ 16.594874] el0t_64_irq_handler+0x10/0x1c [ 16.595232] el0t_64_irq+0x190/0x194 [ 16.595545] ---[ end trace 0000000000000000 ]--- [ 16.595950] ------------[ cut here ]------------
Random piece of data, but I also had some difficulty making things boot on Android when trying Vincent's nr_running accounting series due to a very similar crash/stack trace. Though what I saw went a bit further and actually crashed within task_contending, called from enqueue_dl_entity. Possibly crashed in one of the inlined functions.
Even though the kernel was 6.8 and it was backports, it seems awfully similar to the above.
Quoted 17 lines
It looks like it is trying to enqueue an already queued deadline task. Full serial console log available [1].
We are running the exact same scheduler stress test both on Intel NUCs as well as RADXA ROCK 5B board farms. While so far we have not seen this on amd64 it crashes consistently/reproducible on aarch64.
We haven't had time to do a non-proprietary reproduction case as of yet but I wanted to report our current findings asking for any feedback/suggestions from the community.
Thanks!
Cheers
Marcel