sched/fair: Update rq_clock, cfs_rq before migrating for asym cpu capacity
PATCH 1/1 sched/fair: Update rq_clock, cfs_rq before migrating for asym cpu capacity
The ancient workaround to avoid the cost of updating rq clocks in the middle of a migration causes some issues on asymmetric CPU capacity systems where we use task utilization to determine which cpus fit a task. On quiet systems we can inflate task util after a migration which causes misfit to fire and force-migrate the task.
This occurs when:
(a) a task has util close to the non-overutilized capacity limit of a particular cpu (cpu0 here); and
(b) the prev_cpu was quiet otherwise, such that rq clock is sufficiently out of date (cpu1 here).
e.g.
_____
cpu0: ________________________| |______________
|<- misfit happens
______ ___ ___
cpu1: ____| |______________|___| |_________|->| |<- wakeup migration time
last rq clock update
When the task util is in just the right range for the system, we can end up migrating an unlucky task back and forth many times until we are lucky and the source rq happens to be updated close to the migration time.
In order to address this, lets update both rq_clock and cfs_rq where this could be an issue.
--- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
kernel/sched/fair.c +15 -0
@@ -6545,6 +6545,21 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* wakee task is less decayed, but giving the wakee more load * sounds not bad. */ +if (static_branch_unlikely (&sched_asym_cpucapacity) && + p->state ==TASK_WAKING ) { +/* +* On asymmetric capacity systems task util guides +* wake placement so update rq_clock and cfs_rq +*/ +struct cfs_rq *cfs_rq =task_cfs_rq (p); +struct rq *rq =task_rq (p); +struct rq_flags rf; + +rq_lock_irqsave (rq, &rf); +update_rq_clock (rq); +update_cfs_rq_load_avg (cfs_rq_clock_pelt (cfs_rq), cfs_rq); +rq_unlock_irqrestore (rq, &rf); + }remove_entity_load_avg (&p->se); }
Signature · 1 line
2.17.1
Peter Zijlstra
maintainerQuoted 22 lines · Chris Redpath
The ancient workaround to avoid the cost of updating rq clocks in the middle of a migration causes some issues on asymmetric CPU capacity systems where we use task utilization to determine which cpus fit a task. On quiet systems we can inflate task util after a migration which causes misfit to fire and force-migrate the task.
This occurs when:
(a) a task has util close to the non-overutilized capacity limit of a particular cpu (cpu0 here); and
(b) theprev_cpuwas quiet otherwise, such that rq clock is sufficiently out of date (cpu1 here).e.g.
_____cpu0: ________________________| |______________
|<- misfit happens ______ ___ ___ cpu1: ____| |______________|___| |_________|->| |<- wakeup migration time
Quoted 8 lines
last rq clock update
When the task util is in just the right range for the system, we can end up migrating an unlucky task back and forth many times until we are lucky and the source rq happens to be updated close to the migration time.
In order to address this, lets update both
rq_clockandcfs_rqwhere this could be an issue.
Can you quantify how much of a problem this really is? It is really sad, but this is already the second place where we take rq->lock on migration. We worked so hard to avoid having to acquire it :/
Quoted 10 lines
Signed-off-by Chris Redpath <[email protected]>--- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void migrate_task_rq_fair(struct task_struct *p, int new_cpu) · line 6549
Quoted 5 lines
kernel/sched/fair.c +2
-0@@ -6545,6 +6545,21 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* wakee task is less decayed, but giving the wakee more load * sounds not bad. */ +if (static_branch_unlikely (&sched_asym_cpucapacity) && + p->state ==TASK_WAKING ) {
nit: indent fail.
Quoted 19 lines
kernel/sched/fair.c +13
-0+/* +* On asymmetric capacity systems task util guides +* wake placement so update rq_clock and cfs_rq +*/ +struct cfs_rq *cfs_rq =task_cfs_rq (p); +struct rq *rq =task_rq (p); +struct rq_flags rf; + +rq_lock_irqsave (rq, &rf); +update_rq_clock (rq); +update_cfs_rq_load_avg (cfs_rq_clock_pelt (cfs_rq), cfs_rq); +rq_unlock_irqrestore (rq, &rf); + }remove_entity_load_avg (&p->se); }Signature · 1 line
2.17.1
Hi Peter,
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Chris Redpath
The ancient workaround to avoid the cost of updating rq clocks in the middle of a migration causes some issues on asymmetric CPU capacity systems where we use task utilization to determine which cpus fit a task. On quiet systems we can inflate task util after a migration which causes misfit to fire and force-migrate the task.
This occurs when:
(a) a task has util close to the non-overutilized capacity limit of a particular cpu (cpu0 here); and
(b) theprev_cpuwas quiet otherwise, such that rq clock is sufficiently out of date (cpu1 here).e.g.
_____cpu0: ________________________| |______________
|<- misfit happens
______ ___ ___
cpu1: ____| |______________|___| |_________|->| |<- wakeup migration timelast rq clock update
When the task util is in just the right range for the system, we can end up migrating an unlucky task back and forth many times until we are lucky and the source rq happens to be updated close to the migration time.
Quoted 7 lines
Quoted 2 lines
In order to address this, lets update both
rq_clockandcfs_rqwhere this could be an issue.Can you quantify how much of a problem this really is? It is really sad, but this is already the second place where we take
rq->lockon migration. We worked so hard to avoid having to acquire it :/
I think you're familiar with the way we test the EAS and misfit stuff, but some might not be, so I'll just outline them.
We have performance and placement tests for a suite of simple synthetic scenarios selected to trigger the EAS & misfit mechanisms. The performance tests use rt-app's slack metric, and we try to minimise negative slack (i.e. missed deadlines).
In the placement tests we estimate the minimum energy consumed to run a particular synthetic test job and we calculate the energy consumed in the actual execution according to a trace. We pass the test if our estimate of actual is less than ideal+20%.
We enter this code quite often in our testing, most individual runs of a test which has small tasks involved have at least one hit where we make a change to the clock with this patch in.
That said - despite the relatively high number of hits only about 5% of runs see enough additional energy consumed to trigger a test failure. We do try to keep a quiet system as much as possible and only run for a few seconds so the impact we see in testing is also probably higher than in the real world.
I totally appreciate the reluctance to add this - I don't much like it either, but I was hoping that sticking it behind the asym_cpucapacity key might be a reasonable compromise.
At least only those people who select a CPU using task util and capacity see this cost, and we have smaller systems so in theory the cost is smaller.
I'm very open to exploring alternatives :)
Quoted 5 lines
Quoted 5 lines
Signed-off-by Chris Redpath <[email protected]>--- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
Quoted 13 lines
kernel/sched/fair.c static void migrate_task_rq_fair(struct task_struct *p, int new_cpu) · line 6549
Quoted 10 lines
kernel/sched/fair.c +2
-0@@ -6545,6 +6545,21 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* wakee task is less decayed, but giving the wakee more load * sounds not bad. */ +if (static_branch_unlikely (&sched_asym_cpucapacity) && + p->state ==TASK_WAKING ) {nit: indent fail.
oops, will tweak it
--Chris
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Vincent Guittot
maintainerHi Chris,
Quoted 49 lines · Chris Redpath
Hi Peter,
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Chris Redpath
The ancient workaround to avoid the cost of updating rq clocks in the middle of a migration causes some issues on asymmetric CPU capacity systems where we use task utilization to determine which cpus fit a task. On quiet systems we can inflate task util after a migration which causes misfit to fire and force-migrate the task.
This occurs when:
(a) a task has util close to the non-overutilized capacity limit of a particular cpu (cpu0 here); and
(b) theprev_cpuwas quiet otherwise, such that rq clock is sufficiently out of date (cpu1 here).e.g.
_____cpu0: ________________________| |______________
|<- misfit happens
______ ___ ___
cpu1: ____| |______________|___| |_________|->| |<- wakeup migration timelast rq clock update
When the task util is in just the right range for the system, we can end up migrating an unlucky task back and forth many times until we are lucky and the source rq happens to be updated close to the migration time.
Quoted 7 lines
Quoted 2 lines
In order to address this, lets update both
rq_clockandcfs_rqwhere this could be an issue.Can you quantify how much of a problem this really is? It is really sad, but this is already the second place where we take
rq->lockon migration. We worked so hard to avoid having to acquire it :/I think you're familiar with the way we test the EAS and misfit stuff, but some might not be, so I'll just outline them.
We have performance and placement tests for a suite of simple synthetic scenarios selected to trigger the EAS & misfit mechanisms. The performance tests use rt-app's slack metric, and we try to minimise negative slack (i.e. missed deadlines).
Quoted 8 lines
In the placement tests we estimate the minimum energy consumed to run a particular synthetic test job and we calculate the energy consumed in the actual execution according to a trace. We pass the test if our estimate of actual is less than ideal+20%.
We enter this code quite often in our testing, most individual runs of a test which has small tasks involved have at least one hit where we make a change to the clock with this patch in.
Do you have a rt-app file that you can share ?
Quoted 6 lines
That said - despite the relatively high number of hits only about 5% of runs see enough additional energy consumed to trigger a test failure. We do try to keep a quiet system as much as possible and only run for a few seconds so the impact we see in testing is also probably higher than in the real world.
Yeah, I'm curious to see the impact on a real system which have a 60fps screen update like an android phone
Quoted 33 lines
I totally appreciate the reluctance to add this - I don't much like it either, but I was hoping that sticking it behind the
asym_cpucapacitykey might be a reasonable compromise.At least only those people who select a CPU using task util and capacity see this cost, and we have smaller systems so in theory the cost is smaller.
I'm very open to exploring alternatives :)
Quoted 5 lines
Quoted 5 lines
Signed-off-by Chris Redpath <[email protected]>--- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)Quoted 13 lines
kernel/sched/fair.c static void migrate_task_rq_fair(struct task_struct *p, int new_cpu) · line 6549
Quoted 10 lines
kernel/sched/fair.c +2
-0@@ -6545,6 +6545,21 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* wakee task is less decayed, but giving the wakee more load * sounds not bad. */ +if (static_branch_unlikely (&sched_asym_cpucapacity) && + p->state ==TASK_WAKING ) {nit: indent fail.
oops, will tweak it
--Chris
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 9 lines · Vincent Guittot
Hi Chris,
Quoted 4 lines
We enter this code quite often in our testing, most individual runs of a test which has small tasks involved have at least one hit where we make a change to the clock with this patch in.
Do you have a rt-app file that you can share ?
The ThreeSmallTasks test which is the worst hit produces this:
{
"tasks": {
"small_0": {
"policy": "SCHED_OTHER",
"delay": 0,
"loop": 1,
"phases": {
"p000001": {
"loop": 62,
"run": 2880,
"timer": {
"ref": "small_0",
"period": 16000
}
}
}
},
"small_1": {
"policy": "SCHED_OTHER",
"delay": 0,
"loop": 1,
"phases": {
"p000001": {
"loop": 62,
"run": 2880,
"timer": {
"ref": "small_1",
"period": 16000
}
}
}
},
"small_2": {
"policy": "SCHED_OTHER",
"delay": 0,
"loop": 1,
"phases": {
"p000001": {
"loop": 62,
"run": 2880,
"timer": {
"ref": "small_2",
"period": 16000
}
}
}
}
},
"global": {
"default_policy": "SCHED_OTHER",
"duration": -1,
"calibration": 264,
"logdir": "/root/devlib-target"
}
}when I run it
Quoted 10 lines
Quoted 6 lines
That said - despite the relatively high number of hits only about 5% of runs see enough additional energy consumed to trigger a test failure. We do try to keep a quiet system as much as possible and only run for a few seconds so the impact we see in testing is also probably higher than in the real world.
Yeah, I'm curious to see the impact on a real system which have a 60fps screen update like an android phone
I wouldn't expect much change there but I would on the idle-ish homescreen/day-of-use type benchmarks.
If I had a platform with any kind of representative energy use, I'd test it :)
Vincent Guittot
maintainerQuoted 70 lines · Chris Redpath
Quoted 9 lines · Vincent Guittot
Hi Chris,
Quoted 4 lines
We enter this code quite often in our testing, most individual runs of a test which has small tasks involved have at least one hit where we make a change to the clock with this patch in.
Do you have a rt-app file that you can share ?
The ThreeSmallTasks test which is the worst hit produces this:
{ "tasks": { "small_0": { "policy": "SCHED_OTHER", "delay": 0, "loop": 1, "phases": { "p000001": { "loop": 62, "run": 2880, "timer": { "ref": "small_0", "period": 16000 } } } }, "small_1": { "policy": "SCHED_OTHER", "delay": 0, "loop": 1, "phases": { "p000001": { "loop": 62, "run": 2880, "timer": { "ref": "small_1", "period": 16000 } } } }, "small_2": { "policy": "SCHED_OTHER", "delay": 0, "loop": 1, "phases": { "p000001": { "loop": 62, "run": 2880, "timer": { "ref": "small_2", "period": 16000 } } } } }, "global": { "default_policy": "SCHED_OTHER", "duration": -1, "calibration": 264, "logdir": "/root/devlib-target" } }
Quoted 1 line
when I run it
Thanks I will make it a try on my b.L platform
Quoted 17 lines
Quoted 11 lines
Quoted 7 lines
That said - despite the relatively high number of hits only about 5% of runs see enough additional energy consumed to trigger a test failure. We do try to keep a quiet system as much as possible and only run for a few seconds so the impact we see in testing is also probably higher than in the real world.
Yeah, I'm curious to see the impact on a real system which have a 60fps screen update like an android phone
I wouldn't expect much change there but I would on the idle-ish homescreen/day-of-use type benchmarks.
If I had a platform with any kind of representative energy use, I'd test it :)