sched: fix/add missing update_rq_clock() calls
PATCH 1/1 sched: fix/add missing update_rq_clock() calls
sched: fix/add missing update_rq_clock() calls.
kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.
Aside from preventing potential latency hits, there' a side benefit in that early boot printk time stamps become monotonic.
LKML-Reference: <new-submission>
--- kernel/sched.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
kernel/sched.c +12 -5
@@ -2019,6 +2019,7 @@ void kthread_bind(struct task_struct *p, }spin_lock_irqsave (&rq->lock, flags); +update_rq_clock (rq);set_task_cpu (p, cpu); p->cpus_allowed =cpumask_of_cpu (cpu); p->rt.nr_cpus_allowed =1 ;
@@ -2117,6 +2118,7 @@ migrate_task(struct task_struct *p, int* it is sufficient to simply update the task's cpu field. */ if (!p->se.on_rq && !task_running (rq, p)) { +update_rq_clock (rq);set_task_cpu (p, dest_cpu);return 0 ; }
@@ -2378,14 +2380,15 @@ static int try_to_wake_up(struct task_sttask_rq_unlock (rq, &flags); cpu = p->sched_class->select_task_rq (p,SD_BALANCE_WAKE , wake_flags); -if (cpu != orig_cpu) +if (cpu != orig_cpu) { +local_irq_save (flags); + rq =cpu_rq (cpu); +update_rq_clock (rq);set_task_cpu (p, cpu); - +local_irq_restore (flags); + } rq =task_rq_lock (p, &flags); -if (rq != orig_rq) -update_rq_clock (rq); -WARN_ON (p->state !=TASK_WAKING ); cpu =task_cpu (p);
@@ -2558,6 +2561,7 @@ static void __sched_fork(struct task_strvoid sched_fork (struct task_struct *p,int clone_flags) {int cpu =get_cpu (); +unsigned long flags;__sched_fork (p);
@@ -2594,7 +2598,10 @@ void sched_fork(struct task_struct *p, i#ifdef CONFIG_SMP cpu = p->sched_class->select_task_rq (p,SD_BALANCE_FORK ,0 );#endif +local_irq_save (flags); +update_rq_clock (cpu_rq (cpu));set_task_cpu (p, cpu); +local_irq_restore (flags);#if defined(CONFIG_SCHEDSTATS ) || defined(CONFIG_TASK_DELAY_ACCT )if (likely (sched_info_on ()))
Quoted 4 lines · Mike Galbraith
sched: fix/add missing
update_rq_clock()calls.
kthread_bind(),migrate_task()andsched_forkwere missing updates, andtry_to_wake_up()was updating after having already used the stale clock.
I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith
sched: fix/add missing
update_rq_clock()calls.
kthread_bind(),migrate_task()andsched_forkwere missing updates, andtry_to_wake_up()was updating after having already used the stale clock.I'm almost at the point where I think we should ditch
rq->clockalltogether and simply callcpu_clock()directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.
It's not just about optimization - it's also accounting precision - we want to handle the full act of schedule() at a single timestamp - not a series of timestamps.
But ... we can get rid of it if it can be done sanely.
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith
sched: fix/add missing
update_rq_clock()calls.
kthread_bind(),migrate_task()andsched_forkwere missing updates, andtry_to_wake_up()was updating after having already used the stale clock.I'm almost at the point where I think we should ditch
rq->clockalltogether and simply callcpu_clock()directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.
Quoted 5 lines
It's not just about optimization - it's also accounting precision - we want to handle the full act of
schedule()at a single timestamp - not a series of timestamps.But ... we can get rid of it if it can be done sanely.
Ah, good point, hard to do for the migration paths though.
[tip:sched/urgent] sched: Fix/add missing update_rq_clock() calls
In reply to Mike GalbraithCommit-ID: 055a00865dcfc8e61f3cbefbb879c9577bd36ae5 Gitweb: http://git.kernel.org/tip/055a00865dcfc8e61f3cbefbb879c9577bd36ae5 Author: Mike Galbraith <[email protected]> AuthorDate: Thu, 12 Nov 2009 11:07:44 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 12 Nov 2009 12:28:29 +0100
sched: Fix/add missing update_rq_clock() calls
kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.
Aside from preventing potential latency hits, there' a side benefit in that early boot printk time stamps become monotonic.
LKML-Reference: <[email protected]>
LKML-Reference: <new-submission>
--- kernel/sched.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/sched.c +12 -5
@@ -2017,6 +2017,7 @@ void kthread_bind(struct task_struct *p, unsigned int cpu) }spin_lock_irqsave (&rq->lock, flags); +update_rq_clock (rq);set_task_cpu (p, cpu); p->cpus_allowed =cpumask_of_cpu (cpu); p->rt.nr_cpus_allowed =1 ;
@@ -2115,6 +2116,7 @@ migrate_task(struct task_struct *p, int dest_cpu, struct migration_req *req)* it is sufficient to simply update the task's cpu field. */ if (!p->se.on_rq && !task_running (rq, p)) { +update_rq_clock (rq);set_task_cpu (p, dest_cpu);return 0 ; }
@@ -2376,14 +2378,15 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,task_rq_unlock (rq, &flags); cpu = p->sched_class->select_task_rq (p,SD_BALANCE_WAKE , wake_flags); -if (cpu != orig_cpu) +if (cpu != orig_cpu) { +local_irq_save (flags); + rq =cpu_rq (cpu); +update_rq_clock (rq);set_task_cpu (p, cpu); - +local_irq_restore (flags); + } rq =task_rq_lock (p, &flags); -if (rq != orig_rq) -update_rq_clock (rq); -WARN_ON (p->state !=TASK_WAKING ); cpu =task_cpu (p);
@@ -2545,6 +2548,7 @@ static void __sched_fork(struct task_struct *p)void sched_fork (struct task_struct *p,int clone_flags) {int cpu =get_cpu (); +unsigned long flags;__sched_fork (p);
@@ -2581,7 +2585,10 @@ void sched_fork(struct task_struct *p, int clone_flags)#ifdef CONFIG_SMP cpu = p->sched_class->select_task_rq (p,SD_BALANCE_FORK ,0 );#endif +local_irq_save (flags); +update_rq_clock (cpu_rq (cpu));set_task_cpu (p, cpu); +local_irq_restore (flags);#if defined(CONFIG_SCHEDSTATS ) || defined(CONFIG_TASK_DELAY_ACCT )if (likely (sched_info_on ()))
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith
sched: fix/add missing
update_rq_clock()calls.
kthread_bind(),migrate_task()andsched_forkwere missing updates, andtry_to_wake_up()was updating after having already used the stale clock.I'm almost at the point where I think we should ditch
rq->clockalltogether and simply callcpu_clock()directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.
Quoted 7 lines
Quoted 5 lines
It's not just about optimization - it's also accounting precision - we want to handle the full act of
schedule()at a single timestamp - not a series of timestamps.But ... we can get rid of it if it can be done sanely.
Ah, good point, hard to do for the migration paths though.
yes, migration is always the odd one out anyway, unless the hardware does cross-CPU-synchronous timestamps.
Ingo
Quoted 21 lines · Ingo Molnar
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith
sched: fix/add missing
update_rq_clock()calls.
kthread_bind(),migrate_task()andsched_forkwere missing updates, andtry_to_wake_up()was updating after having already used the stale clock.I'm almost at the point where I think we should ditch
rq->clockalltogether and simply callcpu_clock()directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.It's not just about optimization - it's also accounting precision - we want to handle the full act of
schedule()at a single timestamp - not a series of timestamps.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
But ... we can get rid of it if it can be done sanely.
Ah, good point, hard to do for the migration paths though.
yes, migration is always the odd one out anyway, unless the hardware does cross-CPU-synchronous timestamps.
Even if it has, we currently only update remote rq->clock variables once we end up in a migration path, which is always after updating the current rq->clock variable. So there's always a slight difference between the two rq clock, even if the hw clock is perfect.