Skip to contentSkip to conversation

sched: fix/add missing update_rq_clock() calls

PATCH In review Mike Galbraith 3 participants 7 messages Nov 12, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: fix/add missing update_rq_clock() calls

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

Aside from preventing potential latency hits, there' a side benefit in that early boot printk time stamps become monotonic.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>

LKML-Reference: <new-submission>

---
 kernel/sched.c |   17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
kernel/sched.c +12 -5
Index: linux-2.6.32.git/kernel/sched.c
===================================================================
@@ -2019,6 +2019,7 @@ void kthread_bind(struct task_struct *p,
 	}
 
 	spin_lock_irqsave(&rq->lock, flags);
+	update_rq_clock(rq);
 	set_task_cpu(p, cpu);
 	p->cpus_allowed = cpumask_of_cpu(cpu);
 	p->rt.nr_cpus_allowed = 1;
@@ -2117,6 +2118,7 @@ migrate_task(struct task_struct *p, int
 	 * it is sufficient to simply update the task's cpu field.
 	 */
 	if (!p->se.on_rq && !task_running(rq, p)) {
+		update_rq_clock(rq);
 		set_task_cpu(p, dest_cpu);
 		return 0;
 	}
@@ -2378,14 +2380,15 @@ static int try_to_wake_up(struct task_st
 	task_rq_unlock(rq, &flags);
 
 	cpu = p->sched_class->select_task_rq(p, SD_BALANCE_WAKE, wake_flags);
-	if (cpu != orig_cpu)
+	if (cpu != orig_cpu) {
+		local_irq_save(flags);
+		rq = cpu_rq(cpu);
+		update_rq_clock(rq);
 		set_task_cpu(p, cpu);
-
+		local_irq_restore(flags);
+	}
 	rq = task_rq_lock(p, &flags);
 
-	if (rq != orig_rq)
-		update_rq_clock(rq);
-
 	WARN_ON(p->state != TASK_WAKING);
 	cpu = task_cpu(p);
 
@@ -2558,6 +2561,7 @@ static void __sched_fork(struct task_str
 void sched_fork(struct task_struct *p, int clone_flags)
 {
 	int cpu = get_cpu();
+	unsigned long flags;
 
 	__sched_fork(p);
 
@@ -2594,7 +2598,10 @@ void sched_fork(struct task_struct *p, i
 #ifdef CONFIG_SMP
 	cpu = p->sched_class->select_task_rq(p, SD_BALANCE_FORK, 0);
 #endif
+	local_irq_save(flags);
+	update_rq_clock(cpu_rq(cpu));
 	set_task_cpu(p, cpu);
+	local_irq_restore(flags);
 
 #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 	if (likely(sched_info_on()))
#1 lore raw
Quoted 4 lines · Mike Galbraith

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.

#2 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.

It's not just about optimization - it's also accounting precision - we want to handle the full act of schedule() at a single timestamp - not a series of timestamps.

But ... we can get rid of it if it can be done sanely.

Ingo

#3 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.

Quoted 5 lines

It's not just about optimization - it's also accounting precision - we want to handle the full act of schedule() at a single timestamp - not a series of timestamps.

But ... we can get rid of it if it can be done sanely.

Ah, good point, hard to do for the migration paths though.

#6 lore raw

[tip:sched/urgent] sched: Fix/add missing update_rq_clock() calls

In reply to Mike Galbraith
Commit-ID:  055a00865dcfc8e61f3cbefbb879c9577bd36ae5
Gitweb:     http://git.kernel.org/tip/055a00865dcfc8e61f3cbefbb879c9577bd36ae5
Author:     Mike Galbraith <[email protected]>
AuthorDate: Thu, 12 Nov 2009 11:07:44 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 12 Nov 2009 12:28:29 +0100

sched: Fix/add missing update_rq_clock() calls

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

Aside from preventing potential latency hits, there' a side benefit in that early boot printk time stamps become monotonic.

LKML-Reference: <[email protected]>

LKML-Reference: <new-submission>

---
 kernel/sched.c |   17 ++++++++++++-----
 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/sched.c +12 -5
index 3c11ae0..701eca4 100644
@@ -2017,6 +2017,7 @@ void kthread_bind(struct task_struct *p, unsigned int cpu)
 	}
 
 	spin_lock_irqsave(&rq->lock, flags);
+	update_rq_clock(rq);
 	set_task_cpu(p, cpu);
 	p->cpus_allowed = cpumask_of_cpu(cpu);
 	p->rt.nr_cpus_allowed = 1;
@@ -2115,6 +2116,7 @@ migrate_task(struct task_struct *p, int dest_cpu, struct migration_req *req)
 	 * it is sufficient to simply update the task's cpu field.
 	 */
 	if (!p->se.on_rq && !task_running(rq, p)) {
+		update_rq_clock(rq);
 		set_task_cpu(p, dest_cpu);
 		return 0;
 	}
@@ -2376,14 +2378,15 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,
 	task_rq_unlock(rq, &flags);
 
 	cpu = p->sched_class->select_task_rq(p, SD_BALANCE_WAKE, wake_flags);
-	if (cpu != orig_cpu)
+	if (cpu != orig_cpu) {
+		local_irq_save(flags);
+		rq = cpu_rq(cpu);
+		update_rq_clock(rq);
 		set_task_cpu(p, cpu);
-
+		local_irq_restore(flags);
+	}
 	rq = task_rq_lock(p, &flags);
 
-	if (rq != orig_rq)
-		update_rq_clock(rq);
-
 	WARN_ON(p->state != TASK_WAKING);
 	cpu = task_cpu(p);
 
@@ -2545,6 +2548,7 @@ static void __sched_fork(struct task_struct *p)
 void sched_fork(struct task_struct *p, int clone_flags)
 {
 	int cpu = get_cpu();
+	unsigned long flags;
 
 	__sched_fork(p);
 
@@ -2581,7 +2585,10 @@ void sched_fork(struct task_struct *p, int clone_flags)
 #ifdef CONFIG_SMP
 	cpu = p->sched_class->select_task_rq(p, SD_BALANCE_FORK, 0);
 #endif
+	local_irq_save(flags);
+	update_rq_clock(cpu_rq(cpu));
 	set_task_cpu(p, cpu);
+	local_irq_restore(flags);
 
 #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 	if (likely(sched_info_on()))
#4 lore raw
In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.

Quoted 7 lines
Quoted 5 lines

It's not just about optimization - it's also accounting precision - we want to handle the full act of schedule() at a single timestamp - not a series of timestamps.

But ... we can get rid of it if it can be done sanely.

Ah, good point, hard to do for the migration paths though.

yes, migration is always the odd one out anyway, unless the hardware does cross-CPU-synchronous timestamps.

Ingo

#5 lore raw
Quoted 21 lines · Ingo Molnar
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Ingo Molnar
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Mike Galbraith

sched: fix/add missing update_rq_clock() calls.

kthread_bind(), migrate_task() and sched_fork were missing updates, and try_to_wake_up() was updating after having already used the stale clock.

I'm almost at the point where I think we should ditch rq->clock alltogether and simply call cpu_clock() directly when we need it. This trying to optimize and avoid calling it game has brought way too many head-aches.

It's not just about optimization - it's also accounting precision - we want to handle the full act of schedule() at a single timestamp - not a series of timestamps.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

But ... we can get rid of it if it can be done sanely.

Ah, good point, hard to do for the migration paths though.

yes, migration is always the odd one out anyway, unless the hardware does cross-CPU-synchronous timestamps.

Even if it has, we currently only update remote rq->clock variables once we end up in a migration path, which is always after updating the current rq->clock variable. So there's always a slight difference between the two rq clock, even if the hw clock is perfect.