sched: Initialize rq->age_stamp on processor start

From: Corey Minyard <[email protected]>

If the sched_clock time starts at a large value, the kernel will spin in sched_avg_update for a long time while rq->age_stamp catches up with rq->clock.

The comment in kernel/sched/clock.c says that there is no strict promise that it starts at zero. So initialize rq->age_stamp when a cpu starts up to avoid this.

I was seeing long delays on a simulator that didn't start the clock at zero. This might also be an issue on reboots on processors that don't re-initialize the timer to zero on reset, and when using kexec.

Signed-off-by Corey Minyard <[email protected]>

---
 kernel/sched/core.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

kernel/sched/core.c +0 -0

index b46131e..5be3d4a 100644

kernel/sched/core.c +4 -0

+
 static int sched_cpu_active(struct notifier_block *nfb,
 				      unsigned long action, void *hcpu)
 {
 	switch (action & ~CPU_TASKS_FROZEN) {
 	case CPU_STARTING:
+		set_cpu_rq_start_time();
+		/* fall through */
 	case CPU_DOWN_FAILED:
 		set_cpu_active((long)hcpu, true);
 		return NOTIFY_OK;

@@ -6922,6 +6931,7 @@ void __init sched_init(void)
 	init_sched_fair_class();
 
 	scheduler_running = 1;
+	set_cpu_rq_start_time();
 }
 
 #ifdef CONFIG_DEBUG_ATOMIC_SLEEP

CC maintainer improves patch aerodynamics.

Quoted 24 lines

From: Corey Minyard <[email protected]>

If the sched_clock time starts at a large value, the kernel will spin in sched_avg_update for a long time while rq->age_stamp catches up with rq->clock.

The comment in kernel/sched/clock.c says that there is no strict promise that it starts at zero. So initialize rq->age_stamp when a cpu starts up to avoid this.

I was seeing long delays on a simulator that didn't start the clock at zero. This might also be an issue on reboots on processors that don't re-initialize the timer to zero on reset, and when using kexec.

Signed-off-by Corey Minyard <[email protected]>

---
 kernel/sched/core.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

kernel/sched/core.c +0 -0

index b46131e..5be3d4a 100644

kernel/sched/core.c static struct notifier_block migration_notifier = { · line 5045

Quoted 9 lines

kernel/sched/core.c +6 -0

@@ -5037,11 +5037,20 @@ static struct notifier_block migration_notifier = {
 	.priority = CPU_PRI_MIGRATION,
 };
 
+static void __cpuinit set_cpu_rq_start_time(void)
+{
+	int cpu = smp_processor_id();
+	struct rq *rq = cpu_rq(cpu);
+	rq->age_stamp = sched_clock_cpu(cpu);
+}

rq->age_stamp must lag rq->clock. See scale_rt_power(), and what happens when it munches magic timewarp mushrooms.

Quoted 14 lines

kernel/sched/core.c +3 -0

+
 static int sched_cpu_active(struct notifier_block *nfb,
 				      unsigned long action, void *hcpu)
 {
 	switch (action & ~CPU_TASKS_FROZEN) {
 	case CPU_STARTING:
+		set_cpu_rq_start_time();
+		/* fall through */
 	case CPU_DOWN_FAILED:
 		set_cpu_active((long)hcpu, true);
 		return NOTIFY_OK;

@@ -6922,6 +6931,7 @@ void __init sched_init(void)
 	init_sched_fair_class();
 

kernel/sched/core.c void __init sched_init(void) · line 6934

Quoted 2 lines

kernel/sched/core.c +1 -0

 	scheduler_running = 1;
+	set_cpu_rq_start_time();

Quoted 24 lines · Mike Galbraith

kernel/sched/core.c static struct notifier_block migration_notifier = { · line 5045

Quoted 6 lines

kernel/sched/core.c +6 -0

+static void __cpuinit set_cpu_rq_start_time(void)
+{
+	int cpu = smp_processor_id();
+	struct rq *rq = cpu_rq(cpu);
+	rq->age_stamp = sched_clock_cpu(cpu);
+}

rq->age_stamp must lag rq->clock. See scale_rt_power(), and what happens when it munches magic timewarp mushrooms.

Quoted 14 lines

kernel/sched/core.c +3 -0

+
 static int sched_cpu_active(struct notifier_block *nfb,
 				      unsigned long action, void *hcpu)
 {
 	switch (action & ~CPU_TASKS_FROZEN) {
 	case CPU_STARTING:
+		set_cpu_rq_start_time();
+		/* fall through */
 	case CPU_DOWN_FAILED:
 		set_cpu_active((long)hcpu, true);
 		return NOTIFY_OK;

@@ -6922,6 +6931,7 @@ void __init sched_init(void)
 	init_sched_fair_class();
 

Quoted 2 lines

kernel/sched/core.c void __init sched_init(void) · line 6934

Quoted 2 lines

kernel/sched/core.c +1 -0

 	scheduler_running = 1;
+	set_cpu_rq_start_time();

I would put it one line up; that scheduler_running=1 is the last thing we should do.