Skip to contentSkip to conversation

Fix smpnice high priority task hopping problem

PATCH In review Peter Williams 2 participants 8 messages Feb 16, 2006 – Feb 17, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix smpnice high priority task hopping problem

Suresh B. Siddha has reported:

"on a lightly loaded system, this will result in higher priority job hopping around from one processor to another processor.. This is because of the code in find_busiest_group() which assumes that SCHED_LOAD_SCALE represents a unit process load and with nice_to_bias calculations this is no longer true (in the presence of non nice-0 tasks)"

Analysis of this problem as revealed that the smpnice code results in the weighted load being larger than 1 and this triggers the active load balancing code. However, in active_load_balance(), the migration thread fails to take into account itself when deciding if there are any tasks to be migrated from its run queue. I.e. even if there is only one other task on the run queue other than itself it will still migrate that other task.

The attached patch fixes that anomaly.

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
kernel/sched.c +1 -1
Index: MM-2.6.X/kernel/sched.c
===================================================================
@@ -2406,7 +2406,7 @@ static void active_load_balance(runqueue
 	runqueue_t *target_rq;
 	int target_cpu = busiest_rq->push_cpu;
 
-	if (busiest_rq->nr_running <= 1)
+	if (busiest_rq->nr_running <= 2)
 		/* no task to move */
 		return;
 
#1 lore raw

Andrew, Please don't apply this patch. This breaks the existing HT (and multi-core) scheduler optimizations.

Peter, on a DP system with HT, if we have only two runnable processes and they end up running on the two threads of the same package, with your patch, migration thread will never move one of those processes to the idle package..

To fix my reported problem, we need to make sure that find_busiest_group() doesn't find an imbalance..

thanks,
suresh

Quoted 40 lines · Peter Williams

Suresh B. Siddha has reported:

"on a lightly loaded system, this will result in higher priority job hopping around from one processor to another processor.. This is because of the code in find_busiest_group() which assumes that SCHED_LOAD_SCALE represents a unit process load and with nice_to_bias calculations this is no longer true (in the presence of non nice-0 tasks)"

Analysis of this problem as revealed that the smpnice code results in the weighted load being larger than 1 and this triggers the active load balancing code. However, in active_load_balance(), the migration thread fails to take into account itself when deciding if there are any tasks to be migrated from its run queue. I.e. even if there is only one other task on the run queue other than itself it will still migrate that other task.

The attached patch fixes that anomaly.

Peter

Signature · 7 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce

Index: MM-2.6.X/kernel/sched.c
===================================================================
kernel/sched.c +1 -1
@@ -2406,7 +2406,7 @@ static void active_load_balance(runqueue
 	runqueue_t *target_rq;
 	int target_cpu = busiest_rq->push_cpu;
 
-	if (busiest_rq->nr_running <= 1)
+	if (busiest_rq->nr_running <= 2)
 		/* no task to move */
 		return;
 
#2 lore raw
Quoted 5 lines · Suresh B

Andrew, Please don't apply this patch. This breaks the existing HT (and multi-core) scheduler optimizations.

Peter, on a DP system with HT, if we have only two runnable processes
and they end up running on the two threads of the same package, with your patch, migration thread will never move one of those processes to the idle package..

On a normal system, would either of them be moved anyway?

Quoted 3 lines

To fix my reported problem, we need to make sure that find_busiest_group() doesn't find an imbalance..

I disagree. If this causes a problem with your "optimizations" then I think that you need to fix the "optimizations".

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

Quoted 37 lines

thanks,
suresh

Quoted 31 lines · Peter Williams

Suresh B. Siddha has reported:

"on a lightly loaded system, this will result in higher priority job hopping around from one processor to another processor.. This is because of the code in find_busiest_group() which assumes that SCHED_LOAD_SCALE represents a unit process load and with nice_to_bias calculations this is no longer true (in the presence of non nice-0 tasks)"

Analysis of this problem as revealed that the smpnice code results in the weighted load being larger than 1 and this triggers the active load balancing code. However, in active_load_balance(), the migration thread fails to take into account itself when deciding if there are any tasks to be migrated from its run queue. I.e. even if there is only one other task on the run queue other than itself it will still migrate that other task.

The attached patch fixes that anomaly.

Peter

Signature · 8 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce


Index: MM-2.6.X/kernel/sched.c
===================================================================
kernel/sched.c +1 -1
@@ -2406,7 +2406,7 @@ static void active_load_balance(runqueue
 	runqueue_t *target_rq;
 	int target_cpu = busiest_rq->push_cpu;
 
-	if (busiest_rq->nr_running <= 1)
+	if (busiest_rq->nr_running <= 2)
 		/* no task to move */
 		return;
 
Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#3 lore raw
Quoted 12 lines · Peter Williams
Quoted 5 lines · Suresh B

Andrew, Please don't apply this patch. This breaks the existing HT (and multi-core) scheduler optimizations.

Peter, on a DP system with HT, if we have only two runnable processes
and they end up running on the two threads of the same package, with your patch, migration thread will never move one of those processes to the idle package..

On a normal system, would either of them be moved anyway?

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

To fix my reported problem, we need to make sure that find_busiest_group() doesn't find an imbalance..

I disagree. If this causes a problem with your "optimizations" then I think that you need to fix the "optimizations".

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

OK. The good news is that (my testing shows that) the "sched: fix smpnice abnormal nice anomalies" fixes the imbalance problem and the consequent CPU hopping.

BUT I still think that this patch (modified if necessary to handle any HT special cases) should be applied. On a normal system, it will (as I've already said) stop active_load_balance() from doing a lot of unnecessary work INCLUDING holding the run queue locks for TWO run queues for no good reason.

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#4 lore raw
Quoted 10 lines · Peter Williams
Quoted 5 lines · Peter Williams

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

OK. The good news is that (my testing shows that) the "sched: fix smpnice abnormal nice anomalies" fixes the imbalance problem and the consequent CPU hopping.

Thats because find_busiest_group() is no longer showing the imbalance :) Anyhow if I get time I will review this patch before I start my vacation. Otherwise I assume Nick and Ingo will review this closely..

Quoted 5 lines

BUT I still think that this patch (modified if necessary to handle any HT special cases) should be applied. On a normal system, it will (as I've already said) stop active_load_balance() from doing a lot of unnecessary work INCLUDING holding the run queue locks for TWO run queues for no good reason.

Please see my earlier response to this..

thanks,
suresh

#5 lore raw
Quoted 14 lines · Suresh B
Quoted 5 lines · Peter Williams
Quoted 1 line · Peter Williams

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

OK. The good news is that (my testing shows that) the "sched: fix smpnice abnormal nice anomalies" fixes the imbalance problem and the consequent CPU hopping.

Thats because find_busiest_group() is no longer showing the imbalance :) Anyhow if I get time I will review this patch before I start my vacation. Otherwise I assume Nick and Ingo will review this closely..

Quoted 4 lines
Quoted 1 line

BUT I still think that this patch (modified if necessary to handle any HT special cases) should be applied. On a normal system, it will (as I've already said) stop active_load_balance() from doing a lot of unnecessary work INCLUDING holding the run queue locks for TWO run queues for no good reason.

Please see my earlier response to this..

I saw nothing there to convince me that this patch isn't worthwhile. Perhaps a better explanation would help me?

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#6 lore raw
In reply to Peter Williams
Quoted 1 line · Peter Williams

On a normal system, would either of them be moved anyway?

Possible. Because when the migration thread runs it moves the current running task out of the processor and the checks in can_migrate_task() like "sd->nr_balance_failed > sd->cache_nice_tries" can result in cache hot task move to the idle package.. This is a round about way and we should not depend on this behavior..

Quoted 5 lines
Quoted 4 lines

To fix my reported problem, we need to make sure that find_busiest_group() doesn't find an imbalance..

Quoted 8 lines

I disagree. If this causes a problem with your "optimizations" then I think that you need to fix the "optimizations".

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

active load balance is designed only with HT optimizations in mind. And now multi-core optimizations also use this active load balance. No one else uses active load balance.

thanks,
suresh

#7 lore raw
Quoted 10 lines · Suresh B
Quoted 1 line · Peter Williams

On a normal system, would either of them be moved anyway?

Possible. Because when the migration thread runs it moves the current running task out of the processor and the checks in can_migrate_task() like "sd->nr_balance_failed > sd->cache_nice_tries" can result in cache hot task move to the idle package.. This is a round about way and we should not depend on this behavior..

So why does it need to be retained?

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

To fix my reported problem, we need to make sure that find_busiest_group() doesn't find an imbalance..

Quoted 8 lines
Quoted 3 lines

I disagree. If this causes a problem with your "optimizations" then I think that you need to fix the "optimizations".

There's a rational argument (IMHO) that this patch should be applied even in the absence of the smpnice patches as it prevents active_load_balance() doing unnecessary work. If this isn't good for hypo threading then hypo threading is a special case and needs to handle it as such.

active load balance is designed only with HT optimizations in mind. And now multi-core optimizations also use this active load balance. No one else uses active load balance.

I can see nothing in the source code that will cause active_load_balance() to be only run on hypo threaded systems. Could you please provide some pointers to the mechanism that does this.

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce