Skip to contentSkip to conversation

v2.6.21.4-rt11

DISCUSSION Srivatsa Vaddagiri 5 participants 17 messages Jun 18, 2007 – Jun 19, 2007 10 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 57 lines · Paul E. McKenney
Quoted 56 lines · Srivatsa Vaddagiri
Quoted 22 lines · Paul E. McKenney
Quoted 21 lines · Ingo Molnar
Quoted 6 lines · Paul E. McKenney
Quoted 3 lines

to make sure it's not some effect in -rt causing this. v17 has an updated load balancing code. (which might or might not affect the rcutorture problem.)

Good point! I will try the following:

Quoted 8 lines

1. Stock 2.6.21.5.

2. 2.6.21-rt14.

3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch

And quickly, before everyone else jumps on the machines that show the problem. ;-)

thanks! It's enough to check whether modprobe rcutorture still produces that weird balancing problem. That clearly has to be fixed ...

Quoted 6 lines
Quoted 2 lines

And i've Cc:-ed Dmitry and Srivatsa, who are busy hacking this area of the CFS code as we speak :-)

Well, I am not sure that the info I was able to collect will be all that helpful, but it most certainly does confirm that the balancing problem that rcutorture produces is indeed weird...

Hi Paul,
I tried on two machines in our lab and could not recreate your
problem.

On a 2way x86_64 AMD box and 2.6.21.5+cfsv17:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12395 root      39  19     0    0    0 R 50.3  0.0   0:57.62 rcu_torture_rea
12394 root      39  19     0    0    0 R 49.9  0.0   0:57.29 rcu_torture_rea
12396 root      39  19     0    0    0 R 49.9  0.0   0:56.96 rcu_torture_rea
12397 root      39  19     0    0    0 R 49.9  0.0   0:56.90 rcu_torture_rea

On a 4way x86_64 Intel Xeon box and 2.6.21.5+cfsv17:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
 6258 root      39  19     0    0    0 R   53  0.0  17:29.72 0 rcu_torture_rea
 6252 root      39  19     0    0    0 R   49  0.0  17:49.40 3 rcu_torture_rea
 6257 root      39  19     0    0    0 R   49  0.0  17:22.49 2 rcu_torture_rea
 6256 root      39  19     0    0    0 R   48  0.0  17:50.12 1 rcu_torture_rea
 6254 root      39  19     0    0    0 R   48  0.0  17:26.98 0 rcu_torture_rea
 6255 root      39  19     0    0    0 R   48  0.0  17:25.74 2 rcu_torture_rea
 6251 root      39  19     0    0    0 R   45  0.0  17:47.45 3 rcu_torture_rea
 6253 root      39  19     0    0    0 R   45  0.0  17:48.48 1 rcu_torture_rea
Quoted 6 lines
Quoted 3 lines

I will try this on few more boxes we have on Monday. If I can't recreate, then I may request you to provide me machine details (or even access to the problem box if it is in IBM labs and if I am allowed to login!)

elm3b6, ABAT job 95107. There are others, this particular job uses 2.6.21.5-cfsv17.

Paul,
I logged into elm3b6 and did some investigation. I think I have
a tentative patch to fix your load-balance problem.

First, an explanation of the problem:

This particular machine, elm3b6, is a 4-cpu, (gasp, yes!) 4-node box i.e each CPU is a node by itself. If you don't have CONFIG_NUMA enabled, then we won't have cross-node (i.e cross-cpu) load balancing. Fortunately in your case you had CONFIG_NUMA enabled, but still were hitting the (gross) load imbalance.

The problem seems to be with idle_balance(). This particular routine, invoked by schedule() on a idle cpu, walks up sched-domain hierarchy and tries to balance in each domain that has SD_BALANCE_NEWIDLE flag set. The nodes-level domain (SD_NODE_INIT) however doesn't set this flag, which means idle cpu looks for (im)balance within its own node at most and not beyond. Now, here's the problem, if the idle cpu doesn't find imbalance within its node (pulled_tasks = 0), it resets this_rq->next_balance so that next balancing activity is deferred for upto a minute (next_balance = jiffies + 60 * HZ). If a idle cpu calls idle_balance again in the next minute and finds no imbalance within its node, it -again- resets next_balance. In your case, I think this was happening repetetively, which made other CPUs never look for cross-node (im)balance.

I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.

I have CCed others who have worked in this area and request them to review this patch.

Andrew,
If there is no objection from anyone, request you to pick this
up for next -mm release. It has been tested against 2.6.22-rc4-mm2.

idle_balance() can erroneously cause system-wide imbalance to be overlooked by reseting rq->next_balance. When called sufficient number of times, it can forever defer system-wide load balance. Patch below modifies idle_balance() not to mess with ->next_balance. If indeed it turns out that there is no imbalance even system-wide, rebalance_domains() will anyway set ->next_balance to happen after a minute.

kernel/sched.c +0 -11
Index: linux-2.6.22-rc4/kernel/sched.c
===================================================================
@@ -2490,27 +2490,16 @@
 {
 	struct sched_domain *sd;
 	int pulled_task = 0;
-	unsigned long next_balance = jiffies + 60 *  HZ;
 
 	for_each_domain(this_cpu, sd) {
 		if (sd->flags & SD_BALANCE_NEWIDLE) {
 			/* If we've pulled tasks over stop searching: */
 			pulled_task = load_balance_newidle(this_cpu,
 							this_rq, sd);
-			if (time_after(next_balance,
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
 			if (pulled_task)
 				break;
 		}
 	}
-	if (!pulled_task)
-		/*
-		 * We are going idle. next_balance may be set based on
-		 * a busy processor. So reset next_balance.
-		 */
-		this_rq->next_balance = next_balance;
 }
 
 /*
Signature · 2 lines
Regards,
vatsa
#11 lore raw
Quoted 11 lines · Srivatsa Vaddagiri

This particular machine, elm3b6, is a 4-cpu, (gasp, yes!) 4-node box i.e each CPU is a node by itself. If you don't have CONFIG_NUMA enabled, then we won't have cross-node (i.e cross-cpu) load balancing. Fortunately in your case you had CONFIG_NUMA enabled, but still were hitting the (gross) load imbalance.

The problem seems to be with idle_balance(). This particular routine, invoked by schedule() on a idle cpu, walks up sched-domain hierarchy and tries to balance in each domain that has SD_BALANCE_NEWIDLE flag set. The nodes-level domain (SD_NODE_INIT) however doesn't set this flag, which means idle cpu looks for (im)balance within its own node at most and

The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node. The balancing on the phys_domain level does balance within a node.

#12 lore raw
Quoted 2 lines · Christoph Lameter

The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node.

I was mostly speaking with the example system in mind (4-node 4-cpu box), but yes, node-level domain does look for imbalance across max 16 nodes as you mention.

Both node and all-node domains don't have SD_BALANCE_NEWIDLE set, which means idle_balance() will stop looking for imbalance beyonds its own node. Based on the observed balance within its own node, IMO, idle_balance() should not cause ->next_balance to be reset.

Signature · 2 lines
Regards,
vatsa
#13 lore raw
Quoted 4 lines · Srivatsa Vaddagiri
Quoted 2 lines · Christoph Lameter

The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node.

Quoted 8 lines

I was mostly speaking with the example system in mind (4-node 4-cpu box), but yes, node-level domain does look for imbalance across max 16 nodes as you mention.

Both node and all-node domains don't have SD_BALANCE_NEWIDLE set, which means idle_balance() will stop looking for imbalance beyonds its own node. Based on the observed balance within its own node, IMO, idle_balance() should not cause ->next_balance to be reset.

I think the check in idle_balance needs to be modified.

If the domain *does not* have SD_BALANCE_NEWIDLE set then next_balance must still be set right. Does this patch fix it?

Scheduler: Fix next_interval determination in idle_balance().

The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.

kernel/sched.c +7 -8
Index: linux-2.6.22-rc4-mm2/kernel/sched.c
===================================================================
@@ -2493,17 +2493,16 @@ static void idle_balance(int this_cpu, s
 	unsigned long next_balance = jiffies + 60 *  HZ;
 
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		if (sd->flags & SD_BALANCE_NEWIDLE)
 			/* If we've pulled tasks over stop searching: */
 			pulled_task = load_balance_newidle(this_cpu,
 							this_rq, sd);
-			if (time_after(next_balance,
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
-			if (pulled_task)
-				break;
-		}
+		if (time_after(next_balance,
+			  sd->last_balance + sd->balance_interval))
+			next_balance = sd->last_balance
+					+ sd->balance_interval;
+		if (pulled_task)
+			break;
 	}
 	if (!pulled_task)
 		/*
#14 lore raw
Quoted 4 lines · Christoph Lameter

I think the check in idle_balance needs to be modified.

If the domain *does not* have SD_BALANCE_NEWIDLE set then next_balance must still be set right. Does this patch fix it?

Is the ->next_balance calculation in idle_balance() necessary at all? rebalance_domains() would have programmed ->next_balance anyway, based on the nearest next_balance point of all (load-balance'able) domains. By repeating that calculation in idle_balance, are we covering any corner case?

Signature · 2 lines
Regards,
vatsa
#15 lore raw
Quoted 10 lines · Srivatsa Vaddagiri
Quoted 4 lines · Christoph Lameter

I think the check in idle_balance needs to be modified.

If the domain *does not* have SD_BALANCE_NEWIDLE set then next_balance must still be set right. Does this patch fix it?

Is the ->next_balance calculation in idle_balance() necessary at all? rebalance_domains() would have programmed ->next_balance anyway, based on the nearest next_balance point of all (load-balance'able) domains. By repeating that calculation in idle_balance, are we covering any corner case?

rebalance_domains() have programmed ->next_balance based on 'busy' state. And now, as it is going to 'idle', this routine is recalculating the next_balance based on 'idle' state.

thanks,
suresh

#16 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
kernel/sched.c +1 -2
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		if (sd->flags & SD_BALANCE_NEWIDLE)
 			/* If we've pulled tasks over stop searching: */
 			pulled_task = load_balance_newidle(this_cpu,
 							this_rq, sd);
-			if (time_after(next_balance,

kernel/sched.c static void idle_balance(int this_cpu, s · line 2503

Quoted 10 lines
kernel/sched.c +4 -6
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
-			if (pulled_task)
-				break;
-		}
+		if (time_after(next_balance,
+			  sd->last_balance + sd->balance_interval))
+			next_balance = sd->last_balance
+					+ sd->balance_interval;

don't we have to do, msecs_to_jiffies(sd->balance_interval)?

thanks,
suresh

Quoted 5 lines
kernel/sched.c +2 -0
+		if (pulled_task)
+			break;
 	}
 	if (!pulled_task)
 		/*
#17 lore raw
Quoted 6 lines · Suresh B

Review

Quoted 4 lines
(quoted diff) +4 -0
+		if (time_after(next_balance,
+			  sd->last_balance + sd->balance_interval))
+			next_balance = sd->last_balance
+					+ sd->balance_interval;

don't we have to do, msecs_to_jiffies(sd->balance_interval)?

Well that is certainly a bug here. Is this better?

Scheduler: Fix next_interval determination in idle_balance().

The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.

Siddha also spotted that the conversion of the balance interval to jiffies is missing. Fix that to.

kernel/sched.c +11 -10
Index: linux-2.6.22-rc4-mm2/kernel/sched.c
===================================================================
@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, s
 	unsigned long next_balance = jiffies + 60 *  HZ;
 
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		unsigned long interval;
+
+		if (sd->flags & SD_BALANCE_NEWIDLE)
 			/* If we've pulled tasks over stop searching: */
-			pulled_task = load_balance_newidle(this_cpu,
-							this_rq, sd);
-			if (time_after(next_balance,
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
-			if (pulled_task)
-				break;
-		}
+			pulled_task = load_balance_newidle(this_cpu,this_rq, sd);
+
+		interval = msecs_to_jiffies(sd->balance_interval);
+		if (time_after(next_balance,
+			  sd->last_balance + interval))
+			next_balance = sd->last_balance + interval;
+		if (pulled_task)
+			break;
 	}
 	if (!pulled_task)
 		/*
#18 lore raw

kernel/sched.c static void idle_balance(int this_cpu, s · line 2497

Quoted 7 lines · Christoph Lameter
kernel/sched.c +2 -1
@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, s
 	unsigned long next_balance = jiffies + 60 *  HZ;
 
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		unsigned long interval;
+

Do we need a :

		if (!(sd->flags & SD_LOAD_BALANCE))
			continue;

here?

Otherwise patch look good and fixes the problem Paul observed earlier.

Quoted 24 lines
kernel/sched.c +9 -9
+		if (sd->flags & SD_BALANCE_NEWIDLE)
 			/* If we've pulled tasks over stop searching: */
-			pulled_task = load_balance_newidle(this_cpu,
-							this_rq, sd);
-			if (time_after(next_balance,
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
-			if (pulled_task)
-				break;
-		}
+			pulled_task = load_balance_newidle(this_cpu,this_rq, sd);
+
+		interval = msecs_to_jiffies(sd->balance_interval);
 
 
+		if (time_after(next_balance,
+			  sd->last_balance + interval))
+			next_balance = sd->last_balance + interval;
+		if (pulled_task)
+			break;
 	}
kernel/sched.c +0 -0
 	if (!pulled_task)
 		/*
Signature · 2 lines
Regards,
vatsa
#19 lore raw
Quoted 11 lines · Srivatsa Vaddagiri

Review static void idle_balance(int this_cpu, s · line 2497

Quoted 7 lines · Christoph Lameter
(quoted diff) +2 -1
@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, s
 	unsigned long next_balance = jiffies + 60 *  HZ;
 
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		unsigned long interval;
+

Do we need a :

Quoted 6 lines
		if (!(sd->flags & SD_LOAD_BALANCE))
			continue;

here?

Otherwise patch look good and fixes the problem Paul observed earlier.

great! I've applied the patch below (added your fix and cleaned it up a bit) and have released 2.6.21.5-rt17 with it.

Ingo

------------------------------>
From: Christoph Lameter <[email protected]>
Subject: [patch] sched: fix next_interval determination in idle_balance().

The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.

Siddha also spotted that the conversion of the balance interval to jiffies is missing. Fix that to.

From: Srivatsa Vaddagiri <[email protected]>

also continue the loop if !(sd->flags & SD_LOAD_BALANCE).

---
 kernel/sched.c |   22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
kernel/sched.c +13 -9
Index: linux/kernel/sched.c
===================================================================
@@ -2591,17 +2591,21 @@ static void idle_balance(int this_cpu, s
 	unsigned long next_balance = jiffies + HZ;
 
 	for_each_domain(this_cpu, sd) {
-		if (sd->flags & SD_BALANCE_NEWIDLE) {
+		unsigned long interval;
+
+		if (!(sd->flags & SD_LOAD_BALANCE))
+			continue;
+
+		if (sd->flags & SD_BALANCE_NEWIDLE)
 			/* If we've pulled tasks over stop searching: */
 			pulled_task = load_balance_newidle(this_cpu,
-							this_rq, sd);
-			if (time_after(next_balance,
-				  sd->last_balance + sd->balance_interval))
-				next_balance = sd->last_balance
-						+ sd->balance_interval;
-			if (pulled_task)
-				break;
-		}
+								this_rq, sd);
+
+		interval = msecs_to_jiffies(sd->balance_interval);
+		if (time_after(next_balance, sd->last_balance + interval))
+			next_balance = sd->last_balance + interval;
+		if (pulled_task)
+			break;
 	}
 	if (pulled_task || time_after(jiffies, this_rq->next_balance)) {
 		/*
#20 lore raw
In reply to Srivatsa Vaddagiri
Quoted 2 lines · Srivatsa Vaddagiri

If you don't have CONFIG_NUMA enabled,
then we won't have cross-node (i.e cross-cpu) load balancing.

Mmm ..that is not correct. I found that disabling CONFIG_NUMA leads to better load balance on the problem system (i.e w/o any patches applied 2.6.22-rc4-mm2 leads to good distribution of rcu readers on all 4 cpus).

Anyway, the patch is still needed for scenarios like you originally tested with.

Signature · 2 lines
Regards,
vatsa
#21 lore raw
In reply to Srivatsa Vaddagiri
Quoted 3 lines · Srivatsa Vaddagiri

I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.

thanks for tracking this down! I've applied Christoph's patch (with your suggested modification plus a few small cleanups).

I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

Ingo

#22 lore raw
Quoted 1 line · Ingo Molnar

I'm wondering, why did this trigger under CFS and not on mainline?

I thought Paul had seen the same problem with 2.6.21.5. I will try a more recent mainline (2.6.22-rc5 maybe) after I get hold of the problem machine and report later today.

If there is any difference, it should be because of the reported topology by low-level platform code. In the problem case, each CPU was being reported to be a separate node (CONFIG_NUMA enabled) which caused idle_balance() to stop load-balance lookups at cpu/node level itself.

Quoted 2 lines

Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

Signature · 2 lines
Regards,
vatsa
#23 lore raw
In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

The problem is there in mainline very much. I could recreate the problem with 2.6.22-rc5 (which doesnt have CFS) on that same hardware, with CONFIG_NUMA enabled.

Let me know if you needed anything else to be clarified.

Signature · 2 lines
Regards,
vatsa
#24 lore raw
Quoted 10 lines · Srivatsa Vaddagiri
Quoted 3 lines · Ingo Molnar

I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

The problem is there in mainline very much. I could recreate the problem with 2.6.22-rc5 (which doesnt have CFS) on that same hardware, with CONFIG_NUMA enabled.

Let me know if you needed anything else to be clarified.

This is a bugfix that needs to go into 2.6.22.

#25 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 3 lines · Srivatsa Vaddagiri

I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.

Quoted 6 lines

thanks for tracking this down! I've applied Christoph's patch (with your suggested modification plus a few small cleanups).

I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

It did in fact trigger under all three of mainline, CFS, and -rt including CFS -- see below for a couple of emails from last Friday giving results for these three on the AMD box (where it happened) and on a single-quad NUMA-Q system (where it did not, at least not with such severity).

That said, there certainly was a time when neither mainline nor -rt acted this way!

Thanx, Paul

------------------------------------------------------------------------

Date: Fri, 15 Jun 2007 13:06:17 -0700
From: "Paul E. McKenney" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>,
Dinakar Guniguntala <[email protected]>
Subject: Re: v2.6.21.4-rt11

Quoted 4 lines · Paul E. McKenney
Quoted 3 lines · Ingo Molnar

Paul,

Quoted 8 lines
Quoted 8 lines

do you still see the load-distribution problem with -rt14? (which includes cfsv17) Or rather ... could you try vanilla cfsv17 instead:

http://people.redhat.com/mingo/cfs-scheduler/

to make sure it's not some effect in -rt causing this. v17 has an updated load balancing code. (which might or might not affect the rcutorture problem.)

No joy, see below. Strangely hardware dependent. My next step, left to myself, would be to patch rcutorture.c to cause the readers to dump the CFS state information every ten seconds or so. My guess is that the important per-task stuff is:

	current->sched_info.pcnt
	current->sched_info.cpu_time
	current->sched_info.run_delay
	current->sched_info.last_arrival
	current->sched_info.last_queued

And maybe the runqueue info dumped out by show_schedstat, this last via new per-CPU tasks.

Other thoughts?

Thanx, Paul

Quoted 3 lines

Good point! I will try the following:

1. Stock 2.6.21.5. 64-bit kernel on AMD Opterons.

All eight readers end up on the same CPU, CPU 1 in this case. And they stay there (ten minutes).

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3058 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3059 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3060 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3061 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3062 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3063 root      39  19     0    0    0 R 12.7  0.0   0:06.91 rcu_torture_rea   
 3057 root      39  19     0    0    0 R 12.3  0.0   0:06.91 rcu_torture_rea   
 3064 root      39  19     0    0    0 R 12.3  0.0   0:06.91 rcu_torture_rea   
Quoted 1 line

1. Stock 2.6.21.5. 32-bit kernel on NUMA-Q.

Works just fine(!).

Quoted 1 line

2. 2.6.21-rt14. 64-bit kernel on AMD Opterons.

All eight readers are spread, but over only two CPUs (0 and 3, in this case). Persists, usually with 4/4 split, but sometimes with five tasks on one CPU and three on the other.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3111 root      39  19     0    0    0 R 23.9  0.0   0:27.27 rcu_torture_rea   
 3114 root      39  19     0    0    0 R 23.9  0.0   0:28.58 rcu_torture_rea   
 3117 root      39  19     0    0    0 R 23.9  0.0   0:32.40 rcu_torture_rea   
 3112 root      39  19     0    0    0 R 23.6  0.0   0:28.41 rcu_torture_rea   
 3110 root      39  19     0    0    0 R 22.9  0.0   0:43.46 rcu_torture_rea   
 3113 root      39  19     0    0    0 R 22.9  0.0   0:27.28 rcu_torture_rea   
 3115 root      39  19     0    0    0 R 22.9  0.0   0:33.08 rcu_torture_rea   
 3116 root      39  19     0    0    0 R 22.6  0.0   0:28.10 rcu_torture_rea   
elm3b6:~# for ((i=3110;i<=3117;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
3 3 0 3 0 0 0 3
Quoted 1 line

2. 2.6.21-rt14. 32-bit kernel on NUMA-Q.

Works just fine.

Quoted 1 line

3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 64-bit kernel on

AMD Opteron.

All eight readers end up on the same CPU, CPU 2 in this case. And they stay there (ten minutes).

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3081 root      39  19     0    0    0 R 11.3  0.0   1:31.77 rcu_torture_rea   
 3082 root      39  19     0    0    0 R 11.3  0.0   1:31.77 rcu_torture_rea   
 3085 root      39  19     0    0    0 R 11.3  0.0   1:31.78 rcu_torture_rea   
 3079 root      39  19     0    0    0 R 11.0  0.0   1:31.72 rcu_torture_rea   
 3080 root      39  19     0    0    0 R 11.0  0.0   1:31.76 rcu_torture_rea   
 3083 root      39  19     0    0    0 R 11.0  0.0   1:31.76 rcu_torture_rea   
 3084 root      39  19     0    0    0 R 11.0  0.0   1:31.77 rcu_torture_rea   
 3086 root      39  19     0    0    0 R 11.0  0.0   1:31.75 rcu_torture_rea   

Using "taskset" to pin each process to a pair of CPUs (masks 0x3, 0x6, 0xc, and 0x9) forces them to CPUs 0 and 2 -- previously this had spread them nicely. So I kept pinning tasks to single CPUs (which defeats some rcutorture testing) until they did spread, getting the following:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3079 root      39  19     0    0    0 R 49.9  0.0   3:18.91 rcu_torture_rea   
 3080 root      39  19     0    0    0 R 49.6  0.0   3:15.76 rcu_torture_rea   
 3086 root      39  19     0    0    0 R 49.6  0.0   3:48.82 rcu_torture_rea   
 3083 root      39  19     0    0    0 R 49.3  0.0   2:58.02 rcu_torture_rea   
 3084 root      39  19     0    0    0 R 48.6  0.0   3:00.54 rcu_torture_rea   
 3081 root      39  19     0    0    0 R 47.9  0.0   3:00.55 rcu_torture_rea   
 3082 root      39  19     0    0    0 R 44.6  0.0   3:18.89 rcu_torture_rea   
 3085 root      39  19     0    0    0 R 44.3  0.0   3:07.11 rcu_torture_rea   
elm3b6:~# for ((i=3079;i<=3086;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
0 0 2 1 3 2 1 3
Quoted 1 line

3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 32-bit kernel on

	NUMA-Q.

Some imbalance:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 2263 root      39  19     0    0    0 R 92.4  0.0   2:19.69 rcu_torture_rea   
 2265 root      39  19     0    0    0 R 49.8  0.0   1:41.84 rcu_torture_rea   
 2264 root      39  19     0    0    0 R 49.5  0.0   2:11.69 rcu_torture_rea   
 2261 root      39  19     0    0    0 R 48.8  0.0   2:09.95 rcu_torture_rea   
 2262 root      39  19     0    0    0 R 48.8  0.0   3:01.42 rcu_torture_rea   
 2266 root      39  19     0    0    0 R 30.1  0.0   1:47.02 rcu_torture_rea   
 2260 root      39  19     0    0    0 R 29.8  0.0   2:10.07 rcu_torture_rea   
 2267 root      39  19     0    0    0 R 29.8  0.0   1:57.34 rcu_torture_rea   
elm3b132:~# for ((i=2260;i<=2267;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
0 1 3 1 2 2 2 0

Has persisted (with some shuffling of CPUs, see below) for about five minutes, will let it run for an hour or so to see if it is really serious about this.

3 0 0 2 1 0 2 3

------------------------------------------------------------------------

Date: Fri, 15 Jun 2007 15:00:17 -0700
From: "Paul E. McKenney" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>,
Dinakar Guniguntala <[email protected]>, Srivatsa Vaddagiri <[email protected]>, Dmitry Adamushko <[email protected]>
Subject: Re: v2.6.21.4-rt11

Quoted 4 lines · Ingo Molnar

(forwarding Paul's mail below to other CFS hackers too.)

------------>

[ . . . ]

Quoted 15 lines
Quoted 1 line

3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 32-bit kernel on

	NUMA-Q.

Some imbalance:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2263 root      39  19     0    0    0 R 92.4  0.0   2:19.69 rcu_torture_rea
 2265 root      39  19     0    0    0 R 49.8  0.0   1:41.84 rcu_torture_rea
 2264 root      39  19     0    0    0 R 49.5  0.0   2:11.69 rcu_torture_rea
 2261 root      39  19     0    0    0 R 48.8  0.0   2:09.95 rcu_torture_rea
 2262 root      39  19     0    0    0 R 48.8  0.0   3:01.42 rcu_torture_rea
 2266 root      39  19     0    0    0 R 30.1  0.0   1:47.02 rcu_torture_rea
 2260 root      39  19     0    0    0 R 29.8  0.0   2:10.07 rcu_torture_rea
 2267 root      39  19     0    0    0 R 29.8  0.0   1:57.34 rcu_torture_rea
Quoted 8 lines
elm3b132:~# for ((i=2260;i<=2267;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
0 1 3 1 2 2 2 0

Has persisted (with some shuffling of CPUs, see below) for about five minutes, will let it run for an hour or so to see if it is really serious about this.

3 0 0 2 1 0 2 3

And when I returned after an hour, it had straightened itself out:
1 3 1 2 2 0 3 0

The 64-bit AMD 4-CPU machines have not straightened themselves out in the past, but will try an extended run over the weekend to see if load balancing is just a bit on the slow side. ;-)

But got distracted for an additional hour, and it is imbalanced again:

2 1 2 0 1 1 3 3

Strange...

Thanx, Paul

#26 lore raw
In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?

Right. The patch needs to go into mainline as well.