v2.6.21.4-rt11
Quoted 57 lines · Paul E. McKenney
Quoted 56 lines · Srivatsa Vaddagiri
Quoted 22 lines · Paul E. McKenney
Quoted 21 lines · Ingo Molnar
Quoted 6 lines · Paul E. McKenney
Quoted 3 lines
to make sure it's not some effect in -rt causing this. v17 has an updated load balancing code. (which might or might not affect the rcutorture problem.)
Good point! I will try the following:
Quoted 8 lines
1. Stock 2.6.21.5.
2. 2.6.21-rt14.
3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch
And quickly, before everyone else jumps on the machines that show the problem. ;-)
thanks! It's enough to check whether modprobe rcutorture still produces that weird balancing problem. That clearly has to be fixed ...
Quoted 6 lines
Quoted 2 lines
And i've Cc:-ed Dmitry and Srivatsa, who are busy hacking this area of the CFS code as we speak :-)
Well, I am not sure that the info I was able to collect will be all that helpful, but it most certainly does confirm that the balancing problem that rcutorture produces is indeed weird...
Hi Paul,
I tried on two machines in our lab and could not recreate your
problem.On a 2way
x86_64AMD box and 2.6.21.5+cfsv17:PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12395 root 39 19 0 0 0 R 50.3 0.0 0:57.62 rcu_torture_rea 12394 root 39 19 0 0 0 R 49.9 0.0 0:57.29 rcu_torture_rea 12396 root 39 19 0 0 0 R 49.9 0.0 0:56.96 rcu_torture_rea 12397 root 39 19 0 0 0 R 49.9 0.0 0:56.90 rcu_torture_reaOn a 4way
x86_64Intel Xeon box and 2.6.21.5+cfsv17:PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 6258 root 39 19 0 0 0 R 53 0.0 17:29.72 0 rcu_torture_rea 6252 root 39 19 0 0 0 R 49 0.0 17:49.40 3 rcu_torture_rea 6257 root 39 19 0 0 0 R 49 0.0 17:22.49 2 rcu_torture_rea 6256 root 39 19 0 0 0 R 48 0.0 17:50.12 1 rcu_torture_rea 6254 root 39 19 0 0 0 R 48 0.0 17:26.98 0 rcu_torture_rea 6255 root 39 19 0 0 0 R 48 0.0 17:25.74 2 rcu_torture_rea 6251 root 39 19 0 0 0 R 45 0.0 17:47.45 3 rcu_torture_rea 6253 root 39 19 0 0 0 R 45 0.0 17:48.48 1 rcu_torture_rea
Quoted 6 lines
Quoted 3 lines
I will try this on few more boxes we have on Monday. If I can't recreate, then I may request you to provide me machine details (or even access to the problem box if it is in IBM labs and if I am allowed to login!)
elm3b6, ABAT job 95107. There are others, this particular job uses 2.6.21.5-cfsv17.
Paul,
I logged into elm3b6 and did some investigation. I think I have
a tentative patch to fix your load-balance problem.
First, an explanation of the problem:
This particular machine, elm3b6, is a 4-cpu, (gasp, yes!) 4-node box i.e each CPU is a node by itself. If you don't have CONFIG_NUMA enabled, then we won't have cross-node (i.e cross-cpu) load balancing. Fortunately in your case you had CONFIG_NUMA enabled, but still were hitting the (gross) load imbalance.
The problem seems to be with idle_balance(). This particular routine, invoked by schedule() on a idle cpu, walks up sched-domain hierarchy and tries to balance in each domain that has SD_BALANCE_NEWIDLE flag set. The nodes-level domain (SD_NODE_INIT) however doesn't set this flag, which means idle cpu looks for (im)balance within its own node at most and not beyond. Now, here's the problem, if the idle cpu doesn't find imbalance within its node (pulled_tasks = 0), it resets this_rq->next_balance so that next balancing activity is deferred for upto a minute (next_balance = jiffies + 60 * HZ). If a idle cpu calls idle_balance again in the next minute and finds no imbalance within its node, it -again- resets next_balance. In your case, I think this was happening repetetively, which made other CPUs never look for cross-node (im)balance.
I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.
I have CCed others who have worked in this area and request them to review this patch.
Andrew,
If there is no objection from anyone, request you to pick this
up for next -mm release. It has been tested against 2.6.22-rc4-mm2.
idle_balance() can erroneously cause system-wide imbalance to be overlooked by reseting rq->next_balance. When called sufficient number of times, it can forever defer system-wide load balance. Patch below modifies idle_balance() not to mess with ->next_balance. If indeed it turns out that there is no imbalance even system-wide, rebalance_domains() will anyway set ->next_balance to happen after a minute.
kernel/sched.c +0 -11
@@ -2490,27 +2490,16 @@ {struct sched_domain *sd;int pulled_task =0 ; -unsigned long next_balance = jiffies +60 *HZ ;for_each_domain (this_cpu, sd) {if (sd->flags &SD_BALANCE_NEWIDLE ) {/* If we've pulled tasks over stop searching: */ pulled_task =load_balance_newidle (this_cpu, this_rq, sd); -if (time_after (next_balance, - sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval;if (pulled_task)break ; } } -if (!pulled_task) -/* -* We are going idle. next_balance may be set based on -* a busy processor. So reset next_balance. -*/ - this_rq->next_balance = next_balance; }/*
Signature · 2 lines
Regards, vatsa
Quoted 11 lines · Srivatsa Vaddagiri
This particular machine, elm3b6, is a 4-cpu, (gasp, yes!) 4-node box i.e each CPU is a node by itself. If you don't have
CONFIG_NUMAenabled, then we won't have cross-node (i.e cross-cpu) load balancing. Fortunately in your case you hadCONFIG_NUMAenabled, but still were hitting the (gross) load imbalance.The problem seems to be with
idle_balance(). This particular routine, invoked byschedule()on a idle cpu, walks up sched-domain hierarchy and tries to balance in each domain that hasSD_BALANCE_NEWIDLEflag set. The nodes-level domain (SD_NODE_INIT) however doesn't set this flag, which means idle cpu looks for (im)balance within its own node at most and
The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node. The balancing on the phys_domain level does balance within a node.
Quoted 2 lines · Christoph Lameter
The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node.
I was mostly speaking with the example system in mind (4-node 4-cpu box), but yes, node-level domain does look for imbalance across max 16 nodes as you mention.
Both node and all-node domains don't have SD_BALANCE_NEWIDLE set, which means idle_balance() will stop looking for imbalance beyonds its own node. Based on the observed balance within its own node, IMO, idle_balance() should not cause ->next_balance to be reset.
Signature · 2 lines
Regards, vatsa
Quoted 2 lines · Srivatsa Vaddagiri
If you don't have
CONFIG_NUMAenabled,
then we won't have cross-node (i.e cross-cpu) load balancing.
Mmm ..that is not correct. I found that disabling CONFIG_NUMA leads to better load balance on the problem system (i.e w/o any patches applied 2.6.22-rc4-mm2 leads to good distribution of rcu readers on all 4 cpus).
Anyway, the patch is still needed for scenarios like you originally tested with.
Signature · 2 lines
Regards, vatsa
Quoted 4 lines · Srivatsa Vaddagiri
Quoted 2 lines · Christoph Lameter
The nodes-level domain looks for internode balances between up to 16 nodes. It is not restricted to a single node.
Quoted 8 lines
I was mostly speaking with the example system in mind (4-node 4-cpu box), but yes, node-level domain does look for imbalance across max 16 nodes as you mention.
Both node and all-node domains don't have
SD_BALANCE_NEWIDLEset, which meansidle_balance()will stop looking for imbalance beyonds its own node. Based on the observed balance within its own node, IMO,idle_balance()should not cause ->next_balance to be reset.
I think the check in idle_balance needs to be modified.
If the domain *does not* have SD_BALANCE_NEWIDLE set then next_balance must still be set right. Does this patch fix it?
Scheduler: Fix next_interval determination in idle_balance().
The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.
kernel/sched.c +7 -8
@@ -2493,17 +2493,16 @@ static void idle_balance(int this_cpu, sunsigned long next_balance = jiffies +60 *HZ ;for_each_domain (this_cpu, sd) { -if (sd->flags &SD_BALANCE_NEWIDLE ) { +if (sd->flags &SD_BALANCE_NEWIDLE )/* If we've pulled tasks over stop searching: */ pulled_task =load_balance_newidle (this_cpu, this_rq, sd); -if (time_after (next_balance, - sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval; -if (pulled_task) -break ; - } +if (time_after (next_balance, + sd->last_balance + sd->balance_interval)) + next_balance = sd->last_balance + + sd->balance_interval; +if (pulled_task) +break ; }if (!pulled_task)/*
Quoted 4 lines · Christoph Lameter
I think the check in
idle_balanceneeds to be modified.If the domain *does not* have
SD_BALANCE_NEWIDLEset thennext_balancemust still be set right. Does this patch fix it?
Is the ->next_balance calculation in idle_balance() necessary at all? rebalance_domains() would have programmed ->next_balance anyway, based on the nearest next_balance point of all (load-balance'able) domains. By repeating that calculation in idle_balance, are we covering any corner case?
Signature · 2 lines
Regards, vatsa
Quoted 10 lines · Srivatsa Vaddagiri
Quoted 4 lines · Christoph Lameter
I think the check in
idle_balanceneeds to be modified.If the domain *does not* have
SD_BALANCE_NEWIDLEset thennext_balancemust still be set right. Does this patch fix it?Is the ->next_balance calculation in
idle_balance()necessary at all?rebalance_domains()would have programmed ->next_balance anyway, based on the nearestnext_balancepoint of all (load-balance'able) domains. By repeating that calculation inidle_balance, are we covering any corner case?
rebalance_domains() have programmed ->next_balance based on 'busy' state. And now, as it is going to 'idle', this routine is recalculating the next_balance based on 'idle' state.
thanks,
suresh
Quoted 7 lines · Christoph Lameter
kernel/sched.c +1
-2for_each_domain (this_cpu, sd) { -if (sd->flags &SD_BALANCE_NEWIDLE ) { +if (sd->flags &SD_BALANCE_NEWIDLE )/* If we've pulled tasks over stop searching: */ pulled_task =load_balance_newidle (this_cpu, this_rq, sd); - if (time_after(next_balance,
kernel/sched.c static void idle_balance(int this_cpu, s · line 2503
Quoted 10 lines
kernel/sched.c +4
-6- sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval; -if (pulled_task) -break ; - } +if (time_after (next_balance, + sd->last_balance + sd->balance_interval)) + next_balance = sd->last_balance + + sd->balance_interval;
don't we have to do, msecs_to_jiffies(sd->balance_interval)?
thanks,
suresh
Quoted 5 lines
kernel/sched.c +2
-0+if (pulled_task) +break ; }if (!pulled_task)/*
Quoted 6 lines · Suresh B
Review
Quoted 4 lines
(quoted diff) +4
-0+ if (time_after(next_balance, + sd->last_balance + sd->balance_interval)) + next_balance = sd->last_balance + + sd->balance_interval;don't we have to do,
msecs_to_jiffies(sd->balance_interval)?
Well that is certainly a bug here. Is this better?
Scheduler: Fix next_interval determination in idle_balance().
The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.
Siddha also spotted that the conversion of the balance interval to jiffies is missing. Fix that to.
kernel/sched.c +11 -10
@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, sunsigned long next_balance = jiffies +60 *HZ ;for_each_domain (this_cpu, sd) { -if (sd->flags &SD_BALANCE_NEWIDLE ) { +unsigned long interval; + +if (sd->flags &SD_BALANCE_NEWIDLE )/* If we've pulled tasks over stop searching: */ - pulled_task =load_balance_newidle (this_cpu, - this_rq, sd); -if (time_after (next_balance, - sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval; -if (pulled_task) -break ; - } + pulled_task =load_balance_newidle (this_cpu,this_rq, sd); + + interval =msecs_to_jiffies (sd->balance_interval); +if (time_after (next_balance, + sd->last_balance + interval)) + next_balance = sd->last_balance + interval; +if (pulled_task) +break ; }if (!pulled_task)/*
kernel/sched.c static void idle_balance(int this_cpu, s · line 2497
Quoted 7 lines · Christoph Lameter
kernel/sched.c +2
-1@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, sunsigned long next_balance = jiffies +60 *HZ ;for_each_domain (this_cpu, sd) { -if (sd->flags &SD_BALANCE_NEWIDLE ) { +unsigned long interval; +
Do we need a :
if (!(sd->flags & SD_LOAD_BALANCE)) continue;
here?
Otherwise patch look good and fixes the problem Paul observed earlier.
Quoted 24 lines
kernel/sched.c +9
-9+if (sd->flags &SD_BALANCE_NEWIDLE )/* If we've pulled tasks over stop searching: */ - pulled_task =load_balance_newidle (this_cpu, - this_rq, sd); -if (time_after (next_balance, - sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval; -if (pulled_task) -break ; - } + pulled_task =load_balance_newidle (this_cpu,this_rq, sd); + + interval =msecs_to_jiffies (sd->balance_interval); +if (time_after (next_balance, + sd->last_balance + interval)) + next_balance = sd->last_balance + interval; +if (pulled_task) +break ; }kernel/sched.c +0
-0if (!pulled_task)/*
Signature · 2 lines
Regards, vatsa
Quoted 11 lines · Srivatsa Vaddagiri
Review static void idle_balance(int this_cpu, s · line 2497
Quoted 7 lines · Christoph Lameter
(quoted diff) +2
-1@@ -2493,17 +2493,18 @@ static void idle_balance(int this_cpu, s unsigned long next_balance = jiffies + 60 * HZ; for_each_domain(this_cpu, sd) { - if (sd->flags & SD_BALANCE_NEWIDLE) { + unsigned long interval; +Do we need a :
Quoted 6 lines
if (!(sd->flags & SD_LOAD_BALANCE)) continue;here?
Otherwise patch look good and fixes the problem Paul observed earlier.
great! I've applied the patch below (added your fix and cleaned it up a bit) and have released 2.6.21.5-rt17 with it.
Ingo
------------------------------>
From: Christoph Lameter <[email protected]>
Subject: [patch] sched: fix next_interval determination in idle_balance().
The intervals of domains that do not have SD_BALANCE_NEWIDLE must be considered for the calculation of the time of the next balance. Otherwise we may defer rebalancing forever.
Siddha also spotted that the conversion of the balance interval to jiffies is missing. Fix that to.
From: Srivatsa Vaddagiri <[email protected]>
also continue the loop if !(sd->flags & SD_LOAD_BALANCE).
--- kernel/sched.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
kernel/sched.c +13 -9
@@ -2591,17 +2591,21 @@ static void idle_balance(int this_cpu, sunsigned long next_balance = jiffies +HZ ;for_each_domain (this_cpu, sd) { -if (sd->flags &SD_BALANCE_NEWIDLE ) { +unsigned long interval; + +if (!(sd->flags &SD_LOAD_BALANCE )) +continue ; + +if (sd->flags &SD_BALANCE_NEWIDLE )/* If we've pulled tasks over stop searching: */ pulled_task =load_balance_newidle (this_cpu, - this_rq, sd); -if (time_after (next_balance, - sd->last_balance + sd->balance_interval)) - next_balance = sd->last_balance - + sd->balance_interval; -if (pulled_task) -break ; - } + this_rq, sd); + + interval =msecs_to_jiffies (sd->balance_interval); +if (time_after (next_balance, sd->last_balance + interval)) + next_balance = sd->last_balance + interval; +if (pulled_task) +break ; }if (pulled_task ||time_after (jiffies, this_rq->next_balance)) {/*
Quoted 3 lines · Srivatsa Vaddagiri
I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.
thanks for tracking this down! I've applied Christoph's patch (with your suggested modification plus a few small cleanups).
I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in idle_balance() too, or am i misreading it?
Ingo
Quoted 1 line · Ingo Molnar
I'm wondering, why did this trigger under CFS and not on mainline?
I thought Paul had seen the same problem with 2.6.21.5. I will try a more recent mainline (2.6.22-rc5 maybe) after I get hold of the problem machine and report later today.
If there is any difference, it should be because of the reported topology by low-level platform code. In the problem case, each CPU was being reported to be a separate node (CONFIG_NUMA enabled) which caused idle_balance() to stop load-balance lookups at cpu/node level itself.
Quoted 2 lines
Mainline seems to have a similar problem in
idle_balance()too, or am i misreading it?
Signature · 2 lines
Regards, vatsa
Quoted 3 lines · Ingo Molnar
I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in
idle_balance()too, or am i misreading it?
The problem is there in mainline very much. I could recreate the problem with 2.6.22-rc5 (which doesnt have CFS) on that same hardware, with CONFIG_NUMA enabled.
Let me know if you needed anything else to be clarified.
Signature · 2 lines
Regards, vatsa
Quoted 7 lines · Ingo Molnar
Quoted 3 lines · Srivatsa Vaddagiri
I believe the patch below is correct. With the patch applied, I could not recreate the imbalance with rcutorture. Let me know whether you still see the problem with this patch applied on any other machine.
Quoted 6 lines
thanks for tracking this down! I've applied Christoph's patch (with your suggested modification plus a few small cleanups).
I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in
idle_balance()too, or am i misreading it?
It did in fact trigger under all three of mainline, CFS, and -rt including CFS -- see below for a couple of emails from last Friday giving results for these three on the AMD box (where it happened) and on a single-quad NUMA-Q system (where it did not, at least not with such severity).
That said, there certainly was a time when neither mainline nor -rt acted this way!
Thanx, Paul
------------------------------------------------------------------------
Date: Fri, 15 Jun 2007 13:06:17 -0700
From: "Paul E. McKenney" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>,
Dinakar Guniguntala <[email protected]>
Subject: Re: v2.6.21.4-rt11
Quoted 4 lines · Paul E. McKenney
Quoted 3 lines · Ingo Molnar
Paul,
Quoted 8 lines
Quoted 8 lines
do you still see the load-distribution problem with -rt14? (which includes cfsv17) Or rather ... could you try vanilla cfsv17 instead:
http://people.redhat.com/mingo/cfs-scheduler/
to make sure it's not some effect in -rt causing this. v17 has an updated load balancing code. (which might or might not affect the rcutorture problem.)
No joy, see below. Strangely hardware dependent. My next step, left to myself, would be to patch rcutorture.c to cause the readers to dump the CFS state information every ten seconds or so. My guess is that the important per-task stuff is:
current->sched_info.pcnt current->sched_info.cpu_time current->sched_info.run_delay current->sched_info.last_arrival current->sched_info.last_queued
And maybe the runqueue info dumped out by show_schedstat, this last via new per-CPU tasks.
Other thoughts?
Thanx, Paul
Quoted 3 lines
Good point! I will try the following:
1. Stock 2.6.21.5. 64-bit kernel on AMD Opterons.
All eight readers end up on the same CPU, CPU 1 in this case. And they stay there (ten minutes).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3058 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3059 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3060 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3061 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3062 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3063 root 39 19 0 0 0 R 12.7 0.0 0:06.91 rcu_torture_rea 3057 root 39 19 0 0 0 R 12.3 0.0 0:06.91 rcu_torture_rea 3064 root 39 19 0 0 0 R 12.3 0.0 0:06.91 rcu_torture_rea
Quoted 1 line
1. Stock 2.6.21.5. 32-bit kernel on NUMA-Q.
Works just fine(!).
Quoted 1 line
2. 2.6.21-rt14. 64-bit kernel on AMD Opterons.
All eight readers are spread, but over only two CPUs (0 and 3, in this case). Persists, usually with 4/4 split, but sometimes with five tasks on one CPU and three on the other.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3111 root 39 19 0 0 0 R 23.9 0.0 0:27.27 rcu_torture_rea 3114 root 39 19 0 0 0 R 23.9 0.0 0:28.58 rcu_torture_rea 3117 root 39 19 0 0 0 R 23.9 0.0 0:32.40 rcu_torture_rea 3112 root 39 19 0 0 0 R 23.6 0.0 0:28.41 rcu_torture_rea 3110 root 39 19 0 0 0 R 22.9 0.0 0:43.46 rcu_torture_rea 3113 root 39 19 0 0 0 R 22.9 0.0 0:27.28 rcu_torture_rea 3115 root 39 19 0 0 0 R 22.9 0.0 0:33.08 rcu_torture_rea 3116 root 39 19 0 0 0 R 22.6 0.0 0:28.10 rcu_torture_rea
elm3b6:~# for ((i=3110;i<=3117;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
3 3 0 3 0 0 0 3Quoted 1 line
2. 2.6.21-rt14. 32-bit kernel on NUMA-Q.
Works just fine.
Quoted 1 line
3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 64-bit kernel on
AMD Opteron.
All eight readers end up on the same CPU, CPU 2 in this case. And they stay there (ten minutes).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3081 root 39 19 0 0 0 R 11.3 0.0 1:31.77 rcu_torture_rea 3082 root 39 19 0 0 0 R 11.3 0.0 1:31.77 rcu_torture_rea 3085 root 39 19 0 0 0 R 11.3 0.0 1:31.78 rcu_torture_rea 3079 root 39 19 0 0 0 R 11.0 0.0 1:31.72 rcu_torture_rea 3080 root 39 19 0 0 0 R 11.0 0.0 1:31.76 rcu_torture_rea 3083 root 39 19 0 0 0 R 11.0 0.0 1:31.76 rcu_torture_rea 3084 root 39 19 0 0 0 R 11.0 0.0 1:31.77 rcu_torture_rea 3086 root 39 19 0 0 0 R 11.0 0.0 1:31.75 rcu_torture_rea
Using "taskset" to pin each process to a pair of CPUs (masks 0x3, 0x6, 0xc, and 0x9) forces them to CPUs 0 and 2 -- previously this had spread them nicely. So I kept pinning tasks to single CPUs (which defeats some rcutorture testing) until they did spread, getting the following:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3079 root 39 19 0 0 0 R 49.9 0.0 3:18.91 rcu_torture_rea 3080 root 39 19 0 0 0 R 49.6 0.0 3:15.76 rcu_torture_rea 3086 root 39 19 0 0 0 R 49.6 0.0 3:48.82 rcu_torture_rea 3083 root 39 19 0 0 0 R 49.3 0.0 2:58.02 rcu_torture_rea 3084 root 39 19 0 0 0 R 48.6 0.0 3:00.54 rcu_torture_rea 3081 root 39 19 0 0 0 R 47.9 0.0 3:00.55 rcu_torture_rea 3082 root 39 19 0 0 0 R 44.6 0.0 3:18.89 rcu_torture_rea 3085 root 39 19 0 0 0 R 44.3 0.0 3:07.11 rcu_torture_rea
elm3b6:~# for ((i=3079;i<=3086;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
0 0 2 1 3 2 1 3Quoted 1 line
3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 32-bit kernel on
NUMA-Q.
Some imbalance:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2263 root 39 19 0 0 0 R 92.4 0.0 2:19.69 rcu_torture_rea 2265 root 39 19 0 0 0 R 49.8 0.0 1:41.84 rcu_torture_rea 2264 root 39 19 0 0 0 R 49.5 0.0 2:11.69 rcu_torture_rea 2261 root 39 19 0 0 0 R 48.8 0.0 2:09.95 rcu_torture_rea 2262 root 39 19 0 0 0 R 48.8 0.0 3:01.42 rcu_torture_rea 2266 root 39 19 0 0 0 R 30.1 0.0 1:47.02 rcu_torture_rea 2260 root 39 19 0 0 0 R 29.8 0.0 2:10.07 rcu_torture_rea 2267 root 39 19 0 0 0 R 29.8 0.0 1:57.34 rcu_torture_rea
elm3b132:~# for ((i=2260;i<=2267;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done
0 1 3 1 2 2 2 0Has persisted (with some shuffling of CPUs, see below) for about five minutes, will let it run for an hour or so to see if it is really serious about this.
3 0 0 2 1 0 2 3
------------------------------------------------------------------------
Date: Fri, 15 Jun 2007 15:00:17 -0700
From: "Paul E. McKenney" <[email protected]>
To: Ingo Molnar <[email protected]>
Cc: Thomas Gleixner <[email protected]>,
Dinakar Guniguntala <[email protected]>, Srivatsa Vaddagiri <[email protected]>, Dmitry Adamushko <[email protected]>
Subject: Re: v2.6.21.4-rt11
Quoted 4 lines · Ingo Molnar
(forwarding Paul's mail below to other CFS hackers too.)
------------>
[ . . . ]
Quoted 15 lines
Quoted 1 line
3. 2.6.21.5 + sched-cfs-v2.6.21.5-v17.patch on 32-bit kernel on
NUMA-Q.Some imbalance:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2263 root 39 19 0 0 0 R 92.4 0.0 2:19.69 rcu_torture_rea 2265 root 39 19 0 0 0 R 49.8 0.0 1:41.84 rcu_torture_rea 2264 root 39 19 0 0 0 R 49.5 0.0 2:11.69 rcu_torture_rea 2261 root 39 19 0 0 0 R 48.8 0.0 2:09.95 rcu_torture_rea 2262 root 39 19 0 0 0 R 48.8 0.0 3:01.42 rcu_torture_rea 2266 root 39 19 0 0 0 R 30.1 0.0 1:47.02 rcu_torture_rea 2260 root 39 19 0 0 0 R 29.8 0.0 2:10.07 rcu_torture_rea 2267 root 39 19 0 0 0 R 29.8 0.0 1:57.34 rcu_torture_rea
Quoted 8 lines
elm3b132:~# for ((i=2260;i<=2267;i++)); do cat /proc/$i/stat | awk '{print $(NF-3)}'; done 0 1 3 1 2 2 2 0Has persisted (with some shuffling of CPUs, see below) for about five minutes, will let it run for an hour or so to see if it is really serious about this.
3 0 0 2 1 0 2 3
And when I returned after an hour, it had straightened itself out:
1 3 1 2 2 0 3 0
The 64-bit AMD 4-CPU machines have not straightened themselves out in the past, but will try an extended run over the weekend to see if load balancing is just a bit on the slow side. ;-)
But got distracted for an additional hour, and it is imbalanced again:
2 1 2 0 1 1 3 3
Strange...
Thanx, Paul
Quoted 3 lines · Ingo Molnar
I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in
idle_balance()too, or am i misreading it?
Right. The patch needs to go into mainline as well.
Quoted 10 lines · Srivatsa Vaddagiri
Quoted 3 lines · Ingo Molnar
I'm wondering, why did this trigger under CFS and not on mainline? Mainline seems to have a similar problem in
idle_balance()too, or am i misreading it?The problem is there in mainline very much. I could recreate the problem with 2.6.22-rc5 (which doesnt have CFS) on that same hardware, with
CONFIG_NUMAenabled.Let me know if you needed anything else to be clarified.
This is a bugfix that needs to go into 2.6.22.