O18.1int
PATCH 1/1 O18.1int
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Some high credit tasks were being missed due to their prolonged cpu burn at startup flagging them as low credit tasks.
Low credit tasks can now recover to become high credit.
Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
iD8DBQE/RwHDZUg7+tp6mRURAie7AJ43egdTeSapoX1D0aJQcEksBTkKdwCfcyHZ cD1TMt7oFNXvmSrqnJe7Z+E=
=41zp
-----END PGP SIGNATURE-----
kernel/sched.c +3 -5
@@ -140,9 +140,6 @@#define LOW_CREDIT (p) \ ((p)->interactive_credit < -MAX_SLEEP_AVG) -#define VARYING_CREDIT (p) \ - (!(HIGH_CREDIT(p) || LOW_CREDIT(p))) -#define TASK_PREEMPTS_CURR (p, rq) \ ((p)->prio < (rq)->curr->prio)
@@ -434,7 +431,7 @@ static void recalc_task_prio(task_t *p,if (p->sleep_avg >NS_MAX_SLEEP_AVG ){ p->sleep_avg =NS_MAX_SLEEP_AVG ; - p->interactive_credit +=VARYING_CREDIT (p); + p->interactive_credit += !(HIGH_CREDIT (p)); } } }
@@ -1548,7 +1545,8 @@ switch_tasks: prev->sleep_avg -= run_time;if ((long )prev->sleep_avg <=0 ){ prev->sleep_avg =0 ; - prev->interactive_credit -=VARYING_CREDIT (prev); + prev->interactive_credit -= + !(HIGH_CREDIT (prev) ||LOW_CREDIT (prev)); } prev->timestamp = now;
Quoted 6 lines · Con Kolivas
Some high credit tasks were being missed due to their prolonged cpu burn at startup flagging them as low credit tasks.
Low credit tasks can now recover to become high credit.
Con
Hi Con!
First of all... Your interactive scheduler work is GREAT! I really like it...!
Now I tried to unterstand what exacly the latest patch does, and as far as I can see the first and the third hunk just delete respectively expand the macro VARYING_CREDIT(p). But the second hunk helps processes to get some interactive_credit until they become a HIGH_CREDIT task. This looks reasonable to me...
So, now I wanted to know how a task may lose its interactive_credit again... The only code I saw doing this is exaclty the third hunk of your patch. But if a process is a HIGH_CREDIT task it can never lose its interactive_credit again. Is that intented?
I think the third hunk should look like following:
@@ -1548,7 +1545,7 @@ switch_tasks:
prev->sleep_avg -= run_time;
if ((long)prev->sleep_avg <= 0){
prev->sleep_avg = 0;
- prev->interactive_credit -= VARYING_CREDIT(prev);
+ prev->interactive_credit -= !(LOW_CREDIT(prev));
}
prev->timestamp = now;As an additional idea I think interactive_credit should be allowed to be a bit bigger than MAX_SLEEP_AVG and a bit lower than -MAX_SLEEP_AVG. This would make LOW_CREDIT processes stay LOW_CREDIT even if they do some sleep and HIGH_CREDIT processes star HIGH_CREDIT even if they do some computing...
But of course I may completely miss something...
Thomas
Quoted 30 lines · Thomas Schlichter
Quoted 7 lines · Con Kolivas
Some high credit tasks were being missed due to their prolonged cpu burn at startup flagging them as low credit tasks.
Low credit tasks can now recover to become high credit.
Con
Hi Con!
First of all... Your interactive scheduler work is GREAT! I really like it...!
Now I tried to unterstand what exacly the latest patch does, and as far as I can see the first and the third hunk just delete respectively expand the macro
VARYING_CREDIT(p). But the second hunk helps processes to get someinteractive_credituntil they become aHIGH_CREDITtask. This looks reasonable to me...So, now I wanted to know how a task may lose its
interactive_creditagain... The only code I saw doing this is exaclty the third hunk of your patch. But if a process is aHIGH_CREDITtask it can never lose itsinteractive_creditagain. Is that intented?I think the third hunk should look like following:
kernel/sched.c +1
-1@@ -1548,7 +1545,7 @@ switch_tasks: prev->sleep_avg -= run_time;if ((long )prev->sleep_avg <=0 ){ prev->sleep_avg =0 ; - prev->interactive_credit -=VARYING_CREDIT (prev); + prev->interactive_credit -= !(LOW_CREDIT (prev)); } prev->timestamp = now;As an additional idea I think
interactive_creditshould be allowed to be a bit bigger thanMAX_SLEEP_AVGand a bit lower than -MAX_SLEEP_AVG. This would makeLOW_CREDITprocesses stayLOW_CREDITeven if they do some sleep andHIGH_CREDITprocesses starHIGH_CREDITeven if they do some computing...
Quoted 3 lines
But of course I may completely miss something...
Hi
I don't know what is preferred on lkml, but I dislike mixing booleans and integer arithmetic.
if (!LOW_CREDIT(prev))
prev->interactive_credit--;
Easier to read IMO.
Quoted 8 lines · Nick Piggin
Hi
I don't know what is preferred on lkml, but I dislike mixing booleans and integer arithmetic.if (!LOW_CREDIT(prev))
prev->interactive_credit--;Easier to read IMO.
I agree. I only mixed them because of my (perhaps false) belief that it's less of a hit not having another if.. branch point. Then again today's compilers probably optimise it out anyway.
Con
Quoted 8 lines · Thomas Schlichter
Quoted 4 lines · Con Kolivas
Some high credit tasks were being missed due to their prolonged cpu burn at startup flagging them as low credit tasks.
Low credit tasks can now recover to become high credit.
First of all... Your interactive scheduler work is GREAT! I really like it...!
Thank you. I never assume no news is good news so I appreciate that.
Quoted 10 lines
Now I tried to unterstand what exacly the latest patch does, and as far as I can see the first and the third hunk just delete respectively expand the macro
VARYING_CREDIT(p). But the second hunk helps processes to get someinteractive_credituntil they become aHIGH_CREDITtask. This looks reasonable to me...So, now I wanted to know how a task may lose its
interactive_creditagain... The only code I saw doing this is exaclty the third hunk of your patch. But if a process is aHIGH_CREDITtask it can never lose itsinteractive_creditagain. Is that intented?
Yes indeed it is.
Quoted 1 line
I think the third hunk should look like following:
[snip]
Quoted 7 lines
As an additional idea I think
interactive_creditshould be allowed to be a bit bigger thanMAX_SLEEP_AVGand a bit lower than -MAX_SLEEP_AVG. This would makeLOW_CREDITprocesses stayLOW_CREDITeven if they do some sleep andHIGH_CREDITprocesses starHIGH_CREDITeven if they do some computing...But of course I may completely miss something...
Originally flagged HIGH_CREDIT and LOW_CREDIT tasks had no limit on how much credit they could accumulate or lose. This worked in practice but I needed some cutoff beyond which they would be one or the other and MSA seemed a decent number. At some stage if it was boundless there would be variable overflow as well and checking for this was unecessary overhead. I dont want tasks flagged as high credit to be forgotten though so thats why I dont let them change. Originally the same idea was for low credit tasks but that proved to occasionally choose X during startup as low credit. Higher numbers? perhaps but really this depends on hardware as to what is perfect.
Cheers,
Con
P.S. All this may be moot as it looks like I, or someone else, may have to start again.
We have a problem. See the this analysis from Steve Pratt.
Quoted 36 lines · Steven Pratt
Quoted 5 lines · Mark Peloquin
Been awhile since results where posted, therefore this is a little long.
Nightly Regression Summary for 2.6.0-test3 vs 2.6.0-test3-mm3
Quoted 8 lines
Benchmark Pass/Fail Improvements Regressions
Results Results Summary
--------------- --------- ------------ -----------
----------- ----------- -------
dbench.ext2 P N N 2.6.0-test3
2.6.0-test3-mm3 report
dbench.ext3 P N Y 2.6.0-test3
2.6.0-test3-mm3 reportThe ext3 dbench regression is very significant for multi threaded 193 ->
118. Looks like this regression first showed up in mm1 and does not exist in any of the bk trees.http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.ext3.throughput.plot.16.png
Quoted 3 lines
volanomark P N Y 2.6.0-test3
2.6.0-test3-mm3 reportVolanomark is significant as well. 10% drop in mm tree. This one also appeared to show up in mm1 although it was a 14% drop then so mm3 actually looks a little better. There were build errors on mm2 run so I don't have that data at this time.
Following link illustrates the drop in mm tree for volanomark.http://ltcperf.ncsa.uiuc.edu/data/history-graphs/volanomark.throughput.plot.1.png
Quoted 13 lines
SpecJBB2000 for high warehouses also took a bit hit. Probably the same root cause as volanomark.
Here is the history plot for the 19 warehouse run.http://ltcperf.ncsa.uiuc.edu/data/history-graphs/specjbb.results.avg.plot.19.png
Huge spike in idle time.
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/specjbb.utilization.idle.avg.plot.19.png
Those graphs are woeful.
Steve has done some preliminary testing which indicates that the volanomark and specjbb regressions are due to the CPU scheduler changes.
I have verifed that the ext3 regression is mostly due to setting PF_SYNCWRITE on kjournald. I/O scheduler stuff. I don't know why, but that patch obviously bites the dust. There is still a 10-15% regression on dbench 16 on my 4x Xeon which is due to the CPU scheduler patches.
It's good that the reaim regression mostly went away, but it would be nice to know why. When I was looking into the reaim problem it appeared that setting TIMESLICE_GRANULARITY to MAX_TIMESLICE made no difference, but more careful testing is needed on this.
There really is no point in proceeding with this fine tuning activity when we have these large and not understood regressions floating about.
I suggest that what we need to do is to await some more complete testing of the CPU scheduler patch alone from Steve and co. If it is fully confirmed that the CPU scheduler changes are the culprit we need to either fix it or go back to square one and start again with more careful testing and a less ambitious set of changes.
It could be that we're looking at some sort of tradeoff here, and we're already too far over to one side. I don't know.
It might help if you or a buddy could get set up with volanomark on an OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
Quoted 41 lines · Andrew Morton
We have a problem. See the this analysis from Steve Pratt.
Quoted 36 lines · Steven Pratt
Quoted 11 lines · Mark Peloquin
Been awhile since results where posted, therefore this is a little long.
Nightly Regression Summary for 2.6.0-test3 vs 2.6.0-test3-mm3
Benchmark Pass/Fail Improvements Regressions Results Results Summary --------------- --------- ------------ ----------- ----------- ----------- ------- dbench.ext2 P N N 2.6.0-test3 2.6.0-test3-mm3 report dbench.ext3 P N Y 2.6.0-test3 2.6.0-test3-mm3 reportThe ext3 dbench regression is very significant for multi threaded 193 -> 118. Looks like this regression first showed up in mm1 and does not exist in any of the bk trees.
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/dbench.ext3.throughput.plot.16.png
Quoted 2 lines
volanomark P N Y 2.6.0-test3 2.6.0-test3-mm3 report
Volanomark is significant as well. 10% drop in mm tree. This one also appeared to show up in mm1 although it was a 14% drop then so mm3 actually looks a little better. There were build errors on mm2 run so I don't have that data at this time. Following link illustrates the drop in mm tree for volanomark.
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/volanomark.throughput.plot.1.png
SpecJBB2000 for high warehouses also took a bit hit. Probably the same root cause as volanomark. Here is the history plot for the 19 warehouse run.
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/specjbb.results.avg.plot.19.png
Huge spike in idle time.
http://ltcperf.ncsa.uiuc.edu/data/history-graphs/specjbb.utilization.idle.avg.plot.19.png
Quoted 6 lines
Quoted 3 lines
Those graphs are woeful.
Aren't they.
Quoted 9 lines
Steve has done some preliminary testing which indicates that the volanomark and specjbb regressions are due to the CPU scheduler changes.
I have verifed that the ext3 regression is mostly due to setting
PF_SYNCWRITEon kjournald. I/O scheduler stuff. I don't know why, but that patch obviously bites the dust. There is still a 10-15% regression on dbench 16 on my 4x Xeon which is due to the CPU scheduler patches.
Thats fine. I never measured any improvement with it. Its sad that that it didn't go as I hoped, but that probably tells you I don't know enough about how journalling works.
Quoted 9 lines
It's good that the reaim regression mostly went away, but it would be nice to know why. When I was looking into the reaim problem it appeared that setting
TIMESLICE_GRANULARITYtoMAX_TIMESLICEmade no difference, but more careful testing is needed on this.There really is no point in proceeding with this fine tuning activity when we have these large and not understood regressions floating about.
I think changes in the CPU scheduler cause butterflies to flap their wings or what have you. Good luck pinning it down.
Quoted 7 lines
I suggest that what we need to do is to await some more complete testing of the CPU scheduler patch alone from Steve and co. If it is fully confirmed that the CPU scheduler changes are the culprit we need to either fix it or go back to square one and start again with more careful testing and a less ambitious set of changes.
Quoted 7 lines
It could be that we're looking at some sort of tradeoff here, and we're already too far over to one side. I don't know.
It might help if you or a buddy could get set up with volanomark on an OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
I think you'd be wasting your time until the interactivity side of things is working better. Unless Con has a smaller set of undisputed improvements to test with.
Quoted 9 lines · Nick Piggin
Quoted 8 lines · Andrew Morton
We have a problem. See the this analysis from Steve Pratt.
Steve has done some preliminary testing which indicates that the volanomark and specjbb regressions are due to the CPU scheduler changes.
It might help if you or a buddy could get set up with volanomark on an OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
Ok here goes.
This is on 8way:
Test4:
Average throughput = 11145 messages per second
Test4-O18.1:
Average throughput = 9860 messages per second
Test3-mm3:
Average throughput = 9788 messages per second
So I grabbed test3-mm3 and started peeling back the patches
and found no change in throughput without _any_ of my Oxint patches applied, and just Ingo's A3 patch:
Test3-mm3-A3
Average throughput = 9889 messages per second
Then finally I removed that patch so there were no interactivity patches:
Test3-mm3-ni
Average throughput = 11052 messages per second
I performed each run 3 times and have the results and profiles posted here:
http://kernel.kolivas.org/2.5/volano
wli suggested inlining sched_clock from A3 to see if that helped but at 3am I think it can wait. At least I've been able to track down the drop. Thanks zwane for the iron access.
Con
Quoted 22 lines · Con Kolivas
Quoted 4 lines
Quoted 4 lines
It might help if you or a buddy could get set up with volanomark on an OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
Ok here goes.
This is on 8way:Test4:
Average throughput = 11145 messages per secondTest4-O18.1:
Average throughput = 9860 messages per secondTest3-mm3:
Average throughput = 9788 messages per secondSo I grabbed test3-mm3 and started peeling back the patches
and found no change in throughput without _any_ of my Oxint patches applied, and just Ingo's A3 patch:
Quoted 7 lines
Test3-mm3-A3
Average throughput = 9889 messages per secondThen finally I removed that patch so there were no interactivity patches:
Test3-mm3-ni
Average throughput = 11052 messages per second
Well that was quick, thanks.
Surely the only reason we see more idle time in this sort of workload is because of runqueue imbalance: some CPUs are idle while other CPUs have more than one runnable process. That sounds like a bug more than a tuning/balancing thing: having no runnable tasks is a sort of binary do-something-right-now case.
We should be going across and pullng a task off another CPU synchronously as soon as a runqueue is seen to be empty. The code tries to do that so hrm.
Ingo just sent the below patch which is related, but doesn't look like it will fix it. I'll include this in test4-mm1, RSN.
From: Ingo Molnar <[email protected]>
the attached patch fixes the SMP balancing problem reported by David Mosberger. (the 'yield-ing threads do not get spread out properly' bug).
it turns out that we never really spread out tasks in the busy-rebalance case - contrary to my intention. The most likely incarnation of this balancing bug is via yield() - but in theory pipe users can be affected too.
the patch balances more agressively with a slow frequency, on SMP (or within the same node on NUMA - not between nodes on NUMA).
kernel/sched.c | 3 +-- 1 files changed, 1 insertion(+), 2 deletions(-)
kernel/sched.c renamed from kernel/sched.c~sched-balance-fix-2.6.0-test3-mm3-A0 +1 -2
@@ -1144,7 +1144,6 @@ static void rebalance_tick(runqueue_t *tload_balance (this_rq, idle,cpu_to_node_mask (this_cpu));spin_unlock (&this_rq->lock); } -return ; }#ifdef CONFIG_NUMA if (!(j %BUSY_NODE_REBALANCE_TICK ))
@@ -1152,7 +1151,7 @@ static void rebalance_tick(runqueue_t *t#endif if (!(j %BUSY_REBALANCE_TICK )) {spin_lock (&this_rq->lock); -load_balance (this_rq, idle,cpu_to_node_mask (this_cpu)); +load_balance (this_rq,0 ,cpu_to_node_mask (this_cpu));spin_unlock (&this_rq->lock); } }
_
Quoted 39 lines · Andrew Morton
Quoted 22 lines · Con Kolivas
Quoted 4 lines
Quoted 4 lines
It might help if you or a buddy could get set up with volanomark on an
OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
Ok here goes.
This is on 8way:Test4:
Average throughput = 11145 messages per secondTest4-O18.1:
Average throughput = 9860 messages per secondTest3-mm3:
Average throughput = 9788 messages per secondSo I grabbed test3-mm3 and started peeling back the patches and found no change in throughput without _any_ of my Oxint patches
applied, and just Ingo's A3 patch:Quoted 7 lines
Test3-mm3-A3
Average throughput = 9889 messages per secondThen finally I removed that patch so there were no interactivity
patches: Test3-mm3-ni
Average throughput = 11052 messages per secondWell that was quick, thanks.
Surely the only reason we see more idle time in this sort of workload is because of runqueue imbalance: some CPUs are idle while other CPUs have more than one runnable process. That sounds like a bug more than a tuning/balancing thing: having no runnable tasks is a sort of binary do-something-right-now case.
Quoted 6 lines
We should be going across and pullng a task off another CPU synchronously as soon as a runqueue is seen to be empty. The code tries to do that so hrm.
Ingo just sent the below patch which is related, but doesn't look like it will fix it. I'll include this in test4-mm1, RSN.
Just for the record I also tried inlining sched_clock and removing the rdtsc call entirely (and just getting the value from jiffies) and it made no measurable impact on performance.
Con
Quoted 3 lines · Andrew Morton
It might help if you or a buddy could get set up with volanomark on an OSDL 4-or-8-way so that you can more closely track the effect of your changes on such benchmarks.
Underway.
Con
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
Signature · 2 lines
Måns Rullgård [email protected]
Måns Rullgård, Mon, Aug 25, 2003 11:24:01 +0200:
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 9 lines · Alex Riesen
Måns Rullgård, Mon, Aug 25, 2003 11:24:01 +0200:
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
Con
Con Kolivas, Mon, Aug 25, 2003 12:16:13 +0200:
Quoted 7 lines · Alex Riesen
Quoted 6 lines
Måns Rullgård, Mon, Aug 25, 2003 11:24:01 +0200:
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
Quoted 7 lines
Quoted 3 lines
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
cannot test atm. Will do in 10hours.
RXVT behaved sanely (or probably spin-effect is very rare) in 2.4 (with O(1) alone and your 2.4 patches) and plain 2.6-test1.
Alex Riesen, Mon, Aug 25, 2003 12:21:33 +0200:
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 7 lines
Quoted 2 lines
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
cannot test atm. Will do in 10hours.
RXVT behaved sanely (or probably spin-effect is very rare) in 2.4 (with O(1) alone and your 2.4 patches) and plain 2.6-test1.
Sorry, I have to postpone this investigation. No time on the machine.
I try to describe the behaviour of rxvt as best as I can below.
Afaics, the application (rxvt) just sleeps at the beginning waiting for input from X. As every terminal would do. At some point its inferior process finishes, but it fails to notice this spinning madly in the internal loop calling select, which returns immediately (because other side of pty was closed. That is the error in rxvt). Probably it has accumulated enough "priority" up to this moment to block other applications (window manager, for example) when it suddenly starts running?
-alex
Quoted 22 lines · Alex Riesen
Alex Riesen, Mon, Aug 25, 2003 12:21:33 +0200:
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 7 lines
Quoted 3 lines
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
cannot test atm. Will do in 10hours.
RXVT behaved sanely (or probably spin-effect is very rare) in 2.4 (with O(1) alone and your 2.4 patches) and plain 2.6-test1.Sorry, I have to postpone this investigation. No time on the machine.
I try to describe the behaviour of rxvt as best as I can below.
Quoted 7 lines
Afaics, the application (rxvt) just sleeps at the beginning waiting for input from X. As every terminal would do. At some point its inferior process finishes, but it fails to notice this spinning madly in the internal loop calling select, which returns immediately (because other side of pty was closed. That is the error in rxvt). Probably it has accumulated enough "priority" up to this moment to block other applications (window manager, for example) when it suddenly starts running?
Something like that. Interesting you point out select as wli was profiling/tracing the mozilla/acroread plugin combination that spins on wait and also found select was causing grief. It was calling select with a 15ms timeout and X was getting less than 5ms to do it's work and respond and it was repeatedly timing out. Seems a common link there.
Con
Con Kolivas, Tue, Aug 26, 2003 00:48:23 +0200:
Quoted 8 lines
Quoted 7 lines
Afaics, the application (rxvt) just sleeps at the beginning waiting for input from X. As every terminal would do. At some point its inferior process finishes, but it fails to notice this spinning madly in the internal loop calling select, which returns immediately (because other side of pty was closed. That is the error in rxvt). Probably it has accumulated enough "priority" up to this moment to block other applications (window manager, for example) when it suddenly starts running?
Quoted 6 lines
Something like that. Interesting you point out select as wli was profiling/tracing the mozilla/acroread plugin combination that spins on wait and also found select was causing grief. It was calling select with a 15ms timeout and X was getting less than 5ms to do it's work and respond and it was repeatedly timing out. Seems a common link there.
Yes, looks similar. Probably a simplier test could be to let the program loop using select on stdin with zero timeout (it is a pty usually :)
Alex Riesen, Mon, Aug 25, 2003 12:21:33 +0200:
Quoted 12 lines
Con Kolivas, Mon, Aug 25, 2003 12:16:13 +0200:
Quoted 11 lines · Alex Riesen
Quoted 9 lines
M?ns Rullg?rd, Mon, Aug 25, 2003 11:24:01 +0200:
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 7 lines
Quoted 3 lines
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
cannot test atm. Will do in 10hours.
RXVT behaved sanely (or probably spin-effect is very rare) in 2.4 (with O(1) alone and your 2.4 patches) and plain 2.6-test1.
run on RXVT on plain 2.4-test4. No freezes, dynamic priority started at 23, dropped to 15 after the window lost focus and stayed at it.
As there were no execution bursts this time, vmstat did not show anything interesting. At least I did not see anything.
-alex
Quoted 11 lines · Con Kolivas
Quoted 8 lines
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 10 lines · Måns Rullgård
Quoted 9 lines · Con Kolivas
Quoted 8 lines
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 8 lines
Quoted 3 lines
What does vanilla kernel do with these apps running? Both immediately after the apps have started up and some time (>1 min) after they've been running?
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Con
Quoted 7 lines · Con Kolivas
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 10 lines · Måns Rullgård
Quoted 7 lines · Con Kolivas
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
No, the 16.2 patch is bad. 16.3 backed it out.
Con
Quoted 11 lines · Con Kolivas
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
No, the 16.2 patch is bad. 16.3 backed it out.
OK, but it somehow made XEmacs behave badly.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 6 lines · Måns Rullgård
Quoted 5 lines · Con Kolivas
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
No, the 16.2 patch is bad. 16.3 backed it out.
OK, but it somehow made XEmacs behave badly.
Well it was a generic fix in 16.2 that helped XEmacs as I said. O15 also had a generic fix (child not preempting it's parent) but that too was covering up the real issue, but it wasnt as drastic as 16.2.
Con
Quoted 10 lines · Con Kolivas
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
No, the 16.2 patch is bad. 16.3 backed it out.
OK, but it somehow made XEmacs behave badly.
Well it was a generic fix in 16.2 that helped XEmacs as I said. O15 also had a generic fix (child not preempting it's parent) but that too was covering up the real issue, but it wasnt as drastic as 16.2.
Of the kernels I've tested, only test1 vanilla and O16.3 and later show the problem. Btw, is it related to the XEmacs regexp search problem, or is that a different one?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 15 lines · Måns Rullgård
Quoted 14 lines · Con Kolivas
Quoted 13 lines
Quoted 11 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Vanilla test1 has the spin effect. Test2 doesn't. I haven't tried vanilla test3 or test4. As I've said, the O16.2-O16.3 patch introduced the problem. With that patch reversed, everything is fine. What problem does that patch fix?
It's a generic fix for priority inversion but it induces badness in smp, and latency in task preemption on up so it's not suitable.
Now I'm confused. If that patch is bad, then why is it in O18?
No, the 16.2 patch is bad. 16.3 backed it out.
OK, but it somehow made XEmacs behave badly.
Quoted 7 lines
Quoted 3 lines
Well it was a generic fix in 16.2 that helped XEmacs as I said. O15 also had a generic fix (child not preempting it's parent) but that too was covering up the real issue, but it wasnt as drastic as 16.2.
Of the kernels I've tested, only test1 vanilla and O16.3 and later show the problem. Btw, is it related to the XEmacs regexp search problem, or is that a different one?
Same thing. Both examples of priority inversion. In your case a parent child interaction which was worked around in O15. That interaction seems the more common case.
Con
Quoted 8 lines · Alex Riesen
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
It sounds like the same bug. IMHO, it's rather bad, since a non-privileged process can make the system unusable for a non-zero amount of time.
How should I do to capture some information about this thing? Do you know what causes it, Con?
Signature · 2 lines
Måns Rullgård [email protected]
Måns Rullgård, Mon, Aug 25, 2003 12:17:16 +0200:
Quoted 7 lines
Quoted 5 lines · Alex Riesen
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
Quoted 7 lines
Quoted 3 lines
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
It sounds like the same bug. IMHO, it's rather bad, since a non-privileged process can make the system unusable for a non-zero amount of time.
the source of RXVT looks more like the bug: it does not check for errors, even though it is a bit tricky to get portably.
It is still a problem, though: "_almost_ locked" does not make it nice.
Quoted 1 line
How should I do to capture some information about this thing?
Use "top" and look at the dynamic priority.
Quoted 9 lines · Alex Riesen
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 6 lines
Quoted 3 lines
It sounds like the same bug. IMHO, it's rather bad, since a non-privileged process can make the system unusable for a non-zero amount of time.
the source of RXVT looks more like the bug: it does not check for errors, even though it is a bit tricky to get portably.
A program should never be able to grab more than it's share of the CPU time, no matter how buggy it is.
Quoted 3 lines
Quoted 1 line
How should I do to capture some information about this thing?
Use "top" and look at the dynamic priority.
I'll try, but it could be tricky, since it doesn't usually last very long.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 10 lines · Måns Rullgård
Quoted 8 lines · Alex Riesen
Quoted 4 lines
XEmacs still spins after running a background job like make or grep. It's fine if I reverse patch-O16.2-O16.3. The spinning doesn't happen as often, or as long time as with O16.3, but it's there and it's irritating.
another example is RXVT (an X terminal emulator). Starts spinnig after it's child has exited. Not always, but annoyingly often. System is almost locked while it spins (calling select).
Quoted 6 lines
It sounds like the same bug. IMHO, it's rather bad, since a non-privileged process can make the system unusable for a non-zero amount of time.
How should I do to capture some information about this thing? Do you know what causes it, Con?
Read my rfc on the orthogonal interactivity patches and look under priority inversion. It may be that, in which case it happens to vanilla as well. To capture useful information for me is quite easy. Run a reniced -11 top -d 1 -b in batch mode and reniced -11 vmstat 1 in batch mode to capture it happening. That should be enough information to see what's going on and doesn't need a kernel compile or any special tools. Renice -11 to make sure the tools dont get preempted in that period.
Con
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Alex Riesen, Mon, Aug 25, 2003 12:29:33 +0200:
Quoted 6 lines
Nick Piggin, Mon, Aug 25, 2003 12:27:14 +0200:
Quoted 2 lines
If you have some spare time perhaps you could test my scheduler patch.
i'll try to. Can't promise to have it today, though.
tried 7a. What I noticed first, is that the problem with rxvt eating up all cpu time is gone :) Also applications get less priorities (11-16). Can't say everything is very smooth, but somehow it makes very good impression. No really rough edges, but I have to admit I tried only pure cpu load (bash -c 'while :; do :; done').
Applications feel to start faster (subjective).
X was/is not niced.
Made the kernel to default boot for now, will see how it behaves.
-alex
Quoted 15 lines · Alex Riesen
Alex Riesen, Mon, Aug 25, 2003 12:29:33 +0200:
Quoted 11 lines
Nick Piggin, Mon, Aug 25, 2003 12:27:14 +0200:
Quoted 4 lines
If you have some spare time perhaps you could test my scheduler patch.
i'll try to. Can't promise to have it today, though.
Quoted 9 lines
tried 7a. What I noticed first, is that the problem with rxvt eating up all cpu time is gone :) Also applications get less priorities (11-16). Can't say everything is very smooth, but somehow it makes very good impression. No really rough edges, but I have to admit I tried only pure cpu load (bash -c 'while :; do :; done').
Applications feel to start faster (subjective).
X was/is not niced.
Thanks.. try renicing X to -10 or even -20.