RSDL-mm 0.28
I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the sched_yield syscall return immediately. Xorg and all tests are run at nice 0.
Loads:
memload: constant memcpy of 16MB buffer
execload: constant re-exec of a trivial shell script
forkload: constant fork and exit of a trivial shell script make -j 5: hot-cache kernel build without ccache make -j 5 ccache: hot-cache kernel build with ccache
Tests:
beryl - 3D window manager, wiggle windows, spin desktop, etc. galeon - web browser, rapidly scrolling long web pages by grabbing
the scroll bar
mp3 - XMMS on a FUSE sshfs over wireless (during all tests) terminal - responsiveness of ssh and local terminal sessions mouse - responsiveness of mouse pointer
Results:
great = completely smooth
good = fully responsive
ok = visible latency
bad = becomes difficult to use (or mp3 skips)
awful = make it stop, please
-mm2 -mm2+noyield rsdl+noyield
no load
beryl great great great galeon good good good mp3 good good good terminal good good good mouse good good good memload x10 beryl awful/bad great good galeon good good ok/good mp3 good good good terminal good good good mouse good good good execload x10 beryl awful/bad bad/good good galeon good bad/good ok/good mp3 good bad good terminal good bad/good good mouse good bad/good good forkload x10 beryl good good great galeon good good ok/good mp3 good good good terminal good good ok/good mouse good good good make -j 5 beryl ok good good/great galeon good good ok/good mp3 good good good terminal good good good mouse good good good make -j 5 ccache beryl ok good awful galeon good good bad mp3 good good bad terminal good good bad/ok mouse good good bad/ok
make -j 5
real 8m1.857s 8m50.659s 8m9.282s user 7m19.127s 8m3.494s 7m30.740s sys 0m30.910s 0m33.722s 0m29.542s
make -j 5 ccache
real 2m6.182s 2m19.032s 2m1.832s user 1m39.466s 1m48.787s 1m37.250s sys 0m19.741s 0m22.993s 0m20.109s
There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current sched_yield, you go all the way to the back of the line.
RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.
Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 65 lines · Matt Mackall
I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the
sched_yieldsyscall return immediately. Xorg and all tests are run at nice 0.Loads:
memload: constant memcpy of 16MB buffer
execload: constant re-exec of a trivial shell script
forkload: constant fork and exit of a trivial shell script make -j 5: hot-cache kernel build without ccache make -j 5 ccache: hot-cache kernel build with ccacheTests:
beryl - 3D window manager, wiggle windows, spin desktop, etc. galeon - web browser, rapidly scrolling long web pages by grabbing
the scroll bar
mp3 - XMMS on a FUSE sshfs over wireless (during all tests) terminal - responsiveness of ssh and local terminal sessions mouse - responsiveness of mouse pointerResults:
great = completely smooth
good = fully responsive
ok = visible latency
bad = becomes difficult to use (or mp3 skips)
awful = make it stop, please-mm2 -mm2+noyield rsdl+noyield
no loadberyl great great great galeon good good good mp3 good good good terminal good good good mouse good good good memload x10 beryl awful/bad great good galeon good good ok/good mp3 good good good terminal good good good mouse good good good execload x10 beryl awful/bad bad/good good galeon good bad/good ok/good mp3 good bad good terminal good bad/good good mouse good bad/good good forkload x10 beryl good good great galeon good good ok/good mp3 good good good terminal good good ok/good mouse good good good make -j 5 beryl ok good good/great galeon good good ok/good mp3 good good good terminal good good good mouse good good good make -j 5 ccache beryl ok good awful galeon good good bad mp3 good good bad terminal good good bad/ok mouse good good bad/okmake -j 5
Quoted 8 lines
real 8m1.857s 8m50.659s 8m9.282s user 7m19.127s 8m3.494s 7m30.740s sys 0m30.910s 0m33.722s 0m29.542smake -j 5 ccache
real 2m6.182s 2m19.032s 2m1.832s user 1m39.466s 1m48.787s 1m37.250s sys 0m19.741s 0m22.993s 0m20.109s
Thanks very much for that comprehensive summary and testing!
Quoted 7 lines
There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current
sched_yield, you go all the way to the back of the line.
Well... are you advocating we change sched_yield semantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding using sched_yield instead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).
Quoted 4 lines
RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.
Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time. This is "worked around" in mainline thanks to the testing for sleeping on uninterruptible sleep in the interactivity estimator. What I suspect is happening is kernel threads that are running nice -5 are doing all the work on make's behalf in the setting of ccache since it is mostly i/o bound. The reason for -nice values on kernel threads is questionable anyway. Can you try renicing your kernel threads all to nice 0 and see what effect that has? Obviously this doesn't need a recompile, but is simple enough to implement in kthread code as a new default.
Quoted 3 lines
Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.
Ew. It's probably not a bug but a good example of some of the starvation scenarios we're hitting on mainline (hence the need for a rewrite ;))
Thanks!
---
ck
Quoted 5 lines · On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]> wrote:
Well... are you advocating we change
sched_yieldsemantics to a gentler form?From a practical POV: our present
yield()behaviour is so truly awful that
it's basically always a bug to use it. This probably isn't a good thing.
So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expect yield() to do.
otoh,
a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not use yield(), so perhaps there's little to be gained in changing it now.
b) if we _were_ to change yield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.
Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.
Quoted 18 lines · Andrew Morton
Quoted 5 lines
On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]>
wrote: Well... are you advocating we changesched_yieldsemantics to a gentler form?From a practical POV: our present
yield()behaviour is so truly awful thatit's basically always a bug to use it. This probably isn't a good thing.
So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expect
yield()to do.otoh,
a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not use
yield(), so perhaps there's little to be gained in changing it now.
Quoted 8 lines
b) if we _were_ to change
yield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.Bottom line: we've had a _lot_ of problems with the new
yield()semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.
So... I would take it that's a yes for a recommendation with respect to implementing a new yield() ? A new scheduler is as good a time as any to do it.
Signature · 1 line
-ck
Quoted 8 lines · On Sun, 11 Mar 2007 14:59:28 +1100 Con Kolivas <[email protected]> wrote:
Quoted 3 lines
Bottom line: we've had a _lot_ of problems with the new
yield()semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.So... I would take it that's a yes for a recommendation with respect to implementing a new
yield()? A new scheduler is as good a time as any to do it.
I guess so. We'd, err, need to gather Ingo's input ;)
Perhaps a suitable way of doing this would be to characterise then emulate the 2.4 behaviour. As long as it turns out to be vaguely sensible.
Quoted 2 lines · On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]> wrote:
Quoted 2 lines
Well... are you advocating we change
sched_yieldsemantics to a gentler form?
Quoted 5 lines · Andrew Morton
From a practical POV: our present
yield()behaviour is so truly awful that it's basically always a bug to use it. This probably isn't a good thing. So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expectyield()to do.
ISTR that apps varied wrt. their expectations for yield(). Some, particularly those using it to implement multi-tiered userspace locks, really did expect to go all the way to the back of the queue. (Rumor has it that realtime apps break otherwise.) Others wanted a kinder, gentler, "mistress, please hit me, but not too hard" opportunity to let someone else have a little cpu time, particularly when userspace is spinning in some sort of busywait. In both scenarios something very much against the latest trends of Linux kernel politics is done by userspace.
Quoted 10 lines · Andrew Morton
otoh,
a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not useyield(), so perhaps there's little to be gained in changing it now.
b) if we _were_ to changeyield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.
Bottom line: we've had a _lot_ of problems with the newyield()semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.
My dumb idea would be to break out new syscall. One for the kinder, gentler version, one for the serious version. Or otherwise pass an argument indicating the expected behavior. Then a dumb app can be LD_PRELOAD'd into calling whatever makes it run fastest.
-- wli
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Sun, 11 Mar 2007 14:59:28 +1100 Con Kolivas <[email protected]> wrote:
Quoted 4 lines
Bottom line: we've had a _lot_ of problems with the new
yield()semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.So... I would take it that's a yes for a recommendation with respect to implementing a new
yield()? A new scheduler is as good a time as any to do it.I guess so. We'd, err, need to gather Ingo's input ;)
cc'ed. Don't you hate timezones?
Quoted 2 lines
Perhaps a suitable way of doing this would be to characterise then emulate the 2.4 behaviour. As long as it turns out to be vaguely sensible.
It's really very simple. We just go the end of the current queued priority on the same array instead of swapping to the expired array; ie we do what realtime tasks currently do. It works fine here locally afaict.
Signature · 1 line
-ck
Quoted 6 lines · Con Kolivas
Quoted 6 lines
make -j 5 ccache
beryl ok good awful galeon good good bad mp3 good good bad terminal good good bad/ok mouse good good bad/ok
...
Quoted 11 lines
Quoted 4 lines
RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.
Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.
I don't see significant system time while this is happening.
Quoted 9 lines
This is "worked around" in mainline thanks to the testing for sleeping on uninterruptible sleep in the interactivity estimator. What I suspect is happening is kernel threads that are running nice -5 are doing all the work on make's behalf in the setting of ccache since it is mostly i/o bound. The reason for -nice values on kernel threads is questionable anyway. Can you try renicing your kernel threads all to nice 0 and see what effect that has? Obviously this doesn't need a recompile, but is simple enough to implement in kthread code as a new default.
Sorry, little to no benefit.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 9 lines · Matt Mackall
Quoted 6 lines · Con Kolivas
Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.
I don't see significant system time while this is happening.
Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 9 lines · Matt Mackall
Quoted 8 lines · Matt Mackall
Quoted 6 lines · Con Kolivas
Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.
Quoted 4 lines
Quoted 1 line
I don't see significant system time while this is happening.
Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.
Well I can't reproduce that behaviour here at all whether from disk or the pagecache with ccache, so I'm not entirely sure what's different at your end. However both you and the other person reporting bad behaviour were using ATI drivers. That's about the only commonality? I wonder if they do need to yield... somewhat instead of not at all.
Signature · 1 line
-ck
Quoted 3 lines · Matt Mackall
I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the
sched_yieldsyscall return immediately. Xorg and all tests are run at nice 0.
[skipped long and precise test report]
Quoted 3 lines
Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.
This is not a bug per se, but more a design problem. This is caused by the interactivity booster which is unfair. Mike Galbraith and others spent a lot of time trying to get rid of those problems a few versions ago. In early kernels (around 2.6.11), I could trivially cause pauses more than 30 seconds long by running a few tasks simulating an interactive workload. It is much more difficult to achieve this with recent kernels, and it has absolutely no effect on RSDL, which is one of the reasons I have to find it great !
Regards,
Willy
Re: RSDL for 2.6.21-rc3- 0.29
In reply to Con KolivasQuoted 12 lines · Con Kolivas
Quoted 11 lines · Matt Mackall
Quoted 9 lines · Matt Mackall
Quoted 6 lines · Con Kolivas
Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.
I don't see significant system time while this is happening.
Quoted 8 lines
Quoted 2 lines
Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.
Well I can't reproduce that behaviour here at all whether from disk or the pagecache with ccache, so I'm not entirely sure what's different at your end. However both you and the other person reporting bad behaviour were using ATI drivers. That's about the only commonality? I wonder if they do need to yield... somewhat instead of not at all.
I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.
I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.
What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.
Signature · 7 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) While it may be true that a watched pot never boils, the one you don't keep an eye on can make an awful mess of your stove. -- Edward Stevenson
Re: RSDL for 2.6.21-rc3- 0.29
Hi Gene.
Quoted 11 lines · Gene Heskett
I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.
I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.
What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.
I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(
Signature · 1 line
-ck
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 16 lines · Con Kolivas
Hi Gene.
Quoted 8 lines · Gene Heskett
I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.
I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.
Quoted 4 lines
What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.
Quoted 5 lines
I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(
I have another backup running right now, after building a plain 2.6.21-rc3, and rebooting just now for the test. I don't think its the scheduler itself, but is something post 2.6.20 that is messing with tars mind and making it think the files it just read to do the estimate phase, are all new, so even a level 2 is in effect a level 0. I'll have an answer in about an hour, but its also 2:36am here and I'm headed for the rack to get some zzz's. So I'll report in the morning as to whether or not this backup ran as it was supposed to. I have a feeling its not going to though.
Signature · 6 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) "When it comes to humility, I'm the greatest." -- Bullwinkle Moose
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 23 lines · Gene Heskett
Quoted 16 lines · Con Kolivas
Hi Gene.
Quoted 8 lines · Gene Heskett
I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.
I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.
Quoted 4 lines
What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.
Quoted 5 lines
I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(
Quoted 9 lines
I have another backup running right now, after building a plain 2.6.21-rc3, and rebooting just now for the test. I don't think its the scheduler itself, but is something post 2.6.20 that is messing with tars mind and making it think the files it just read to do the estimate phase, are all new, so even a level 2 is in effect a level 0. I'll have an answer in about an hour, but its also 2:36am here and I'm headed for the rack to get some zzz's. So I'll report in the morning as to whether or not this backup ran as it was supposed to. I have a feeling its not going to though.
I can confirm that a plain 21-rc3 still suffers from this problem. This run of amanda terminated after 13 of the 32 dle's after writing just short of 12GB to the vtape. 8 were level 0's, 5 were level 1's, all were gzipped, achieving a compression ratio of 40% of original size. There is about 45GB of data here to backup, on a 5 day dumpcycle.
If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage into c-times when tar does its estimate scans where the output goes to /dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.
I love your patches Con, but I'll leave them out of this next testing. No use pointing fingers at good code.
Signature · 8 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Mal: "Well, you were right about this being a bad idea." Zoe: "Thanks for sayin', sir." --Episode #1, "Serenity"
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 11 lines · Gene Heskett
If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage into c-times when tar does its estimate scans where the output goes to
/dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.
You may find 'strace' useful to track down this sort of thing (though the output can be voluminous).
-Doug
Quoted 9 lines
Quoted 8 lines
There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current
sched_yield, you go all the way to the back of the line.
Quoted 5 lines
Well... are you advocating we change
sched_yieldsemantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding usingsched_yieldinstead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).
The basic point of sched_yield is to allow every other process at the same static priority level a chance to use the CPU before you get it back. It is generally an error to use sched_yield to be nice. It's nice to get your work done when the scheduler gives you the CPU, that's why it gave it to you.
It is proper to use sched_yield as an optimization when it more efficient to allow another process/thread to run than you, for example, when you encounter a task you cannot do efficiently at that time because another thread holds a lock.
It's also useful prior to doing something that can most efficiently be done without interruption. So a thread that returns from 'sched_yield' should ideally be given a full timeslice if possible. This may not be sensible if the 'sched_yield' didn't actuall yield, but then again, if nothing else wants to run, why not give the only task that does a full slice?
In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)
Note that processes that call sched_yield should be rewarded for doing so just as process that block on I/O are, assuming they do in fact wind up giving up the CPU when they would otherwise have had it.
DS
Quoted 2 lines · David Schwartz
In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)
Perhaps if sched_yield()'s effects were cumulative inside a timeslice, then eventually the calling task would get pushed far enough down that others would run.
Ray
Re: RSDL for 2.6.21-rc3- 0.29
In reply to Douglas McNaughtQuoted 14 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage
Quoted 8 lines
into c-times when tar does its estimate scans where the output goes to
/dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.
Quoted 4 lines
You may find 'strace' useful to track down this sort of thing (though the output can be voluminous).
-Doug
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
Signature · 5 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) I'm definitely not in Omaha!
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ff
-Doug
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 9 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ff-Doug
Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:
running /usr/local/sbin//amcheck Daily
Amanda Tape Server Host Check
-----------------------------
Holding disk /dumps: 72597 MB disk space available, using 72097 MB slot 5: read label Dailys-5', date 20070223002502'
NOTE: skipping tape-writable test
Tape Dailys-5 label ok
Server check took 4.556 seconds
Amanda Backup Client Hosts Check
--------------------------------
WARNING: coyote: selfcheck request failed: unable to bind to a reserved port (got port 32789)
Client check: 1 host checked in 34.733 seconds, 1 problem found
(brought to you by Amanda 2.5.1p3-20070222)
------------------eofsnip------------
The script quit by request shortly after as I didn't want it to further dirty the amanda database system.
Signature · 5 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Safety Third.
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ff
Quoted 8 lines
Quoted 1 line
-Doug
Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through
/etc/servicesaccompanied by endless complaints of-EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:
Can we start a new thread for this amanda issue? It does not seem to have anything to do with Con's scheduler...
Lee
Re: RSDL for 2.6.21-rc3- 0.29
In reply to Gene HeskettQuoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ff
Quoted 8 lines
Quoted 1 line
-Doug
Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through
/etc/servicesaccompanied by endless complaints of-EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:
I'd do 'strace -ff -o /tmp/amanda-strace <command>', which will give you a set of files in /tmp, one for each PID created by fork(). Then find the one that has the 'tar' invocation you're looking for.
-Doug
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 20 lines · Douglas McNaught
Quoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ffQuoted 8 lines
Quoted 1 line
-Doug
Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through
/etc/servicesaccompanied by endless complaints of-EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:
Quoted 3 lines
I'd do 'strace -ff -o
/tmp/amanda-strace<command>', which will give you a set of files in /tmp, one for each PID created byfork(). Then find the one that has the 'tar' invocation you're looking for.
Hi,
I hope you don't mind me jumping in ...
Why not temporarly replace "/bin/tar" with a shell script that does:
#!/bin/sh
exec strace -f -o output /bin/real.tar $@
That should be working, shouldn't it ?
Cheers,
Patrick
Re: RSDL for 2.6.21-rc3- 0.29
Quoted 31 lines · Patrick Mau
Quoted 20 lines · Douglas McNaught
Quoted 11 lines · Gene Heskett
Quoted 9 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett
I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.
$ strace -ff-Doug
Quoted 7 lines
Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through
/etc/servicesaccompanied by endless complaints of-EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:Quoted 3 lines
I'd do 'strace -ff -o
/tmp/amanda-strace<command>', which will give you a set of files in /tmp, one for each PID created byfork(). Then find the one that has the 'tar' invocation you're looking for.Hi,
I hope you don't mind me jumping in ...
Why not temporarly replace "
/bin/tar" with a shell script that does:
Quoted 7 lines
#!/bin/sh
exec strace -f -o output/bin/real.tar$@That should be working, shouldn't it ?
Cheers,
Patrick
It should I'd think, for those instances where these scripts are run by hand and therefore have a shell to output to. I'll give that a try when I get my script problem resolved, its a basic diff in the operation of real tapes and vtapes I need to get right for both scenarios.
Good Idea(TM), Thanks.
Signature · 6 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) "Ignorance is the soil in which belief in miracles grows." -- Robert G. Ingersoll
Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
In reply to Patrick MauQuoted 4 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
Quoted 6 lines · Douglas McNaught
Quoted 4 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@
Quoted 7 lines
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.
But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.
Signature · 6 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) rugged, adj.: Too heavy to lift.
Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
Quoted 15 lines · Gene Heskett
Quoted 6 lines · Douglas McNaught
Quoted 4 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@Quoted 7 lines
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
Quoted 7 lines
In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.
But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.
buried indeed:
"Special Parameters:
...
$ Expands to the process ID of the shell. In a () subshell, it
expands to the process ID of the current shell, not the sub‐
shell.
"Thanks,
Nish
Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@Quoted 8 lines
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.
Quoted 3 lines
But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.
Quoted 8 lines
buried indeed:
"Special Parameters:
...$ Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the current shell, not the sub‐ shell. "
Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.
Quoted 2 lines
Thanks,
Nish
Signature · 6 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) "Say yur prayers, yuh flea-pickin' varmint!" -- Yosemite Sam
Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
Quoted 38 lines · Gene Heskett
Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@Quoted 8 lines
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.
Quoted 3 lines
But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.
Quoted 8 lines
buried indeed:
"Special Parameters:
...$ Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the current shell, not the sub?$B!> shell. "
Quoted 8 lines
Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.
Yes there a risk of wrapping, but it is very small. You can add the command line arguments to the file name if you want, like this :
#!/bin/sh
exec strace -f -o "output.$$.${*//\//_}" /bin/real.tar $@
It will name the output file "output.<pid>.<args>", replacing slashes with underscores. This is very dirty but can help.
Cheers,
Willy
Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)
Quoted 48 lines · Willy Tarreau
Quoted 38 lines · Gene Heskett
Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau
Why not temporarly replace "
/bin/tar" with a shell script that does:#!/bin/sh
exec strace -f -o output/bin/real.tar$@Quoted 8 lines
You beat me to it. :) I've done that before; it's a great suggestion.
Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.
-Doug
In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.
Quoted 3 lines
But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.
Quoted 8 lines
buried indeed:
"Special Parameters:
...$ Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the current shell, not the sub?$B!> shell. "Quoted 8 lines
Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.
Quoted 9 lines
Yes there a risk of wrapping, but it is very small. You can add the command line arguments to the file name if you want, like this :
#!/bin/sh
exec strace -f -o "output.$$.${*//\//_}"/bin/real.tar$@It will name the output file "output.<pid>.<args>", replacing slashes with underscores. This is very dirty but can help.
Excellent Willy, thanks.
Quoted 2 lines
Cheers,
Willy
Signature · 7 lines
Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Whatever doesn't succeed in two months and a half in California will never succeed. -- Rev. Henry Durant, founder of the University of California
Quoted 48 lines · David Schwartz
Quoted 9 lines
Quoted 7 lines
There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current
sched_yield, you go all the way to the back of the line.Quoted 5 lines
Well... are you advocating we change
sched_yieldsemantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding usingsched_yieldinstead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).The basic point of
sched_yieldis to allow every other process at the same static priority level a chance to use the CPU before you get it back. It is generally an error to usesched_yieldto be nice. It's nice to get your work done when the scheduler gives you the CPU, that's why it gave it to you.It is proper to use
sched_yieldas an optimization when it more efficient to allow another process/thread to run than you, for example, when you encounter a task you cannot do efficiently at that time because another thread holds a lock.It's also useful prior to doing something that can most efficiently be done without interruption. So a thread that returns from '
sched_yield' should ideally be given a full timeslice if possible. This may not be sensible if the 'sched_yield' didn't actuall yield, but then again, if nothing else wants to run, why not give the only task that does a full slice?In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)
Note that processes that call
sched_yieldshould be rewarded for doing so just as process that block on I/O are, assuming they do in fact wind up giving up the CPU when they would otherwise have had it.DS
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com