Skip to contentSkip to conversation

RSDL-mm 0.28

DISCUSSION Matt Mackall 13 participants 32 messages Mar 11, 2007 – Mar 13, 2007

lore
Share & tools
lore
#0 lore raw

I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the sched_yield syscall return immediately. Xorg and all tests are run at nice 0.

Loads:
memload: constant memcpy of 16MB buffer
execload: constant re-exec of a trivial shell script
forkload: constant fork and exit of a trivial shell script make -j 5: hot-cache kernel build without ccache make -j 5 ccache: hot-cache kernel build with ccache

Tests:
beryl - 3D window manager, wiggle windows, spin desktop, etc. galeon - web browser, rapidly scrolling long web pages by grabbing
the scroll bar
mp3 - XMMS on a FUSE sshfs over wireless (during all tests) terminal - responsiveness of ssh and local terminal sessions mouse - responsiveness of mouse pointer

Results:
great = completely smooth
good = fully responsive
ok = visible latency
bad = becomes difficult to use (or mp3 skips)
awful = make it stop, please

-mm2 -mm2+noyield rsdl+noyield
no load

 beryl    great       great          great
 galeon   good        good           good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
memload x10
 beryl    awful/bad   great          good
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
execload x10
 beryl    awful/bad   bad/good       good
 galeon   good        bad/good       ok/good
 mp3      good        bad            good
 terminal good        bad/good       good
 mouse    good        bad/good       good
forkload x10
 beryl    good        good           great
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           ok/good
 mouse    good        good           good
make -j 5
 beryl    ok          good           good/great
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
make -j 5 ccache
 beryl    ok          good           awful
 galeon   good        good           bad
 mp3      good        good           bad
 terminal good        good           bad/ok
 mouse    good        good           bad/ok

make -j 5

real      8m1.857s    8m50.659s      8m9.282s
user      7m19.127s   8m3.494s       7m30.740s
sys       0m30.910s   0m33.722s      0m29.542s

make -j 5 ccache

real      2m6.182s    2m19.032s      2m1.832s
user      1m39.466s   1m48.787s      1m37.250s
sys       0m19.741s   0m22.993s      0m20.109s

There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current sched_yield, you go all the way to the back of the line.

RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.

Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#1 lore raw
Quoted 65 lines · Matt Mackall

I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the sched_yield syscall return immediately. Xorg and all tests are run at nice 0.

Loads:
memload: constant memcpy of 16MB buffer
execload: constant re-exec of a trivial shell script
forkload: constant fork and exit of a trivial shell script make -j 5: hot-cache kernel build without ccache make -j 5 ccache: hot-cache kernel build with ccache

Tests:
beryl - 3D window manager, wiggle windows, spin desktop, etc. galeon - web browser, rapidly scrolling long web pages by grabbing
the scroll bar
mp3 - XMMS on a FUSE sshfs over wireless (during all tests) terminal - responsiveness of ssh and local terminal sessions mouse - responsiveness of mouse pointer

Results:
great = completely smooth
good = fully responsive
ok = visible latency
bad = becomes difficult to use (or mp3 skips)
awful = make it stop, please

-mm2 -mm2+noyield rsdl+noyield
no load

 beryl    great       great          great
 galeon   good        good           good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
memload x10
 beryl    awful/bad   great          good
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
execload x10
 beryl    awful/bad   bad/good       good
 galeon   good        bad/good       ok/good
 mp3      good        bad            good
 terminal good        bad/good       good
 mouse    good        bad/good       good
forkload x10
 beryl    good        good           great
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           ok/good
 mouse    good        good           good
make -j 5
 beryl    ok          good           good/great
 galeon   good        good           ok/good
 mp3      good        good           good
 terminal good        good           good
 mouse    good        good           good
make -j 5 ccache
 beryl    ok          good           awful
 galeon   good        good           bad
 mp3      good        good           bad
 terminal good        good           bad/ok
 mouse    good        good           bad/ok

make -j 5

Quoted 8 lines
real      8m1.857s    8m50.659s      8m9.282s
user      7m19.127s   8m3.494s       7m30.740s
sys       0m30.910s   0m33.722s      0m29.542s

make -j 5 ccache

real      2m6.182s    2m19.032s      2m1.832s
user      1m39.466s   1m48.787s      1m37.250s
sys       0m19.741s   0m22.993s      0m20.109s

Thanks very much for that comprehensive summary and testing!

Quoted 7 lines

There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current sched_yield, you go all the way to the back of the line.

Well... are you advocating we change sched_yield semantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding using sched_yield instead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).

Quoted 4 lines

RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.

Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time. This is "worked around" in mainline thanks to the testing for sleeping on uninterruptible sleep in the interactivity estimator. What I suspect is happening is kernel threads that are running nice -5 are doing all the work on make's behalf in the setting of ccache since it is mostly i/o bound. The reason for -nice values on kernel threads is questionable anyway. Can you try renicing your kernel threads all to nice 0 and see what effect that has? Obviously this doesn't need a recompile, but is simple enough to implement in kthread code as a new default.

Quoted 3 lines

Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.

Ew. It's probably not a bug but a good example of some of the starvation scenarios we're hitting on mainline (hence the need for a rewrite ;))

Thanks!

---
ck

#2 lore raw
Quoted 5 lines · On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]> wrote:

Well... are you advocating we change sched_yield semantics to a gentler form?

From a practical POV: our present yield() behaviour is so truly awful that

it's basically always a bug to use it. This probably isn't a good thing.

So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expect yield() to do.

otoh,

a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not use yield(), so perhaps there's little to be gained in changing it now.

b) if we _were_ to change yield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.

Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.

#3 lore raw
Quoted 18 lines · Andrew Morton
Quoted 5 lines

On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]>
wrote: Well... are you advocating we change sched_yield semantics to a gentler form?

From a practical POV: our present yield() behaviour is so truly awful that

it's basically always a bug to use it. This probably isn't a good thing.

So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expect yield() to do.

otoh,

a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not use yield(), so perhaps there's little to be gained in changing it now.

Quoted 8 lines

b) if we _were_ to change yield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.

Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.

So... I would take it that's a yes for a recommendation with respect to implementing a new yield() ? A new scheduler is as good a time as any to do it.

Signature · 1 line
-ck
#4 lore raw
Quoted 8 lines · On Sun, 11 Mar 2007 14:59:28 +1100 Con Kolivas <[email protected]> wrote:
Quoted 3 lines

Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.

So... I would take it that's a yes for a recommendation with respect to implementing a new yield() ? A new scheduler is as good a time as any to do it.

I guess so. We'd, err, need to gather Ingo's input ;)

Perhaps a suitable way of doing this would be to characterise then emulate the 2.4 behaviour. As long as it turns out to be vaguely sensible.

#5 lore raw
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Sun, 11 Mar 2007 14:59:28 +1100 Con Kolivas <[email protected]> wrote:
Quoted 4 lines

Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.

So... I would take it that's a yes for a recommendation with respect to implementing a new yield() ? A new scheduler is as good a time as any to do it.

I guess so. We'd, err, need to gather Ingo's input ;)

cc'ed. Don't you hate timezones?

Quoted 2 lines

Perhaps a suitable way of doing this would be to characterise then emulate the 2.4 behaviour. As long as it turns out to be vaguely sensible.

It's really very simple. We just go the end of the current queued priority on the same array instead of swapping to the expired array; ie we do what realtime tasks currently do. It works fine here locally afaict.

Signature · 1 line
-ck
#6 lore raw
In reply to Andrew Morton
Quoted 2 lines · On Sun, 11 Mar 2007 13:28:22 +1100 "Con Kolivas" <[email protected]> wrote:
Quoted 2 lines

Well... are you advocating we change sched_yield semantics to a gentler form?

Quoted 5 lines · Andrew Morton

From a practical POV: our present yield() behaviour is so truly awful that it's basically always a bug to use it. This probably isn't a good thing. So yes, I do think that we should have a rethink and try to come up with behaviour which is more in accord with what application developers expect yield() to do.

ISTR that apps varied wrt. their expectations for yield(). Some, particularly those using it to implement multi-tiered userspace locks, really did expect to go all the way to the back of the queue. (Rumor has it that realtime apps break otherwise.) Others wanted a kinder, gentler, "mistress, please hit me, but not too hard" opportunity to let someone else have a little cpu time, particularly when userspace is spinning in some sort of busywait. In both scenarios something very much against the latest trends of Linux kernel politics is done by userspace.

Quoted 10 lines · Andrew Morton

otoh,
a) we should have done this five years ago. Instead, we've spent that time training userspace programmers to not use yield(), so perhaps there's little to be gained in changing it now.
b) if we _were_ to change yield(), people would use it more, and their applications would of course suck bigtime when run on earlier 2.6 kernels.
Bottom line: we've had a _lot_ of problems with the new yield() semantics. We effectively broke back-compatibility by changing its behaviour a lot, and we can't really turn around and blame application developers for that.

My dumb idea would be to break out new syscall. One for the kinder, gentler version, one for the serious version. Or otherwise pass an argument indicating the expected behavior. Then a dumb app can be LD_PRELOAD'd into calling whatever makes it run fastest.

-- wli

#7 lore raw
In reply to Con Kolivas
Quoted 6 lines · Con Kolivas
Quoted 6 lines

make -j 5 ccache

beryl    ok          good           awful
galeon   good        good           bad
mp3      good        good           bad
terminal good        good           bad/ok
mouse    good        good           bad/ok

...

Quoted 11 lines
Quoted 4 lines

RSDL makes most of the noyield hit back in normal make and then some with ccache. Impressive. But ccache is still destroying interactivity somehow. The ccache effect is fairly visible even with non-parallel 'make'.

Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.

I don't see significant system time while this is happening.

Quoted 9 lines

This is "worked around" in mainline thanks to the testing for sleeping on uninterruptible sleep in the interactivity estimator. What I suspect is happening is kernel threads that are running nice -5 are doing all the work on make's behalf in the setting of ccache since it is mostly i/o bound. The reason for -nice values on kernel threads is questionable anyway. Can you try renicing your kernel threads all to nice 0 and see what effect that has? Obviously this doesn't need a recompile, but is simple enough to implement in kthread code as a new default.

Sorry, little to no benefit.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#8 lore raw
Quoted 9 lines · Matt Mackall
Quoted 6 lines · Con Kolivas

Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.

I don't see significant system time while this is happening.

Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#9 lore raw
Quoted 9 lines · Matt Mackall
Quoted 8 lines · Matt Mackall
Quoted 6 lines · Con Kolivas

Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.

Quoted 4 lines
Quoted 1 line

I don't see significant system time while this is happening.

Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.

Well I can't reproduce that behaviour here at all whether from disk or the pagecache with ccache, so I'm not entirely sure what's different at your end. However both you and the other person reporting bad behaviour were using ATI drivers. That's about the only commonality? I wonder if they do need to yield... somewhat instead of not at all.

Signature · 1 line
-ck
#10 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 12 lines · Con Kolivas
Quoted 11 lines · Matt Mackall
Quoted 9 lines · Matt Mackall
Quoted 6 lines · Con Kolivas

Ok I don't think there's any actual accounting problem here per se (although I did just recently post a bugfix for rsdl however I think that's unrelated). What I think is going on in the ccache testcase is that all the work is being offloaded to kernel threads reading/writing to/from the filesystem and the make is not getting any actual cpu time.

I don't see significant system time while this is happening.

Quoted 8 lines
Quoted 2 lines

Also, it's running pretty much entirely out of page cache so there wouldn't be a whole lot for kernel threads to do.

Well I can't reproduce that behaviour here at all whether from disk or the pagecache with ccache, so I'm not entirely sure what's different at your end. However both you and the other person reporting bad behaviour were using ATI drivers. That's about the only commonality? I wonder if they do need to yield... somewhat instead of not at all.

I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.

I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.

What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.

Signature · 7 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
While it may be true that a watched pot never boils, the one you don't
keep an eye on can make an awful mess of your stove.
		-- Edward Stevenson
#11 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Hi Gene.

Quoted 11 lines · Gene Heskett

I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.

I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.

What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.

I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(

Signature · 1 line
-ck
#12 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 16 lines · Con Kolivas

Hi Gene.

Quoted 8 lines · Gene Heskett

I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.

I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.

Quoted 4 lines

What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.

Quoted 5 lines

I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(

I have another backup running right now, after building a plain 2.6.21-rc3, and rebooting just now for the test. I don't think its the scheduler itself, but is something post 2.6.20 that is messing with tars mind and making it think the files it just read to do the estimate phase, are all new, so even a level 2 is in effect a level 0. I'll have an answer in about an hour, but its also 2:36am here and I'm headed for the rack to get some zzz's. So I'll report in the morning as to whether or not this backup ran as it was supposed to. I have a feeling its not going to though.

Signature · 6 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
"When it comes to humility, I'm the greatest."
-- Bullwinkle Moose
#13 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 23 lines · Gene Heskett
Quoted 16 lines · Con Kolivas

Hi Gene.

Quoted 8 lines · Gene Heskett

I hate to say it Con, but this one seems to have broken the amanda-tar symbiosis.

I haven't tried a plain 21-rc3, so the problem may exist there, and in fact it did for 21-rc1, but I don't recall if it was true for -rc2. But I will have a plain 21-rc3 running by tomorrow nights amanda run to test.

Quoted 4 lines

What happens is that when amanda tells tar to do a level 1 or 2, tar still thinks its doing a level 0. The net result is that the tape is filled completely and amanda does an EOT exit in about 10 of my 42 dle's. This is tar-1.15-1 for fedora core 6.

Quoted 5 lines

I'm sorry but I have to say I have no idea what any of this means. I gather you're making an association between some application combination failing and RSDL cpu scheduler. Unfortunately the details of what the problem is, or how the cpu scheduler is responsible, escape me :(

Quoted 9 lines

I have another backup running right now, after building a plain 2.6.21-rc3, and rebooting just now for the test. I don't think its the scheduler itself, but is something post 2.6.20 that is messing with tars mind and making it think the files it just read to do the estimate phase, are all new, so even a level 2 is in effect a level 0. I'll have an answer in about an hour, but its also 2:36am here and I'm headed for the rack to get some zzz's. So I'll report in the morning as to whether or not this backup ran as it was supposed to. I have a feeling its not going to though.

I can confirm that a plain 21-rc3 still suffers from this problem. This run of amanda terminated after 13 of the 32 dle's after writing just short of 12GB to the vtape. 8 were level 0's, 5 were level 1's, all were gzipped, achieving a compression ratio of 40% of original size. There is about 45GB of data here to backup, on a 5 day dumpcycle.

If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage into c-times when tar does its estimate scans where the output goes to /dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.

I love your patches Con, but I'll leave them out of this next testing. No use pointing fingers at good code.

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Mal: "Well, you were right about this being a bad idea."

Zoe: "Thanks for sayin', sir."
				--Episode #1, "Serenity"
#14 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 11 lines · Gene Heskett

If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage into c-times when tar does its estimate scans where the output goes to /dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.

You may find 'strace' useful to track down this sort of thing (though the output can be voluminous).

-Doug

#15 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 14 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

If, and I have previously, I revert to a 2.6.20-ck1 patching, this does not occur. So my contention is that someplace in this recent progression from 2.6.20 to 2.6.21-rc3, there is a patch which acts to change how c-time is being reported to tar. Or there is a spillage

Quoted 8 lines

into c-times when tar does its estimate scans where the output goes to /dev/null. Or possibly even this version of tar is doing it differently. I just looked up how to get the c-times out of ls, and they, as far as ls is concerned, look sane. But tars actions while running a 2.6.21-rcX kernel certainly are not. I do have a plain -rc2 I can try, so that will be the next test. If that also fails in this manner, I'll build a later 2.6.20-2 or whatever to verify that it doesn't so suffer.

Quoted 4 lines

You may find 'strace' useful to track down this sort of thing (though the output can be voluminous).

-Doug

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

Signature · 5 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
I'm definitely not in Omaha!
#16 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff 

-Doug

#17 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 9 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff

-Doug

Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:

running /usr/local/sbin//amcheck Daily
Amanda Tape Server Host Check
-----------------------------
Holding disk /dumps: 72597 MB disk space available, using 72097 MB slot 5: read label Dailys-5', date 20070223002502'
NOTE: skipping tape-writable test
Tape Dailys-5 label ok
Server check took 4.556 seconds

Amanda Backup Client Hosts Check
--------------------------------
WARNING: coyote: selfcheck request failed: unable to bind to a reserved port (got port 32789)
Client check: 1 host checked in 34.733 seconds, 1 problem found

(brought to you by Amanda 2.5.1p3-20070222)
------------------eofsnip------------

The script quit by request shortly after as I didn't want it to further dirty the amanda database system.

Signature · 5 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Safety Third.
#18 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff
Quoted 8 lines
Quoted 1 line

-Doug

Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:

Can we start a new thread for this amanda issue? It does not seem to have anything to do with Con's scheduler...

Lee

#19 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

In reply to Gene Heskett
Quoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff
Quoted 8 lines
Quoted 1 line

-Doug

Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:

I'd do 'strace -ff -o /tmp/amanda-strace <command>', which will give you a set of files in /tmp, one for each PID created by fork(). Then find the one that has the 'tar' invocation you're looking for.

-Doug

#20 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 20 lines · Douglas McNaught
Quoted 9 lines · Gene Heskett
Quoted 8 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff
Quoted 8 lines
Quoted 1 line

-Doug

Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:

Quoted 3 lines

I'd do 'strace -ff -o /tmp/amanda-strace <command>', which will give you a set of files in /tmp, one for each PID created by fork(). Then find the one that has the 'tar' invocation you're looking for.

Hi,

I hope you don't mind me jumping in ...

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

That should be working, shouldn't it ?

Cheers,
Patrick

#21 lore raw

Re: RSDL for 2.6.21-rc3- 0.29

Quoted 31 lines · Patrick Mau
Quoted 20 lines · Douglas McNaught
Quoted 11 lines · Gene Heskett
Quoted 9 lines · Douglas McNaught
Quoted 4 lines · Gene Heskett

I'd considered it, but with 32 dle entries, the whole strace output would be terrabytes & I don't have THAT much disk. Not to mention it traces only the parent process, so tar would be merrily marching along to its own drummer and not traced I'm afraid.

$ strace -ff

-Doug

Quoted 7 lines

Someone else suggested the single -f, and I tried that, but even with the shell history set for 100,000 lines, i can't get back to the start, and I think its mucking with the shell arguments numbering as what I can see is about 5 reads through /etc/services accompanied by endless complaints of -EBADFD, the the logfile it generates says the port it was given was rejected when amcheck was run, here is that snip:

Quoted 3 lines

I'd do 'strace -ff -o /tmp/amanda-strace <command>', which will give you a set of files in /tmp, one for each PID created by fork(). Then find the one that has the 'tar' invocation you're looking for.

Hi,

I hope you don't mind me jumping in ...

Why not temporarly replace "/bin/tar" with a shell script that does:

Quoted 7 lines

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

That should be working, shouldn't it ?

Cheers,
Patrick

It should I'd think, for those instances where these scripts are run by hand and therefore have a shell to output to. I'll give that a try when I get my script problem resolved, its a basic diff in the operation of real tapes and vtapes I need to get right for both scenarios.

Good Idea(TM), Thanks.

Signature · 6 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
"Ignorance is the soil in which belief in miracles grows."
-- Robert G. Ingersoll
#22 lore raw

Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

In reply to Patrick Mau
Quoted 4 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

#23 lore raw

Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

Quoted 6 lines · Douglas McNaught
Quoted 4 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

Quoted 7 lines

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.

But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.

Signature · 6 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
rugged, adj.:
	Too heavy to lift.
#24 lore raw

Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

Quoted 15 lines · Gene Heskett
Quoted 6 lines · Douglas McNaught
Quoted 4 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

Quoted 7 lines

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

Quoted 7 lines

In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.

But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.

buried indeed:

"Special Parameters:
...

      $      Expands to the process ID of the shell.  In a  ()  subshell,  it
             expands  to  the  process  ID of the current shell, not the sub‐
             shell.
"

Thanks,
Nish

#25 lore raw

Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

Quoted 8 lines

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.

Quoted 3 lines

But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.

Quoted 8 lines

buried indeed:

"Special Parameters:
...

      $      Expands to the process ID of the shell.  In a  () 
subshell,  it expands  to  the  process  ID of the current shell, not
the sub‐ shell.
"

Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.

Quoted 2 lines

Thanks,
Nish

Signature · 6 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
"Say yur prayers, yuh flea-pickin' varmint!"
-- Yosemite Sam
#26 lore raw

Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

Quoted 38 lines · Gene Heskett
Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

Quoted 8 lines

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.

Quoted 3 lines

But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.

Quoted 8 lines

buried indeed:

"Special Parameters:
...

      $      Expands to the process ID of the shell.  In a  () 
subshell,  it expands  to  the  process  ID of the current shell, not
the sub?$B!> shell.
"
Quoted 8 lines

Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.

Yes there a risk of wrapping, but it is very small. You can add the command line arguments to the file name if you want, like this :

#!/bin/sh
exec strace -f -o "output.$$.${*//\//_}" /bin/real.tar $@

It will name the output file "output.<pid>.<args>", replacing slashes with underscores. This is very dirty but can help.

Cheers,
Willy

#27 lore raw

Re: Stracing Amanda (was: RSDL for 2.6.21-rc3- 0.29)

Quoted 48 lines · Willy Tarreau
Quoted 38 lines · Gene Heskett
Quoted 28 lines · Nish Aravamudan
Quoted 23 lines · Gene Heskett
Quoted 7 lines · Douglas McNaught
Quoted 5 lines · Patrick Mau

Why not temporarly replace "/bin/tar" with a shell script that does:

#!/bin/sh
exec strace -f -o output /bin/real.tar $@

Quoted 8 lines

You beat me to it. :) I've done that before; it's a great suggestion.

Except that if you expect 'tar' to be invoked multiple times in a run, you should probably use 'output.$$' for the output filename so things don't get clobbered.

-Doug

In my case, Doug, it will get invoked 64 times, amanda does a dummy run to get an estimate, calculates what to do based on that output which is 32 runs, 1 per disklist entry and I have 32, and then reruns tar with the appropriate level options against each individual disklist entry.

Quoted 3 lines

But I'm puzzled a bit, what does the double $$ do?, or it buried someplace in the bash manpage? Its not something I've stumbled over yet.

Quoted 8 lines

buried indeed:

"Special Parameters:
...

      $      Expands to the process ID of the shell.  In a  ()
subshell,  it expands  to  the  process  ID of the current shell,
not the sub?$B!> shell.
"
Quoted 8 lines

Well, that's clear enough, but what of the double $$ case? Would this them make a PID unique to each invocation untill it finally wraps a 16 bit value, or will the kernel re-use them because they won't all be running simultainiously, but limited by the number of unique 'spindle' numbers on the system, this to prevent as best as it can, the thrashing of a drive by having tar working on 2 separate (or more) partitions at the same time. In my case 2 are possible, as /var is on a separate drive.

Quoted 9 lines

Yes there a risk of wrapping, but it is very small. You can add the command line arguments to the file name if you want, like this :

#!/bin/sh
exec strace -f -o "output.$$.${*//\//_}" /bin/real.tar $@

It will name the output file "output.<pid>.<args>", replacing slashes with underscores. This is very dirty but can help.

Excellent Willy, thanks.

Quoted 2 lines

Cheers,
Willy

Signature · 7 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Whatever doesn't succeed in two months and a half in California will
never succeed.
		-- Rev. Henry Durant, founder of the University of California
#28 lore raw
In reply to Con Kolivas
Quoted 9 lines
Quoted 8 lines

There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current sched_yield, you go all the way to the back of the line.

Quoted 5 lines

Well... are you advocating we change sched_yield semantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding using sched_yield instead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).

The basic point of sched_yield is to allow every other process at the same static priority level a chance to use the CPU before you get it back. It is generally an error to use sched_yield to be nice. It's nice to get your work done when the scheduler gives you the CPU, that's why it gave it to you.

It is proper to use sched_yield as an optimization when it more efficient to allow another process/thread to run than you, for example, when you encounter a task you cannot do efficiently at that time because another thread holds a lock.

It's also useful prior to doing something that can most efficiently be done without interruption. So a thread that returns from 'sched_yield' should ideally be given a full timeslice if possible. This may not be sensible if the 'sched_yield' didn't actuall yield, but then again, if nothing else wants to run, why not give the only task that does a full slice?

In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)

Note that processes that call sched_yield should be rewarded for doing so just as process that block on I/O are, assuming they do in fact wind up giving up the CPU when they would otherwise have had it.

DS

#29 lore raw
Quoted 2 lines · David Schwartz

In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)

Perhaps if sched_yield()'s effects were cumulative inside a timeslice, then eventually the calling task would get pushed far enough down that others would run.

Ray

#30 lore raw
In reply to David Schwartz
Quoted 48 lines · David Schwartz
Quoted 9 lines
Quoted 7 lines

There's a substantial performance hit for not yield, so we probably want to investigate alternate semantics for it. It seems reasonable for apps to say "let me not hog the CPU" without completely expiring them. Imagine you're in the front of the line (aka queue) and you spend a moment fumbling for your wallet. The polite thing to do is to let the next guy in front. But with the current sched_yield, you go all the way to the back of the line.

Quoted 5 lines

Well... are you advocating we change sched_yield semantics to a gentler form? This is a cinch to implement but I know how Ingo feels about this. It will only encourage more lax coding using sched_yield instead of proper blocking (see huge arguments with the ldap people on this one who insist it's impossible not to use yield).

The basic point of sched_yield is to allow every other process at the same static priority level a chance to use the CPU before you get it back. It is generally an error to use sched_yield to be nice. It's nice to get your work done when the scheduler gives you the CPU, that's why it gave it to you.

It is proper to use sched_yield as an optimization when it more efficient to allow another process/thread to run than you, for example, when you encounter a task you cannot do efficiently at that time because another thread holds a lock.

It's also useful prior to doing something that can most efficiently be done without interruption. So a thread that returns from 'sched_yield' should ideally be given a full timeslice if possible. This may not be sensible if the 'sched_yield' didn't actuall yield, but then again, if nothing else wants to run, why not give the only task that does a full slice?

In no case is much of anything guaranteed, of course. (What can you do if there's no other process to yield to?)

Note that processes that call sched_yield should be rewarded for doing so just as process that block on I/O are, assuming they do in fact wind up giving up the CPU when they would otherwise have had it.

DS

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#31 lore raw
In reply to Matt Mackall
Quoted 3 lines · Matt Mackall

I've tested -mm2 against -mm2+noyield and -mm2+rsdl+noyield. The noyield patch simply makes the sched_yield syscall return immediately. Xorg and all tests are run at nice 0.

[skipped long and precise test report]

Quoted 3 lines

Also note I could occassionally trigger nasty multi-second pauses with -mm2+noyield under exectest that didn't show up elsewhere. That's probably a bug in the mainline scheduler.

This is not a bug per se, but more a design problem. This is caused by the interactivity booster which is unfair. Mike Galbraith and others spent a lot of time trying to get rid of those problems a few versions ago. In early kernels (around 2.6.11), I could trivially cause pauses more than 30 seconds long by running a few tasks simulating an interactive workload. It is much more difficult to achieve this with recent kernels, and it has absolutely no effect on RSDL, which is one of the reasons I have to find it great !

Regards,
Willy