2.6.0-test6 scheduling(?) oddness

P4 2G
1G PC2700 RAM
ASUS P4S533

Large tasks (like raytrace rendering) take double the amount of time they used to take, although the system is nicer to the user while they run. In 2.6.0-test5 had a little trouble with it and Piggin pointed me to a patch that fixed it and is now in -test6, however the patch didn't slow the rendering as much as it does in test6. (Con Koliva's patch, I believe it was). For example - rendering an image that took 15 minutes in 2.5.65 takes 20 minutes in 2.6.0-test5 (with patch) and 30 minutes in 2.6.0-test6 (raw from kernel.org). Same config options (everything I use builtin - no modules).

A new issue (which also doesn't happen in -test5 with the patch):
When running cpu intense tasks, new (large) tasks will not start till the first one finishes.
For example, using POV-Ray 3.5 to render an image that takes 30 minutes when it is the only program running, start oowriter.
The render finishes in the same 30 minutes, then oowriter starts. oowriter takes about 3 seconds to load if no rendering is going on. I can use apps that are already open but can't start new ones while rendering. In 2.6.0-test5 (with patch) opening oowriter while rendering takes about 1 minute.
In 2.5.65 opening oowriter while rendering takes about 2 minutes (and X gets very hard to use till oowriter is completely done opening).

Murray J. Root

Hi Murray!
Con Kolivas' patch has been included in test6. You might have tried that or maybe my patch?

Anyway, lets just clarify your report:
1 CPU, hyperthreading on or off?

test5 interactivity under load isn't as good as test6.

povray takes 150% more time in test6 vs tset5. How many compute threads does povray use? Is anything else running?

oowriter takes 3 seconds to load when nothing is running oowriter takes 30(or more) minutes to load when povray is runnnig, takes 1 minute in test5 plus patch (could you tell us exactly what the patch is)

Sorry, 150% _the_ time, 50% more time.

It was Con Kolivas' patch - I never got to yours since his fixed my -test5 issue.

1 CPU, no ht

Hard to measure - they're both good from the user's point of view.

povray uses 1 process to compute. As far as I can tell it's not threading at all.
as for other things running - X with fluxbox 0.1.14 is the only constant. I vary other apps to test but there seems to be no influence unless the other app is also a cpu/memory hog.

correct

that 1 minute in test5 is *while rendering* the same image I'm rendering in this test.
with no load in test5 oowriter takes about 3 seconds to load.

oowriter doesn't load at all until the render is finished. 30 minutes was the image I was working with. tried it with a 1 hour render, and oowriter didn't start till the render finished.

Murray J. Root

OpenOffice uses sched_yield() in strange ways which causes it to get hopelessly starved on 2.6 kernels. I think RH have a fixed version, but I don't know if that has propagated into the upstream yet.

So... Don't worry about OpenOffice too much. Is the problem reproducible with other applications?

OK. Aside from the OpenOffice issue, you still have povray taking 50% longer to complete, which is quite remarkable if its single threaded and nothing else is running. Maybe its not the scheduler change? Anyway, Con will want to know exactly which version of his scheduler you used in test5 to check for possibilities.

patch-test5-O20int

Not sure what caused the increase from 2.6.0-test5 to 2.6.0-test6. One noticeable thing - I get a noticeable speed increase if I turn off rendering to the screen, so that it only goes to a file. I don't get that speed improvement in 2.5.65 - only in 2.6.0-test5 and 2.6.0-test6 test5 goes from 20 mins to about 15 mins (the same speed as 2.5.65 with or without screen rendering)
test6 goes from 30 mins to 24 mins - still worse than test5 by a lot.

Murray J. Root

(the linux-kernel email convention is "reply to all", btw)

What sort of context switch rate are you seeing during this run? Running `vmstat 1' will tell.

I have no idea how to read it, but the lines look like this.

Without render to screen:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  0  12264   6572  79004 747208    0    0     0     0 1032  1712 99  1  0  0
 2  0  12264   6444  79004 747336    0    0   128     0 1028  1654 99  1  0  0
 2  0  12264   6444  79004 747336    0    0     0     0 1041  1679 99  1  0  0
 2  0  12264   6316  79004 747464    0    0   128     0 1024  1638 98  2  0  0
 2  0  12264   6252  79040 747464    0    0     0   124 1089  1650 99  1  0  0
 1  0  12264   6124  79040 747592    0    0   128     0 1073  1679 99  1  0  0
 3  0  12264   6124  79040 747592    0    0     0     0 1025  1689 99  1  0  0
 2  0  12264   6124  79040 747592    0    0     0     0 1130  1691 99  1  0  0
 1  0  12264   5996  79040 747720    0    0   128     0 1149  1684 98  2  0  0
 2  0  12264   5996  79076 747720    0    0     0    64 1095  1720 100  0  0  0
 1  0  12264   5868  79076 747848    0    0   128     0 1205  2002 98  2  0  0

with render to screen:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 6  0  12264   8988  79528 744060    0    0    42    50   13   159 24  1 75  1
 1  0  12264   8988  79616 744060    0    0     0   164 1050  1830 99  1  0  0
 3  0  12264   8988  79616 744076    0    0    16     0 1031  1763 98  2  0  0
 1  0  12264   8988  79616 744076    0    0     0     0 1025  1771 99  1  0  0
 1  0  12264   8620  79616 744440    0    0   364     0 1059  1781 99  1  0  0
 2  0  12264   8620  79616 744440    0    0     0     0 1029  1721 99  1  0  0
 1  0  12264   8620  79652 744440    0    0     0    64 1035  1742 98  2  0  0
 3  0  12264   8492  79652 744568    0    0   128     0 1028  1703 99  1  0  0
 2  0  12264   8492  79652 744568    0    0     0     0 1028  1690 99  1  0  0
 3  0  12264   8364  79652 744696    0    0   128     0 1029  1727 97  3  0  0
 1  0  12264   8364  79652 744696    0    0     0     0 1023  1670 99  1  0  0
 1  0  12264   8236  79688 744824    0    0   128    64 1036  1705 99  1  0  0
 1  0  12264   8236  79688 744824    0    0     0     0 1024  1657 99  1  0  0
 2  0  12264   8108  79688 744952    0    0   128     0 1031  1658 98  2  0  0

Render ended:

0 0 12264 6952 80948 745608 0 0 0 136 1044 1819 17 2 81 0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  12264   6812  80948 745736    0    0   128     0 1025  1571  2  0 98  0
 2  0  12264   6812  80948 745736    0    0     0     0 1050  1532  1  1 98  0
 0  0  12264   6812  80948 745736    0    0     0     0 1172  1639  1  1 98  0
 0  0  12264   6672  80948 745864    0    0   128     0 1199  1659  3  0 97  0
 0  0  12264   6688  80972 745864    0    0     0    64 1110  1621  1  1 98  0
 0  0  12264   6688  80972 745864    0    0     0     0 1024  1582  2  0 98  0
 0  0  12264   6576  80972 745992    0    0   128     0 1024  1547  0  1 99  0
 0  0  12264   6576  80972 745992    0    0     0     0 1042  1574  2  0 98  0

Murray J. Root

Nope - even tried it with KDE apps.
Write it off to OpenOffice, not test6.

That doesn't explain the major time increase of the render, though. 200% for 2.5.65 vs 2.6.0-test6 or 150% for 2.6.0-test5 vs 2.6.0-test6 is a bit extreme.

Murray J. Root

In article <20031001051008.GD1416@Master>,
Murray J. Root <[email protected]> wrote:

| On Tue, Sep 30, 2003 at 09:55:12PM -0700, Andrew Morton wrote:
| > "Murray J. Root" <[email protected]> wrote:
| > >
| > > The render finishes in the same 30 minutes, then oowriter starts.
| > >  oowriter takes about 3 seconds to load if no rendering is going on.
| > 
| > OpenOffice uses sched_yield() in strange ways which causes it to
| > get hopelessly starved on 2.6 kernels.  I think RH have a fixed version,
| > but I don't know if that has propagated into the upstream yet.
| > 
| > So...  Don't worry about OpenOffice too much.  Is the problem reproducible
| > with other applications?
| 
| Nope - even tried it with KDE apps.
| Write it off to OpenOffice, not test6.

I wish I could just write off programs like that, but if a program is running, and doing legitimate system calls, and it stops running (totally or usefully), I'd like to be sure that the kernel doesn't have some unintended behaviour before I just pass on the program.

Particularly when OO is what allows lots of people to avoid running that other operating system.

| 
| That doesn't explain the major time increase of the render, though.
| 200% for 2.5.65 vs 2.6.0-test6 or 150% for 2.6.0-test5 vs 2.6.0-test6 is a 
| bit extreme.

The vmstat sure didn't show a big increase in contenxt switches, but if there's nothing elese wanting the CPU I would expect the render to be what's running, and at the same speed as test5.

Suggestion: could you run 'top' in the text output mode to see if for some reason the CPU is going to some odd process. The revised nice handling can't make a user process lower priority than the idle loop or something odd like that, can it?

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

it isn't doing something legitimate since as he said, it was the only program that exibited the behavior. Perhaps openoffice was exploiting a characteristic of the old schedular to increase it's performance, perhaps it's just the way they ended up coding it. But if it's the only one then that's that.

Why would you trust what top says about cpu usage? It lies it lies it lies. At best a rough estimate.

Con's scheduler fixes are quite nice. Make sure if you want something to hog the cpu to nice it down, otherwise it has to play nice. Just because programs use very little cpu, doesn't mean they wont need some in a given time, and that of course will slow down any app that needs cpu but is being forced to share, the stop and go even though it's for a short time takes time and happens a lot with the more processes being run. This is how a multi-user OS should behave, a render process shouldn't be allowed to lag the rest of the system for the sake of it's own performance unless you explicitly allow it. Having a loss of speed even at nice -20 however, is a problem.

In article <[email protected]>,
Ed Sweetman <[email protected]> wrote:
| bill davidsen wrote:

| > I wish I could just write off programs like that, but if a program is
| > running, and doing legitimate system calls, and it stops running
| > (totally or usefully), I'd like to be sure that the kernel doesn't have
| > some unintended behaviour before I just pass on the program.
| > 
| > Particularly when OO is what allows lots of people to avoid running that
| > other operating system.
| 
| it isn't doing something legitimate since as he said, it was the only 
| program that exibited the behavior. Perhaps openoffice was exploiting a 
| characteristic of the old schedular to increase it's performance, 
| perhaps it's just the way they ended up coding it.  But if it's the only 
| one then that's that.

I see nothing to indicate that any illegal system calls were made, in what way is it not doing something legitimate?

One program which has always worked suddenly stopping is a symptom of a problem, and assuming that there is no problem seems optimistic. Particularly when it works on BSD, Solaris, all previous Linux and even Windows.

If this is the sched_yeild() stuff again, I thought that was beaten into the ground before, and it was agreed that SUS allows it to work the way it has always worked and the way it works elsewhere. Hopefully this is not the reason performance is so grim, and a solution can be found.

BTW: I'm told that StarOffice (commercial release) also doesn't work usefully on test6, can anyone confirm? The test system is not overly stable and I don't trust negative results there.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

OOo works just fine - it just won't *start* while POVRay is rendering. Once it's started it runs fine, even when rendering.

Murray J. Root