Skip to contentSkip to conversation

Nick's scheduler v16

DISCUSSION Nick Piggin 4 participants 9 messages Oct 18, 2003 – Nov 3, 2003

lore
Share & tools
lore
#0 lore raw

Hi

http://www.kerneltrap.org/~npiggin/v16/

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its possible they could get included if they turn out well. If you find any problems with it (high end or interactivity), please let me know.

#1 lore raw
Quoted 8 lines · Nick Piggin

Hi

http://www.kerneltrap.org/~npiggin/v16/

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

Oh, if anyone can suggest other benchmarks I could use it would be good (I'm currently getting osdl's dbt2 set up). System is 16-way 16GB ia32 NUMA and very minimum disk bandwidth. Thanks.

#2 lore raw
In reply to Nick Piggin
Quoted 7 lines

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its possible they could get included if they turn out well. If you find any problems with it (high end or interactivity), please let me know.

Interesting ... some things get getter, some worse:

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)

                              Elapsed      System        User         CPU
              2.6.0-test8       45.20      100.97      566.65     1476.25
         2.6.0-test8-nick       44.81       93.98      568.49     1477.50
        2.6.0-test8-nick2       44.78       94.69      568.81     1482.00

elapsed is a tiny bit faster, system is significantly less, but with higher parallelism:

Kernbench: (make -j vmlinux, maximal tasks)

                              Elapsed      System        User         CPU
              2.6.0-test8       45.86      119.41      569.66     1502.00
         2.6.0-test8-nick       47.00      112.75      590.40     1495.00
        2.6.0-test8-nick2       47.11      112.86      590.31     1491.50

elapsed is definitely worse now.
SDET is a happy bunny though:

SDET 128 (see disclaimer)

                           Throughput    Std. Dev
              2.6.0-test8       100.0%         0.3%
         2.6.0-test8-nick       109.9%         0.2%

Much of the changes there might just be backing out Con's interactivity changes ...

M.

#3 lore raw
Quoted 21 lines · Martin J. Bligh
Quoted 8 lines

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its possible they could get included if they turn out well. If you find any problems with it (high end or interactivity), please let me know.

Interesting ... some things get getter, some worse:

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)

                             Elapsed      System        User         CPU
             2.6.0-test8       45.20      100.97      566.65     1476.25
        2.6.0-test8-nick       44.81       93.98      568.49     1477.50
       2.6.0-test8-nick2       44.78       94.69      568.81     1482.00

elapsed is a tiny bit faster, system is significantly less, but with higher parallelism:

Quoted 8 lines

Kernbench: (make -j vmlinux, maximal tasks)

                             Elapsed      System        User         CPU
             2.6.0-test8       45.86      119.41      569.66     1502.00
        2.6.0-test8-nick       47.00      112.75      590.40     1495.00
       2.6.0-test8-nick2       47.11      112.86      590.31     1491.50

elapsed is definitely worse now.

I'll see if I can reproduce this and work out what is going on. Thanks Martin.

#4 lore raw

Re: Nick's scheduler v16 - reaim

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its possible they could get included if they turn out well. If you find any problems with it (high end or interactivity), please let me know.

Results from reaim aren't encouraging. patch was applied against 2.6.0-test8 - the result is PLM #2232

STP id  kernal name       Max JPM   Max User Pct    elevator
281932	nick_v16          4923.08    60	     0.00   AS
281933	nick_v16          5196.06    68	     5.54   deadline
281722	linux-2.6.0-test8 5432.77    92	     9.38   deadline
281792	2.6.0-test8-mm1   5384.41    92	     8.56   deadline
281790	2.6.0-test8-mm1   5392.65    88      8.7    AS

The kernel doesn't perform well at larger user numbers. Notice the different in max users.
Compare the graphs for jobs per minute, the usual graph is quite flat, see:
http://khack.osdl.org/stp/281932/results/jpm.png

With the v16 scheduler, jobs per minute falls off rapidly as user number increases giving graph with a steep slope, not good.
http://khack.osdl.org/stp/281790/results/jpm.png
Further results: http://www.developer.osdl.org/reaim/index.html

cliffw

#5 lore raw

Re: Nick's scheduler v16 - reaim

Quoted 29 lines · Cliff White
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite a bit. It isn't too bad, although I'm only testing dbench, tbench, and volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its possible they could get included if they turn out well. If you find any problems with it (high end or interactivity), please let me know.

Results from reaim aren't encouraging. patch was applied against 2.6.0-test8 - the result is PLM #2232

STP id  kernal name       Max JPM   Max User Pct    elevator
281932	nick_v16          4923.08    60	     0.00   AS
281933	nick_v16          5196.06    68	     5.54   deadline
281722	linux-2.6.0-test8 5432.77    92	     9.38   deadline
281792	2.6.0-test8-mm1   5384.41    92	     8.56   deadline
281790	2.6.0-test8-mm1   5392.65    88      8.7    AS

The kernel doesn't perform well at larger user numbers. Notice the different in max users.
Compare the graphs for jobs per minute, the usual graph is quite flat, see:
http://khack.osdl.org/stp/281932/results/jpm.png

Quoted 5 lines

With the v16 scheduler, jobs per minute falls off rapidly as user number increases giving graph with a steep slope, not good.
http://khack.osdl.org/stp/281790/results/jpm.png
Further results: http://www.developer.osdl.org/reaim/index.html

Hi Cliff,

Yeah, unfortunately this graph has been a feature of my scheduler for quite a while :( I think it is caused either by too much balancing or timeslices getting too small because there are a lot of threads blocking. Anyway, if you had the time to test this one would be really good.

http://www.kerneltrap.org/~npiggin/v16/sched-rollup-nopolicy-v16.gz

Nick

#7 lore raw

Re: Nick's scheduler v17a

Quoted 7 lines · On Thu, 30 Oct 2003 14:41:07 +1100 Nick Piggin <[email protected]> wrote:

http://www.kerneltrap.org/~npiggin/v17a/

More balancing fixes. I also incorporated some of Andrew Theurer's ideas. I'm generally getting good numbers now, but using fairly synthetic benchmarks.

Now would be a good time to test if anyone is interested. Thanks.

well i didn't have the time to make extensive tests yet, but the behaviour improved a lot since v16, and *looks* quite better compared to test9 on a 2 way xeon p4 with hyperthreading enabled, seeing 4 cpu's.

when i get back home, I'll try to make a few benchmarks.

#8 lore raw

Re: Nick's scheduler v17a

Quoted 10 lines · Rhino
Quoted 6 lines · On Thu, 30 Oct 2003 14:41:07 +1100 Nick Piggin <[email protected]> wrote:

http://www.kerneltrap.org/~npiggin/v17a/

More balancing fixes. I also incorporated some of Andrew Theurer's ideas. I'm generally getting good numbers now, but using fairly synthetic benchmarks.

Quoted 9 lines
Quoted 2 lines

Now would be a good time to test if anyone is interested. Thanks.

well i didn't have the time to make extensive tests yet, but the behaviour improved a lot since v16, and *looks* quite better compared to test9 on a 2 way xeon p4 with hyperthreading enabled, seeing 4 cpu's.

when i get back home, I'll try to make a few benchmarks.

That would be nice. There still isn't a lot of work done in the HT department, but something simple and easy like the shared runqueues patch might be reasonable for 2.6.