Skip to contentSkip to conversation

benh performance problem

DISCUSSION Mikolaj Krzewicki 4 participants 7 messages Feb 3, 2004 – Feb 7, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

hi all,
starting 2.6.0-test9-benh3 i noticed an increased system overhead on my ibook 500. Each process eats much more cpu time than in 2.4 or even vanilla 2.6.
gkrellm takes 1% instead of 0.1, xmms 5% instead of ~1% and so on. I benchmarked the lot with octave and it indeed runs ~10% slower on the benh kernels.
I would use vanilla 2.6.1 but it won't sleep/wake correctly (at all). All pre-2.6.0-test9-benh3 kernels worked fast but would't sleep/wake either. I tried to isolate the patch responsible but failed.
Has anybody come across similar behaviour?
greetings, M.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#1 lore raw
Quoted 4 lines · Mikolaj Krzewicki

hi all,
starting 2.6.0-test9-benh3 i noticed an increased system overhead on my ibook 500. Each process eats much more cpu time than in 2.4 or even vanilla 2.6.

Quoted 8 lines

gkrellm takes 1% instead of 0.1, xmms 5% instead of ~1% and so on. I benchmarked the lot with octave and it indeed runs ~10% slower on the benh kernels.
I would use vanilla 2.6.1 but it won't sleep/wake correctly (at all). All pre-2.6.0-test9-benh3 kernels worked fast but would't sleep/wake either. I tried to isolate the patch responsible but failed.
Has anybody come across similar behaviour?
greetings, M

I switched to HZ=1000, the userland procps utilities (like ps and top) tend to not properly deal with that, at least earlier versions, afaik.

I suspect it's just display crap.

Alos, you can find a more recent kernel than that :) My bk tree is currently at 2.6.2-rc3-ben1

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#2 lore raw
Quoted 20 lines · Mikolaj Krzewicki
Quoted 13 lines
Quoted 13 lines

hi all,
starting 2.6.0-test9-benh3 i noticed an increased system overhead on my ibook 500. Each process eats much more cpu time than in 2.4 or even vanilla 2.6.
gkrellm takes 1% instead of 0.1, xmms 5% instead of ~1% and so on. I benchmarked the lot with octave and it indeed runs ~10% slower on the benh kernels.
I would use vanilla 2.6.1 but it won't sleep/wake correctly (at all). All pre-2.6.0-test9-benh3 kernels worked fast but would't sleep/wake either. I tried to isolate the patch responsible but failed.
Has anybody come across similar behaviour?
greetings, M

I switched to HZ=1000, the userland procps utilities (like ps and top) tend to not properly deal with that, at least earlier versions, afaik.

Quoted 6 lines

I suspect it's just display crap.

Alos, you can find a more recent kernel than that My bk tree is currently at 2.6.2-rc3-ben1

Ben.

i tested 2.6.1-benh1 now with HZ=100
the benchmark i used shows a speedup back to the values i'm used to. The benchmark itself is an octave script the execution time of which i tested with different kernels under similar circumstances, so here are the details:

2.6.1-benh1, HZ=1000: timing=36.5s
2.6.1-benh1, HZ=100 : timing=32s

this is with X running and a lot more processes(not running). the weird thing is it executes slightly faster (on average) with pbbuttonsd off.
The machine is g3 500 ibook, the octave script is:

  tic;
    a = abs(randn(1500, 1500)/10);
    b = a';
    c= a*b;
    a = reshape(b, 750, 3000);
    b = a';
  timing=toc;

so lots of system calls and cache flushing is in order.

Mikolaj.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 21 lines · Mikolaj Krzewicki

the benchmark i used shows a speedup back to the values i'm used to. The benchmark itself is an octave script the execution time of which i tested with different kernels under similar circumstances, so here are the details:

2.6.1-benh1, HZ=1000: timing=36.5s
2.6.1-benh1, HZ=100 : timing=32s

this is with X running and a lot more processes(not running). the weird thing is it executes slightly faster (on average) with pbbuttonsd off.
The machine is g3 500 ibook, the octave script is:

   tic;
     a = abs(randn(1500, 1500)/10);
     b = a';
     c= a*b;
     a = reshape(b, 750, 3000);
     b = a';
   timing=toc;
Quoted 3 lines

so lots of system calls and cache flushing is in order.

Mikolaj.

Well, I don't knwo what the above means, I don't talk that language anyway :)

The fact that pbbuttons makes a difference makes me think the interrupt handling is taking way too much time on your setup, and pbbuttons is loading the machine with PMU interrupts...

Not sure if I can fix any of this at this point without doing a major rewrite of the exception handling code, I suspect those CPUs don't like running in real mode and our exception handling happens mostly in that mode in ppc32...

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#5 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 3 lines · Mikolaj Krzewicki

2.6.1-benh1, HZ=1000: timing=36.5s
2.6.1-benh1, HZ=100 : timing=32s

The fact that pbbuttons makes a difference makes me think the interrupt handling is taking way too much time on your setup, and pbbuttons is loading the machine with PMU interrupts...

pbbuttons is loading the machine? What about all those extra decrementer ticks? I assume benchmarks showed no performance degradation on some class of ppc32; what was the slowest system tested? For that matter, couldn't this hurt 4xx, 8xx(x), as well?

Perhaps HZ should be user-configurable? Although that doesn't help distributions who'd like one kernel to work everywhere... How does Linux handle increased HZ on old i386 anyways?

Quoted 4 lines

Not sure if I can fix any of this at this point without doing a major rewrite of the exception handling code, I suspect those CPUs don't like running in real mode and our exception handling happens mostly in that mode in ppc32...

I didn't realize CPUs go slower in real mode..?

--
Hollis Blanchard
IBM Linux Technology Center

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#6 lore raw
Quoted 5 lines

pbbuttons is loading the machine? What about all those extra decrementer ticks? I assume benchmarks showed no performance degradation on some class of ppc32; what was the slowest system tested? For that matter, couldn't this hurt 4xx, 8xx(x), as well?

Heh, well, I did see any obvious degradation on my powerbook but I didn't benchmark much actually. Maybe I should, or make kernel HZ a config option.

Quoted 3 lines

Perhaps HZ should be user-configurable? Although that doesn't help distributions who'd like one kernel to work everywhere... How does Linux handle increased HZ on old i386 anyways?

I don't know :)

Quoted 6 lines
Quoted 4 lines

Not sure if I can fix any of this at this point without doing a major rewrite of the exception handling code, I suspect those CPUs don't like running in real mode and our exception handling happens mostly in that mode in ppc32...

I didn't realize CPUs go slower in real mode..?

It is the case with 970's and I think with IBM G3s, I'm not sure about Motorola G4s and other CPUs. In real mode, all memory is treated like it has the G bit set, preventing speculation (among others, I suppose prefetch gets killed too).

I's very visible on lmbench with a G5: The null syscall overhead of the ppc64 kernel is lower than the one of the ppc32 kernel despite actually running more code and saving more & bigger registers.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#7 lore raw
In reply to Benjamin Herrenschmidt
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 4 lines · Mikolaj Krzewicki

hi all,
starting 2.6.0-test9-benh3 i noticed an increased system overhead on my ibook 500. Each process eats much more cpu time than in 2.4 or even vanilla 2.6.

Quoted 8 lines

gkrellm takes 1% instead of 0.1, xmms 5% instead of ~1% and so on. I benchmarked the lot with octave and it indeed runs ~10% slower on the benh kernels.
I would use vanilla 2.6.1 but it won't sleep/wake correctly (at all). All pre-2.6.0-test9-benh3 kernels worked fast but would't sleep/wake either. I tried to isolate the patch responsible but failed.
Has anybody come across similar behaviour?
greetings, M

Quoted 4 lines

I switched to HZ=1000, the userland procps utilities (like ps and top) tend to not properly deal with that, at least earlier versions, afaik.

I suspect it's just display crap.

The other problem is that measuring CPU time just by sampling which process is running at timer interrupt is a crappy method bound to be inaccurate, espcially on very bursty workloads, or one which depend on other frequencies which beat with the timer.

Didn't somebody implemented CPU process accounting based on the timebase? Or was it for another architecture (maybe s390)?

Regards,
Gabriel

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/