Skip to contentSkip to conversation

gcc3.2 v 2.95.3 (contest and linux-2.5.38)

BENCHMARK Con Kolivas 6 participants 12 messages Sep 23, 2002 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

No Load:

Kernel                  Time            CPU
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%

Process Load:

Kernel                  Time            CPU
2.5.38                  71.60           95%
2.5.38-gcc32            112.98          91%

IO Half Load:

Kernel                  Time            CPU
2.5.38                  81.26           90%
2.5.38-gcc32            168.25          70%

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

Mem Load:

Kernel                  Time            CPU
2.5.38                  104.22          70%
2.5.38-gcc32            153.00          74%

Both kernels had identical configs and contest was run using gcc2.95.3 (I can't see how this would affect it but tell me if you think it will). The only difference was the original 2.5.38 was compiled with gcc2.95.3 and -gcc32 was compiled with gcc3.2

Note the lower performance even with noload, and the totally abysmal performance under full load. See the logs below for the page fault differences too.

Full logs follow:
2.5.38:
noload Time: 68.25 CPU: 99% Major Faults: 204613 Minor Faults: 255906 process_load Time: 71.60 CPU: 95% Major Faults: 204019 Minor Faults: 255238 io_halfmem Time: 81.26 CPU: 90% Major Faults: 204019 Minor Faults: 255325 Was writing number 4 of a 112Mb sized io_load file after 90 seconds io_fullmem Time: 170.21 CPU: 42% Major Faults: 204019 Minor Faults: 255272 Was writing number 6 of a 224Mb sized io_load file after 194 seconds mem_load Time: 104.22 CPU: 70% Major Faults: 204120 Minor Faults: 256271

2.5.38-gcc32:
noload Time: 103.03 CPU: 99% Major Faults: 237750 Minor Faults: 439982 process_load Time: 112.98 CPU: 91% Major Faults: 236958 Minor Faults: 439030 io_halfmem Time: 168.25 CPU: 70% Major Faults: 236958 Minor Faults: 439261 Was writing number 16 of a 112Mb sized io_load file after 202 seconds io_fullmem Time: 1405.25 CPU: 8% Major Faults: 237199 Minor Faults: 440233 Was writing number 59 of a 224Mb sized io_load file after 1496 seconds mem_load Time: 153.00 CPU: 74% Major Faults: 237863 Minor Faults: 440524

These are disturbing to me.

Comments?
Con.

#1 lore raw
Quoted 4 lines · Con Kolivas

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

Ugh?? Something is _seriously_ messed up here.

The CPU utilization is only 8% but the time is nearly 10x worse. You sure the only difference was the compiler? I could think gcc-3.2 makes some poorer choices wrt code optimization, but nothing feasible can come to mind that would produce such terrible results.

Also, I believe RedHat is compiling their kernel in 8.0 with gcc-3.2, unless they reintroduced kgcc. Surely that are not seeing these abysmal numbers.

Robert Love

#2 lore raw

Quoting Robert Love <[email protected]>:

Quoted 8 lines · Con Kolivas
Quoted 5 lines

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

Ugh?? Something is _seriously_ messed up here.

Agreed!

Quoted 4 lines

The CPU utilization is only 8% but the time is nearly 10x worse. You sure the only difference was the compiler? I could think gcc-3.2 makes some poorer choices wrt code optimization, but nothing feasible can come to mind that would produce such terrible results.

Absolutely certain. I'm shaking from the results still... hard to type...

Quoted 3 lines

Also, I believe RedHat is compiling their kernel in 8.0 with gcc-3.2, unless they reintroduced kgcc. Surely that are not seeing these abysmal numbers.

contest is a new benchmark. Noone has ever done anything like this before so it wouldn't have shown up in ordinary benchmarks. Mandrake has done the same with LM9.0 I believe

Con

#3 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Con Kolivas
Quoted 2 lines

Ugh?? Something is _seriously_ messed up here.

The most important question to ask here is: What flags did you compile both ?!? I wouldn't count on the flags that were designed for gcc 2.95 to be any good for 3.2... Could the original poster comment on this ?

Any GCC maintainers on this list to comment ? Is there any set of flags to be passed to gcc 3.2 to replicate 2.95 behaviour ? I wouldn't rule out gcc 3.2 having a totaly different set of optimizations geared towards user space C++. Again, any gcc maintainers comments ?!?

Since most of the code in gcc is for C++ most of the changes in gcc should have been geared towards C++ (yes - quite a monstrous language). It seems to me that the changes in C compilation between 2.95 and 3.2 should be minor EXCEPT in terms of C optimization. Can anyone with assembly knowledge take apart two identical drivers and see the better machine code produced by 2.95 as compared to 3.2 ? If so - can this be reported to the gcc folk ?

It seems to me that the difference is so huge that even user space applications could show the difference. I suggest compiling a large C program (emphasis on the C) in user space and doing the comparison... I would guess that this should have been done by the gcc folk but because of the hideousness of the C++ language I would guess that they mostly concentrated on C++ and didn't bother to benchmark regular C optimization. This is quite awful as the bulk of lower level open source code is in C and not C++ so this kind of test has a lot of meaning for any distribution that is going to be based on gcc 3.2...

If this benchmark turns out to be right then it seems to me that the only conclusion is that the gcc folk let their interest in aesoteric features of C++ (which has about 1/2 a billion of those) override the basic need for strong C optimization. Yes - it now seems that the C++ language (which is quite an abomination in terms of engineering and the KISS principle) is actually hurting open source (which has been my conclusion for quite some time).

Mark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9jvWZxlxDIcceXTgRAnxpAKDYz61RWvceqD13Z889rwtZLOaomwCggmmj ixt6x1e+zXewlrYCCHbiN9Y=
=snOl
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 35 lines · Mark Veltzer

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Con Kolivas
Quoted 2 lines

Ugh?? Something is _seriously_ messed up here.

The most important question to ask here is: What flags did you compile both ?!? I wouldn't count on the flags that were designed for gcc 2.95 to be any good for 3.2... Could the original poster comment on this ?

Any GCC maintainers on this list to comment ? Is there any set of flags to be passed to gcc 3.2 to replicate 2.95 behaviour ? I wouldn't rule out gcc 3.2 having a totaly different set of optimizations geared towards user space C++. Again, any gcc maintainers comments ?!?

Since most of the code in gcc is for C++ most of the changes in gcc should have been geared towards C++ (yes - quite a monstrous language). It seems to me that the changes in C compilation between 2.95 and 3.2 should be minor EXCEPT in terms of C optimization. Can anyone with assembly knowledge take apart two identical drivers and see the better machine code produced by 2.95 as compared to 3.2 ? If so - can this be reported to the gcc folk ?

It seems to me that the difference is so huge that even user space applications could show the difference. I suggest compiling a large C program (emphasis on the C) in user space and doing the comparison... I would guess that this should have been done by the gcc folk but because of the hideousness of the C++ language I would guess that they mostly concentrated on C++ and didn't bother to benchmark regular C optimization. This is quite awful as the bulk of lower level open source code is in C and not C++ so this kind of test has a lot of meaning for any distribution that is going to be based on gcc 3.2...

Quoted 7 lines

If this benchmark turns out to be right then it seems to me that the only conclusion is that the gcc folk let their interest in aesoteric features of C++ (which has about 1/2 a billion of those) override the basic need for strong C optimization. Yes - it now seems that the C++ language (which is quite an abomination in terms of engineering and the KISS principle) is actually hurting open source (which has been my conclusion for quite some time).

Mark, if you followed the GCC development process you'd realize that all of your above ranting about C++ is completely unfounded. Most people doing performance work - and there are a good number of them - focus on language-independent optimizations and check them primarily in C.

And I've no idea what you mean by "EXCEPT in terms of C optimization". First of all you're completely wrong - 3.2 supports most of C99, which is substantial - and secondly, of course the bulk of changes in support for a language are optimization. And those are substantial too.

Signature · 2 lines
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer
#5 lore raw
In reply to Con Kolivas
Quoted 4 lines · Con Kolivas

I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

Daniel Jacobowitz, a stunning gcc hacker, suggests you retry with gcc-3.2 from CVS. Apparently some fixes have gone in since 3.2 release.

If the problem still occurs, there is either a serious problem in gcc or we have some code that is miscompiled by gcc-3.2 (perhaps not its fault but the kernel's).

Robert Love

#6 lore raw
In reply to Con Kolivas
Quoted 11 lines · Con Kolivas

I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

No Load:

Kernel                  Time            CPU
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%

Try gcc-2.91.66. It might break the 45 second mark.

Quoted 4 lines

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

The streaming write is stalling gcc's read for long enough for gcc's working set to be evicted. And the working set cannot be reestablished because the streaming write prevents it. Meltdown.

I have fixed this. Hang around.

#7 lore raw
Quoted 20 lines · Andrew Morton
Quoted 11 lines · Con Kolivas

I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

No Load:

Kernel                  Time            CPU
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%

Try gcc-2.91.66. It might break the 45 second mark.

Quoted 4 lines

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%
Quoted 5 lines

The streaming write is stalling gcc's read for long enough for gcc's working set to be evicted. And the working set cannot be reestablished because the streaming write prevents it. Meltdown.

I have fixed this. Hang around.

That sounds like a slowdown in _running_ GCC... The worrisome part of Con's post is that he was talking about which compiler he'd built the kernel with. Right?

Signature · 2 lines
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer
#8 lore raw

Quoting Daniel Jacobowitz <[email protected]>:

Quoted 26 lines · Andrew Morton
Quoted 3 lines
Quoted 2 lines · Con Kolivas

I performed contest benchmarks on kernel 2.5.38 when the kernel is

compiled with

Quoted 21 lines
Quoted 9 lines

gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

No Load:

Kernel                  Time            CPU
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%

Try gcc-2.91.66. It might break the 45 second mark.

Quoted 4 lines

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

The streaming write is stalling gcc's read for long enough for gcc's working set to be evicted. And the working set cannot be reestablished because the streaming write prevents it. Meltdown.

Quoted 5 lines
Quoted 1 line

I have fixed this. Hang around.

That sounds like a slowdown in _running_ GCC... The worrisome part of Con's post is that he was talking about which compiler he'd built the kernel with. Right?

Correct. contest was run with gcc2.95.3 only. The kernels were compiled with 2.95.3 and 3.2 respectively.

Con.

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw

Quoting Andrew Morton <[email protected]>:

Quoted 27 lines
Quoted 6 lines · Con Kolivas

Quoting Andrew Morton <[email protected]>:

Quoted 3 lines
Quoted 2 lines · Con Kolivas

Correct. contest was run with gcc2.95.3 only. The kernels were

compiled

Quoted 9 lines
Quoted 5 lines

with

Quoted 1 line

2.95.3 and 3.2 respectively.

I think you made a mistake. Please rerun. Just one data point will do.

Ok here are two points to confirm the results and their reproducibility:

No Load:

Quoted 9 lines
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%
2.5.38-gcc32a           103.47          99%

Process Load:

2.5.38                  71.60           95%
2.5.38-gcc32            112.98          91%
2.5.38-gcc32a           113.60          91%
Quoted 4 lines

beats the hell out of me. Frankly, I *still* think you made a mistake (at least on the io load thing) because the CPU time went down by a mile - it was waiting on disk all the time.

If you think I've made a mistake then you're probably correct. I'm investigating this further. Please do NOT pass judgement on these benchmarks until I completely retest everything, ensuring gcc is fixed for everything except the kernel being tested. Disregard until I have a fresh set of confirmed results.

Con.

#13 lore raw
Quoted 4 lines · Con Kolivas

If you think I've made a mistake then you're probably correct. I'm investigating this further. Please do NOT pass judgement on these benchmarks until I completely retest everything, ensuring gcc is fixed for everything except the kernel being tested. Disregard until I have a fresh set of confirmed results.

And do not forget to use gcc from CVS, they're at 3.2.1 (prerelease) right now. Or even checkout 3.3 while you're at it.

Regards,
Axel

#14 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton

Try gcc-2.91.66. It might break the 45 second mark.

Quoted 4 lines

IO Full Load:

Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%
Quoted 5 lines

The streaming write is stalling gcc's read for long enough for gcc's working set to be evicted. And the working set cannot be reestablished because the streaming write prevents it. Meltdown.

I have fixed this. Hang around.

Ehh, I was under the impression he was benchmarking kernels compiled WITH these compilers, using contest?

Your post seems to imply he was using the compilers as the benchmark. If so, I retract my previous post - I know gcc 3.x is slow as puke. I think, however, he is comparing the resulting kernels, in which case there is a serious issue at hand...

Robert Love