Itanium2@900MHz slower than alpha@666MHz ?
Hello,
I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org
However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.
Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.
Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when QPROF_HW_EVENT is set).
Thanks a lot,
Ionut
Signature · 7 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do."
Quoted 7 lines · Ionut Georgescu
I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org
Quoted 5 lines
However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.
This strikes me as possibly being a cache size thing. Do you have the 1.5MB cache or 3MB cache version of the 900MHz Itanium? The DS10 specs I found at
http://www.spec.org/cpu2000/results/res2000q3/cpu2000-20000630-00134.asc
say it has a 2MB cache, so if your data set fits in a 2MB cache and not in a 1.5MB cache, that would be a possible cause.
What kind of fluctuations do you see between runs? Linux doesn't do cache-colouring, so high variation between runs could indicate a close-to-edge-of-cache scenario.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Yes, it is a 1.5MB cache CPU.
But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.
The variations are within +-0.02s between runs (for the 32x32 case).
Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.
Thanks,
Ionut
Quoted 30 lines · Matthew Wilcox
Quoted 7 lines · Ionut Georgescu
I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org
Quoted 5 lines
However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.
This strikes me as possibly being a cache size thing. Do you have the 1.5MB cache or 3MB cache version of the 900MHz Itanium? The DS10 specs I found at
http://www.spec.org/cpu2000/results/res2000q3/cpu2000-20000630-00134.asc
say it has a 2MB cache, so if your data set fits in a 2MB cache and not in a 1.5MB cache, that would be a possible cause.What kind of fluctuations do you see between runs? Linux doesn't do cache-colouring, so high variation between runs could indicate a close-to-edge-of-cache scenario.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Signature · 7 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do."
Quoted 8 lines · Ionut Georgescu
Yes, it is a 1.5MB cache CPU.
But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.
Quoted 4 lines
The variations are within +-0.02s between runs (for the 32x32 case).
Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.
I don't know about testing cache bandwidth, but /proc/pal/cpu0/cache_info gives various stats on how the different levels of cache play together on ia64.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 2 lines · Ionut Georgescu
Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.
lmbench I think has a section for this.
grant
Ionut,
Quoted 8 lines · Ionut Georgescu
Yes, it is a 1.5MB cache CPU.
But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.
Another thing to check is the syslog. Check to see if your program is not getting "floating point software assist" fault. That can slow you down significantly.
Signature · 1 line
-Stephane
It wasn't clear from the description whether you actually turned on profile guided optimization with electron compiler. It is a two pass compilation, once with -prof_gen to generate execution profile and then once with -prof_use to complete PGO optimization.
One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting that break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles in the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.
See electron compiler user's guide for PGO methodology:
http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx
.pdf
Cycle accounting is described in Intel Itanium Software developer's manual.
- Ken
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ionut Georgescu
Sent: Friday, October 24, 2003 8:15 AM
To: [email protected]
Subject: Itanium2@900MHz slower than alpha@666MHz ?
Hello,
I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org
However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.
Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.
Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when QPROF_HW_EVENT is set).
Thanks a lot,
Ionut
Signature · 13 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do." - To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Stephane Eranian
Stephane> Ionut, On Fri, Oct 24, 2003 at 06:26:55PM +0200, Ionut
Quoted 7 lines · Stephane> Georgescu wrote:
Quoted 7 lines
Yes, it is a 1.5MB cache CPU.
But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.
Stephane> Another thing to check is the syslog. Check to see if your Stephane> program is not getting "floating point software assist" Stephane> fault. That can slow you down significantly.
Note that on Red Hat you'd have to raise the dmesg level above WARNING to see such messages (dmesg -n8 should do the trick).
--david
Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And the name of the .dpy file is rather related to the pid of the process than to the name of the program.
Quoted 11 lines · Kenneth W
It wasn't clear from the description whether you actually turned on profile guided optimization with electron compiler. It is a two pass compilation, once with -prof_gen to generate execution profile and then once with -prof_use to complete PGO optimization.
One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting that break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles in the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.
Are qprof and pfmon enough to do this ?
Quoted 4 lines
See electron compiler user's guide for PGO methodology:
http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx
Is this the same for 7.x ?
Quoted 4 lines
Cycle accounting is described in Intel Itanium Software developer's manual.
Thank you for the info. I'll try to find the bottleneck.
Ionut
Quoted 52 lines
- Ken
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ionut Georgescu
Sent: Friday, October 24, 2003 8:15 AM
To: [email protected]
Subject: Itanium2@900MHz slower than alpha@666MHz ?Hello,
I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org
However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.
Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.
Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when
QPROF_HW_EVENTis set).Thanks a lot,
IonutSignature · 13 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do." - To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 7 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do."
Quoted 3 lines · Ionut Georgescu
Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And the name of the .dpy file is rather related to the pid of the process than to the name
Sorry, meant the .dyn file.
Signature · 7 lines
*************** * Ionut Georgescu * http://www.physik.tu-cottbus.de/~george/ * Registered Linux User #244479 * * "In Windows you can do everything Microsoft wants you to do; in Unix you * can do anything the computer is able to do."
Quoted 10 lines · Ionut Georgescu
Quoted 2 lines
Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And
the name of the
Quoted 1 line
.dpy file is rather related to the pid of the process than
to the name
Sorry, meant the .dyn file.
You need to use profmerge to convert *.dyn into pgopti.dpi information. Name of the *.dyn file is related to the pid because it will enable collecting information from multiple/parallel runs.
thanks,
suresh
Quoted 2 lines
Quoted 2 lines
One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting
that
Quoted 2 lines
Quoted 2 lines
break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles
in
Quoted 4 lines
Quoted 2 lines
the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.
Are qprof and pfmon enough to do this ?
Yes.
Quoted 2 lines
Quoted 2 lines
See electron compiler user's guide for PGO methodology:
http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx
Quoted 3 lines
Quoted 1 line
Is this the same for 7.x ?
I think so.
- Ken