Skip to contentSkip to conversation

Itanium2@900MHz slower than alpha@666MHz ?

DISCUSSION Ionut Georgescu 7 participants 12 messages Oct 24, 2003

lore
Share & tools
lore
#0 lore raw

Hello,

I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org

However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.

Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.

Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when QPROF_HW_EVENT is set).

Thanks a lot,
Ionut

Signature · 7 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix you
*                can do anything the computer is able to do."
#1 lore raw
Quoted 7 lines · Ionut Georgescu

I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org

Quoted 5 lines

However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.

This strikes me as possibly being a cache size thing. Do you have the 1.5MB cache or 3MB cache version of the 900MHz Itanium? The DS10 specs I found at
http://www.spec.org/cpu2000/results/res2000q3/cpu2000-20000630-00134.asc
say it has a 2MB cache, so if your data set fits in a 2MB cache and not in a 1.5MB cache, that would be a possible cause.

What kind of fluctuations do you see between runs? Linux doesn't do cache-colouring, so high variation between runs could indicate a close-to-edge-of-cache scenario.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#2 lore raw
In reply to Ionut Georgescu

Yes, it is a 1.5MB cache CPU.

But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.

The variations are within +-0.02s between runs (for the 32x32 case).

Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.

Thanks,
Ionut

Quoted 30 lines · Matthew Wilcox
Quoted 7 lines · Ionut Georgescu

I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org

Quoted 5 lines

However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.

This strikes me as possibly being a cache size thing. Do you have the 1.5MB cache or 3MB cache version of the 900MHz Itanium? The DS10 specs I found at
http://www.spec.org/cpu2000/results/res2000q3/cpu2000-20000630-00134.asc
say it has a 2MB cache, so if your data set fits in a 2MB cache and not in a 1.5MB cache, that would be a possible cause.

What kind of fluctuations do you see between runs? Linux doesn't do cache-colouring, so high variation between runs could indicate a close-to-edge-of-cache scenario.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Signature · 7 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix you
*                can do anything the computer is able to do."
#3 lore raw
In reply to Ionut Georgescu
Quoted 8 lines · Ionut Georgescu

Yes, it is a 1.5MB cache CPU.

But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.

Quoted 4 lines

The variations are within +-0.02s between runs (for the 32x32 case).

Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.

I don't know about testing cache bandwidth, but /proc/pal/cpu0/cache_info gives various stats on how the different levels of cache play together on ia64.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#4 lore raw
In reply to Ionut Georgescu
Quoted 2 lines · Ionut Georgescu

Is there a way to test the bandwidth of the cache ? Because I think the alphas have actually 2MB of L2 cache, not L3.

lmbench I think has a section for this.

grant

#5 lore raw
In reply to Ionut Georgescu

Ionut,

Quoted 8 lines · Ionut Georgescu

Yes, it is a 1.5MB cache CPU.

But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.

Another thing to check is the syslog. Check to see if your program is not getting "floating point software assist" fault. That can slow you down significantly.

Signature · 1 line
-Stephane
#6 lore raw
In reply to Ionut Georgescu

It wasn't clear from the description whether you actually turned on profile guided optimization with electron compiler. It is a two pass compilation, once with -prof_gen to generate execution profile and then once with -prof_use to complete PGO optimization.

One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting that break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles in the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.

See electron compiler user's guide for PGO methodology:
http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx
.pdf

Cycle accounting is described in Intel Itanium Software developer's manual.

- Ken

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ionut Georgescu
Sent: Friday, October 24, 2003 8:15 AM
To: [email protected]
Subject: Itanium2@900MHz slower than alpha@666MHz ?

Hello,

I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org

However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.

Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.

Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when QPROF_HW_EVENT is set).

Thanks a lot,
Ionut

Signature · 13 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix
you
*                can do anything the computer is able to do."

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw
In reply to Ionut Georgescu
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Stephane Eranian

Stephane> Ionut, On Fri, Oct 24, 2003 at 06:26:55PM +0200, Ionut

Quoted 7 lines · Stephane> Georgescu wrote:
Quoted 7 lines

Yes, it is a 1.5MB cache CPU.

But it might be more than just cache misses. When running with a 256x256 grid, I am actually using 3 256x257 matrices, which is slightly over 1.5MB. I just made a comparison with a 32x32 grid and the difference is the same: alpha 10.294s, zx2000 14.777s . -O3 is only by 0.4s faster than -O2 in this case.

Stephane> Another thing to check is the syslog. Check to see if your Stephane> program is not getting "floating point software assist" Stephane> fault. That can slow you down significantly.

Note that on Red Hat you'd have to raise the dmesg level above WARNING to see such messages (dmesg -n8 should do the trick).

	--david
#8 lore raw
In reply to Ionut Georgescu

Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And the name of the .dpy file is rather related to the pid of the process than to the name of the program.

Quoted 11 lines · Kenneth W

It wasn't clear from the description whether you actually turned on profile guided optimization with electron compiler. It is a two pass compilation, once with -prof_gen to generate execution profile and then once with -prof_use to complete PGO optimization.

One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting that break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles in the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.

Are qprof and pfmon enough to do this ?

Quoted 4 lines

See electron compiler user's guide for PGO methodology:
http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx
.pdf

Is this the same for 7.x ?

Quoted 4 lines

Cycle accounting is described in Intel Itanium Software developer's manual.

Thank you for the info. I'll try to find the bottleneck.

Ionut

Quoted 52 lines

- Ken

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ionut Georgescu
Sent: Friday, October 24, 2003 8:15 AM
To: [email protected]
Subject: Itanium2@900MHz slower than alpha@666MHz ?

Hello,

I am puzzled about the speed of a zx2000 workstation with a 900Mhz CPU. According to the SPECfp2000 benchmarks, this workstation should be about twice as fast as a DS10 alpha workstation and according to the fftw2 benchmarks at least 50% faster (double precision, real data, 256x256 FFT transforms). I ran the fftw2 benchmark myself and I could reproduce the data on fftw.org

However, my program is about 40% slower on the zx2000 as on the alpha. It only does some Fourier transforms (fftw2, 256x256) and some matrix operations (sort of an inner product). Both fftw2 and the program have been compiled with ecc -O2 -ipo -limf. ecc is Version 7.1, Build 20030307.

Both the alpha and the Itanium2 run Debian stable and kernel 2.4.20.

Is there anything else I can do to improve performance ? I tried to some profiling (CFLAGS="-g -p -Ob0 -O0 -inline_debug_info"), but the report is missing the call-graph and a lot of other information, so that I can't trust the quality of those data. Right now I'm trying to dig my way through qprof and pfmon (for the moment qprof fails when QPROF_HW_EVENT is set).

Thanks a lot,
Ionut

Signature · 13 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix
you
*                can do anything the computer is able to do."

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 7 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix you
*                can do anything the computer is able to do."
#9 lore raw
In reply to Ionut Georgescu
Quoted 3 lines · Ionut Georgescu

Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And the name of the .dpy file is rather related to the pid of the process than to the name

Sorry, meant the .dyn file.

Signature · 7 lines
***************
* Ionut Georgescu
* http://www.physik.tu-cottbus.de/~george/
* Registered Linux User #244479
*
* "In Windows you can do everything Microsoft wants you to do; in Unix you
*                can do anything the computer is able to do."
#10 lore raw
In reply to Ionut Georgescu
Quoted 10 lines · Ionut Georgescu
Quoted 2 lines

Thanks, but I still need to learn about this kind of optimization. After running the program I don't get any .dpi file. And

the name of the

Quoted 1 line

.dpy file is rather related to the pid of the process than

to the name

Sorry, meant the .dyn file.

You need to use profmerge to convert *.dyn into pgopti.dpi information. Name of the *.dyn file is related to the pid because it will enable collecting information from multiple/parallel runs.

thanks,
suresh

#11 lore raw
In reply to Ionut Georgescu
Quoted 2 lines
Quoted 2 lines

One other neat thing about Itanium architecture is the capability of it's performance counter. It has capability to do cycle accounting

that

Quoted 2 lines
Quoted 2 lines

break-down the number of cycles that are lost due to various kinds of micro-architectural events, it is based on CPU's actual stall cycles

in

Quoted 4 lines
Quoted 2 lines

the pipeline so you can see exactly where the stall is coming from to eliminate any guess work.

Are qprof and pfmon enough to do this ?

Yes.

Quoted 2 lines
Quoted 2 lines

See electron compiler user's guide for PGO methodology:

http://www.intel.com/software/products/compilers/c60l/resources/c_ug_lnx

Quoted 3 lines
Quoted 1 line

.pdf

Is this the same for 7.x ?

I think so.

- Ken