Skip to contentSkip to conversation

HPL benchmarking linux kernel for shared memory performance.

DISCUSSION Allan Menezes 3 participants 3 messages Mar 16, 2008

lore
Share & tools
lore
#0 lore raw

Hi,
I eliminated in which kernel verion this begins to happen. I am
benchmarking a single node with hpl and openmpi beta 1.3 and gotoblas v1.24
for personal noncommercial reasons.
I tried a single node with the command $ mpirun -np 1 ./xhpl
and with kernel ver 2.6.23.14 i get over 38Gflops
but with kernel ver 2.6.23.15 compiling with the same .config i get 7.xx Gflops which is 1/5th that of the other kernel.
Keep in mind that only the kernels have changed from kernel.org not any hardware or anything else as all else is same! but the performace drops to 1/5th There is no network involved in this single node quad core intel test but just shared memory.
So the shared memory or smp performance of the newer kernels is far far worse than upto 2.6.23.14!
Even with 2.6.25.rc5 the performance is degraded!
Can you please help me find why this is occurring? Please advise!
mY set up Quad core Q6600 intel overclocked stably to 2.88 GHZ , 6 gig ddr2 800 mhz dual channel ram,
Allan Menezes

#1 lore raw
Quoted 12 lines · Allan Menezes

Hi,
I eliminated in which kernel verion this begins to happen. I am
benchmarking a single node with hpl and openmpi beta 1.3 and gotoblas v1.24 for personal noncommercial reasons.
I tried a single node with the command $ mpirun -np 1 ./xhpl and with kernel ver 2.6.23.14 i get over 38Gflops
but with kernel ver 2.6.23.15 compiling with the same .config i get 7.xx Gflops which is 1/5th that of the other kernel.
Keep in mind that only the kernels have changed from kernel.org not any hardware or anything else as all else is same! but the performace drops to 1/5th
There is no network involved in this single node quad core intel test but

Quoted 8 lines

just shared memory.
So the shared memory or smp performance of the newer kernels is far far worse than upto 2.6.23.14!
Even with 2.6.25.rc5 the performance is degraded!
Can you please help me find why this is occurring? Please advise! mY set up Quad core Q6600 intel overclocked stably to 2.88 GHZ , 6 gig ddr2 800 mhz dual channel ram,
Allan Menezes

If you want to speed up bug resolution, you can bisect it to only one patch with git (google for git bisect) on this tree:
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.23.y.git

ps: please post your .config and dmesg output

Marcin

#2 lore raw
In reply to Allan Menezes
Quoted 6 lines · Allan Menezes

Hi,
I eliminated in which kernel verion this begins to happen. I am
benchmarking a single node with hpl and openmpi beta 1.3 and gotoblas v1.24 for personal noncommercial reasons.
I tried a single node with the command $ mpirun -np 1 ./xhpl and with kernel ver 2.6.23.14 i get over 38Gflops

Quoted 8 lines

but with kernel ver 2.6.23.15 compiling with the same .config i get 7.xx Gflops which is 1/5th that of the other kernel.
Keep in mind that only the kernels have changed from kernel.org not any hardware or anything else as all else is same! but the performace drops to 1/5th There is no network involved in this single node quad core intel test but just shared memory.
So the shared memory or smp performance of the newer kernels is far far worse than upto 2.6.23.14!
Even with 2.6.25.rc5 the performance is degraded!
Can you please help me find why this is occurring? Please advise!
mY set up Quad core Q6600 intel overclocked stably to 2.88 GHZ , 6 gig ddr2 800 mhz dual channel ram,
Allan Menezes

Make sure that it is actually using multiple processes or threads, if it was only using a single thread the number could be that low, and a number of really trivial things being wrong could cause it to only be using one thread. "top -H" will show threads.

Is the HPL executable identical (the same executable should work just fine so long as the underlying HW is the same, so no need to recompile/relink xhpl)?

And you might also try Intel's MKL with hpl and see if that is better or worse than gotoblas (this probably won't affect the problem, but it would be good to know if the Intel MKL is better or worse than gotoblas).

Also, as the other person mentioned use git bisect and figure out which patch did it, if you can identify one patch someone can probably figure it out.

Roger