How to detect multi-core and/or HT-enabled CPUs in 2.4.x and 2.6.x kernels
Hi, (please CC on replies, thanks)
for the ganglia project (http://ganglia.sourceforge.net/) we are
trying to find a heuristics to determine the number of physical CPU "cores" as opposed to virtual processors added by enabling HT. The method should work on 2.4 and 2.6 kernels.
So far it seems that looking at the "physical id", "core id" and "cpu
cores" of /proc/cpuinfo is the way to go.
In 2.6 I would try to find the distinct "physical id"s and and sum
up the corresponding "cpu cores". The question is whether this would work for 2.4 based systems.
Does anybody recall when the "physical id", "core id" and "cpu cores"
were added to /proc/cpuinfo ?
Cheers
Martin
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 6 lines · Martin Knoblauch
Hi, (please CC on replies, thanks)
for the ganglia project (http://ganglia.sourceforge.net/) we are
trying to find a heuristics to determine the number of physical CPU "cores" as opposed to virtual processors added by enabling HT. The method should work on 2.4 and 2.6 kernels.
I have a counter question for you.. what are you trying to do with the "these two are SMT sibblings" information ?
Because I suspect "HT" is the wrong level of detection for what you really want to achieve....
If you want to decide "shares caches" then at least 2.6 kernels directly export that (and HT is just the wrong way to go about this).
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 3 lines · Martin Knoblauch
So far it seems that looking at the "physical id", "core id" and "cpu cores" of
/proc/cpuinfois the way to go.
Possibly, but it does not catch all cases.
$grep '"physical id' /erk/kernel/linux-2.6.20-rc2/ -r
returns exactly three lines, for /erk/kernel/linux-2.6.20-rc2/arch/i386/kernel/cpu/proc.c /erk/kernel/linux-2.6.20-rc2/arch/ia64/kernel/setup.c /erk/kernel/linux-2.6.20-rc2/arch/x86_64/kernel/setup.c
So what'cha doing about, say, sparc64? Here is its procinfo of a standard SMP one:
15:31 ares:~ # cat /proc/cpuinfo
cpu : TI UltraSparc II (BlackBird)
fpu : UltraSparc II integrated FPU
prom : OBP 3.30.0 2003/11/11 10:37
type : sun4u
ncpus probed : 2
ncpus active : 2
D$ parity tl1 : 0 I$ parity tl1 : 0 Cpu0Bogo : 800.49 Cpu0ClkTck : 0000000017d78400 Cpu1Bogo : 800.05 Cpu1ClkTck : 0000000017d78400 MMU Type : Spitfire State: CPU0: online CPU1: online
Quoted 6 lines
In 2.6 I would try to find the distinct "physical id"s and and sum up the corresponding "cpu cores". The question is whether this would work for 2.4 based systems.
Does anybody recall when the "physical id", "core id" and "cpu cores" were added to
/proc/cpuinfo?
Why don't you check it out? 2.4.34 only has the "physical id" string for x86_64. It does not seem to have CONFIG_SCHED_SMT at all. (Time to leave the dead horse alone.)
-`J'
Quoted 12 lines · --- Arjan van de Ven
Quoted 6 lines · Martin Knoblauch
Hi, (please CC on replies, thanks)
for the ganglia project (http://ganglia.sourceforge.net/) we are
trying to find a heuristics to determine the number of physical CPU "cores" as opposed to virtual processors added by enabling HT. The method should work on 2.4 and 2.6 kernels.I have a counter question for you.. what are you trying to do with the
"these two are SMT sibblings" information ?
Quoted 7 lines
Because I suspect "HT" is the wrong level of detection for what you really want to achieve....
If you want to decide "shares caches" then at least 2.6 kernels directly
export that (and HT is just the wrong way to go about this).
Hi Arjan,
one piece of information that Ganglia collects for a node is the
"number of CPUs", originally meaning "physical CPUs". With the introduction of HT and multi-core things are a bit more complex now. We have decided that HT sibblings do not qualify as "real" CPUs, while multi-cores do.
Currently we are doing "sysconf(_SC_NPROCESSORS_ONLN)". But this
includes both physical and virtual (HT) cores. We are looking for a method that only shows "real iron" and works on 2.6 and 2.4 kernels. Whether this has any practial valus is a completely different question.
Cheers
Martin
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 4 lines
one piece of information that Ganglia collects for a node is the
"number of CPUs", originally meaning "physical CPUs".
Ok I was afraid of that.
Quoted 4 lines
With the
introduction of HT and multi-core things are a bit more complex now. We have decided that HT sibblings do not qualify as "real" CPUs, while multi-cores do.
I think that decision is a mistake, and is probably based on experiences with the first generation of HT capable Pentium 4 processors.
The original p4 HT to a large degree suffered from a too small cache that now was shared. SMT in general isn't per se all that different in performance than dual core, at least not on a fundamental level, it's all a matter of how many resources each thread has on average. With dual core sharing the cache for example, that already is part HT. Putting the "boundary" at HT-but-not-dual-core is going to be highly artificial and while it may work for the current hardware, in general it's not a good way of separating things (just look at the PowerPC processors, those are highly SMT as well), and I suspect that your distinction is just going to break all the time over the next 10 years ;) Or even today on the current "large cache" P4 processors with HT it already breaks. (just those tend to be the expensive models so more rare)
I would strongly urge you to reconsider this decision; if you want to show "sockets" that sounds reasonable, or even if you want to do it on the "bus sharing" level (FSB/HT), but HT.. just sounds wrong.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 4 lines · Arjan van de Ven
The original p4 HT to a large degree suffered from a too small cache that now was shared. SMT in general isn't per se all that different in performance than dual core, at least not on a fundamental level, it's all a matter of how many resources each thread has on average. With dual
Quoted 9 lines
core sharing the cache for example, that already is part HT. Putting the "boundary" at HT-but-not-dual-core is going to be highly artificial and while it may work for the current hardware, in general it's not a good way of separating things (just look at the PowerPC processors, those are highly SMT as well), and I suspect that your distinction is just going to break all the time over the next 10 years ;) Or even today on the current "large cache" P4 processors with HT it already breaks. (just those tend to be the expensive models so more rare)
If I run two threads that are doing only calculations and very little or no IO at all on the same socket will modern HT and dual core be the same (or close) performance wise?
--
Gleb.
Quoted 4 lines
If I run two threads that are doing only calculations and very little or no IO at all on the same socket will modern HT and dual core be the same (or close) performance wise?
it depends on how cache/memory bandwidth sensitive your calculation is.... if your calculation is memory bandwidth sensitive then they're the same. If your calculation is very sensitive to have the dataset fit in cache.. it's another different ballgame again, because then it depends if the cores in your dual core share the cache or not.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 21 lines · --- Gleb Natapov
Quoted 1 line · Arjan van de Ven
The original p4 HT to a large degree suffered from a too small
cache
Quoted 1 line
that now was shared. SMT in general isn't per se all that different
in
Quoted 1 line
performance than dual core, at least not on a fundamental level,
it's
Quoted 1 line
all a matter of how many resources each thread has on average. With
dual
Quoted 1 line
core sharing the cache for example, that already is part HT.
Putting the
Quoted 1 line
"boundary" at HT-but-not-dual-core is going to be highly artificial
and
Quoted 1 line
while it may work for the current hardware, in general it's not a
good
Quoted 1 line
way of separating things (just look at the PowerPC processors,
those are
Quoted 1 line
highly SMT as well), and I suspect that your distinction is just
going
Quoted 1 line
to break all the time over the next 10 years ;) Or even today on
the
Quoted 9 lines
Quoted 1 line
current "large cache" P4 processors with HT it already breaks.
(just
Quoted 2 lines
those tend to be the expensive models so more rare)
If I run two threads that are doing only calculations and very little or no
IO at all on the same socket will modern HT and dual core be the same (or close) performance wise?
Hi Gleb,
this is a real interesting question. Ganglia is coming [originally]
from the HPC side of computing. At least in the past HT as implemented on XEONs did help a lot. Running two CPU+memory-bandwith intensive processes on the same physical CPU would at best result in a 50/50 performance split. So, knowing how many "real" CPUs are in a system is interesting to us.
Other workloads (like lots of java threads doing mixed IO and CPU
stuff) of course can benefit from HT.
Cheers
Martin
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 6 lines · --- Gleb Natapov
Quoted 1 line
If I run two threads that are doing only calculations and very little or no
IO at all on the same socket will modern HT and dual core be the same (or close) performance wise?
actually I wanted to write that "HT as implemented on XEONs did not
help a lot for HPC workloads in the past"....
Cheers
Martin
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 8 lines
this is a real interesting question. Ganglia is coming [originally]
from the HPC side of computing. At least in the past HT as implemented on XEONs did help a lot. Running two CPU+memory-bandwith intensive processes on the same physical CPU would at best result in a 50/50 performance split. So, knowing how many "real" CPUs are in a system is interesting to us.
but this 50/50 split is most likely because of either a cache or a bandwidth bottleneck, at which point "HT" is the wrong measure.
(also if you go to things like SUN's Niagara CPU then it's even more clear that SMT isn't the right measure for performance capability)
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 4 lines
actually I wanted to write that "HT as implemented on XEONs did not
help a lot for HPC workloads in the past"....
btw this is exactly the problem I am trying to point out: ".. as implemented in generation XYZ model ABC of processor DEF". that's going to be really fragile and in fact won't work even for processors you can buy today (power5 and sparc niagara for example, and depending on the workload, even on todays 16Mb cache Xeons).
once your program (and many others) have such a check, then the next step will be pressure on the kernel code to "fake" the old situation when there is a processor where <vague criteria of the day> no longer holds. It's basically a road to madness :-(
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 4 lines · In article <[email protected]> you wrote:
once your program (and many others) have such a check, then the next step will be pressure on the kernel code to "fake" the old situation when there is a processor where <vague criteria of the day> no longer holds. It's basically a road to madness :-(
I agree that for HPC sizing a benchmark with various levels of parallelity are better. The question is, if the code in question only is for inventory reasons. In that case I would do something like x sockets, y cores and z cm threads.
Bernd
Quoted 1 line
In article <1167235772.3281.3977.camel@xxxxxxxxxxxxxxxxxxxxx> you
Quoted 3 lines · wrote:
Quoted 3 lines
once your program (and many others) have such a check, then the next step will be pressure on the kernel code to "fake" the old situation when there is a processor where <vague criteria of the day> no
longer
Quoted 8 lines
Quoted 1 line
holds. It's basically a road to madness :-(
I agree that for HPC sizing a benchmark with various levels of parallelity are better. The question is, if the code in question only is for inventory reasons. In that case I would do something like x sockets, y cores and z cm threads.
Bernd
For sizing purposes, doing benchmarks is the only way. For the purpose
of Ganglia the sockets/cores/threads info is purely for inventory. And we are likely going to add the new information to our metrics.
But - we still need to find a way to extract the infor :-)
Cheers
Martin
PS: I have likely killed the CC this time. Sorry.
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 5 lines · Martin Knoblauch
For sizing purposes, doing benchmarks is the only way. For the purpose
of Ganglia the sockets/cores/threads info is purely for inventory. And we are likely going to add the new information to our metrics.But - we still need to find a way to extract the infor :-)
Only the 2.4 x86_64 kernels are exporting limited info("physical id", "siblings") through /proc/cpuinfo.
Some of the distos based on 2.4 kernels have the complete topology (physical id, core id, cpu cores, siblings) exported through /proc/cpuinfo.
thanks,
suresh