Skip to contentSkip to conversation

NUMA processor numbering

DISCUSSION Stephan von Krawczynski 3 participants 4 messages Oct 3, 2013

lore
Share & tools
lore
#0 lore raw

Hello all,

I have a box with this kind of processor (0-31) and 128 GB RAM:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2486.000
cache size : 20480 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfp u pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_tim er aes xsave avx lahf_lm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 4400.12
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

Now, numactl --hardware shows this:

available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 node 0 size: 64581 MB
node 0 free: 12676 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 node 1 size: 64637 MB
node 1 free: 10660 MB
node distances:
node 0 1

  0:  10  20 
  1:  20  10 

Physically these are two processors with 8 Cores and 8 HT each.
Does the above output mean that the cores are numbered right across the two physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in "short distance" to node 0 RAM?
If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning. Is there a way to achieve this?
Please cc me when answering.

Signature · 2 lines
Regards,
Stephan
#1 lore raw
Quoted 3 lines · Stephan von Krawczynski

Does the above output mean that the cores are numbered right across the two physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in "short distance" to node 0 RAM?

...

Quoted 2 lines

If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning. Is there a way to achieve this?

Yes, use hwloc to get the pinning masks for whatever property you want (e.g. all threads in a given core, all threads in a given node, all threads that share a given L3 cache...).

http://www.open-mpi.org/projects/hwloc/

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#2 lore raw
Quoted 7 lines · On Thu, 3 Oct 2013 07:22:55 -0300 Henrique de Moraes Holschuh <[email protected]>…
Quoted 3 lines · Stephan von Krawczynski

Does the above output mean that the cores are numbered right across the two physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in "short distance" to node 0 RAM?

...

Quoted 8 lines
Quoted 2 lines

If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning. Is there a way to achieve this?

Yes, use hwloc to get the pinning masks for whatever property you want (e.g. all threads in a given core, all threads in a given node, all threads that share a given L3 cache...).

http://www.open-mpi.org/projects/hwloc/

Ok, let me re-phrase the question a bit.
Is it really possible what you see here:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2002.000
cache size : 20480 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
[...]

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 1518.000
cache size : 20480 KB
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
apicid : 32
initial apicid : 32
[...]

These are the first two in the cpu list. If you look at that they are both on core id 0, but have different physical ids. Up to now I thought that processor 1 is the HT of core id 0. But with a different physical id this would mean that they are different NUMA nodes, right? How can that be? Someone from Intel with a hint?

Signature · 2 lines
Regards,
Stephan
#3 lore raw
Quoted 37 lines · Stephan von Krawczynski

Ok, let me re-phrase the question a bit.
Is it really possible what you see here:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2002.000
cache size : 20480 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
[...]

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping : 7
microcode : 0x70d
cpu MHz : 1518.000
cache size : 20480 KB
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
apicid : 32
initial apicid : 32
[...]

Quoted 5 lines

These are the first two in the cpu list. If you look at that they are both on core id 0, but have different physical ids. Up to now I thought that processor 1 is the HT of core id 0. But with a different physical id this would mean that they are different NUMA nodes, right? How can that be? Someone from Intel with a hint?

Such "unexpected" numberings are very common. The BIOS is free to number processors in many different orders, including NUMA first, core first, HT first, etc.

Having the two hyperthreads of the first core physically numbered 0 and 1 doesn't seem very common on current Intel platforms. Most Xeon E5 machines I've seen put the second hyperthreads of all cores at the end of range. But there's no guarantee about this.

Use hwloc just like Henrique said, it will take care of virtually reordering objects in a logical manner.

Brice