Odd performance results

Hello!

So I ran a quick benchmark which showed stair-step results. I immediately thought "Ah, this is due to CPU 0 and 1, 2 and 3, 4 and 5, and 6 and 7 being threads in a core." Then I thought "Wait, this is an x86!" Then I dumped out cpu*/topology/thread_siblings_list, getting the following:

cpu0/topology/thread_siblings_list: 0-1
cpu1/topology/thread_siblings_list: 0-1
cpu2/topology/thread_siblings_list: 2-3
cpu3/topology/thread_siblings_list: 2-3
cpu4/topology/thread_siblings_list: 4-5
cpu5/topology/thread_siblings_list: 4-5
cpu6/topology/thread_siblings_list: 6-7
cpu7/topology/thread_siblings_list: 6-7

Is this now expected behavior or a fluke of my particular laptop? Here is hoping for expected behavior, as it makes NUMA locality the default for a great many workloads.

Enlightenment?

Thanx, Paul

I'm guessing this is an AMD bulldozer like machine?

Sent from my Android device with K-9 Mail. Please excuse my brevity.

/proc/cpuinfo thinks otherwise:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
stepping : 3
microcode : 0x1c
cpu MHz : 800.000
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid
bogomips : 4988.59
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

Thanx, Paul

Weird, I've never seen an Intel box do that before... hpa, any idea? or is this just one weird BIOS.

;-)

It is a Lenovo W541 laptop, for whatever that might be worth. Roughly on year old.

Thanx, Paul

Well, the obvious thing here is that CPUs 0-1, 2-3, 4-5, and 6-7 *are* indeed threads in a core... Intel x86 products have supported multithreading since the Pentium 4. So the "wait, this is an x86!" bit is strange to me.

The CPU in question (and /proc/cpuinfo should show this) has four cores with a total of eight threads. The "siblings" and "cpu cores" fields in /proc/cpuinfo should show the same thing. So I am utterly confused about what is unexpected here?

Also, you mentioned absolutely nothing about what kind of benchmark it was, or what the "stairstepping" results imply, so it doesn't really make it any easier...

-hpa

My prior experience with Intel x86 systems led me to expect that the hardware-thread pairs would instead be 0 and 4, 1 and 5, 2 and 6, and 3 and 7. This would result in a graph with a two-segment line, having higher slope for the lower-numbered CPUs and a lower slope for the higher-numbered CPUs, and I have in fact seen this behavior on older Intel x86 systems. See for example slides 64-67 of:

http://www.rdrop.com/users/paulmck/scalability/paper/Updates.2016.06.05a.TUDresden.pdf

But don't get me wrong, I do very much prefer the CPU-numbering approach that my laptop uses, where the hardware threads in a given core have consecutive numbers.

The benchmark was a POSIX-threads multithreaded benchmark with each thread repeatedly searching a small linked list, which should fit into the nearest-to-CPU cache. The "stairstepping" results suggest to me that a no-cache-miss pointer-following workload allows a single hardware thread to consume most of a given core's relevant hardware resources, at least on this particular chip. Which is fine -- this sort of thing always has been workload-specific.

If you want to see an example plot, take a look at:

CodeSamples/defer/perf-rcu-qsbr.eps

within:

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git

Thanx, Paul

Typically threads are enumerated differently on Intel parts. Namely:

cpu_id = code_id + nr_cores * smt_id

which gives, for a 4 core, 2 thread part:

	0-3: core 0-3, smt0
	4-7: core 0-3, smt1

My Core i7-2600k for example has:

$ cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list
0,4
1,5
2,6
3,7
0,4
1,5
2,6
3,7

The ordering Paul has, namely 0,1 for core0,smt{0,1} is not something I've ever seen on an Intel part. AMD otoh does enumerate their CMT stuff like what Paul has.

Paul's isn't unique:
cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list 0-1
0-1
2-3
2-3

i7-3520M CPU @ 2.90GHz (Dual core with hyperthread, Thinkpad t530, fedora 24)

Dave

 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Glad that it is not just me! ;-)

Thanx, Paul

Got one too:

aldebaran:~> grep -i . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -t u -k +3 -n /sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0-1 /sys/devices/system/cpu/cpu1/topology/thread_siblings_list:0-1 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2-3 /sys/devices/system/cpu/cpu3/topology/thread_siblings_list:2-3 /sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4-5 /sys/devices/system/cpu/cpu5/topology/thread_siblings_list:4-5 /sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6-7 /sys/devices/system/cpu/cpu7/topology/thread_siblings_list:6-7 /sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8-9 /sys/devices/system/cpu/cpu9/topology/thread_siblings_list:8-9 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10-11 /sys/devices/system/cpu/cpu11/topology/thread_siblings_list:10-11 /sys/devices/system/cpu/cpu12/topology/thread_siblings_list:12-13 /sys/devices/system/cpu/cpu13/topology/thread_siblings_list:12-13 /sys/devices/system/cpu/cpu14/topology/thread_siblings_list:14-15 /sys/devices/system/cpu/cpu15/topology/thread_siblings_list:14-15

Thanks,

Ingo

Yeah, they are 'interleaved' at the thread/core level - I suppose to 'mix' them on OS schedulers that don't know about SMT.

(Fortunately this interleaving is not done across NUMA domains.)

Btw., this command will print out the mappings in order even on larger systems and shows the CPU # as well:

 $ grep -i . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -t u -k +3 -n

/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,60 /sys/devices/system/cpu/cpu1/topology/thread_siblings_list:1,61 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2,62 /sys/devices/system/cpu/cpu3/topology/thread_siblings_list:3,63 /sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4,64 /sys/devices/system/cpu/cpu5/topology/thread_siblings_list:5,65 /sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6,66 /sys/devices/system/cpu/cpu7/topology/thread_siblings_list:7,67 /sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8,68 /sys/devices/system/cpu/cpu9/topology/thread_siblings_list:9,69 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10,70 /sys/devices/system/cpu/cpu11/topology/thread_siblings_list:11,71
...
/sys/devices/system/cpu/cpu116/topology/thread_siblings_list:56,116 /sys/devices/system/cpu/cpu117/topology/thread_siblings_list:57,117 /sys/devices/system/cpu/cpu118/topology/thread_siblings_list:58,118 /sys/devices/system/cpu/cpu119/topology/thread_siblings_list:59,119

That's more the natural 'direct' mapping from CPU internal topology to CPU id:
what's close to each other physically is close to each other in the CPU id space as well.

Thanks,

Ingo

But does it correctly reflect the hardware? That seems to be the real question...

  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Its just enumeration afaict. But its weird that this changed. Some BIOS team somewhere changed things and I want to know why (and how widespread this is).

Here is what that gets me on the x86 test system I usually use:

/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,32 /sys/devices/system/cpu/cpu1/topology/thread_siblings_list:1,33 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2,34 /sys/devices/system/cpu/cpu3/topology/thread_siblings_list:3,35 /sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4,36 /sys/devices/system/cpu/cpu5/topology/thread_siblings_list:5,37 /sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6,38 /sys/devices/system/cpu/cpu7/topology/thread_siblings_list:7,39 /sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8,40 /sys/devices/system/cpu/cpu9/topology/thread_siblings_list:9,41 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10,42 /sys/devices/system/cpu/cpu11/topology/thread_siblings_list:11,43

[ . . . ]

/sys/devices/system/cpu/cpu56/topology/thread_siblings_list:24,56 /sys/devices/system/cpu/cpu57/topology/thread_siblings_list:25,57 /sys/devices/system/cpu/cpu58/topology/thread_siblings_list:26,58 /sys/devices/system/cpu/cpu59/topology/thread_siblings_list:27,59 /sys/devices/system/cpu/cpu60/topology/thread_siblings_list:28,60 /sys/devices/system/cpu/cpu61/topology/thread_siblings_list:29,61 /sys/devices/system/cpu/cpu62/topology/thread_siblings_list:30,62 /sys/devices/system/cpu/cpu63/topology/thread_siblings_list:31,63

On my laptop:

/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0-1 /sys/devices/system/cpu/cpu1/topology/thread_siblings_list:0-1 /sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2-3 /sys/devices/system/cpu/cpu3/topology/thread_siblings_list:2-3 /sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4-5 /sys/devices/system/cpu/cpu5/topology/thread_siblings_list:4-5 /sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6-7 /sys/devices/system/cpu/cpu7/topology/thread_siblings_list:6-7

Agreed!

Thanx, Paul