Skip to content

RFC fix topology for multi-NUMA-node CPUs

Dave Hansen linux-arch lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest6 patches 7 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6topology: rename topology_core_cpumask() to topology_package_cpumask()arch/arm/include/asm/topology.h arch/arm64/include/asm/topology.h arch/ia64/include/asm/topology.h +14 more+21 -210 replies
2/6x86: introduce cpumask specifically for the packagearch/x86/include/asm/smp.h arch/x86/include/asm/topology.h arch/x86/kernel/smpboot.c +1 more+21 -01 reply
3/6x86: use package_map instead of core_map for sysfsarch/x86/include/asm/topology.h+1 -10 replies
4/6sched: eliminate "DIE" domain level when NUMA presentkernel/sched/core.c+2 -01 reply
5/6sched: keep MC domain from crossing nodes OR packagesarch/x86/kernel/smpboot.c+20 -60 replies
6/6sched: consolidate config optionsarch/arm/Kconfig arch/arm64/Kconfig arch/ia64/Kconfig +9 more+63 -1113 replies

Reviews trailers per patch

None of the 6 patches has a Reviewed-by or Acked-by.

PatchAravind Gopalakrishnan
SeriesT-b
1/6
2/6
3/6
4/6
5/6
6/6

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC][PATCH 0/6] fix topology for multi-NUMA-node CPUs

This is a big fat RFC. It takes quite a few liberties with the multi-core topology level that I'm not completely comfortable with.

It has only been tested lightly.

Full dmesg for a Cluster-on-Die system with this set applied, and sched_debug on the command-line is here:

	http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt

---

I'm getting the spew below when booting with Haswell (Xeon E5-2699 v3) CPUs and the "Cluster-on-Die" (CoD) feature enabled in the BIOS. It seems similar to the issue that some folks from AMD ran in to on their systems and addressed in this commit:

	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=161270fc1f9ddfc17154e0d49291472a9cdef7db

Both these Intel and AMD systems break an assumption which is being enforced by topology_sane(): a socket may not contain more than one NUMA node.

AMD special-cased their system by looking for a cpuid flag. The Intel mode is dependent on BIOS options and I do not know of a way which it is enumerated other than the tables being parsed during the CPU bringup process.

This also fixes sysfs because CPUs with the same 'physical_package_id' in /sys/devices/system/cpu/cpu*/topology/ are not listed together in the same 'core_siblings_list'. This violates a statement from Documentation/ABI/testing/sysfs-devices-system-cpu:

core_siblings: internal kernel map of cpu#'s hardware threads within the same physical_package_id.

core_siblings_list: human-readable list of the logical CPU numbers within the same physical_package_id as cpu#.

The sysfs effects here cause an issue with the hwloc tool where it gets confused and thinks there are more sockets than are physically present.

Before this set, there are two packages:

# cd /sys/devices/system/cpu/
# cat cpu*/topology/physical_package_id | sort | uniq -c
     18 0
     18 1

But 4 _sets_ of core siblings:

# cat cpu*/topology/core_siblings_list | sort | uniq -c
      9 0-8
      9 18-26
      9 27-35
      9 9-17

After this set, there are only 2 sets of core siblings, which is what we expect for a 2-socket system.

# cat cpu*/topology/physical_package_id | sort | uniq -c
     18 0
     18 1
# cat cpu*/topology/core_siblings_list | sort | uniq -c
     18 0-17
     18 18-35

Example spew:
...

	NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
	 #2  #3  #4  #5  #6  #7  #8
	.... node  #1, CPUs:    #9
	------------[ cut here ]------------
	WARNING: CPU: 9 PID: 0 at /home/ak/hle/linux-hle-2.6/arch/x86/kernel/smpboot.c:306 topology_sane.isra.2+0x74/0x90()
	sched: CPU #9's mc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
	Modules linked in:
	CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.17.0-rc1-00293-g8e01c4d-dirty #631
	Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0036.R05.1407140519 07/14/2014
	0000000000000009 ffff88046ddabe00 ffffffff8172e485 ffff88046ddabe48
	ffff88046ddabe38 ffffffff8109691d 000000000000b001 0000000000000009
	ffff88086fc12580 000000000000b020 0000000000000009 ffff88046ddabe98
	Call Trace:
	[<ffffffff8172e485>] dump_stack+0x45/0x56
	[<ffffffff8109691d>] warn_slowpath_common+0x7d/0xa0
	[<ffffffff8109698c>] warn_slowpath_fmt+0x4c/0x50
	[<ffffffff81074f94>] topology_sane.isra.2+0x74/0x90
	[<ffffffff8107530e>] set_cpu_sibling_map+0x31e/0x4f0
	[<ffffffff8107568d>] start_secondary+0x1ad/0x240
	---[ end trace 3fe5f587a9fcde61 ]---
	#10 #11 #12 #13 #14 #15 #16 #17
	.... node  #2, CPUs:   #18 #19 #20 #21 #22 #23 #24 #25 #26
	.... node  #3, CPUs:   #27 #28 #29 #30 #31 #32 #33 #34 #35

Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: "H. Peter Anvin" <[email protected]>
rc: [email protected]
Cc: [email protected]
Cc: [email protected]