fix topology for multi-NUMA-node CPUs
Subsystems (16)
cover fix topology for multi-NUMA-node CPUs
This is a big fat RFC. It takes quite a few liberties with the multi-core topology level that I'm not completely comfortable with.
It has only been tested lightly.
Full dmesg for a Cluster-on-Die system with this set applied, and sched_debug on the command-line is here:
http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt
---
I'm getting the spew below when booting with Haswell (Xeon E5-2699 v3) CPUs and the "Cluster-on-Die" (CoD) feature enabled in the BIOS. It seems similar to the issue that some folks from AMD ran in to on their systems and addressed in this commit:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=161270fc1f9ddfc17154e0d49291472a9cdef7db
Both these Intel and AMD systems break an assumption which is being enforced by topology_sane(): a socket may not contain more than one NUMA node.
AMD special-cased their system by looking for a cpuid flag. The Intel mode is dependent on BIOS options and I do not know of a way which it is enumerated other than the tables being parsed during the CPU bringup process.
This also fixes sysfs because CPUs with the same 'physical_package_id' in /sys/devices/system/cpu/cpu*/topology/ are not listed together in the same 'core_siblings_list'. This violates a statement from Documentation/ABI/testing/sysfs-devices-system-cpu:
core_siblings: internal kernel map of cpu#'s hardware threads within the same physical_package_id.
core_siblings_list: human-readable list of the logical CPU numbers within the same physical_package_id as cpu#.
The sysfs effects here cause an issue with the hwloc tool where it gets confused and thinks there are more sockets than are physically present.
Before this set, there are two packages:
# cd /sys/devices/system/cpu/
# cat cpu*/topology/physical_package_id | sort | uniq -c
18 0
18 1But 4 _sets_ of core siblings:
# cat cpu*/topology/core_siblings_list | sort | uniq -c
9 0-8
9 18-26
9 27-35
9 9-17After this set, there are only 2 sets of core siblings, which is what we expect for a 2-socket system.
# cat cpu*/topology/physical_package_id | sort | uniq -c
18 0
18 1
# cat cpu*/topology/core_siblings_list | sort | uniq -c
18 0-17
18 18-35Example spew:
...
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. #2 #3 #4 #5 #6 #7 #8 .... node #1, CPUs: #9 ------------[ cut here ]------------ WARNING: CPU: 9 PID: 0 at /home/ak/hle/linux-hle-2.6/arch/x86/kernel/smpboot.c:306 topology_sane.isra.2+0x74/0x90() sched: CPU #9's mc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. Modules linked in: CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.17.0-rc1-00293-g8e01c4d-dirty #631 Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0036.R05.1407140519 07/14/2014 0000000000000009 ffff88046ddabe00 ffffffff8172e485 ffff88046ddabe48 ffff88046ddabe38 ffffffff8109691d 000000000000b001 0000000000000009 ffff88086fc12580 000000000000b020 0000000000000009 ffff88046ddabe98 Call Trace: [<ffffffff8172e485>] dump_stack+0x45/0x56 [<ffffffff8109691d>] warn_slowpath_common+0x7d/0xa0 [<ffffffff8109698c>] warn_slowpath_fmt+0x4c/0x50 [<ffffffff81074f94>] topology_sane.isra.2+0x74/0x90 [<ffffffff8107530e>] set_cpu_sibling_map+0x31e/0x4f0 [<ffffffff8107568d>] start_secondary+0x1ad/0x240 ---[ end trace 3fe5f587a9fcde61 ]--- #10 #11 #12 #13 #14 #15 #16 #17 .... node #2, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26 .... node #3, CPUs: #27 #28 #29 #30 #31 #32 #33 #34 #35
Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: "H. Peter Anvin" <[email protected]>
rc: [email protected]
Cc: [email protected]
Cc: [email protected]
RFC PATCH 1/6 topology: rename topology_core_cpumask() to topology_package_cpumask()
From: Dave Hansen <[email protected]>
topology_core_cpumask() is a function that arch-independent code uses. Each (non-generic) architecture will define it in order to map over to the arch-specific variables storing the actual mask.
I think topology_core_cpumask() is a bad name. It makes it sound like it is generating a cpumask for *A* core. It is, in fact, generating the mask of all cores inside a CPU package. Let's make that more clear with the naming.
For the non-x86 architectures, I'd appreciate a sanity check of this. It definitely makes sense on x86, but I'm less confident about the others.
---
b/arch/arm/include/asm/topology.h | 2 +- b/arch/arm64/include/asm/topology.h | 2 +- b/arch/ia64/include/asm/topology.h | 2 +- b/arch/metag/include/asm/topology.h | 2 +- b/arch/mips/include/asm/smp.h | 2 +- b/arch/powerpc/include/asm/topology.h | 2 +- b/arch/s390/include/asm/topology.h | 2 +- b/arch/sh/include/asm/topology.h | 2 +- b/arch/sparc/include/asm/topology_64.h | 2 +- b/arch/tile/include/asm/topology.h | 2 +- b/arch/x86/include/asm/topology.h | 2 +- b/drivers/base/topology.c | 8 ++++---- b/drivers/block/nvme-core.c | 2 +- b/drivers/cpufreq/arm_big_little.c | 2 +- b/drivers/infiniband/hw/qib/qib_iba7322.c | 2 +- b/include/linux/topology.h | 4 ++-- b/lib/cpu_rmap.c | 2 +- 17 files changed, 21 insertions(+), 21 deletions(-)
arch/arm64/include/asm/topology.h renamed from arch/arm64/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -17,7 +17,7 @@ extern struct cpu_topology cpu_topology[#define topology_physical_package_id (cpu) (cpu_topology[cpu].cluster_id)#define topology_core_id (cpu) (cpu_topology[cpu].core_id) -#define topology_core_cpumask (cpu) (&cpu_topology[cpu].core_sibling) +#define topology_package_cpumask (cpu) (&cpu_topology[cpu].core_sibling)#define topology_thread_cpumask (cpu) (&cpu_topology[cpu].thread_sibling)void init_cpu_topology (void );
arch/arm/include/asm/topology.h renamed from arch/arm/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -17,7 +17,7 @@ extern struct cputopo_arm cpu_topology[N#define topology_physical_package_id (cpu) (cpu_topology[cpu].socket_id)#define topology_core_id (cpu) (cpu_topology[cpu].core_id) -#define topology_core_cpumask (cpu) (&cpu_topology[cpu].core_sibling) +#define topology_package_cpumask (cpu) (&cpu_topology[cpu].core_sibling)#define topology_thread_cpumask (cpu) (&cpu_topology[cpu].thread_sibling)void init_cpu_topology (void );
arch/ia64/include/asm/topology.h renamed from arch/ia64/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -52,7 +52,7 @@ void build_cpu_to_node_map(void);#ifdef CONFIG_SMP #define topology_physical_package_id (cpu) (cpu_data(cpu)->socket_id)#define topology_core_id (cpu) (cpu_data(cpu)->core_id) -#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_package_cpumask (cpu) (&cpu_core_map[cpu])#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu))#endif
arch/metag/include/asm/topology.h renamed from arch/metag/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -21,7 +21,7 @@ const struct cpumask *cpu_coregroup_maskextern cpumask_t cpu_core_map[NR_CPUS ]; -#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_package_cpumask (cpu) (&cpu_core_map[cpu])#include <asm-generic/topology.h>
arch/mips/include/asm/smp.h renamed from arch/mips/include/asm/smp.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -39,7 +39,7 @@ extern int __cpu_logical_map[NR_CPUS];#define topology_physical_package_id (cpu) (cpu_data[cpu].package)#define topology_core_id (cpu) (cpu_data[cpu].core) -#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_package_cpumask (cpu) (&cpu_core_map[cpu])#define topology_thread_cpumask (cpu) (&cpu_sibling_map[cpu])#define SMP_RESCHEDULE_YOURSELF 0x1/* XXX braindead */
arch/powerpc/include/asm/topology.h renamed from arch/powerpc/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -88,7 +88,7 @@ static inline int prrn_is_enabled(void)#define topology_physical_package_id (cpu) (cpu_to_chip_id(cpu))#define topology_thread_cpumask (cpu) (per_cpu(cpu_sibling_map, cpu)) -#define topology_core_cpumask (cpu) (per_cpu(cpu_core_map, cpu)) +#define topology_package_cpumask (cpu) (per_cpu(cpu_core_map, cpu))#define topology_core_id (cpu) (cpu_to_core_id(cpu))#endif #endif
arch/s390/include/asm/topology.h renamed from arch/s390/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -20,7 +20,7 @@ extern struct cpu_topology_s390 cpu_topo#define topology_physical_package_id (cpu) (cpu_topology[cpu].socket_id)#define topology_core_id (cpu) (cpu_topology[cpu].core_id) -#define topology_core_cpumask (cpu) (&cpu_topology[cpu].core_mask) +#define topology_package_cpumask (cpu) (&cpu_topology[cpu].core_mask)#define topology_book_id (cpu) (cpu_topology[cpu].book_id)#define topology_book_cpumask (cpu) (&cpu_topology[cpu].book_mask)
arch/sh/include/asm/topology.h renamed from arch/sh/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -21,7 +21,7 @@ const struct cpumask *cpu_coregroup_maskextern cpumask_t cpu_core_map[NR_CPUS ]; -#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_package_cpumask (cpu) (&cpu_core_map[cpu])#include <asm-generic/topology.h>
arch/sparc/include/asm/topology_64.h renamed from arch/sparc/include/asm/topology_64.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -40,7 +40,7 @@ static inline int pcibus_to_node(struct#ifdef CONFIG_SMP #define topology_physical_package_id (cpu) (cpu_data(cpu).proc_id)#define topology_core_id (cpu) (cpu_data(cpu).core_id) -#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_package_cpumask (cpu) (&cpu_core_map[cpu])#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu))#endif /* CONFIG_SMP */
arch/tile/include/asm/topology.h renamed from arch/tile/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -54,7 +54,7 @@ static inline const struct cpumask *cpum#ifdef CONFIG_SMP #define topology_physical_package_id (cpu) ((void)(cpu), 0)#define topology_core_id (cpu) (cpu) -#define topology_core_cpumask (cpu) ((void)(cpu), cpu_online_mask) +#define topology_package_cpumask (cpu) ((void)(cpu), cpu_online_mask)#define topology_thread_cpumask (cpu) cpumask_of(cpu)#endif
arch/x86/include/asm/topology.h renamed from arch/x86/include/asm/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -123,7 +123,7 @@ extern const struct cpumask *cpu_coregro#define topology_core_id (cpu) (cpu_data(cpu).cpu_core_id)#ifdef ENABLE_TOPO_DEFINES -#define topology_core_cpumask (cpu) (per_cpu(cpu_core_map, cpu)) +#define topology_package_cpumask (cpu) (per_cpu(cpu_core_map, cpu))#define topology_thread_cpumask (cpu) (per_cpu(cpu_sibling_map, cpu))#endif
drivers/base/topology.c renamed from drivers/base/topology.c~rename-topology_cpu_cpumask-topology_package_cpumask +4 -4
@@ -42,7 +42,7 @@ static ssize_t show_##name(struct device return sprintf(buf, "%d\n", topology_##name(dev->id)); \ } -#if defined(topology_thread_cpumask) || defined(topology_core_cpumask) || \ +#if defined(topology_thread_cpumask) || defined(topology_package_cpumask) || \ defined(topology_book_cpumask)static ssize_t show_cpumap (int type,const struct cpumask *mask,char *buf) {
@@ -88,9 +88,9 @@ define_siblings_show_func(thread_cpumaskdefine_one_ro_named (thread_siblings, show_thread_cpumask);define_one_ro_named (thread_siblings_list, show_thread_cpumask_list); -define_siblings_show_func (core_cpumask); -define_one_ro_named (core_siblings, show_core_cpumask); -define_one_ro_named (core_siblings_list, show_core_cpumask_list); +define_siblings_show_func (package_cpumask); +define_one_ro_named (core_siblings, show_package_cpumask); +define_one_ro_named (core_siblings_list, show_package_cpumask_list);#ifdef CONFIG_SCHED_BOOK define_id_show_func (book_id);
drivers/block/nvme-core.c renamed from drivers/block/nvme-core.c~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -2054,7 +2054,7 @@ static void nvme_assign_io_queues(struct nvmeq, cpus_per_queue);if (cpus_weight (mask) < cpus_per_queue)nvme_add_cpus (&mask, unassigned_cpus, -topology_core_cpumask (cpu), +topology_package_cpumask (cpu), nvmeq, cpus_per_queue);if (cpus_weight (mask) < cpus_per_queue) nvme_add_cpus(&mask, unassigned_cpus,
drivers/cpufreq/arm_big_little.c renamed from drivers/cpufreq/arm_big_little.c~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -449,7 +449,7 @@ static int bL_cpufreq_init(struct cpufreif (cur_cluster <MAX_CLUSTERS ) {int cpu; -cpumask_copy (policy->cpus,topology_core_cpumask (policy->cpu)); +cpumask_copy (policy->cpus,topology_package_cpumask (policy->cpu));for_each_cpu (cpu, policy->cpus)per_cpu (physical_cluster, cpu) = cur_cluster;
drivers/infiniband/hw/qib/qib_iba7322.c renamed from drivers/infiniband/hw/qib/qib_iba7322.c~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -3449,7 +3449,7 @@ try_intx: firstcpu =cpumask_first (local_mask);if (firstcpu >= nr_cpu_ids ||cpumask_weight (local_mask) ==num_online_cpus ()) { - local_mask =topology_core_cpumask (0 ); + local_mask =topology_package_cpumask (0 ); firstcpu =cpumask_first (local_mask); }if (firstcpu < nr_cpu_ids) {
include/linux/topology.h renamed from include/linux/topology.h~rename-topology_cpu_cpumask-topology_package_cpumask +2 -2
@@ -177,8 +177,8 @@ static inline int cpu_to_mem(int cpu)#ifndef topology_thread_cpumask#define topology_thread_cpumask (cpu) cpumask_of(cpu)#endif -#ifndef topology_core_cpumask -#define topology_core_cpumask (cpu) cpumask_of(cpu) +#ifndef topology_package_cpumask +#define topology_package_cpumask (cpu) cpumask_of(cpu)#endif #ifdef CONFIG_SCHED_SMT
lib/cpu_rmap.c renamed from lib/cpu_rmap.c~rename-topology_cpu_cpumask-topology_package_cpumask +1 -1
@@ -194,7 +194,7 @@ int cpu_rmap_update(struct cpu_rmap *rma topology_thread_cpumask(cpu), 1))continue ;if (cpu_rmap_copy_neigh (rmap, cpu, -topology_core_cpumask (cpu),2 )) +topology_package_cpumask (cpu),2 ))continue ;if (cpu_rmap_copy_neigh (rmap, cpu,cpumask_of_node (cpu_to_node (cpu)),3 ))
_
RFC PATCH 2/6 x86: introduce cpumask specifically for the package
In reply to Dave HansenFrom: Dave Hansen <[email protected]>
As noted by multiple reports:
https://lkml.org/lkml/2014/9/15/1240 https://lkml.org/lkml/2014/7/28/442
the sched domains code has some assumptions that break on newer AMD and Intel CPUs. Namely, the code assumes that NUMA node boundaries always lie outside of a CPU package. That assumption is no longer true with Intel's Cluster-on-Die found in Haswell CPUs (with a special BIOS config knob) and AMD's DCM feature.
Essentially, the 'cpu_core_map' is no longer suitable for enumerating all the CPUs in a physical package.
This patch introduces a new map which is specifically built by consulting the the physical package ids instead of inferring the information from NUMA nodes.
This still leaves us with a broken 'core_siblings_list' in sysfs, but a later patch will fix that up too.
---
b/arch/x86/include/asm/smp.h | 6 ++++++ b/arch/x86/include/asm/topology.h | 1 + b/arch/x86/kernel/smpboot.c | 13 +++++++++++++ b/arch/x86/xen/smp.c | 1 + 4 files changed, 21 insertions(+)
arch/x86/include/asm/smp.h renamed from arch/x86/include/asm/smp.h~introduce-package-sd-level +6 -0
@@ -32,6 +32,7 @@ static inline bool cpu_has_ht_siblings(vDECLARE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_sibling_map);DECLARE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_core_map); +DECLARE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_package_map);/* cpus sharing the last level cache: */ DECLARE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_llc_shared_map);DECLARE_PER_CPU_READ_MOSTLY (u16, cpu_llc_id);
@@ -52,6 +53,11 @@ static inline struct cpumask *cpu_llc_shreturn per_cpu (cpu_llc_shared_map, cpu); } +static inline struct cpumask *cpu_package_mask (int cpu) +{ +return per_cpu (cpu_package_map, cpu); +} +DECLARE_EARLY_PER_CPU_READ_MOSTLY (u16, x86_cpu_to_apicid);DECLARE_EARLY_PER_CPU_READ_MOSTLY (u16, x86_bios_cpu_apicid);#if defined(CONFIG_X86_LOCAL_APIC ) && defined(CONFIG_X86_32 )
arch/x86/include/asm/topology.h renamed from arch/x86/include/asm/topology.h~introduce-package-sd-level +1 -0
@@ -118,6 +118,7 @@ static inline void setup_node_to_cpumask#include <asm-generic/topology.h> extern const struct cpumask *cpu_coregroup_mask (int cpu); +extern const struct cpumask *cpu_package_mask_func (int cpu);#define topology_physical_package_id (cpu) (cpu_data(cpu).phys_proc_id)#define topology_core_id (cpu) (cpu_data(cpu).cpu_core_id)
arch/x86/kernel/smpboot.c renamed from arch/x86/kernel/smpboot.c~introduce-package-sd-level +13 -0
@@ -98,6 +98,8 @@ EXPORT_PER_CPU_SYMBOL(cpu_core_map);DEFINE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_llc_shared_map); +DEFINE_PER_CPU_READ_MOSTLY (cpumask_var_t, cpu_package_map); +/* Per CPU bogomips and other parameters */ DEFINE_PER_CPU_SHARED_ALIGNED(struct cpuinfo_x86, cpu_info);EXPORT_PER_CPU_SYMBOL (cpu_info);
@@ -352,6 +354,13 @@ static bool match_mc(struct cpuinfo_x86return false; } +static bool match_pkg (struct cpuinfo_x86 *c,struct cpuinfo_x86 *o) +{ +if (c->phys_proc_id == o->phys_proc_id) +return true; +return false; +} +void set_cpu_sibling_map (int cpu) {bool has_smt = smp_num_siblings >1 ;
@@ -365,6 +374,7 @@ void set_cpu_sibling_map(int cpu)if (!has_mp) {cpumask_set_cpu (cpu,cpu_sibling_mask (cpu));cpumask_set_cpu (cpu,cpu_llc_shared_mask (cpu)); +cpumask_set_cpu (cpu,cpu_package_mask (cpu));cpumask_set_cpu (cpu,cpu_core_mask (cpu)); c->booted_cores =1 ;return ;
@@ -410,6 +420,9 @@ void set_cpu_sibling_map(int cpu) }else if (i != cpu && !c->booted_cores) c->booted_cores =cpu_data (i).booted_cores; } +if ((i == cpu) || (has_mp &&match_pkg (c, o))) { +link_mask (package, cpu, i); + } } }
arch/x86/xen/smp.c renamed from arch/x86/xen/smp.c~introduce-package-sd-level +1 -0
@@ -331,6 +331,7 @@ static void __init xen_smp_prepare_cpus(zalloc_cpumask_var (&per_cpu (cpu_sibling_map, i),GFP_KERNEL );zalloc_cpumask_var (&per_cpu (cpu_core_map, i),GFP_KERNEL );zalloc_cpumask_var (&per_cpu (cpu_llc_shared_map, i),GFP_KERNEL ); +zalloc_cpumask_var (&per_cpu (cpu_package_map, i),GFP_KERNEL ); }set_cpu_sibling_map (0 );
_
Peter Zijlstra
maintainerRe: [RFC][PATCH 2/6] x86: introduce cpumask specifically for the package
Quoted 17 lines · Dave Hansen
From: Dave Hansen <[email protected]>
As noted by multiple reports:
https://lkml.org/lkml/2014/9/15/1240 https://lkml.org/lkml/2014/7/28/442the sched domains code has some assumptions that break on newer AMD and Intel CPUs. Namely, the code assumes that NUMA node boundaries always lie outside of a CPU package. That assumption is no longer true with Intel's Cluster-on-Die found in Haswell CPUs (with a special BIOS config knob) and AMD's DCM feature.
Essentially, the '
cpu_core_map' is no longer suitable for enumerating all the CPUs in a physical package.
Quoted 6 lines
This patch introduces a new map which is specifically built by consulting the the physical package ids instead of inferring the information from NUMA nodes.
This still leaves us with a broken '
core_siblings_list' in sysfs, but a later patch will fix that up too.
If we do dynamic topology layout we don't need a second mask I think. The machines that have multiple packages per node will simply present a different sched_domain_topology than the machines that have multiple nodes per package.
Specifically, in the former we include the package_mask as DIE level, in the other case we leave it out entirely and rely on the SLIT table to build the right domain topology.
RFC PATCH 3/6 x86: use package_map instead of core_map for sysfs
In reply to Dave HansenFrom: Dave Hansen <[email protected]>
The /sys/devices/system/cpu/cpu*/topology/core_siblings* files were previously built from the "cpu_core_map". That mask is deeply connected to the sched domains internal multi-core (MC) level, which is now become disconnected from the actual CPU package.
We have a new "cpu_package_map" which has the sole purpose of tracking which package the CPU is in and is unconnected to the scheduler. We will now build those sysfs with information from the new package map.
Note: this also realigns the sysfs files with their documentation in Documentation/ABI.
---
b/arch/x86/include/asm/topology.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/topology.h renamed from arch/x86/include/asm/topology.h~x86-use-package-map-instead-of-core-map +1 -1
@@ -124,7 +124,7 @@ extern const struct cpumask *cpu_package#define topology_core_id (cpu) (cpu_data(cpu).cpu_core_id)#ifdef ENABLE_TOPO_DEFINES -#define topology_package_cpumask (cpu) (per_cpu(cpu_core_map, cpu)) +#define topology_package_cpumask (cpu) (per_cpu(cpu_package_map, cpu))#define topology_thread_cpumask (cpu) (per_cpu(cpu_sibling_map, cpu))#endif
_
RFC PATCH 4/6 sched: eliminate "DIE" domain level when NUMA present
In reply to Dave HansenFrom: Dave Hansen <[email protected]>
The "DIE" topology level is currently defined like this:
static inline const struct cpumask *cpu_cpu_mask(int cpu)
{
return cpumask_of_node(cpu_to_node(cpu));
}But that makes very little sense on a NUMA system since the lowest-domain NUMA node is guaranteed to be essentially the same as this level.
We leave this for systems that are !CONFIG_NUMA and that might need a top-level domain.
This also keeps us from having screwy topologies when the smallest NUMA node is only _part_ of the die.
---
b/kernel/sched/core.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/core.c renamed from kernel/sched/core.c~die-is-NUMA-based-and-screwed-up +2 -0
@@ -6141,7 +6141,9 @@ static struct sched_domain_topology_leve #ifdef CONFIG_SCHED_MC { cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) }, #endif +#ifndef CONFIG_NUMA { cpu_cpu_mask, SD_INIT_NAME(DIE) }, +#endif { NULL, }, };
_
Peter Zijlstra
maintainerRe: [RFC][PATCH 4/6] sched: eliminate "DIE" domain level when NUMA present
Quoted 30 lines · Dave Hansen
From: Dave Hansen <[email protected]>
The "DIE" topology level is currently defined like this:
static inline const struct cpumask *cpu_cpu_mask(int cpu)
{ return cpumask_of_node(cpu_to_node(cpu)); }But that makes very little sense on a NUMA system since the lowest-domain NUMA node is guaranteed to be essentially the same as this level.
We leave this for systems that are !CONFIG_NUMA and that might need a top-level domain.
This also keeps us from having screwy topologies when the smallest NUMA node is only _part_ of the die.
Signed-off-by Dave Hansen <[email protected]>---
b/kernel/sched/core.c | 2 ++ 1 file changed, 2 insertions(+)diff -puN
kernel/sched/core.c~die-is-NUMA-based-and-screwed-upkernel/sched/core.c
kernel/sched/core.c static struct sched_domain_topology_leve · line 6148
Quoted 8 lines
kernel/sched/core.c renamed from kernel/sched/core.c~die-is-NUMA-based-and-screwed-up +2
-0@@ -6141,7 +6141,9 @@ static struct sched_domain_topology_leve #ifdef CONFIG_SCHED_MC { cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) }, #endif +#ifndef CONFIG_NUMA { cpu_cpu_mask, SD_INIT_NAME(DIE) }, +#endif { NULL, }, };
Yeah, no. Also, looking at it now, I see why it worked and how its been wrong :-) As you say it returns the node mask, not the PKG mask as it should have been doing.
So don't change the default topology, in general I'd say its still true that you get one or more packages inside a node. Change override the default topology in arch code.
RFC PATCH 5/6 sched: keep MC domain from crossing nodes OR packages
In reply to Dave HansenFrom: Dave Hansen <[email protected]>
The MC (MultiCore) sched domain was originally intended to represent the groupings of cores in a multi-core CPU.
The sched domains code has essentially two kinds of topology levels:
1. CPU levels, like hyperthreads or cores that share cache
2. NUMA levels derived from the system's NUMA topology
The domains are built by first going through the CPU levels, then through the NUMA levels. However, we now have at least two instances of systems where a single CPU "package" has multiple NUMA nodes.
To fix this, we redefine the multi-core level. Previously, it was defined as stopping at the CPU package. Now, we define it as grouping similar CPUs that are both in the same package *and* that have the same access to some set of memory. Essentially an MC group must be in the same package *and* be on the same NUMA node.
This does no harm because there is a NUMA precisely at the level which "MC" used to represent.
---
b/arch/x86/kernel/smpboot.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
arch/x86/kernel/smpboot.c renamed from arch/x86/kernel/smpboot.c~keep-mc-from-crossing-nodes-or-package +20 -6
@@ -345,13 +345,27 @@ static bool match_llc(struct cpuinfo_x86static bool match_mc (struct cpuinfo_x86 *c,struct cpuinfo_x86 *o) { -if (c->phys_proc_id == o->phys_proc_id) { -if (cpu_has (c,X86_FEATURE_AMD_DCM )) -return true; +/* +* Do not allow a multi-core group +* ouside of a package. +*/ +if (c->phys_proc_id != o->phys_proc_id) +return false; -return topology_sane (c, o,"mc" ); - } -return false; +/* +* Do not allow a multi-core group +* ouside of a NUMA node. +*/ +if (cpu_to_node (c->cpu_index) != +cpu_to_node (o->cpu_index)) +return false; + +/* +* This pretty much repeats the NUMA node +* check above, but leave it here for +* consistency. +*/ +return topology_sane (c, o,"mc" ); }static bool match_pkg (struct cpuinfo_x86 *c,struct cpuinfo_x86 *o)
_
RFC PATCH 6/6 sched: consolidate config options
In reply to Dave HansenI originally did this when renaming CONFIG_SCHED_MC. I ended up not renaming it, but I still think it's nice to have all the Kconfigs consolidated like this.
--
From: Dave Hansen <[email protected]>
We have 2 config options (SCHED_MC and SCHED_SMT) which are used across a few architectures. We have one (SCHED_BOOK) only used on s390.
The Kconfig text for MC/SMT are copied verbatim across each of the architectures that use it. This consolidates them down to a single Kconfig location.
This gives us a centrally-defined set of config options which architectures can 'select' when needed.
---
b/arch/arm/Kconfig | 18 ++----------- b/arch/arm64/Kconfig | 18 ++----------- b/arch/ia64/Kconfig | 9 ------ b/arch/mips/Kconfig | 14 +--------- b/arch/powerpc/Kconfig | 8 ----- b/arch/powerpc/platforms/Kconfig.cputype | 1 b/arch/s390/Kconfig | 14 ++-------- b/arch/sh/Kconfig | 1 b/arch/sh/mm/Kconfig | 9 ------ b/arch/sparc/Kconfig | 20 ++------------ b/arch/x86/Kconfig | 20 ++------------ b/kernel/sched/Kconfig | 42 +++++++++++++++++++++++++++++++ 12 files changed, 63 insertions(+), 111 deletions(-)
arch/arm64/Kconfig renamed from arch/arm64/Kconfig~consolidate-config-SCHED_MC +3 -15
@@ -68,6 +68,8 @@ config ARM64 select SPARSE_IRQ select SYSCTL_EXCEPTION_TRACE select HAVE_CONTEXT_TRACKING + select ARCH_ENABLE_SCHED_MC + select ARCH_ENABLE_SCHED_SMT help ARM 64-bit (AArch64) Linux support.
@@ -235,21 +237,7 @@ config SMP If you don't know what to do here, say N. -config SCHED_MC - bool "Multi-core scheduler support" - depends on SMP - help - Multi-core scheduler support improves the CPU scheduler's decision - making when dealing with multi-core CPU chips at a cost of slightly - increased overhead in some places. If unsure say N here. - -config SCHED_SMT - bool "SMT scheduler support" - depends on SMP - help - Improves the CPU scheduler's decision making when dealing with - MultiThreading at a cost of slightly increased overhead in some - places. If unsure say N here. +source kernel/sched/Kconfig config NR_CPUS int "Maximum number of CPUs (2-32)"
arch/arm/Kconfig renamed from arch/arm/Kconfig~consolidate-config-SCHED_MC +3 -15
@@ -1355,27 +1355,15 @@ config SMP_ON_UP config ARM_CPU_TOPOLOGY bool "Support cpu topology definition" depends on SMP && CPU_V7 + select ARCH_ENABLE_SCHED_MC + select ARCH_ENABLE_SCHED_SMT default y help Support ARM cpu topology definition. The MPIDR register defines affinity between processors which is then used to describe the cpu topology of an ARM System. -config SCHED_MC - bool "Multi-core scheduler support" - depends on ARM_CPU_TOPOLOGY - help - Multi-core scheduler support improves the CPU scheduler's decision - making when dealing with multi-core CPU chips at a cost of slightly - increased overhead in some places. If unsure say N here. - -config SCHED_SMT - bool "SMT scheduler support" - depends on ARM_CPU_TOPOLOGY - help - Improves the CPU scheduler's decision making when dealing with - MultiThreading at a cost of slightly increased overhead in some - places. If unsure say N here. +source kernel/sched/Kconfig config HAVE_ARM_SCU bool
arch/ia64/Kconfig renamed from arch/ia64/Kconfig~consolidate-config-SCHED_MC +1 -8
@@ -49,6 +49,7 @@ config IA64 select MODULES_USE_ELF_RELA select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_AUDITSYSCALL + select ARCH_ENABLE_SCHED_SMT default y help The Itanium Processor Family is Intel's 64-bit successor to
@@ -382,14 +383,6 @@ config ARCH_ENABLE_MEMORY_HOTPLUG config ARCH_ENABLE_MEMORY_HOTREMOVE def_bool y -config SCHED_SMT - bool "SMT scheduler support" - depends on SMP - help - Improves the CPU scheduler's decision making when dealing with - Intel IA64 chips with MultiThreading at a cost of slightly increased - overhead in some places. If unsure say N here. - config PERMIT_BSP_REMOVE bool "Support removal of Bootstrap Processor" depends on HOTPLUG_CPU
arch/mips/Kconfig renamed from arch/mips/Kconfig~consolidate-config-SCHED_MC +2 -12
@@ -1948,7 +1948,7 @@ config MIPS_MT_SMP select SMP select SMP_UP select SYS_SUPPORTS_SMP - select SYS_SUPPORTS_SCHED_SMT + select ARCH_ENABLE_SCHED_SMT select MIPS_PERF_SHARED_TC_COUNTERS help This is a kernel model which is known as SMVP. This is supported
@@ -1960,17 +1960,7 @@ config MIPS_MT_SMP config MIPS_MT bool -config SCHED_SMT - bool "SMT (multithreading) scheduler support" - depends on SYS_SUPPORTS_SCHED_SMT - default n - help - SMT scheduler support improves the CPU scheduler's decision making - when dealing with MIPS MT enabled cores at a cost of slightly - increased overhead in some places. If unsure say N here. - -config SYS_SUPPORTS_SCHED_SMT - bool +source "kernel/sched/Kconfig" config SYS_SUPPORTS_MULTITHREADING bool
arch/powerpc/Kconfig renamed from arch/powerpc/Kconfig~consolidate-config-SCHED_MC +0 -8
@@ -603,14 +603,6 @@ config PPC_SUBPAGE_PROT to set access permissions (read/write, readonly, or no access) on the 4k subpages of each 64k page. -config SCHED_SMT - bool "SMT (Hyperthreading) scheduler support" - depends on PPC64 && SMP - help - SMT scheduler support improves the CPU scheduler's decision making - when dealing with POWER5 cpus at a cost of slightly increased - overhead in some places. If unsure say N here. - config PPC_DENORMALISATION bool "PowerPC denormalisation exception handling" depends on PPC_BOOK3S_64
arch/powerpc/platforms/Kconfig.cputype renamed from arch/powerpc/platforms/Kconfig.cputype~consolidate-config-SCHED_MC +1 -0
@@ -2,6 +2,7 @@ config PPC64 bool "64-bit kernel" default n select HAVE_VIRT_CPU_ACCOUNTING + select SCHED_SMT if SMP help This option selects whether a 32-bit or a 64-bit kernel will be built.
arch/s390/Kconfig renamed from arch/s390/Kconfig~consolidate-config-SCHED_MC +3 -11
@@ -146,6 +146,8 @@ config S390 select VIRT_CPU_ACCOUNTING select VIRT_TO_BUS select ARCH_HAS_SG_CHAIN + select ARCH_ENABLE_SCHED_BOOK + select ARCH_ENABLE_SCHED_MC config SCHED_OMIT_FRAME_POINTER def_bool y
@@ -372,17 +374,7 @@ config HOTPLUG_CPU can be controlled through /sys/devices/system/cpu/cpu#. Say N if you want to disable CPU hotplug. -config SCHED_MC - def_bool n - -config SCHED_BOOK - def_bool y - prompt "Book scheduler support" - depends on SMP - select SCHED_MC - help - Book scheduler support improves the CPU scheduler's decision making - when dealing with machines that have several books. +source kernel/sched/Kconfig source kernel/Kconfig.preempt
arch/sh/Kconfig renamed from arch/sh/Kconfig~consolidate-config-SCHED_MC +1 -0
@@ -43,6 +43,7 @@ config SUPERH select OLD_SIGSUSPEND select OLD_SIGACTION select HAVE_ARCH_AUDITSYSCALL + select ARCH_ENABLE_SCHED_MC help The SuperH is a RISC processor targeted for use in embedded systems and consumer electronics; it was also used in the Sega Dreamcast
arch/sh/mm/Kconfig renamed from arch/sh/mm/Kconfig~consolidate-config-SCHED_MC +1 -8
@@ -226,14 +226,7 @@ endchoice source "mm/Kconfig" -config SCHED_MC - bool "Multi-core scheduler support" - depends on SMP - default y - help - Multi-core scheduler support improves the CPU scheduler's decision - making when dealing with multi-core CPU chips at a cost of slightly - increased overhead in some places. If unsure say N here. +source "kernel/sched/Kconfig" endmenu
arch/sparc/Kconfig renamed from arch/sparc/Kconfig~consolidate-config-SCHED_MC +3 -17
@@ -79,6 +79,8 @@ config SPARC64 select NO_BOOTMEM select HAVE_ARCH_AUDITSYSCALL select ARCH_SUPPORTS_ATOMIC_RMW + select ARCH_ENABLE_SCHED_MC if SMP + select ARCH_ENABLE_SCHED_SMT if SMP config ARCH_DEFCONFIG string
@@ -306,23 +308,7 @@ if SPARC64 source "kernel/power/Kconfig" endif -config SCHED_SMT - bool "SMT (Hyperthreading) scheduler support" - depends on SPARC64 && SMP - default y - help - SMT scheduler support improves the CPU scheduler's decision making - when dealing with SPARC cpus at a cost of slightly increased overhead - in some places. If unsure say N here. - -config SCHED_MC - bool "Multi-core scheduler support" - depends on SPARC64 && SMP - default y - help - Multi-core scheduler support improves the CPU scheduler's decision - making when dealing with multi-core CPU chips at a cost of slightly - increased overhead in some places. If unsure say N here. +source "kernel/sched/Kconfig" source "kernel/Kconfig.preempt"
arch/x86/Kconfig renamed from arch/x86/Kconfig~consolidate-config-SCHED_MC +3 -17
@@ -255,6 +255,8 @@ config X86_64_SMP config X86_HT def_bool y depends on SMP + select ARCH_ENABLE_SCHED_MC + select ARCH_ENABLE_SCHED_SMT config X86_32_LAZY_GS def_bool y
@@ -789,23 +791,7 @@ config NR_CPUS This is purely to save memory - each supported CPU adds approximately eight kilobytes to the kernel image. -config SCHED_SMT - bool "SMT (Hyperthreading) scheduler support" - depends on X86_HT - ---help--- - SMT scheduler support improves the CPU scheduler's decision making - when dealing with Intel Pentium 4 chips with HyperThreading at a - cost of slightly increased overhead in some places. If unsure say - N here. - -config SCHED_MC - def_bool y - prompt "Multi-core scheduler support" - depends on X86_HT - ---help--- - Multi-core scheduler support improves the CPU scheduler's decision - making when dealing with multi-core CPU chips at a cost of slightly - increased overhead in some places. If unsure say N here. +source "kernel/sched/Kconfig" source "kernel/Kconfig.preempt"
kernel/sched/Kconfig new file +42 -0
42 lines not on this page · raw message
_
Peter Zijlstra
maintainerRe: [RFC][PATCH 6/6] sched: consolidate config options
Quoted 28 lines · Dave Hansen
diff -puN
/dev/nullkernel/sched/Kconfigkernel/sched/Kconfig new file +24
-0@@ -0,0 +1,42 @@ +config ARCH_ENABLE_SCHED_MC + depends on SMP + def_bool n + +config ARCH_ENABLE_SCHED_BOOK + depends on SMP + def_bool n + +config ARCH_ENABLE_SCHED_SMT + depends on SMP + def_bool n + +config SCHED_MC + bool "Multi-core scheduler support" + default n if s390 + default y + depends on ARCH_ENABLE_SCHED_MC + help + Multi-core scheduler support improves the CPU scheduler's decision + making when dealing with multi-core CPU chips at a cost of slightly + increased overhead in some places. If unsure say N here. + +config SCHED_BOOK + def_bool y
kernel/sched/Kconfig line 34
Quoted 10 lines
kernel/sched/Kconfig new file +10
-0+ prompt "Book scheduler support" + depends on ARCH_ENABLE_SCHED_BOOK + select SCHED_MC + help + Book scheduler support improves the CPU scheduler's decision making + when dealing with machines that have several books. + + Currenltly only used on s390 which has only a single NUMA node. + Books are collections of CPUs that are grouped similarly to a NUMA + node, but without the same memory properites that NUMA nodes have.
Nothing outside of s390 knows about SCHED_BOOK, it doesn't make sense to have that here.
Quoted 8 lines
kernel/sched/Kconfig new file +7
-0+config SCHED_SMT + bool "SMT scheduler support" + depends on ARCH_ENABLE_SCHED_SMT + help + Improves the CPU scheduler's decision making when dealing with + MultiThreading at a cost of slightly increased overhead in some + places. If unsure say N here._
Re: [RFC][PATCH 6/6] sched: consolidate config options
Quoted 6 lines · Peter Zijlstra
Quoted 6 lines
Quoted 6 lines
kernel/sched/Kconfig new file +6
-0+config SCHED_BOOK + def_bool y + prompt "Book scheduler support" + depends on ARCH_ENABLE_SCHED_BOOK + select SCHED_MC + help
Quoted 8 lines
Quoted 6 lines
kernel/sched/Kconfig line 34
Quoted 6 lines
kernel/sched/Kconfig new file +6
-0+ Book scheduler support improves the CPU scheduler's decision making + when dealing with machines that have several books. + + Currenltly only used on s390 which has only a single NUMA node. + Books are collections of CPUs that are grouped similarly to a NUMA + node, but without the same memory properites that NUMA nodes have.Nothing outside of s390 knows about
SCHED_BOOK, it doesn't make sense to have that here.
By sticking all of them together, my hope was that folks who were going to add a topology level could see all of the existing options in a single place.
But, just say the word and I'll yank it out and repost.
Peter Zijlstra
maintainerRe: [RFC][PATCH 6/6] sched: consolidate config options
Quoted 16 lines · Dave Hansen
Quoted 6 lines · Peter Zijlstra
Quoted 6 lines
Quoted 6 lines
kernel/sched/Kconfig new file +6
-0+config SCHED_BOOK + def_bool y + prompt "Book scheduler support" + depends on ARCH_ENABLE_SCHED_BOOK + select SCHED_MC + helpQuoted 8 lines
Quoted 6 lines
kernel/sched/Kconfig line 34
Quoted 6 lines
kernel/sched/Kconfig new file +6
-0+ Book scheduler support improves the CPU scheduler's decision making + when dealing with machines that have several books. + + Currenltly only used on s390 which has only a single NUMA node. + Books are collections of CPUs that are grouped similarly to a NUMA + node, but without the same memory properites that NUMA nodes have.Nothing outside of s390 knows about
SCHED_BOOK, it doesn't make sense to have that here.
Quoted 5 lines
By sticking all of them together, my hope was that folks who were going to add a topology level could see all of the existing options in a single place.
But, just say the word and I'll yank it out and repost.
Yeah, I think its best to leave it a s390 private afair.
Borislav Petkov
maintainerQuoted 10 lines · Dave Hansen
This is a big fat RFC. It takes quite a few liberties with the multi-core topology level that I'm not completely comfortable with.
It has only been tested lightly.
Full dmesg for a Cluster-on-Die system with this set applied, and
sched_debugon the command-line is here:http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt
So how do I find out what topology this system has?
[ 0.175294] .... node #0, CPUs: #1 [ 0.190970] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.191813] #2 #3 #4 #5 #6 #7 #8 [ 0.290753] .... node #1, CPUs: #9 #10 #11 #12 #13 #14 #15 #16 #17 [ 0.436162] .... node #2, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26 [ 0.660795] .... node #3, CPUs: #27 #28 #29 #30 #31 #32 #33 #34 #35 [ 0.806365] .... node #0, CPUs: #36 #37 #38 #39 #40 #41 #42 #43 #44 [ 0.933573] .... node #1, CPUs: #45 #46 #47 #48 #49 #50 #51 #52 #53 [ 1.061079] .... node #2, CPUs: #54 #55 #56 #57 #58 #59 #60 #61 #62 [ 1.188491] .... node #3, CPUs: #63 [ 1.202620] x86: Booted up 4 nodes, 64 CPUs
SRAT says 4 nodes but I'm guessing from the context those 4 nodes are actually in pairs in two physical sockets, right?
Btw, you'd need to increase NR_CPUS because you obviously have more APICs than 64.
So if we pick a cpu at random:
[ 1.350640] CPU49 attaching sched-domain: [ 1.350641] domain 0: span 13,49 level SMT [ 1.350642] groups: 49 (cpu_capacity = 588) 13 (cpu_capacity = 588) [ 1.350644] domain 1: span 9-17,45-53 level MC [ 1.350645] groups: 13,49 (cpu_capacity = 1176) 14,50 (cpu_capacity = 1176) 15,51 (cpu_capacity = 1176) 16,52 (cpu_capa city = 1176) 17,53 (cpu_capacity = 1176) 9,45 (cpu_capacity = 1176) 10,46 (cpu_capacity = 1177) 11,47 (cpu_capacity = 1176) 1 2,48 (cpu_capacity = 1176) [ 1.350654] domain 2: span 0-17,36-53 level NUMA [ 1.350655] groups: 9-17,45-53 (cpu_capacity = 10585) 0-8,36-44 (cpu_capacity = 10589) [ 1.350659] domain 3: span 0-63 level NUMA [ 1.350660] groups: 0-17,36-53 (cpu_capacity = 21174) 18-35,54-63 (cpu_capacity = 19944)
domain level 1 MC is what tells me which cores are on the internal nodes of a socket? Or how do we find that out? Or even, do we need that info at all...?
It might be useful for RAS and when we want to disable cores or whatever...
Thanks.
Signature · 3 lines
Regards/Gruss,
Boris.
--A message that is not in the archive (sent off-list or not yet received): CAOjmkp8EGO0jicmdO=p6ATHz-hUJmWb+xoBLjOdLBUwwGzyhhg@mail.gmail.com
Aravind Gopalakrishnan
Tested-byQuoted 96 lines · Aravind Gopalakrishnan
This is a big fat RFC. It takes quite a few liberties with the multi-core topology level that I'm not completely comfortable with.
It has only been tested lightly.
Full dmesg for a Cluster-on-Die system with this set applied, and
sched_debugon the command-line is here:http://sr71.net/~dave/intel/full-dmesg-hswep-20140917.txt <http://sr71.net/%7Edave/intel/full-dmesg-hswep-20140917.txt>
---
I'm getting the spew below when booting with Haswell (Xeon E5-2699 v3) CPUs and the "Cluster-on-Die" (CoD) feature enabled in the BIOS. It seems similar to the issue that some folks from AMD ran in to on their systems and addressed in this commit:
Both these Intel and AMD systems break an assumption which is being enforced by
topology_sane(): a socket may not contain more than one NUMA node.AMD special-cased their system by looking for a cpuid flag. The Intel mode is dependent on BIOS options and I do not know of a way which it is enumerated other than the tables being parsed during the CPU bringup process.
This also fixes sysfs because CPUs with the same '
physical_package_id' in/sys/devices/system/cpu/cpu*/topology/are not listed together in the same 'core_siblings_list'. This violates a statement fromDocumentation/ABI/testing/sysfs-devices-system-cpu:
core_siblings: internal kernel map of cpu#'s hardware threads within the samephysical_package_id.
core_siblings_list: human-readable list of the logical CPU numbers within the samephysical_package_idas cpu#.The sysfs effects here cause an issue with the hwloc tool where it gets confused and thinks there are more sockets than are physically present.
Before this set, there are two packages:
# cd /sys/devices/system/cpu/ # cat cpu*/topology/physical_package_id | sort | uniq -c 18 0 18 1But 4 _sets_ of core siblings:
# cat cpu*/topology/core_siblings_list | sort | uniq -c 9 0-8 9 18-26 9 27-35 9 9-17After this set, there are only 2 sets of core siblings, which is what we expect for a 2-socket system.
# cat cpu*/topology/physical_package_id | sort | uniq -c 18 0 18 1 # cat cpu*/topology/core_siblings_list | sort | uniq -c 18 0-17 18 18-35Example spew:
...
NMI watchdog: enabled on all CPUs, permanently consumes one
hw-PMU counter.#2 #3 #4 #5 #6 #7 #8 .... node #1, CPUs: #9 ------------[ cut here ]------------ WARNING: CPU: 9 PID: 0 at /home/ak/hle/linux-hle-2.6/arch/x86/kernel/smpboot.c:306 topology_sane.isra.2+0x74/0x90() sched: CPU #9's mc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. Modules linked in: CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.17.0-rc1-00293-g8e01c4d-dirty #631 Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1.86B.0036.R05.1407140519 07/14/2014 0000000000000009 ffff88046ddabe00 ffffffff8172e485 ffff88046ddabe48 ffff88046ddabe38 ffffffff8109691d 000000000000b001 0000000000000009 ffff88086fc12580 000000000000b020 0000000000000009 ffff88046ddabe98 Call Trace: [<ffffffff8172e485>] dump_stack+0x45/0x56 [<ffffffff8109691d>] warn_slowpath_common+0x7d/0xa0
Quoted 8 lines
[<ffffffff8109698c>] warn_slowpath_fmt+0x4c/0x50
[<ffffffff81074f94>] topology_sane.isra.2+0x74/0x90
[<ffffffff8107530e>] set_cpu_sibling_map+0x31e/0x4f0
[<ffffffff8107568d>] start_secondary+0x1ad/0x240
---[ end trace 3fe5f587a9fcde61 ]---
#10 #11 #12 #13 #14 #15 #16 #17
.... node #2, CPUs: #18 #19 #20 #21 #22 #23 #24 #25 #26
.... node #3, CPUs: #27 #28 #29 #30 #31 #32 #33 #34 #35Hi,
I looked at the topology info from sysfs both w/ and w/o the patch series and they are identical. So, the patches seem to work fine on an AMD MCM part.
Thanks,
-Aravind.