Skip to contentSkip to conversation

Add CONFIG_PERMANENT_CPU_TOPOLOGY

PATCH In review Prarit Bhargava 2 participants 5 messages Oct 19, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add CONFIG_PERMANENT_CPU_TOPOLOGY

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid.

This patchset adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists for the lifetime of the CPU. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a down and added on an up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

This patchset combines drivers/base/topology.c and drivers/base/cpu.c to implement CONFIG_PERMANENT_CPU_TOPOLOGY, and leaves all arches except x86 with the current behavior, and adds an additional minor cleanup to drivers/base/cpu.c.

Before patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

Down a cpu

[root@prarit cpu143]# echo 0 > online

[root@intel-brickland-05 cpu143]# ls

cpuidle      crash_notes_size  firmware_node  online  subsystem
crash_notes  driver            node3          power   uevent

[root@intel-brickland-05 cpu143]# ls -l topology
ls: cannot access topology: No such file or directory

After patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

Down a cpu

[root@prarit cpu143]# echo 0 > online
[root@intel-brickland-05 cpu143]# ls

cpuidle      crash_notes_size  firmware_node  online  subsystem  uevent
crash_notes  driver            node3          power   topology

[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

I did some light testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Len Brown <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Zhu Guihua <[email protected]>
Cc Denys Vlasenko <[email protected]>
Cc "Jan H. Schönherr" <[email protected]>
Cc Boris Ostrovsky <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Thomas Renninger <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>

Prarit Bhargava (2):
cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY and keep topology
directory for lifetime of CPU
base, cpu, remove hotplugable_cpu_attr_groups

 arch/x86/kernel/smpboot.c |   28 --------
 drivers/base/Kconfig      |   13 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  141 ++++++++++++++++++++++++++++++++++++++---
 drivers/base/topology.c   |  155 ---------------------------------------------
 5 files changed, 146 insertions(+), 193 deletions(-)
 delete mode 100644 drivers/base/topology.c
Signature · 1 line
1.7.9.3
#1 lore raw

PATCH 1/2 cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY and keep topology directory for lifetime of CPU

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid.

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists for the lifetime of the CPU. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a down and added on an up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

This patch combines drivers/base/topology.c and drivers/base/cpu.c to implement CONFIG_PERMANENT_CPU_TOPOLOGY, and leaves all arches except x86 with the current behavior.

Before patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

Down a cpu

[root@prarit cpu143]# echo 0 > online

[root@intel-brickland-05 cpu143]# ls

cpuidle      crash_notes_size  firmware_node  online  subsystem
crash_notes  driver            node3          power   uevent

[root@intel-brickland-05 cpu143]# ls -l topology
ls: cannot access topology: No such file or directory

After patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

Down a cpu

[root@prarit cpu143]# echo 0 > online
[root@intel-brickland-05 cpu143]# ls
cpuidle crash_notes_size firmware_node online subsystem uevent
crash_notes driver node3 power topology
[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

I did some light testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Len Brown <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Zhu Guihua <[email protected]>
Cc Denys Vlasenko <[email protected]>
Cc "Jan H. Schönherr" <[email protected]>
Cc Boris Ostrovsky <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Thomas Renninger <[email protected]>
---
 arch/x86/kernel/smpboot.c |   28 --------
 drivers/base/Kconfig      |   13 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  135 +++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c   |  155 ---------------------------------------------
 5 files changed, 149 insertions(+), 184 deletions(-)
 delete mode 100644 drivers/base/topology.c
arch/x86/kernel/smpboot.c +0 -28
index e0c198e..19082c7 100644
@@ -1322,32 +1322,6 @@ __init void prefill_possible_map(void)
 
 #ifdef CONFIG_HOTPLUG_CPU
 
-static void remove_siblinginfo(int cpu)
-{
-	int sibling;
-	struct cpuinfo_x86 *c = &cpu_data(cpu);
-
-	for_each_cpu(sibling, topology_core_cpumask(cpu)) {
-		cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));
-		/*/
-		 * last thread sibling in this cpu core going down
-		 */
-		if (cpumask_weight(topology_sibling_cpumask(cpu)) == 1)
-			cpu_data(sibling).booted_cores--;
-	}
-
-	for_each_cpu(sibling, topology_sibling_cpumask(cpu))
-		cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling));
-	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
-		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
-	cpumask_clear(cpu_llc_shared_mask(cpu));
-	cpumask_clear(topology_sibling_cpumask(cpu));
-	cpumask_clear(topology_core_cpumask(cpu));
-	c->phys_proc_id = 0;
-	c->cpu_core_id = 0;
-	cpumask_clear_cpu(cpu, cpu_sibling_setup_mask);
-}
-
 static void remove_cpu_from_maps(int cpu)
 {
 	set_cpu_online(cpu, false);
@@ -1362,8 +1336,6 @@ void cpu_disable_common(void)
 {
 	int cpu = smp_processor_id();
 
-	remove_siblinginfo(cpu);
-
 	/* It's now safe to remove this processor from the online map */
 	lock_vector_lock();
 	remove_cpu_from_maps(cpu);
drivers/base/Kconfig +13 -0
index 98504ec..321d261 100644
@@ -324,4 +324,17 @@ config CMA_ALIGNMENT
 
 endif
 
+config PERMANENT_CPU_TOPOLOGY
+	bool "Permanent CPU Topology"
+	depends on HOTPLUG_CPU
+	default 1 if X86
+	default 0
+	help
+	  This option configures CPU topology to be permanent for the lifetime
+	  of the CPU (until it is physically removed).  Selecting Y here
+	  results in the kernel reporting the physical location for offlined
+	  CPUs.
+
+	  If unsure, leave the default value as is.
+
 endmenu
drivers/base/Makefile +1 -1
index 6b2a84e..567ab7c 100644
@@ -4,7 +4,7 @@ obj-y			:= component.o core.o bus.o dd.o syscore.o \
 			   driver.o class.o platform.o \
 			   cpu.o firmware.o init.o map.o devres.o \
 			   attribute_container.o transport_class.o \
-			   topology.o container.o property.o cacheinfo.o
+			   container.o property.o cacheinfo.o
 obj-$(CONFIG_DEVTMPFS)	+= devtmpfs.o
 obj-$(CONFIG_DMA_CMA) += dma-contiguous.o
 obj-y			+= power/
drivers/base/cpu.c +135 -0

162 lines not on this page · raw message

drivers/base/topology.c deleted +0 -155

155 lines not on this page · raw message

Signature · 1 line
1.7.9.3
#2 lore raw

Re: [PATCH 1/2] cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY and keep topology directory for lifetime of CPU

reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Prarit,

[auto build test ERROR on driver-core/driver-core-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]

url:    https://github.com/0day-ci/linux/commits/Prarit-Bhargava/Add-CONFIG_PERMANENT_CPU_TOPOLOGY/20151019-230927
config: mips-jz4740 (attached as .config)

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=mips

All errors (new ones prefixed by >>):

>> drivers/base/cpu.c:249:11: error: 'topology_attrs' undeclared here (not in a function)

.attrs = topology_attrs,
^

vim +/topology_attrs +249 drivers/base/cpu.c

243 static struct attribute_group crash_note_cpu_attr_group = {
244 .attrs = crash_note_cpu_attrs,
245 };
246 #endif
247
248 static struct attribute_group topology_attr_group = {

 > 249		.attrs = topology_attrs,

250 .name = "topology"
251 };
252

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#3 lore raw

Re: [PATCH 1/2] cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY and keep topology directory for lifetime of CPU

In reply to Prarit Bhargava
Quoted 115 lines · Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid.

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists for the lifetime of the CPU. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a down and added on an up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

This patch combines drivers/base/topology.c and drivers/base/cpu.c to implement CONFIG_PERMANENT_CPU_TOPOLOGY, and leaves all arches except x86 with the current behavior.

Before patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

Down a cpu

[root@prarit cpu143]# echo 0 > online

[root@intel-brickland-05 cpu143]# ls

cpuidle      crash_notes_size  firmware_node  online  subsystem
crash_notes  driver            node3          power   uevent

[root@intel-brickland-05 cpu143]# ls -l topology
ls: cannot access topology: No such file or directory

After patch:

[root@prarit cpu143]# ls

cache    crash_notes       firmware_node  online     thermal_throttle
cpufreq  crash_notes_size  microcode      power      topology
cpuidle  driver            node3          subsystem  uevent

[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

Down a cpu

[root@prarit cpu143]# echo 0 > online
[root@intel-brickland-05 cpu143]# ls
cpuidle crash_notes_size firmware_node online subsystem uevent
crash_notes driver node3 power topology
[root@prarit cpu143]# cat topology/*
27
ffff,c0000000,000000ff,ffc00000,00000000
54-71,126-143
3
8000,00000000,00000080,00000000,00000000
71,143

I did some light testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Len Brown <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Zhu Guihua <[email protected]>
Cc Denys Vlasenko <[email protected]>
Cc "Jan H. Schönherr" <[email protected]>
Cc Boris Ostrovsky <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
Cc Thomas Renninger <[email protected]>
---
 arch/x86/kernel/smpboot.c |   28 --------
 drivers/base/Kconfig      |   13 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  135 +++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c   |  155 ---------------------------------------------
 5 files changed, 149 insertions(+), 184 deletions(-)
 delete mode 100644 drivers/base/topology.c
arch/x86/kernel/smpboot.c +0 -11
index e0c198e..19082c7 100644
@@ -1322,32 +1322,6 @@ __init void prefill_possible_map(void)
 
 #ifdef CONFIG_HOTPLUG_CPU
 
-static void remove_siblinginfo(int cpu)
-{
-	int sibling;
-	struct cpuinfo_x86 *c = &cpu_data(cpu);
-
-	for_each_cpu(sibling, topology_core_cpumask(cpu)) {
-		cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));
-		/*/
-		 * last thread sibling in this cpu core going down
-		 */
-		if (cpumask_weight(topology_sibling_cpumask(cpu)) == 1)

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1345

Quoted 10 lines
arch/x86/kernel/smpboot.c +0 -10
-			cpu_data(sibling).booted_cores--;
-	}
-
-	for_each_cpu(sibling, topology_sibling_cpumask(cpu))
-		cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling));
-	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
-		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
-	cpumask_clear(cpu_llc_shared_mask(cpu));
-	cpumask_clear(topology_sibling_cpumask(cpu));
-	cpumask_clear(topology_core_cpumask(cpu));

It was just pointed out to me that the these masks tend to be used in the kernel as "sibling & online mask", and "core & online_mask". The other values are indeed static and are safe to use.

I'll "self-nak" this and come up with a slightly better patchset.

P.

#4 lore raw

PATCH 2/2 base, cpu, remove hotplugable_cpu_attr_groups

In reply to Prarit Bhargava

hotplugable_cpu_attr_groups is not different from common_cpu_attr_groups, and can be removed. This patchset renames common_cpu_attr_groups to cpu_attr_groups.

Cc Greg Kroah-Hartman <[email protected]>
CC Thomas Renninger <[email protected]>
---
 drivers/base/cpu.c |   16 ++--------------
 1 file changed, 2 insertions(+), 14 deletions(-)
drivers/base/cpu.c +2 -14
index 9c30782..8520c68 100644
@@ -309,17 +309,7 @@ out:
 device_initcall(topology_sysfs_init);
 #endif
 
-static const struct attribute_group *common_cpu_attr_groups[] = {
-#ifdef CONFIG_KEXEC
-	&crash_note_cpu_attr_group,
-#endif
-#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
-	&topology_attr_group,
-#endif
-	NULL
-};
-
-static const struct attribute_group *hotplugable_cpu_attr_groups[] = {
+static const struct attribute_group *cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
@@ -502,9 +492,7 @@ int register_cpu(struct cpu *cpu, int num)
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	cpu->dev.bus->uevent = cpu_uevent;
 #endif
-	cpu->dev.groups = common_cpu_attr_groups;
-	if (cpu->hotpluggable)
-		cpu->dev.groups = hotplugable_cpu_attr_groups;
+	cpu->dev.groups = cpu_attr_groups;
 	error = device_register(&cpu->dev);
 	if (!error)
 		per_cpu(cpu_sys_devices, num) = &cpu->dev;
Signature · 1 line
1.7.9.3