Add CONFIG_PERMANENT_CPU_TOPOLOGY

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid and the cpu's location is now lost to userspace.

This patchset adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

Before patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# ls /sys/devices/system/cpu/cpu10/topology
ls: cannot access topology: No such file or directory

After patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

I did some testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Notes:

core_siblings, and thread_siblings are "numa siblings that are online" and "thread siblings that are online" and are used as such within the kernel They must be zero'd out when the CPU is offline.

Patch 1/3 simply combines topology.c and cpu.c and introduces no new functionality. It is a staging patch for 2/3. Patch 3/3 is a simple cleanup.

Prarit Bhargava (3):
drivers/base, Combine topology.c and cpu.c cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY drivers/base, cpu, remove hotplugable_cpu_attr_groups

 arch/x86/kernel/smpboot.c |    3 -
 drivers/base/Kconfig      |   12 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  141 ++++++++++++++++++++++++++++++++++++++---
 drivers/base/topology.c   |  155 ---------------------------------------------
 5 files changed, 145 insertions(+), 168 deletions(-)
 delete mode 100644 drivers/base/topology.c

1.7.9.3

The topology.c file contains sysfs files that describe a cpu's location and its siblings. This patch combines topology.c into cpu.c to make the next set of changes easier to understand.

There are no functional changes with this patch.

---
 drivers/base/Makefile   |    2 +-
 drivers/base/cpu.c      |  127 ++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c |  155 -----------------------------------------------
 3 files changed, 128 insertions(+), 156 deletions(-)
 delete mode 100644 drivers/base/topology.c

index 6b2a84e..567ab7c 100644
@@ -4,7 +4,7 @@ obj-y			:= component.o core.o bus.o dd.o syscore.o \
 			   driver.o class.o platform.o \
 			   cpu.o firmware.o init.o map.o devres.o \
 			   attribute_container.o transport_class.o \
-			   topology.o container.o property.o cacheinfo.o
+			   container.o property.o cacheinfo.o
 obj-$(CONFIG_DEVTMPFS)	+= devtmpfs.o
 obj-$(CONFIG_DMA_CMA) += dma-contiguous.o
 obj-y			+= power/

1.7.9.3

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid and the cpu's location is now lost to userspace.

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

Before patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# ls /sys/devices/system/cpu/cpu10/topology
ls: cannot access topology: No such file or directory

After patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

I did some testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Note: core_siblings, and thread_siblings are "numa siblings that are online" and "thread siblings that are online" and are used as such within the kernel. They must be zero'd out when the CPU is offline.

---
 arch/x86/kernel/smpboot.c |    3 ---
 drivers/base/Kconfig      |   12 ++++++++++++
 drivers/base/cpu.c        |    8 ++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)

index 892ee2e5..6591195 100644
@@ -1332,7 +1332,6 @@ __init void prefill_possible_map(void)
 static void remove_siblinginfo(int cpu)
 {
 	int sibling;
-	struct cpuinfo_x86 *c = &cpu_data(cpu);
 
 	for_each_cpu(sibling, topology_core_cpumask(cpu)) {
 		cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));

@@ -1350,8 +1349,6 @@ static void remove_siblinginfo(int cpu)
 	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(topology_sibling_cpumask(cpu));
 	cpumask_clear(topology_core_cpumask(cpu));
-	c->phys_proc_id = 0;
-	c->cpu_core_id = 0;
 	cpumask_clear_cpu(cpu, cpu_sibling_setup_mask);
 }
 

index 98504ec..b3935a2 100644
@@ -324,4 +324,16 @@ config CMA_ALIGNMENT
 
 endif
 
+config PERMANENT_CPU_TOPOLOGY
+	bool "Permanent CPU Topology"
+	depends on HOTPLUG_CPU
+	def_bool y if X86_64
+	help
+	  This option configures CPU topology to be permanent for the lifetime
+	  of the CPU (until it is physically removed).  Selecting Y here
+	  results in the kernel reporting the physical location for offlined
+	  CPUs.
+
+	  If unsure, leave the default value as is.
+
 endmenu

index b939c98..9c30782 100644
@@ -250,6 +250,7 @@ static struct attribute_group topology_attr_group = {
 	.name = "topology"
 };
 
+#ifndef CONFIG_PERMANENT_CPU_TOPOLOGY
 /* Add/Remove cpu_topology interface for CPU device */
 static int topology_add_dev(unsigned int cpu)
 {

@@ -306,11 +307,15 @@ out:
 }
 
 device_initcall(topology_sysfs_init);
+#endif
 
 static const struct attribute_group *common_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 

@@ -318,6 +323,9 @@ static const struct attribute_group *hotplugable_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 

1.7.9.3

hotplugable_cpu_attr_groups is not different from common_cpu_attr_groups, and can be removed. This patchset renames common_cpu_attr_groups to cpu_attr_groups.

---
 drivers/base/cpu.c |   16 ++--------------
 1 file changed, 2 insertions(+), 14 deletions(-)

index 9c30782..8520c68 100644
@@ -309,17 +309,7 @@ out:
 device_initcall(topology_sysfs_init);
 #endif
 
-static const struct attribute_group *common_cpu_attr_groups[] = {
-#ifdef CONFIG_KEXEC
-	&crash_note_cpu_attr_group,
-#endif
-#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
-	&topology_attr_group,
-#endif
-	NULL
-};
-
-static const struct attribute_group *hotplugable_cpu_attr_groups[] = {
+static const struct attribute_group *cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif

@@ -502,9 +492,7 @@ int register_cpu(struct cpu *cpu, int num)
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	cpu->dev.bus->uevent = cpu_uevent;
 #endif
-	cpu->dev.groups = common_cpu_attr_groups;
-	if (cpu->hotpluggable)
-		cpu->dev.groups = hotplugable_cpu_attr_groups;
+	cpu->dev.groups = cpu_attr_groups;
 	error = device_register(&cpu->dev);
 	if (!error)
 		per_cpu(cpu_sys_devices, num) = &cpu->dev;

1.7.9.3