Skip to contentSkip to conversation

cpu hotplug, keep topology directory after soft remove event

PATCH Superseded Prarit Bhargava 3 participants 6 messages Sep 7, 2016 – Sep 8, 2016

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover cpu hotplug, keep topology directory after soft remove event

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid. IOW, the cpu is still present but the kernel has removed the topology directory making it very difficult to determine exactly where the cpu is located.

This patchset adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists for the lifetime of the CPU. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a down and added on an up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

This patchset combines drivers/base/topology.c and drivers/base/cpu.c to implement CONFIG_PERMANENT_CPU_TOPOLOGY and leaves all arches except x86 with the current behavior.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>

Prarit Bhargava (2):
drivers/base: Combine topology.c and cpu.c
cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY

 arch/x86/kernel/smpboot.c |    3 -
 drivers/base/Kconfig      |   12 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  146 +++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c   |  168 ---------------------------------------------
 5 files changed, 159 insertions(+), 172 deletions(-)
 delete mode 100644 drivers/base/topology.c
Signature · 1 line
1.7.9.3
#1 lore raw

PATCH 1/2 drivers/base: Combine topology.c and cpu.c

The topology.c file contains sysfs files that describe a cpu's location and its siblings. There is no purpose that this file is separate from the core cpu code. This patch combines topology.c into cpu.c to make the next set of changes easier to understand.

There are no functional changes with this patch.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>
---
 drivers/base/Makefile   |    2 +-
 drivers/base/cpu.c      |  138 ++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c |  168 -----------------------------------------------
 3 files changed, 139 insertions(+), 169 deletions(-)
 delete mode 100644 drivers/base/topology.c
drivers/base/Makefile +1 -1
index 2609ba20b396..e50491bb3d60 100644
@@ -4,7 +4,7 @@ obj-y			:= component.o core.o bus.o dd.o syscore.o \
 			   driver.o class.o platform.o \
 			   cpu.o firmware.o init.o map.o devres.o \
 			   attribute_container.o transport_class.o \
-			   topology.o container.o property.o cacheinfo.o
+			   container.o property.o cacheinfo.o
 obj-$(CONFIG_DEVTMPFS)	+= devtmpfs.o
 obj-$(CONFIG_DMA_CMA) += dma-contiguous.o
 obj-y			+= power/
drivers/base/cpu.c +138 -0

149 lines not on this page · raw message

drivers/base/topology.c deleted +0 -168

168 lines not on this page · raw message

Signature · 1 line
1.7.9.3
#2 lore raw

Re: [PATCH 1/2] drivers/base: Combine topology.c and cpu.c

url: https://github.com/0day-ci/linux/commits/Prarit-Bhargava/cpu-hotplug-keep-topology-directory-after-soft-remove-event/20160907-210020 config: x86_64-randconfig-x015-201636 (attached as .config)…

Hi Prarit,

[auto build test ERROR on driver-core/driver-core-testing]
[also build test ERROR on v4.8-rc5 next-20160907]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
[Suggest to use git(>=2.9.0) format-patch --base=<commit> (or --base=auto for convenience) to record what (public, well-known) commit your patch series was built on] [Check https://git-scm.com/docs/git-format-patch for more information]

url: https://github.com/0day-ci/linux/commits/Prarit-Bhargava/cpu-hotplug-keep-topology-directory-after-soft-remove-event/20160907-210020
config: x86_64-randconfig-x015-201636 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705 reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

>> drivers/base/cpu.c:260:11: error: 'topology_attrs' undeclared here (not in a function)

.attrs = topology_attrs,
^~~~~~~~~~~~~~

vim +/topology_attrs +260 drivers/base/cpu.c

254 static struct attribute_group crash_note_cpu_attr_group = {
255 .attrs = crash_note_cpu_attrs,
256 };
257 #endif
258
259 static struct attribute_group topology_attr_group = {

 > 260		.attrs = topology_attrs,

261 .name = "topology"
262 };
263

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#3 lore raw

PATCH 2/2 cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY

In reply to Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid and the cpu's location is now lost to userspace.

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

Before patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# ls /sys/devices/system/cpu/cpu10/topology
ls: cannot access topology: No such file or directory

After patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

I did some testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd CPUs in sequence, randomly, by thread group, by socket group and didn't see any issues.

Note: core_siblings, and thread_siblings are "numa siblings that are online" and "thread siblings that are online" and are used as such within the kernel. They must be zero'd out when the CPU is offline.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>
---
 arch/x86/kernel/smpboot.c |    3 ---
 drivers/base/Kconfig      |   12 ++++++++++++
 drivers/base/cpu.c        |    8 ++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
arch/x86/kernel/smpboot.c +0 -3
index 4296beb8fdd3..ae82f8f45b61 100644
@@ -1472,7 +1472,6 @@ static void recompute_smt_state(void)
 static void remove_siblinginfo(int cpu)
 {
 	int sibling;
-	struct cpuinfo_x86 *c = &cpu_data(cpu);
 
 	for_each_cpu(sibling, topology_core_cpumask(cpu)) {
 		cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));
@@ -1490,8 +1489,6 @@ static void remove_siblinginfo(int cpu)
 	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(topology_sibling_cpumask(cpu));
 	cpumask_clear(topology_core_cpumask(cpu));
-	c->phys_proc_id = 0;
-	c->cpu_core_id = 0;
 	cpumask_clear_cpu(cpu, cpu_sibling_setup_mask);
 	recompute_smt_state();
 }
drivers/base/Kconfig +12 -0
index 98504ec99c7d..b3935a272c3c 100644
@@ -324,4 +324,16 @@ config CMA_ALIGNMENT
 
 endif
 
+config PERMANENT_CPU_TOPOLOGY
+	bool "Permanent CPU Topology"
+	depends on HOTPLUG_CPU
+	def_bool y if X86_64
+	help
+	  This option configures CPU topology to be permanent for the lifetime
+	  of the CPU (until it is physically removed).  Selecting Y here
+	  results in the kernel reporting the physical location for offlined
+	  CPUs.
+
+	  If unsure, leave the default value as is.
+
 endmenu
drivers/base/cpu.c +8 -0
index e8c8c1155b13..c84cbd1defd5 100644
@@ -261,6 +261,7 @@ static struct attribute_group topology_attr_group = {
 	.name = "topology"
 };
 
+#ifndef CONFIG_PERMANENT_CPU_TOPOLOGY
 /* Add/Remove cpu_topology interface for CPU device */
 static int topology_add_dev(unsigned int cpu)
 {
@@ -317,11 +318,15 @@ out:
 }
 
 device_initcall(topology_sysfs_init);
+#endif
 
 static const struct attribute_group *common_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 
@@ -329,6 +334,9 @@ static const struct attribute_group *hotplugable_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 
Signature · 1 line
1.7.9.3
#4 lore raw

Re: [PATCH 2/2] cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY

Quoted 10 lines · Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid and the cpu's location is now lost to userspace.

Quoted 9 lines

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

So how do you physically remove things and how does the information then get removed?

Also, why is that an x86 only feature?

#5 lore raw

Re: [PATCH 2/2] cpu hotplug, add CONFIG_PERMANENT_CPU_TOPOLOGY

Quoted 21 lines · Peter Zijlstra
Quoted 10 lines · Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid and the cpu's location is now lost to userspace.

Quoted 9 lines

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

Quoted 3 lines

So how do you physically remove things and how does the information then get removed?

Physical hotremove was done with a toggle switch enabled test box.

The topology directory gets removed when either common_cpu_attr_groups (!CONFIG_HOTPLUG_CPU) or hotplugable_cpu_attr_groups (CONFIG_HOTPLUG_CPU) are destroyed. The cpu device's attributes are set in register_cpu() and the device is destroyed in unregister_cpu().

IOW, the patch expands the lifetime of the topology directories to match that of the device.

Quoted 1 line

Also, why is that an x86 only feature?

The only arch that I could test on was x86 and I wasn't able to test to see if topology fields are valid for other arches on a soft remove. So x86 by default for now and other arches can make their decisions.

P.