Skip to contentSkip to conversation

cpu hotplug: Preserve topology directory after soft remove event

PATCH In review Prarit Bhargava 4 participants 20 messages Sep 21, 2016 – Sep 28, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover cpu hotplug: Preserve topology directory after soft remove event

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid. IOW, the cpu is still present but the kernel has removed the topology directory making it very difficult to determine exactly where the cpu is located.

This patchset adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists for the lifetime of the CPU. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a down and added on an up). Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

This patchset combines drivers/base/topology.c and drivers/base/cpu.c to implement CONFIG_PERMANENT_CPU_TOPOLOGY and leaves all arches except x86 with the current behavior.

peterz asked when the topology directory is destroyed when CONFIG_PERMANENT_CPU_TOPOLOGY=y. The topology directory's lifetime will change when CONFIG_PERMANENT_CPU_TOPOLOGY=y from existing when the thread is online, to being created when the struct device associated with the thread is created, and similarly being destroyed when the struct device is destroyed.

[v2 & v3]: Fix ktest build robot build failures

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>

Prarit Bhargava (2):
drivers/base: Combine topology.c and cpu.c
cpu hotplug: add CONFIG_PERMANENT_CPU_TOPOLOGY

 arch/x86/kernel/smpboot.c |    3 -
 drivers/base/Kconfig      |   12 ++++
 drivers/base/Makefile     |    2 +-
 drivers/base/cpu.c        |  148 +++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c   |  168 ---------------------------------------------
 5 files changed, 161 insertions(+), 172 deletions(-)
 delete mode 100644 drivers/base/topology.c
Signature · 1 line
1.7.9.3
#1 lore raw

PATCH v3 1/2 drivers/base: Combine topology.c and cpu.c

The topology.c file contains sysfs files that describe a cpu's location and its siblings. There is no purpose that this file is separate from the core cpu code. This patch combines topology.c into cpu.c to make the next set of changes easier to understand.

There are no functional changes with this patch.

[v2]: fix error from ktest build robot with CONFIG_KEXEC=n [v3]: fix error from ktest build robot with ARCH=sparc

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>
---
 drivers/base/Makefile   |    2 +-
 drivers/base/cpu.c      |  140 +++++++++++++++++++++++++++++++++++++++
 drivers/base/topology.c |  168 -----------------------------------------------
 3 files changed, 141 insertions(+), 169 deletions(-)
 delete mode 100644 drivers/base/topology.c
drivers/base/Makefile +1 -1
index 2609ba20b396..e50491bb3d60 100644
@@ -4,7 +4,7 @@ obj-y			:= component.o core.o bus.o dd.o syscore.o \
 			   driver.o class.o platform.o \
 			   cpu.o firmware.o init.o map.o devres.o \
 			   attribute_container.o transport_class.o \
-			   topology.o container.o property.o cacheinfo.o
+			   container.o property.o cacheinfo.o
 obj-$(CONFIG_DEVTMPFS)	+= devtmpfs.o
 obj-$(CONFIG_DMA_CMA) += dma-contiguous.o
 obj-y			+= power/
drivers/base/cpu.c +140 -0

152 lines not on this page · raw message

drivers/base/topology.c deleted +0 -168

168 lines not on this page · raw message

Signature · 1 line
1.7.9.3
#2 lore raw

PATCH v3 2/2 cpu hotplug: add CONFIG_PERMANENT_CPU_TOPOLOGY

In reply to Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a thread the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the thread's core has not been physically removed and the data in the topology directory is still valid and the core's location is now lost to userspace.

This patch adds CONFIG_PERMANENT_CPU_TOPOLOGY, and is Y by default for x86, an N for all other arches. When enabled the kernel is modified so that the topology directory is added to the core cpu sysfs files so that the topology directory exists while the CPU is physically present. When disabled, the behavior of the current kernel is maintained (that is, the topology directory is removed on a soft down and added on an soft up of a thread).

Adding CONFIG_PERMANENT_CPU_TOPOLOGY may require additional architecture so that the cpumask data the CPU's topology is not cleared during a CPU down.

Before patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# ls /sys/devices/system/cpu/cpu10/topology
ls: cannot access topology: No such file or directory

After patch:

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:ffff /sys/devices/system/cpu/cpu10/topology/core_siblings_list:0-15 /sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0404 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:2,10

Down a cpu

[root@hp-z620-01 ~]# echo 0 > /sys/devices/system/cpu/cpu10/online

[root@hp-z620-01 ~]# grep ^ /sys/devices/system/cpu/cpu10/topology/* /sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

I did some testing with and without BOOTPARAM_HOTPLUG_CPU0 enabled, and up'd and down'd threads in sequence, randomly, by thread group, by socket group and didn't see any issues.

core_siblings and thread_siblings are "numa siblings that are online" and "thread siblings that are online" and are used as such within the kernel. They must be zero'd out when the thread is offline.

CONFIG_PERMANENT_CPU_TOPOLOGY=y changes the lifetime of the topology directory from existing when a thread is online to when a thread is created and destroyed.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Len Brown <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Juergen Gross <[email protected]>
---
 arch/x86/kernel/smpboot.c |    3 ---
 drivers/base/Kconfig      |   12 ++++++++++++
 drivers/base/cpu.c        |    8 ++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
arch/x86/kernel/smpboot.c +0 -3
index 4296beb8fdd3..ae82f8f45b61 100644
@@ -1472,7 +1472,6 @@ static void recompute_smt_state(void)
 static void remove_siblinginfo(int cpu)
 {
 	int sibling;
-	struct cpuinfo_x86 *c = &cpu_data(cpu);
 
 	for_each_cpu(sibling, topology_core_cpumask(cpu)) {
 		cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));
@@ -1490,8 +1489,6 @@ static void remove_siblinginfo(int cpu)
 	cpumask_clear(cpu_llc_shared_mask(cpu));
 	cpumask_clear(topology_sibling_cpumask(cpu));
 	cpumask_clear(topology_core_cpumask(cpu));
-	c->phys_proc_id = 0;
-	c->cpu_core_id = 0;
 	cpumask_clear_cpu(cpu, cpu_sibling_setup_mask);
 	recompute_smt_state();
 }
drivers/base/Kconfig +12 -0
index 98504ec99c7d..b3935a272c3c 100644
@@ -324,4 +324,16 @@ config CMA_ALIGNMENT
 
 endif
 
+config PERMANENT_CPU_TOPOLOGY
+	bool "Permanent CPU Topology"
+	depends on HOTPLUG_CPU
+	def_bool y if X86_64
+	help
+	  This option configures CPU topology to be permanent for the lifetime
+	  of the CPU (until it is physically removed).  Selecting Y here
+	  results in the kernel reporting the physical location for offlined
+	  CPUs.
+
+	  If unsure, leave the default value as is.
+
 endmenu
drivers/base/cpu.c +8 -0
index ce722fdf48c6..891afc94c149 100644
@@ -263,6 +263,7 @@ static struct attribute_group topology_attr_group = {
 	.name = "topology"
 };
 
+#ifndef CONFIG_PERMANENT_CPU_TOPOLOGY
 /* Add/Remove cpu_topology interface for CPU device */
 static int topology_add_dev(unsigned int cpu)
 {
@@ -319,11 +320,15 @@ out:
 }
 
 device_initcall(topology_sysfs_init);
+#endif
 
 static const struct attribute_group *common_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 
@@ -331,6 +336,9 @@ static const struct attribute_group *hotplugable_cpu_attr_groups[] = {
 #ifdef CONFIG_KEXEC
 	&crash_note_cpu_attr_group,
 #endif
+#ifdef CONFIG_PERMANENT_CPU_TOPOLOGY
+	&topology_attr_group,
+#endif
 	NULL
 };
 
Signature · 1 line
1.7.9.3
#3 lore raw
In reply to Prarit Bhargava
Quoted 10 lines · Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid. IOW, the cpu is still present but the kernel has removed the topology directory making it very difficult to determine exactly where the cpu is located.

So I'm afraid I still don't understand what the problem here is.

And the commit message of 2/2 doesn't make it any clearer. Can you please give a concrete example what the problem is and what you're trying to achieve.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#4 lore raw
Quoted 12 lines · Borislav Petkov
Quoted 10 lines · Prarit Bhargava

The information in /sys/devices/system/cpu/cpuX/topology directory is useful for userspace monitoring applications and in-tree utilities like cpupower & turbostat.

When down'ing a CPU the /sys/devices/system/cpu/cpuX/topology directory is removed during the CPU_DEAD hotplug callback in the kernel. The problem with this model is that the CPU has not been physically removed and the data in the topology directory is still valid. IOW, the cpu is still present but the kernel has removed the topology directory making it very difficult to determine exactly where the cpu is located.

Quoted 5 lines

So I'm afraid I still don't understand what the problem here is.

And the commit message of 2/2 doesn't make it any clearer. Can you please give a concrete example what the problem is and what you're trying to achieve.

Sorry, I'll try again....

Right now, if you removed thread 29 from your system you would do:

echo 0 > /sys/devices/system/cpu/cpu29/online

Quoted 1 line

From the userspace side, this results in the removing of the

/sys/devices/system/cpu/cpu29/topology

directory.

This is not the right thing to do [1]. The topology directory should exist as long as the thread is present in the system. The thread (and its core) are still physically there, it's just that the thread is not available to the scheduler. The topology of the thread hasn't changed due to it being soft offlined this way.

turbostat was modified to deal with the missing topology directory, and in tree utility cpupower prints out significantly less information when a thread is offline. ISTR a powertop bug due to hotplug too. This makes these monitoring utilities a problem for users who want only one thread per core.

The patchset does two things. The first patch unifies the topology.c and cpu.c code. The second patch introduces a config option to change the lifetime of the topology directory to exist as long as the thread's device struct exists in the device subsystem.

This now means that

echo 0 > /sys/devices/system/cpu/cpu29/online

will result in the thread's topology directory staying around until the struct device associated with it is destroyed upon a physical socket hotplug event.

This patchset will result in cleanups to turbostat, and make fixes to cpupower *much* easier to deal with.

[1] I cannot say with any certainty that other arches do or do not require this change. That is the only reason the change is restricted to x86 right now.

P.

Quoted 3 lines

Thanks.

#5 lore raw
Quoted 5 lines · Prarit Bhargava

This is not the right thing to do [1]. The topology directory should exist as long as the thread is present in the system. The thread (and its core) are still physically there, it's just that the thread is not available to the scheduler. The topology of the thread hasn't changed due to it being soft offlined this way.

So far so good.

Quoted 3 lines

turbostat was modified to deal with the missing topology directory, and in tree utility cpupower prints out significantly less information when a thread is offline.

Why does it do that? Why does an offlined core change that info?

Concrete details please.

Quoted 2 lines

ISTR a powertop bug due to hotplug too. This makes these monitoring utilities a problem for users who want only one thread per core.

one thread per core? What does that mean?

Quoted 6 lines

This now means that

echo 0 > /sys/devices/system/cpu/cpu29/online

will result in the thread's topology directory staying around until the struct device associated with it is destroyed upon a physical socket hotplug event.

So your 2/2 says that on an offlined CPU, you have

/sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

and this information is bollocks. core_siblings is 0, thread_siblings is 0. You can just as well not have them there at all.

So is this whole jumping around just so that you can have a /sys/devices/system/cpu/cpu10/topology directory and so that tools don't get confused by it missing?

So again, what exactly are those tools accessing and how does the offlined cores puzzle them?

A concrete example please:

"turbostat tries to access X and it is gone when the CPU is offlined so this is a problem because it can't do Y"

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#6 lore raw
Quoted 13 lines · Borislav Petkov
Quoted 5 lines · Prarit Bhargava

This is not the right thing to do [1]. The topology directory should exist as long as the thread is present in the system. The thread (and its core) are still physically there, it's just that the thread is not available to the scheduler. The topology of the thread hasn't changed due to it being soft offlined this way.

So far so good.

Quoted 3 lines

turbostat was modified to deal with the missing topology directory, and in tree utility cpupower prints out significantly less information when a thread is offline.

Quoted 8 lines

Why does it do that? Why does an offlined core change that info?

Concrete details please.

Quoted 2 lines

ISTR a powertop bug due to hotplug too. This makes these monitoring utilities a problem for users who want only one thread per core.

one thread per core? What does that mean?

System boots with (usually) with 2 threads/core. Some performance users want one thread per core. Since there is no "noht" option anymore, users use /sys to disable a thread on each core.

Quoted 17 lines
Quoted 7 lines

This now means that

echo 0 > /sys/devices/system/cpu/cpu29/online

will result in the thread's topology directory staying around until the struct device associated with it is destroyed upon a physical socket hotplug event.

So your 2/2 says that on an offlined CPU, you have

/sys/devices/system/cpu/cpu10/topology/core_id:3 /sys/devices/system/cpu/cpu10/topology/core_siblings:0000 /sys/devices/system/cpu/cpu10/topology/core_siblings_list:
/sys/devices/system/cpu/cpu10/topology/physical_package_id:0 /sys/devices/system/cpu/cpu10/topology/thread_siblings:0000 /sys/devices/system/cpu/cpu10/topology/thread_siblings_list:

Quoted 2 lines

and this information is bollocks. core_siblings is 0, thread_siblings is 0. You can just as well not have them there at all.

core_siblings and thread_siblings are the online thread's sibling cores and threads that are available to the scheduler, and should be 0 when the thread is offline. That comes directly from reading the code.

Quoted 4 lines

So is this whole jumping around just so that you can have a /sys/devices/system/cpu/cpu10/topology directory and so that tools don't get confused by it missing?

Yes.

Quoted 6 lines

So again, what exactly are those tools accessing and how does the offlined cores puzzle them?

A concrete example please:

See commit 20102ac5bee3 ("cpupower: cpupower monitor reports uninitialized values for offline cpus"). That patch papers over the bug of not being able to find core_id and physical_package_id for an offline thread.

P.

#7 lore raw
Quoted 3 lines · Prarit Bhargava

System boots with (usually) with 2 threads/core. Some performance users want one thread per core. Since there is no "noht" option anymore, users use /sys to disable a thread on each core.

I see.

Quoted 2 lines

core_siblings and thread_siblings are the online thread's sibling cores and threads that are available to the scheduler

Hmm, I see something else:

<Documentation/cputopology.txt>:
7) /sys/devices/system/cpu/cpuX/topology/core_siblings:

internal kernel map of cpuX's hardware threads within the same physical_package_id.

Quoted 2 lines

and should be 0 when the thread is offline. That comes directly from reading the code.

But then code which reads those will have to *know* that those cores are offline - otherwise it would be confused by what it is reading there.

For example, the core siblings of an offlined core are still the same, they don't change. It is just the core that is offline.

Quoted 3 lines

See commit 20102ac5bee3 ("cpupower: cpupower monitor reports uninitialized values for offline cpus"). That patch papers over the bug of not being able to find core_id and physical_package_id for an offline thread.

Right, and this is *exactly* the *right* thing to do - tools should handle the case gracefully when cores are offline.

And we already state that explicitly in /sys/devices/system/cpu/online.

And for offlined cores they should show exactly that - cores are offlined and either say that the measurement is going to be wrong/innacurate or if they're showing per-core info, they don't show it for the offlined cores or show special symbols like stars and whatnot.

So far I still see no need for changing anything in the kernel.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#8 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Prarit Bhargava

System boots with (usually) with 2 threads/core. Some performance users want one thread per core. Since there is no "noht" option anymore, users use /sys to disable a thread on each core.

I see.

Quoted 2 lines

core_siblings and thread_siblings are the online thread's sibling cores and threads that are available to the scheduler

Quoted 7 lines

Hmm, I see something else:

<Documentation/cputopology.txt>:
7) /sys/devices/system/cpu/cpuX/topology/core_siblings:

internal kernel map of cpuX's hardware threads within the same physical_package_id.

Thanks. I'll send a patch to modify the Documentation which is out-of-date.

Quoted 6 lines
Quoted 3 lines

and should be 0 when the thread is offline. That comes directly from reading the code.

But then code which reads those will have to *know* that those cores are offline - otherwise it would be confused by what it is reading there.

When offline, /sys/devices/system/cpuX/cpu/online is 0. The problem is that when online is 0, topology disappears so there is no way to determine _the location_ of the offline'd thread.

Quoted 3 lines

For example, the core siblings of an offlined core are still the same, they don't change. It is just the core that is offline.

Please see (in latest linux.git)

arch/x86/kernel/smpboot.c:1493 function remove_siblinginfo()

Specifically,

        cpumask_clear(topology_sibling_cpumask(cpu));
        cpumask_clear(topology_core_cpumask(cpu));
Quoted 7 lines
Quoted 4 lines

See commit 20102ac5bee3 ("cpupower: cpupower monitor reports uninitialized values for offline cpus"). That patch papers over the bug of not being able to find core_id and physical_package_id for an offline thread.

Right, and this is *exactly* the *right* thing to do - tools should handle the case gracefully when cores are offline.

cpupower should still print out all asterisks for down'd threads. It does not because the topology directory is incorrectly removed.

IOW how does userspace know the _location_ of the thread? The topology directory no longer exists when the thread is downed, so core_id and physical_package_id (both of which would be effectively static) do not exist. The whole point of this patchset is to know where the offline'd thread actually is.

P.

#9 lore raw
Quoted 3 lines · Prarit Bhargava

When offline, /sys/devices/system/cpuX/cpu/online is 0. The problem is that when online is 0, topology disappears so there is no way to determine _the location_ of the offline'd thread.

What does "the location" mean exactly?

Quoted 7 lines

cpupower should still print out all asterisks for down'd threads. It does not because the topology directory is incorrectly removed.

IOW how does userspace know the _location_ of the thread? The topology directory no longer exists when the thread is downed, so core_id and physical_package_id (both of which would be effectively static) do not exist. The whole point of this patchset is to know where the offline'd thread actually is.

What do you mean "where"?

$ echo 0 > /sys/devices/system/cpu/cpu2/online
$ cat /sys/devices/system/cpu/online
0-1,3-7

So core 2 is right between 1 and 3.

If you need to show the package id, you still iterate over the core numbers in an increasing order and show '*' for the offlined ones.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#10 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Prarit Bhargava

When offline, /sys/devices/system/cpuX/cpu/online is 0. The problem is that when online is 0, topology disappears so there is no way to determine _the location_ of the offline'd thread.

What does "the location" mean exactly?

Quoted 3 lines

cpupower should still print out all asterisks for down'd threads. It does not because the topology directory is incorrectly removed.

Quoted 6 lines
Quoted 4 lines

IOW how does userspace know the _location_ of the thread? The topology directory no longer exists when the thread is downed, so core_id and physical_package_id (both of which would be effectively static) do not exist. The whole point of this patchset is to know where the offline'd thread actually is.

What do you mean "where"?

The socket and core location.

Look at it this way (and let's get the terminology straight at the same time).

You have a socket CPU. That socket has cores on it. Each core (at least on Intel) has two threads.

I down a thread (as you did):

Quoted 2 lines
$ echo 0 > /sys/devices/system/cpu/cpu2/online

This results in the topology directory being destroyed. It shouldn't be -- the socket and core are still there. If you could open up your computer you could touch them. This is similar to downing a PCI device, or removing !kernel memory DIMM from a system. The device is still physically there.

Quoted 4 lines
$ cat /sys/devices/system/cpu/online
0-1,3-7

So core 2 is right between 1 and 3.

Yes. But *where* is it relative to the cores and socket(s)?

Quoted 4 lines

If you need to show the package id, you still iterate over the core numbers in an increasing order and show '*' for the offlined ones.

Explain this in more detail please?

P.

#11 lore raw
Quoted 7 lines · Prarit Bhargava
Quoted 1 line · Borislav Petkov
$ echo 0 > /sys/devices/system/cpu/cpu2/online

This results in the topology directory being destroyed. It shouldn't be -- the socket and core are still there. If you could open up your computer you could touch them. This is similar to downing a PCI device, or removing !kernel memory DIMM from a system. The device is still physically there.

If you remove a PCI device from the system, by turning it off from the PCI hotplug interface, it will go away from sysfs, just like this CPU device is going away, no matter if you physically remove the device or not.

Thanks for validating that everything is working the same and correctly :)

greg k-h

#12 lore raw
Quoted 9 lines · Greg Kroah-Hartman
Quoted 7 lines · Prarit Bhargava
Quoted 1 line · Borislav Petkov
$ echo 0 > /sys/devices/system/cpu/cpu2/online

This results in the topology directory being destroyed. It shouldn't be -- the socket and core are still there. If you could open up your computer you could touch them. This is similar to downing a PCI device, or removing !kernel memory DIMM from a system. The device is still physically there.

Quoted 5 lines

If you remove a PCI device from the system, by turning it off from the PCI hotplug interface, it will go away from sysfs, just like this CPU device is going away, no matter if you physically remove the device or not.

While similar, the thread's device struct, link to structures, and masks in the kernel are kept in place. In the PCI case these links are all destroyed which results in requiring a bus rescan to find the devcie. The sysfs (soft) removal of a thread has a very different outcome from PCI.

I see now that the issue is not understanding the difference between physical and soft thread removal. I will write that up and get back to everyone.

P.

Quoted 5 lines

Thanks for validating that everything is working the same and correctly :)

greg k-h

#13 lore raw
Quoted 2 lines · Prarit Bhargava

I see now that the issue is not understanding the difference between physical and soft thread removal. I will write that up and get back to everyone.

No need - we understand the issue.

What I don't understand is what information you need *exactly* in sysfs from the offlined cores and why. Something like "I need to know the id of the thread on core X on socket Y because..."

I've been trying to get it out of you but I've failed so far at it.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#14 lore raw
In reply to Prarit Bhargava
Quoted 2 lines · Prarit Bhargava

I see now that the issue is not understanding the difference between physical and soft thread removal. I will write that up and get back to everyone.

You don't seem to understand that from the kernels POV there is no such distinction.

There is only one unplug operation that caters to both cases. Therefore unplug needs to assume the most stringent (ie. physical) for things to work right.

There is no 'soft thread removal', except maybe in the userspace API, you're the one that is confused.

#15 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Prarit Bhargava

I see now that the issue is not understanding the difference between physical and soft thread removal. I will write that up and get back to everyone.

You don't seem to understand that from the kernels POV there is no such distinction.

And I'm saying you're wrong. Unlike PCI, CPU sysfs unplug does not completely remove the device from the kernel's knowledge. If that were the case then

/sys/devices/system/cpu/cpuX/online (and other files left after the unplug) wouldn't exist.

If you do a physical removal (signaled via ACPI or interrupt) the entire device is removed from the kernel's POV.

Yes, the sysfs removal is a subset of the physical removal. But the end result is very different.

Quoted 3 lines

There is only one unplug operation that caters to both cases. Therefore unplug needs to assume the most stringent (ie. physical) for things to work right.

See last comment above.

P.

#16 lore raw
In reply to Prarit Bhargava
Quoted 1 line · Prarit Bhargava

Yes. But *where* is it relative to the cores and socket(s)?

And you need that information because...

Quoted 5 lines
Quoted 3 lines

If you need to show the package id, you still iterate over the core numbers in an increasing order and show '*' for the offlined ones.

Explain this in more detail please?

Just s/package id/core id/ and then it makes sense - I meant "core id".

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#17 lore raw
In reply to Prarit Bhargava
Quoted 6 lines · Prarit Bhargava
Quoted 2 lines

But then code which reads those will have to *know* that those cores are offline - otherwise it would be confused by what it is reading there.

When offline, /sys/devices/system/cpuX/cpu/online is 0. The problem is that when online is 0, topology disappears so there is no way to determine _the location_ of the offline'd thread.

As far as all that code is concerned, that CPU doesn't even have a location anymore.

While there might be some distinction between hotplug and physical hotplug on the user API side (I really wouldn't know), there isn't on the kernel side.

Once you unplug a CPU, its _gone_. There isn't another hotplug operation once you really take the CPU out.

Offline means out gone, vamoosh.

And it doesn't make sense to talk about the location of a resource that's not there.

#18 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Prarit Bhargava
Quoted 2 lines

But then code which reads those will have to *know* that those cores are offline - otherwise it would be confused by what it is reading there.

When offline, /sys/devices/system/cpuX/cpu/online is 0. The problem is that when online is 0, topology disappears so there is no way to determine _the location_ of the offline'd thread.

As far as all that code is concerned, that CPU doesn't even have a location anymore.

Quoted 6 lines

While there might be some distinction between hotplug and physical hotplug on the user API side (I really wouldn't know), there isn't on the kernel side.

Once you unplug a CPU, its _gone_. There isn't another hotplug operation once you really take the CPU out.

There's a difference between soft remove (via sysfs) and a true hot remove operation (where the whole thing is physically removed). Soft remove only results in the processor being made "not available" to the scheduler.

Quoted 2 lines

Offline means out gone, vamoosh.

No, that is incorrect. The socket that contains the cores (and threads) is still plugged in.

Quoted 4 lines

And it doesn't make sense to talk about the location of a resource that's not there.

Again, it is _physically_ there.

P.

#19 lore raw
Quoted 3 lines · Prarit Bhargava

There's a difference between soft remove (via sysfs) and a true hot remove operation (where the whole thing is physically removed). Soft remove only results in the processor being made "not available" to the scheduler.

How is it different? We do _ONE_ CPU unplug operation. We do not touch the thing anymore after that. It _can_ be taken out after that.

Therefore hotplug wipes the topology information and clears the CPU from relevant bitmasks.

Quoted 1 line

From the kernel's POV there really is no distinction.