Skip to contentSkip to conversation

ARM: smp: Only expose /sys/.../cpuX/online if hotpluggable

PATCH Superseded Stephen Boyd 4 participants 7 messages Feb 14, 2015 – Apr 6, 2015

Subsystems (4)
Arm (32-bit) Arm sub-architectures SuperH Arm/RISC-V/Renesas architecture

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Stephen Boyd

author
#0 lore raw

PATCH v3 1/1 ARM: smp: Only expose /sys/.../cpuX/online if hotpluggable

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)
arch/arm/common/mcpm_platsmp.c +4 -8
index 92e54d7c6f46..dc91ed04c25a 100644
@@ -65,14 +65,10 @@ static int mcpm_cpu_kill(unsigned int cpu)
 	return !mcpm_wait_for_cpu_powerdown(pcpu, pcluster);
 }
 
-static int mcpm_cpu_disable(unsigned int cpu)
+static int mcpm_cpu_can_disable(unsigned int cpu)
 {
-	/*
-	 * We assume all CPUs may be shut down.
-	 * This would be the hook to use for eventual Secure
-	 * OS migration requests as described in the PSCI spec.
-	 */
-	return 0;
+	/* We assume all CPUs may be shut down. */
+	return 1;
 }
 
 static void mcpm_cpu_die(unsigned int cpu)

@@ -92,7 +88,7 @@ static struct smp_operations __initdata mcpm_smp_ops = {
 	.smp_secondary_init	= mcpm_secondary_init,
 #ifdef CONFIG_HOTPLUG_CPU
 	.cpu_kill		= mcpm_cpu_kill,
-	.cpu_disable		= mcpm_cpu_disable,
+	.cpu_can_disable	= mcpm_cpu_can_disable,
 	.cpu_die		= mcpm_cpu_die,
 #endif
 };
arch/arm/include/asm/smp.h +10 -0
index 18f5a554134f..525c006fdf7c 100644
@@ -104,6 +104,7 @@ struct smp_operations {
 #ifdef CONFIG_HOTPLUG_CPU
 	int  (*cpu_kill)(unsigned int cpu);
 	void (*cpu_die)(unsigned int cpu);
+	int  (*cpu_can_disable)(unsigned int cpu);
 	int  (*cpu_disable)(unsigned int cpu);
 #endif
 #endif

@@ -123,4 +124,13 @@ struct of_cpu_method {
  */
 extern void smp_set_ops(struct smp_operations *);
 
+#ifdef CONFIG_HOTPLUG_CPU
+extern int platform_can_hotplug_cpu(unsigned int cpu);
+#else
+static inline int platform_can_hotplug_cpu(unsigned int cpu)
+{
+	return 0;
+}
+#endif
+
 #endif /* ifndef __ASM_ARM_SMP_H */
arch/arm/kernel/setup.c +1 -1
index 715ae19bc7c8..c61c09defc78 100644
@@ -974,7 +974,7 @@ static int __init topology_init(void)
 
 	for_each_possible_cpu(cpu) {
 		struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu);
-		cpuinfo->cpu.hotpluggable = 1;
+		cpuinfo->cpu.hotpluggable = platform_can_hotplug_cpu(cpu);
 		register_cpu(&cpuinfo->cpu, cpu);
 	}
arch/arm/kernel/smp.c +14 -1
index fe0386c751b2..152f65ef0494 100644
@@ -179,13 +179,26 @@ static int platform_cpu_disable(unsigned int cpu)
 	if (smp_ops.cpu_disable)
 		return smp_ops.cpu_disable(cpu);
 
+	return 0;
+}
+
+int platform_can_hotplug_cpu(unsigned int cpu)
+{
+	/* cpu_die must be specified to support hotplug */
+	if (!smp_ops.cpu_die)
+		return 0;
+
+	if (smp_ops.cpu_can_disable)
+		return smp_ops.cpu_can_disable(cpu);
+
 	/*
 	 * By default, allow disabling all CPUs except the first one,
 	 * since this is special on a lot of platforms, e.g. because
 	 * of clock tick interrupts.
 	 */
-	return cpu == 0 ? -EPERM : 0;
+	return cpu == 0 ? 0 : 1;
 }
+
 /*
  * __cpu_disable runs on the processor to be shutdown.
  */
arch/arm/mach-shmobile/common.h +1 -1
index 309025efd4cf..e124df267edc 100644
@@ -13,7 +13,7 @@ extern void shmobile_smp_boot(void);
 extern void shmobile_smp_sleep(void);
 extern void shmobile_smp_hook(unsigned int cpu, unsigned long fn,
 			      unsigned long arg);
-extern int shmobile_smp_cpu_disable(unsigned int cpu);
+extern int shmobile_smp_cpu_can_disable(unsigned int cpu);
 extern void shmobile_invalidate_start(void);
 extern void shmobile_boot_scu(void);
 extern void shmobile_smp_scu_prepare_cpus(unsigned int max_cpus);
arch/arm/mach-shmobile/platsmp.c +2 -2
index 3923e09e966d..a614cef18db1 100644
@@ -31,8 +31,8 @@ void shmobile_smp_hook(unsigned int cpu, unsigned long fn, unsigned long arg)
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
-int shmobile_smp_cpu_disable(unsigned int cpu)
+int shmobile_smp_cpu_can_disable(unsigned int cpu)
 {
-	return 0; /* Hotplug of any CPU is supported */
+	return 1; /* Hotplug of any CPU is supported */
 }
 #endif
arch/arm/mach-shmobile/smp-r8a7790.c +1 -1
index 9c3da1345b8b..67b86c434fb8 100644
@@ -63,7 +63,7 @@ struct smp_operations r8a7790_smp_ops __initdata = {
 	.smp_prepare_cpus	= r8a7790_smp_prepare_cpus,
 	.smp_boot_secondary	= shmobile_smp_apmu_boot_secondary,
 #ifdef CONFIG_HOTPLUG_CPU
-	.cpu_disable		= shmobile_smp_cpu_disable,
+	.cpu_can_disable	= shmobile_smp_cpu_can_disable,
 	.cpu_die		= shmobile_smp_apmu_cpu_die,
 	.cpu_kill		= shmobile_smp_apmu_cpu_kill,
 #endif
arch/arm/mach-shmobile/smp-r8a7791.c +1 -1
index 7e49e0a52e32..24b7f12e20bb 100644
@@ -58,7 +58,7 @@ struct smp_operations r8a7791_smp_ops __initdata = {
 	.smp_prepare_cpus	= r8a7791_smp_prepare_cpus,
 	.smp_boot_secondary	= r8a7791_smp_boot_secondary,
 #ifdef CONFIG_HOTPLUG_CPU
-	.cpu_disable		= shmobile_smp_cpu_disable,
+	.cpu_can_disable	= shmobile_smp_cpu_can_disable,
 	.cpu_die		= shmobile_smp_apmu_cpu_die,
 	.cpu_kill		= shmobile_smp_apmu_cpu_kill,
 #endif
arch/arm/mach-shmobile/smp-sh73a0.c +1 -1
index c16dbfe9836c..4dddbd3844dd 100644
@@ -68,7 +68,7 @@ struct smp_operations sh73a0_smp_ops __initdata = {
 	.smp_prepare_cpus	= sh73a0_smp_prepare_cpus,
 	.smp_boot_secondary	= sh73a0_boot_secondary,
 #ifdef CONFIG_HOTPLUG_CPU
-	.cpu_disable		= shmobile_smp_cpu_disable,
+	.cpu_can_disable	= shmobile_smp_cpu_can_disable,
 	.cpu_die		= shmobile_smp_scu_cpu_die,
 	.cpu_kill		= shmobile_smp_scu_cpu_kill,
 #endif
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#1 lore raw

arch/arm/include/asm/smp.h struct smp_operations { · line 107

Quoted 7 lines · Stephen Boyd
arch/arm/include/asm/smp.h +1 -0
@@ -104,6 +104,7 @@ struct smp_operations {
 #ifdef CONFIG_HOTPLUG_CPU
        int  (*cpu_kill)(unsigned int cpu);
        void (*cpu_die)(unsigned int cpu);
+       int  (*cpu_can_disable)(unsigned int cpu);

Perhaps this should be changed to return bool while you're adding the function? That saves us from an atomic update later when someone does janitorial work.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#2 lore raw
In reply to Stephen Boyd
Quoted 34 lines · Stephen Boyd

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

Quoted 10 lines
 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)

I think it would make sense to separate the ARM-core changes from the mach-shmobile integration changes.

author
#3 lore raw
Quoted 45 lines · Simon Horman
Quoted 34 lines · Stephen Boyd

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

Quoted 10 lines
 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)
Quoted 2 lines

I think it would make sense to separate the ARM-core changes from the mach-shmobile integration changes.

Are you saying two (three?) patches to add the op, and then move over each struct smp_operations? It's all going through rmk's tree so I'll leave that up to him.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#4 lore raw
Quoted 46 lines · Stephen Boyd
Quoted 45 lines · Simon Horman
Quoted 44 lines · Stephen Boyd

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)
Quoted 6 lines
Quoted 2 lines

I think it would make sense to separate the ARM-core changes from the mach-shmobile integration changes.

Are you saying two (three?) patches to add the op, and then move over each struct smp_operations? It's all going through rmk's tree so I'll leave that up to him.

I'm also happy to let RMK to decide what he thinks is best.

#5 lore raw
Quoted 47 lines · Simon Horman
Quoted 46 lines · Stephen Boyd
Quoted 45 lines · Simon Horman
Quoted 44 lines · Stephen Boyd

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I think it would make sense to separate the ARM-core changes from the mach-shmobile integration changes.

Are you saying two (three?) patches to add the op, and then move over each struct smp_operations? It's all going through rmk's tree so I'll leave that up to him.

I'm also happy to let RMK to decide what he thinks is best.

Apologies for bringing up an older thread, but was this change ever picked up? I recently hit this issue when I was running the kselftest suite (specifically the cpu-hotplug test case) on the ARM boards from kernelci.org. I don't see it in -next so I've tested the core changes and confirmed it the fixes the cpu-hotplug behavior on platforms that do not support it.

Quoted 5 lines

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

- Tyler

author
#6 lore raw
Quoted 53 lines · Tyler Baker
Quoted 52 lines · Simon Horman
Quoted 51 lines · Stephen Boyd
Quoted 45 lines · Simon Horman
Quoted 34 lines · Stephen Boyd

Writes to /sys/.../cpuX/online fail if we determine the platform doesn't support hotplug for that CPU. Furthermore, if the cpu_die op isn't specified the system hangs when we try to offline a CPU and it comes right back online unexpectedly. Let's figure this stuff out before we make the sysfs nodes so that the online file doesn't even exist if it isn't (at least sometimes) possible to hotplug the CPU.

Add a new cpu_can_disable op and repoint all cpu_disable implementations at it because all current users use the op to indicate if a CPU can be hotplugged or not in a static fashion. With PSCI we may need to introduce a cpu_disable op so that the secure OS can be migrated off the CPU we're trying to hotplug. In this case, the cpu_can_disable op will indicate that all CPUs are hotpluggable by returning 1, but the cpu_disable op will make a PSCI migration call and occasionally fail, denying the hotplug of a CPU. This shouldn't be any worse than x86 where we may indicate that all CPUs are hotpluggable but occasionally we can't offline a CPU due to check_irq_vectors_for_cpu_disable() failing to find a CPU to move vectors to.

Cc Mark Rutland <[email protected]>
Cc Nicolas Pitre <[email protected]>
Cc Dave Martin <[email protected]>
Cc Simon Horman <[email protected]>
Cc Magnus Damm <[email protected]>
Cc <[email protected]>
Signed-off-by Stephen Boyd <[email protected]>

---

Changes since v2:
* Left cpu_disable op in place
* Split out shmobile function deletion

Quoted 10 lines
 arch/arm/common/mcpm_platsmp.c       | 12 ++++--------
 arch/arm/include/asm/smp.h           | 10 ++++++++++
 arch/arm/kernel/setup.c              |  2 +-
 arch/arm/kernel/smp.c                | 15 ++++++++++++++-
 arch/arm/mach-shmobile/common.h      |  2 +-
 arch/arm/mach-shmobile/platsmp.c     |  4 ++--
 arch/arm/mach-shmobile/smp-r8a7790.c |  2 +-
 arch/arm/mach-shmobile/smp-r8a7791.c |  2 +-
 arch/arm/mach-shmobile/smp-sh73a0.c  |  2 +-
 9 files changed, 35 insertions(+), 16 deletions(-)
Quoted 2 lines

I think it would make sense to separate the ARM-core changes from the mach-shmobile integration changes.

Are you saying two (three?) patches to add the op, and then move over each struct smp_operations? It's all going through rmk's tree so I'll leave that up to him.

Quoted 7 lines
Quoted 1 line

I'm also happy to let RMK to decide what he thinks is best.

Apologies for bringing up an older thread, but was this change ever picked up? I recently hit this issue when I was running the kselftest suite (specifically the cpu-hotplug test case) on the ARM boards from kernelci.org. I don't see it in -next so I've tested the core changes and confirmed it the fixes the cpu-hotplug behavior on platforms that do not support it.

No it hasn't gone anywhere. It would be good if Mark Rutland could ack the patch, which I hope would give enough confidence to Russell that the patch is good. Your tested-by would also be welcome. I'll go make the bool change that Geert suggested and resend.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project