Skip to contentSkip to conversation

cpu_hotplug on IBM JS20 system

RFC PATCH In review Srinivasa Ds 5 participants 8 messages Nov 13, 2006 – Nov 15, 2006

Subsystems (2)
PowerPC CPU hotplug
lore
Share & tools
lore

Srinivasa Ds

author
#0 lore raw

RFC PATCH 1/1 cpu_hotplug on IBM JS20 system

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped in to xmon. On analyzing the problem,I found out that "self-stop" token is not exported to the OS through rtas(Could be verified by looking in to /proc/device-tree/rtas file).

1:mon> e
cpu 0x1: Vector: 700 (Program Check) at [c00000000ff1bab0]
pc: c00000000001b144: .rtas_stop_self+0x34/0x70
lr: c0000000000439c0: .pSeries_mach_cpu_die+0x34/0x40
sp: c00000000ff1bd30
msr: 8000000000021032
current = 0xc00000000ff050b0
paca = 0xc0000000005ec500
pid = 0, comm = swapper
kernel BUG in rtas_stop_self at arch/powerpc/kernel/rtas.c:829!
===========================================
void rtas_stop_self(void)

{
      struct rtas_args *rtas_args = &rtas_stop_self_args;

      local_irq_disable();

      BUG_ON(rtas_args->token == RTAS_UNKNOWN_SERVICE);
===================================================
#ifdef CONFIG_HOTPLUG_CPU
      rtas_stop_self_args.token = rtas_token("stop-self");
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_RTAS_ERROR_LOGGING
      rtas_last_error_token = rtas_token("rtas-last-error");

===================================================

Since we are not supported by hardware for cpu hotplug. I have developed the patch which will disable cpu hotplug on IBM bladecentre JS20. Please let me know your comments on this please.

Signed-off-by Srinivasa DS <[email protected]>

Thanks
Srinivasa DS

 arch/powerpc/kernel/rtas.c |    9 +++++++++
 kernel/cpu.c               |    4 ++--
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/powerpc/kernel/rtas.c +9 -0
Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
@@ -53,6 +53,9 @@
 
 unsigned long rtas_rmo_buf;
 
+extern  int cpu_hotplug_disabled;
+extern  struct mutex cpu_add_remove_lock;
+
 /*
  * If non-NULL, this gets called when the kernel terminates.
  * This is done like this so rtas_flash can be a module.

@@ -881,6 +884,12 @@
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
+		mutex_lock(&cpu_add_remove_lock);
+		cpu_hotplug_disabled = 1;
+		mutex_unlock(&cpu_add_remove_lock);
+	}
+
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
kernel/cpu.c +2 -2
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
@@ -16,7 +16,7 @@
 #include <linux/mutex.h>
 
 /* This protects CPUs going up and down... */
-static DEFINE_MUTEX(cpu_add_remove_lock);
+DEFINE_MUTEX(cpu_add_remove_lock);
 static DEFINE_MUTEX(cpu_bitmask_lock);
 
 static __cpuinitdata RAW_NOTIFIER_HEAD(cpu_chain);

@@ -24,7 +24,7 @@
 /* If set, cpu_up and cpu_down will return -EBUSY and do nothing.
  * Should always be manipulated under cpu_add_remove_lock
  */
-static int cpu_hotplug_disabled;
+int cpu_hotplug_disabled;
 
 #ifdef CONFIG_HOTPLUG_CPU
#1 lore raw
Quoted 6 lines

Since we are not supported by hardware for cpu hotplug. I have developed the patch which will disable cpu hotplug on IBM bladecentre JS20. Please let me know your comments on this please.

arch/powerpc/kernel/rtas.c +2 -0
+extern  int cpu_hotplug_disabled;
+extern  struct mutex cpu_add_remove_lock;

[...]

arch/powerpc/kernel/rtas.c line 895

Quoted 9 lines
arch/powerpc/kernel/rtas.c +6 -0
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
+		mutex_lock(&cpu_add_remove_lock);
+		cpu_hotplug_disabled = 1;
+		mutex_unlock(&cpu_add_remove_lock);
+	}
+
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");

You should add a function to kernel/cpu.c which you can call in order to disable cpu hotplug instead of exporting its private data structures.

#2 lore raw
Quoted 7 lines · Heiko Carstens
Quoted 6 lines

Since we are not supported by hardware for cpu hotplug. I have developed the patch which will disable cpu hotplug on IBM bladecentre JS20. Please let me know your comments on this please.

arch/powerpc/kernel/rtas.c +2 -0
+extern  int cpu_hotplug_disabled;
+extern  struct mutex cpu_add_remove_lock;

[...]

Quoted 12 lines

arch/powerpc/kernel/rtas.c line 895

Quoted 9 lines
arch/powerpc/kernel/rtas.c +6 -0
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE) {
+		mutex_lock(&cpu_add_remove_lock);
+		cpu_hotplug_disabled = 1;
+		mutex_unlock(&cpu_add_remove_lock);
+	}
+
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");

You should add a function to kernel/cpu.c which you can call in order to disable cpu hotplug instead of exporting its private data structures.

Yup. Also, considering the fact that enable_nonboot_cpus() can reset the cpu_hotplug_disabled flag, I would suggest the following:

a) create one additional state for cpu_hotplug_disabled, something like

#define PERMANENTLY_DISABLED -1

b) Define a function kernel/cpu.c

void disable_cpu_hotplug_perm()

{
	cpu_hotplug_disabled = PERMANENTLY_DISABLED;
}

and call it in rtas.c

c) Check for status of cpu_hotplug_disabled in functions cpu_up, cpu_down, enable_nonboot_cpus and disable_nonboot_cpus and if cpu_hotplug_disabled == PERMANENTLY_DISABLED, return bypassing further code.

Thoughts?

Regards,
gautham.

Signature · 5 lines
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"
#3 lore raw
In reply to Srinivasa Ds
Quoted 6 lines · Srinivasa Ds

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped in to xmon. On analyzing the problem,I found out that "self-stop" token is not exported
to the OS through rtas(Could be verified by looking in to /proc/device-tree/rtas file).

Please CC powerpc related patches to [email protected]

author
#4 lore raw

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped in to xmon. On analyzing the problem,I found out that "self-stop" token is not exported to the OS through rtas(Could be verified by looking in to /proc/device-tree/rtas file).

1:mon> e
cpu 0x1: Vector: 700 (Program Check) at [c00000000ff1bab0]
pc: c00000000001b144: .rtas_stop_self+0x34/0x70
lr: c0000000000439c0: .pSeries_mach_cpu_die+0x34/0x40
sp: c00000000ff1bd30
msr: 8000000000021032
current = 0xc00000000ff050b0
paca = 0xc0000000005ec500
pid = 0, comm = swapper
kernel BUG in rtas_stop_self at arch/powerpc/kernel/rtas.c:829!
===========================================
void rtas_stop_self(void)

{
     struct rtas_args *rtas_args = &rtas_stop_self_args;

     local_irq_disable();

     BUG_ON(rtas_args->token == RTAS_UNKNOWN_SERVICE);
===================================================
#ifdef CONFIG_HOTPLUG_CPU
     rtas_stop_self_args.token = rtas_token("stop-self");
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_RTAS_ERROR_LOGGING
     rtas_last_error_token = rtas_token("rtas-last-error");

===================================================

Since we are not supported by hardware for cpu hotplug. I have developed the patch which will disable cpu hotplug on IBM bladecentre JS20. Please let me know your comments on this please.

Signed-off-by Srinivasa DS <[email protected]>
 arch/powerpc/kernel/rtas.c |    3 +++
 include/linux/cpu.h        |    4 ++++
 kernel/cpu.c               |   24 ++++++++++++++++++++++--
 3 files changed, 29 insertions(+), 2 deletions(-)
arch/powerpc/kernel/rtas.c +3 -0
Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
@@ -19,6 +19,7 @@
 #include <linux/init.h>
 #include <linux/capability.h>
 #include <linux/delay.h>
+#include <linux/cpu.h>
 
 #include <asm/prom.h>
 #include <asm/rtas.h>

@@ -881,6 +882,8 @@
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE)
+		disable_cpu_hotplug_perm();
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
kernel/cpu.c +22 -2
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
@@ -63,6 +63,18 @@
 }
 EXPORT_SYMBOL_GPL(unlock_cpu_hotplug);
 
+void disable_cpu_hotplug_perm(void)
+{
+	mutex_lock(&cpu_add_remove_lock);
+	cpu_hotplug_disabled = PERM_DISABLED_CPU_HOTPLUG;
+	mutex_unlock(&cpu_add_remove_lock);
+}
+
+static int is_cpu_hotplug_perm_disabled()
+{
+	return cpu_hotplug_disabled == PERM_DISABLED_CPU_HOTPLUG;
+}
+
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */

@@ -193,7 +205,7 @@
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_down(cpu);

@@ -244,7 +256,7 @@
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_up(cpu);

@@ -261,6 +273,10 @@
 	int cpu, first_cpu, error;
 
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		error = -EBUSY;
+		goto out;
+	}
 	first_cpu = first_cpu(cpu_present_map);
 	if (!cpu_online(first_cpu)) {
 		error = _cpu_up(first_cpu);

@@ -311,6 +327,10 @@
 
 	/* Allow everyone to use the CPU hotplug again */
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		mutex_unlock(&cpu_add_remove_lock);
+		return;
+	}
 	cpu_hotplug_disabled = 0;
 	mutex_unlock(&cpu_add_remove_lock);
include/linux/cpu.h +4 -0
Index: linux-2.6.19-rc5/include/linux/cpu.h
===================================================================
@@ -31,6 +31,8 @@
 	struct sys_device sysdev;
 };
 
+#define  PERM_DISABLED_CPU_HOTPLUG -1
+
 extern int register_cpu(struct cpu *cpu, int num);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU

@@ -68,6 +70,7 @@
 /* Stop CPUs going up and down. */
 extern void lock_cpu_hotplug(void);
 extern void unlock_cpu_hotplug(void);
+extern void disable_cpu_hotplug_perm(void);
 #define hotcpu_notifier(fn, pri) {				\
 	static struct notifier_block fn##_nb =			\
 		{ .notifier_call = fn, .priority = pri };	\

@@ -80,6 +83,7 @@
 #else
 #define lock_cpu_hotplug()	do { } while (0)
 #define unlock_cpu_hotplug()	do { } while (0)
+#define disable_cpu_hotplug_perm()     do { } while (0)
 #define lock_cpu_hotplug_interruptible() 0
 #define hotcpu_notifier(fn, pri)	do { } while (0)
 #define register_hotcpu_notifier(nb)	do { } while (0)
author
#5 lore raw
Quoted 3 lines · Srinivasa Ds

Since we are not supported by hardware for cpu hotplug. I have developed the patch which will disable cpu hotplug on IBM bladecentre JS20. Please let me know your comments on this please.

Iam sorry, Just resending the patch after formatting it again.

Signed-off-by Srinivasa DS <[email protected]>
 arch/powerpc/kernel/rtas.c |    3 +++
 include/linux/cpu.h        |    4 ++++
 kernel/cpu.c               |   24 ++++++++++++++++++++++--
 3 files changed, 29 insertions(+), 2 deletions(-)
arch/powerpc/kernel/rtas.c +3 -0
Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
@@ -19,6 +19,7 @@
 #include <linux/init.h>
 #include <linux/capability.h>
 #include <linux/delay.h>
+#include <linux/cpu.h>
 
 #include <asm/prom.h>
 #include <asm/rtas.h>

@@ -881,6 +882,8 @@ void __init rtas_initialize(void)
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE)
+		disable_cpu_hotplug_perm();
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
kernel/cpu.c +22 -2
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
@@ -63,8 +63,20 @@ void unlock_cpu_hotplug(void)
 }
 EXPORT_SYMBOL_GPL(unlock_cpu_hotplug);
 
+void disable_cpu_hotplug_perm(void)
+{
+	mutex_lock(&cpu_add_remove_lock);
+	cpu_hotplug_disabled = PERM_DISABLED_CPU_HOTPLUG;
+	mutex_unlock(&cpu_add_remove_lock);
+}
+
 #endif	/* CONFIG_HOTPLUG_CPU */
 
+static int is_cpu_hotplug_perm_disabled()
+{
+	return cpu_hotplug_disabled == PERM_DISABLED_CPU_HOTPLUG;
+}
+
 /* Need to know about CPUs going up/down? */
 int __cpuinit register_cpu_notifier(struct notifier_block *nb)
 {

@@ -193,7 +205,7 @@ int cpu_down(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_down(cpu);

@@ -244,7 +256,7 @@ int __devinit cpu_up(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_up(cpu);

@@ -261,6 +273,10 @@ int disable_nonboot_cpus(void)
 	int cpu, first_cpu, error;
 
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		error = -EBUSY;
+		goto out;
+	}
 	first_cpu = first_cpu(cpu_present_map);
 	if (!cpu_online(first_cpu)) {
 		error = _cpu_up(first_cpu);

@@ -311,6 +327,10 @@ void enable_nonboot_cpus(void)
 
 	/* Allow everyone to use the CPU hotplug again */
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		mutex_unlock(&cpu_add_remove_lock);
+		return;
+	}
 	cpu_hotplug_disabled = 0;
 	mutex_unlock(&cpu_add_remove_lock);
include/linux/cpu.h +4 -0
Index: linux-2.6.19-rc5/include/linux/cpu.h
===================================================================
@@ -31,6 +31,8 @@ struct cpu {
 	struct sys_device sysdev;
 };
 
+#define  PERM_DISABLED_CPU_HOTPLUG -1
+
 extern int register_cpu(struct cpu *cpu, int num);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU

@@ -68,6 +70,7 @@ extern struct sysdev_class cpu_sysdev_cl
 /* Stop CPUs going up and down. */
 extern void lock_cpu_hotplug(void);
 extern void unlock_cpu_hotplug(void);
+extern void disable_cpu_hotplug_perm(void);
 #define hotcpu_notifier(fn, pri) {				\
 	static struct notifier_block fn##_nb =			\
 		{ .notifier_call = fn, .priority = pri };	\

@@ -80,6 +83,7 @@ int cpu_down(unsigned int cpu);
 #else
 #define lock_cpu_hotplug()	do { } while (0)
 #define unlock_cpu_hotplug()	do { } while (0)
+#define disable_cpu_hotplug_perm()     do { } while (0)
 #define lock_cpu_hotplug_interruptible() 0
 #define hotcpu_notifier(fn, pri)	do { } while (0)
 #define register_hotcpu_notifier(nb)	do { } while (0)
#6 lore raw
In reply to Srinivasa Ds
Quoted 3 lines · Srinivasa Ds

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped in to xmon.

How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
author
#7 lore raw
Quoted 8 lines · Michael Ellerman
Quoted 3 lines · Srinivasa Ds

Hi
when I tried to hot plug a cpu on IBM bladecentre JS20 system,it dropped in to xmon.

How did you try to hot plug a cpu? Through sysfs/hmc/somethingelse ?

echo 0 > /sys/devices/system/cpu/cpu1/online

Quoted 3 lines

cheers