Skip to contentSkip to conversation

cpu hotplug on power based systems.

RFC PATCH In review Srinivasa Ds 6 participants 15 messages Nov 16, 2006 – Nov 21, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 cpu hotplug on power based systems.

Linux kernel uses some of the rtas token to perform cpu hotplug on power systems. Some of the systems may not provide all the rtas services,which are required to perform cpu hotplug. Like for example
1) JS20 doesn't provide "stop-self" token and cpu hotplug operations
on these systems causes system to crash.
2) some of the p630 systems doesn't provide "query-cpu-stopped-state"
token and we are not sure of whether cpu is under stopped state or not or stop-self is still in progress .
So we can't take decision on whether cpu really has gone offline or not.

So we need to make sure that all required rtas tokens for cpu hotplug are available during rtas initialization phase and to disable cpu hotplug if they are not available. I have developed the patch which does the above thing. Please let me know your comments on this.

 arch/powerpc/kernel/rtas.c |    6 ++++++
 include/linux/cpu.h        |    4 ++++
 kernel/cpu.c               |   24 ++++++++++++++++++++++--
 3 files changed, 32 insertions(+), 2 deletions(-)
arch/powerpc/kernel/rtas.c +6 -0
Index: linux-2.6.19-rc5/arch/powerpc/kernel/rtas.c
===================================================================
@@ -19,6 +19,7 @@
 #include <linux/init.h>
 #include <linux/capability.h>
 #include <linux/delay.h>
+#include <linux/cpu.h>
 
 #include <asm/prom.h>
 #include <asm/rtas.h>
@@ -843,6 +844,7 @@ void rtas_stop_self(void)
 void __init rtas_initialize(void)
 {
 	unsigned long rtas_region = RTAS_INSTANTIATE_MAX;
+	int qcss_tok;
 
 	/* Get RTAS dev node and fill up our "rtas" structure with infos
 	 * about it.
@@ -881,6 +883,10 @@ void __init rtas_initialize(void)
 
 #ifdef CONFIG_HOTPLUG_CPU
 	rtas_stop_self_args.token = rtas_token("stop-self");
+	qcss_tok = rtas_token("query-cpu-stopped-state");
+	if(rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE ||
+				 qcss_tok == RTAS_UNKNOWN_SERVICE)
+		disable_cpu_hotplug_perm();
 #endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
kernel/cpu.c +22 -2
Index: linux-2.6.19-rc5/kernel/cpu.c
===================================================================
@@ -63,8 +63,20 @@ void unlock_cpu_hotplug(void)
 }
 EXPORT_SYMBOL_GPL(unlock_cpu_hotplug);
 
+void disable_cpu_hotplug_perm(void)
+{
+	mutex_lock(&cpu_add_remove_lock);
+	cpu_hotplug_disabled = PERM_DISABLED_CPU_HOTPLUG;
+	mutex_unlock(&cpu_add_remove_lock);
+}
+
 #endif	/* CONFIG_HOTPLUG_CPU */
 
+static int is_cpu_hotplug_perm_disabled()
+{
+	return cpu_hotplug_disabled == PERM_DISABLED_CPU_HOTPLUG;
+}
+
 /* Need to know about CPUs going up/down? */
 int __cpuinit register_cpu_notifier(struct notifier_block *nb)
 {
@@ -193,7 +205,7 @@ int cpu_down(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_down(cpu);
@@ -244,7 +256,7 @@ int __devinit cpu_up(unsigned int cpu)
 	int err = 0;
 
 	mutex_lock(&cpu_add_remove_lock);
-	if (cpu_hotplug_disabled)
+	if (cpu_hotplug_disabled || is_cpu_hotplug_perm_disabled())
 		err = -EBUSY;
 	else
 		err = _cpu_up(cpu);
@@ -261,6 +273,10 @@ int disable_nonboot_cpus(void)
 	int cpu, first_cpu, error;
 
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		error = -EBUSY;
+		goto out;
+	}
 	first_cpu = first_cpu(cpu_present_map);
 	if (!cpu_online(first_cpu)) {
 		error = _cpu_up(first_cpu);
@@ -311,6 +327,10 @@ void enable_nonboot_cpus(void)
 
 	/* Allow everyone to use the CPU hotplug again */
 	mutex_lock(&cpu_add_remove_lock);
+	if(is_cpu_hotplug_perm_disabled()) {
+		mutex_unlock(&cpu_add_remove_lock);
+		return;
+	}
 	cpu_hotplug_disabled = 0;
 	mutex_unlock(&cpu_add_remove_lock);
 
include/linux/cpu.h +4 -0
Index: linux-2.6.19-rc5/include/linux/cpu.h
===================================================================
@@ -31,6 +31,8 @@ struct cpu {
 	struct sys_device sysdev;
 };
 
+#define  PERM_DISABLED_CPU_HOTPLUG -1
+
 extern int register_cpu(struct cpu *cpu, int num);
 extern struct sys_device *get_cpu_sysdev(unsigned cpu);
 #ifdef CONFIG_HOTPLUG_CPU
@@ -68,6 +70,7 @@ extern struct sysdev_class cpu_sysdev_cl
 /* Stop CPUs going up and down. */
 extern void lock_cpu_hotplug(void);
 extern void unlock_cpu_hotplug(void);
+extern void disable_cpu_hotplug_perm(void);
 #define hotcpu_notifier(fn, pri) {				\
 	static struct notifier_block fn##_nb =			\
 		{ .notifier_call = fn, .priority = pri };	\
@@ -80,6 +83,7 @@ int cpu_down(unsigned int cpu);
 #else
 #define lock_cpu_hotplug()	do { } while (0)
 #define unlock_cpu_hotplug()	do { } while (0)
+#define disable_cpu_hotplug_perm()     do { } while (0)
 #define lock_cpu_hotplug_interruptible() 0
 #define hotcpu_notifier(fn, pri)	do { } while (0)
 #define register_hotcpu_notifier(nb)	do { } while (0)
#1 lore raw
Quoted 9 lines · Srinivasa Ds

Linux kernel uses some of the rtas token to perform cpu hotplug on power systems. Some of the systems may not provide all the rtas services,which are required to perform cpu hotplug. Like for example
1) JS20 doesn't provide "stop-self" token and cpu hotplug operations
on these systems causes system to crash.
2) some of the p630 systems doesn't provide "query-cpu-stopped-state"
token and we are not sure of whether cpu is under stopped state or not or stop-self is still in progress .

Neither of these systems (well, the p630 in "SMP" mode) have hypervisors so their firmwares don't provide the rtas primitives for CPU offline.

Quoted 7 lines

So we can't take decision on whether cpu really has gone offline or not.

So we need to make sure that all required rtas tokens for cpu hotplug are available during rtas initialization phase and to disable cpu hotplug if they are not available.
I have developed the patch which does the above thing. Please let me know your comments on this.

Would be better to either a) inhibit creation of the sysfs 'online' cpu attributes (I thought we used to handle this correctly on these systems) or b) use the generic cpu hotplug operations which have no dependency on RTAS.

#2 lore raw
Quoted 23 lines · Nathan Lynch
Quoted 9 lines · Srinivasa Ds

Linux kernel uses some of the rtas token to perform cpu hotplug on power systems. Some of the systems may not provide all the rtas services,which are required to perform cpu hotplug. Like for example
1) JS20 doesn't provide "stop-self" token and cpu hotplug operations
on these systems causes system to crash.
2) some of the p630 systems doesn't provide "query-cpu-stopped-state"
token and we are not sure of whether cpu is under stopped state or not or stop-self is still in progress .

Neither of these systems (well, the p630 in "SMP" mode) have hypervisors so their firmwares don't provide the rtas primitives for CPU offline.

Quoted 7 lines

So we can't take decision on whether cpu really has gone offline or not.

So we need to make sure that all required rtas tokens for cpu hotplug are available during rtas initialization phase and to disable cpu hotplug if they are not available.
I have developed the patch which does the above thing. Please let me know your comments on this.

Quoted 4 lines

Would be better to either a) inhibit creation of the sysfs 'online' cpu attributes (I thought we used to handle this correctly on these systems) or b) use the generic cpu hotplug operations which have no dependency on RTAS.

Yeah, good point. It seemes cleaner to do this, than to add a new state ("per disabled") that adds complexity to the generic kernel code.

Perhaps we could set this up so that, if rtas "stop-self" is not defined, then generic_cpu_die() is installed instead of pseries_cpu_die()?

--linas

#3 lore raw

PATCH 1/1 Reorganise and then fixup the pseries cpu hotplug code

In reply to Nathan Lynch

The pseries cpu hotplug code is currently spread between ./kernel/rtas.c, ./platforms/pseries/smp.c and ./platforms/pseries/setup.c. Some of it is not #ifdef CONFIG_HOTPLUG_CPU, but it should be.

So move all the cpu hotplug code into platforms/pseries/hotplug-cpu.c While we're moving, rename studly caps functions to normal caps, they're all static so no harm done. Fixup some long lines also, and make things static that can be, now we're all in the same file.

Currently we unconditionally hookup pSeries_mach_cpu_die to ppc_md.cpu_die, even if we don't have CONFIG_HOTPLUG_CPU enabled. This is wrong, as it signals the sysfs code to create the online attribute for cpu nodes, allowing the user to attempt an offline when it's not actually supported.

There is also a problem on systems that don't have the correct RTAS tokens available to do RTAS-based cpu hotplug, we still indicate via sysfs that we support cpu hotplug - and then attempt to do so with missing RTAS tokens.

Both problems are solved by conditionally registering the cpu hotplug callbacks, only when CONFIG_HOTPLUG_CPU is enabled, and only after we've found the requisite RTAS tokens.

---

OK, what do people think of this? I think it solves the problems we've seen lately. This supersedes Linas patch to wrap the pSeries bits in CONFIG_HOTPLUG_CPU.

I haven't tested this - if someone can that'd be great - otherwise I'll have a go on Monday.

---
 arch/powerpc/kernel/rtas.c                   |   28 --
 arch/powerpc/platforms/pseries/Makefile      |    2 
 arch/powerpc/platforms/pseries/hotplug-cpu.c |  272 +++++++++++++++++++++++++++
 arch/powerpc/platforms/pseries/setup.c       |   12 -
 arch/powerpc/platforms/pseries/smp.c         |  199 -------------------
 include/asm-powerpc/rtas.h                   |    4 
 6 files changed, 274 insertions(+), 243 deletions(-)
arch/powerpc/kernel/rtas.c +0 -28
Index: powerpc/arch/powerpc/kernel/rtas.c
===================================================================
@@ -810,31 +810,6 @@ asmlinkage int ppc_rtas(struct rtas_args
 	return 0;
 }
 
-/* This version can't take the spinlock, because it never returns */
-
-struct rtas_args rtas_stop_self_args = {
-	/* The token is initialized for real in setup_system() */
-	.token = RTAS_UNKNOWN_SERVICE,
-	.nargs = 0,
-	.nret = 1,
-	.rets = &rtas_stop_self_args.args[0],
-};
-
-void rtas_stop_self(void)
-{
-	struct rtas_args *rtas_args = &rtas_stop_self_args;
-
-	local_irq_disable();
-
-	BUG_ON(rtas_args->token == RTAS_UNKNOWN_SERVICE);
-
-	printk("cpu %u (hwid %u) Ready to die...\n",
-	       smp_processor_id(), hard_smp_processor_id());
-	enter_rtas(__pa(rtas_args));
-
-	panic("Alas, I survived.\n");
-}
-
 /*
  * Call early during boot, before mem init or bootmem, to retrieve the RTAS
  * informations from the device-tree and allocate the RMO buffer for userland
@@ -879,9 +854,6 @@ void __init rtas_initialize(void)
 #endif
 	rtas_rmo_buf = lmb_alloc_base(RTAS_RMOBUF_MAX, PAGE_SIZE, rtas_region);
 
-#ifdef CONFIG_HOTPLUG_CPU
-	rtas_stop_self_args.token = rtas_token("stop-self");
-#endif /* CONFIG_HOTPLUG_CPU */
 #ifdef CONFIG_RTAS_ERROR_LOGGING
 	rtas_last_error_token = rtas_token("rtas-last-error");
 #endif
arch/powerpc/platforms/pseries/Makefile +2 -0
Index: powerpc/arch/powerpc/platforms/pseries/Makefile
===================================================================
@@ -10,6 +10,8 @@ obj-$(CONFIG_XICS)	+= xics.o
 obj-$(CONFIG_SCANLOG)	+= scanlog.o
 obj-$(CONFIG_EEH)	+= eeh.o eeh_cache.o eeh_driver.o eeh_event.o
 
+obj-$(CONFIG_HOTPLUG_CPU)	+= hotplug-cpu.o
+
 obj-$(CONFIG_HVC_CONSOLE)	+= hvconsole.o
 obj-$(CONFIG_HVCS)		+= hvcserver.o
 obj-$(CONFIG_HCALL_STATS)	+= hvCall_inst.o
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +272 -0

272 lines not on this page · raw message

arch/powerpc/platforms/pseries/setup.c +0 -12
Index: powerpc/arch/powerpc/platforms/pseries/setup.c
===================================================================
@@ -347,17 +347,6 @@ static int __init pSeries_init_panel(voi
 }
 arch_initcall(pSeries_init_panel);
 
-static void pSeries_mach_cpu_die(void)
-{
-	local_irq_disable();
-	idle_task_exit();
-	xics_teardown_cpu(0);
-	rtas_stop_self();
-	/* Should never get here... */
-	BUG();
-	for(;;);
-}
-
 static int pseries_set_dabr(unsigned long dabr)
 {
 	return plpar_hcall_norets(H_SET_DABR, dabr);
@@ -558,7 +547,6 @@ define_machine(pseries) {
 	.power_off		= rtas_power_off,
 	.halt			= rtas_halt,
 	.panic			= rtas_os_term,
-	.cpu_die		= pSeries_mach_cpu_die,
 	.get_boot_time		= rtas_get_boot_time,
 	.get_rtc_time		= rtas_get_rtc_time,
 	.set_rtc_time		= rtas_set_rtc_time,
arch/powerpc/platforms/pseries/smp.c +0 -199

211 lines not on this page · raw message

include/asm-powerpc/rtas.h +0 -4
Index: powerpc/include/asm-powerpc/rtas.h
===================================================================
@@ -54,8 +54,6 @@ struct rtas_args {
 	rtas_arg_t *rets;     /* Pointer to return values in args[]. */
 };  
 
-extern struct rtas_args rtas_stop_self_args;
-
 struct rtas_t {
 	unsigned long entry;		/* physical address pointer */
 	unsigned long base;		/* physical address pointer */
@@ -223,8 +221,6 @@ extern int rtas_get_error_log_max(void);
 extern spinlock_t rtas_data_buf_lock;
 extern char rtas_data_buf[RTAS_DATA_BUF_SIZE];
 
-extern void rtas_stop_self(void);
-
 /* RMO buffer reserved for user-space RTAS use */
 extern unsigned long rtas_rmo_buf;
 
#4 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 23 lines · Michael Ellerman

The pseries cpu hotplug code is currently spread between ./kernel/rtas.c, ./platforms/pseries/smp.c and ./platforms/pseries/setup.c. Some of it is not #ifdef CONFIG_HOTPLUG_CPU, but it should be.

So move all the cpu hotplug code into platforms/pseries/hotplug-cpu.c While we're moving, rename studly caps functions to normal caps, they're all static so no harm done. Fixup some long lines also, and make things static that can be, now we're all in the same file.

Currently we unconditionally hookup pSeries_mach_cpu_die to ppc_md.cpu_die, even if we don't have CONFIG_HOTPLUG_CPU enabled. This is wrong, as it signals the sysfs code to create the online attribute for cpu nodes, allowing the user to attempt an offline when it's not actually supported.

There is also a problem on systems that don't have the correct RTAS tokens available to do RTAS-based cpu hotplug, we still indicate via sysfs that we support cpu hotplug - and then attempt to do so with missing RTAS tokens.

Both problems are solved by conditionally registering the cpu hotplug callbacks, only when CONFIG_HOTPLUG_CPU is enabled, and only after we've found the requisite RTAS tokens.

Quoted 8 lines

---

OK, what do people think of this? I think it solves the problems we've seen lately. This supersedes Linas patch to wrap the pSeries bits in CONFIG_HOTPLUG_CPU.

I haven't tested this - if someone can that'd be great - otherwise I'll have a go on Monday.

I should add .. The callbacks now get setup in an arch_initcall() - this is not as early, by a long shot, as it used to be - however AFAICT it should make no difference as there's no way to trigger a cpu hotplug until later on anyway.

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#5 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

In reply to Michael Ellerman

One small nit ...

arch/powerpc/platforms/pseries/hotplug-cpu.c line 260

Quoted 9 lines · On Fri, 17 Nov 2006 14:36:35 +1100 Michael Ellerman <[email protected]>…
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +8 -0
+static int __init pseries_cpu_hotplug_init(void)
+{
+	rtas_stop_self_args.token = rtas_token("stop-self");
+	qcss_tok = rtas_token("query-cpu-stopped-state");
+
+	if (rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE ||
+			qcss_tok == RTAS_UNKNOWN_SERVICE)
+		return 1;

initcall fucntions should return 0 or -<error>; -ENODEV is ignored, other nonzero values cause a log message if initcall debugging is enabled.

--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/

#6 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 4 lines · Stephen Rothwell

One small nit ...

Quoted 1 line · On Fri, 17 Nov 2006 14:36:35 +1100 Michael Ellerman <[email protected]>…
Quoted 11 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 260

Quoted 8 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +8 -0
+static int __init pseries_cpu_hotplug_init(void)
+{
+	rtas_stop_self_args.token = rtas_token("stop-self");
+	qcss_tok = rtas_token("query-cpu-stopped-state");
+
+	if (rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE ||
+			qcss_tok == RTAS_UNKNOWN_SERVICE)
+		return 1;

initcall fucntions should return 0 or -<error>; -ENODEV is ignored, other nonzero values cause a log message if initcall debugging is enabled.

OK. I wanted to return an error, so there's something in the log to show that cpu hotplug was disabled - but I didn't check what to return. I'll fix it up on Monday to return -ENOENT.

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#7 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 7 lines · On Fri, 17 Nov 2006 15:44:00 +1100 Michael Ellerman <[email protected]>…
Quoted 3 lines

initcall fucntions should return 0 or -<error>; -ENODEV is ignored, other nonzero values cause a log message if initcall debugging is enabled.

OK. I wanted to return an error, so there's something in the log to show that cpu hotplug was disabled - but I didn't check what to return. I'll fix it up on Monday to return -ENOENT.

Which also won't do anything unless initcall debugging is enabled and, even then, will just give an obscure message in the boot logs. If you want to log something, then you should explicitly call printk().

--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/

#8 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

In reply to Michael Ellerman
Quoted 17 lines · Michael Ellerman
Quoted 4 lines · Stephen Rothwell

One small nit ...

Quoted 1 line · On Fri, 17 Nov 2006 14:36:35 +1100 Michael Ellerman <[email protected]>…
Quoted 11 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 260

Quoted 8 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +8 -0
+static int __init pseries_cpu_hotplug_init(void)
+{
+	rtas_stop_self_args.token = rtas_token("stop-self");
+	qcss_tok = rtas_token("query-cpu-stopped-state");
+
+	if (rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE ||
+			qcss_tok == RTAS_UNKNOWN_SERVICE)
+		return 1;

initcall fucntions should return 0 or -<error>; -ENODEV is ignored, other nonzero values cause a log message if initcall debugging is enabled.

Quoted 3 lines

OK. I wanted to return an error, so there's something in the log to show that cpu hotplug was disabled - but I didn't check what to return. I'll fix it up on Monday to return -ENOENT.

I presume the default config from RedHat/SuSE is to have CONFIG_HOTPLUG_CPU turned on. In this case, all sorts of pseries boxes will spew a warning. I suppose this is a form of subliminal advertising: "Hey Schmo, upgrade your box to something that supports CPU hotplug!".

--linas

#9 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 22 lines · Linas Vepstas
Quoted 17 lines · Michael Ellerman
Quoted 4 lines · Stephen Rothwell

One small nit ...

Quoted 1 line · On Fri, 17 Nov 2006 14:36:35 +1100 Michael Ellerman <[email protected]>…
Quoted 11 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 260

Quoted 8 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +8 -0
+static int __init pseries_cpu_hotplug_init(void)
+{
+	rtas_stop_self_args.token = rtas_token("stop-self");
+	qcss_tok = rtas_token("query-cpu-stopped-state");
+
+	if (rtas_stop_self_args.token == RTAS_UNKNOWN_SERVICE ||
+			qcss_tok == RTAS_UNKNOWN_SERVICE)
+		return 1;

initcall fucntions should return 0 or -<error>; -ENODEV is ignored, other nonzero values cause a log message if initcall debugging is enabled.

Quoted 3 lines

OK. I wanted to return an error, so there's something in the log to show that cpu hotplug was disabled - but I didn't check what to return. I'll fix it up on Monday to return -ENOENT.

Quoted 5 lines

I presume the default config from RedHat/SuSE is to have CONFIG_HOTPLUG_CPU turned on. In this case, all sorts of pseries boxes will spew a warning. I suppose this is a form of subliminal advertising: "Hey Schmo, upgrade your box to something that supports CPU hotplug!".

Yeah the distros would have it on for everything. I'm not sure what machines support it, but I think it'd best to have some indication in the log that we disabled it.

I'll change it to printk, and always return success. As an aside, I don't really see the point of having return codes from init routines if nothing is done with them by default ... but whatever.

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#10 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

In reply to Michael Ellerman
Quoted 5 lines · Michael Ellerman

The pseries cpu hotplug code is currently spread between ./kernel/rtas.c, ./platforms/pseries/smp.c and ./platforms/pseries/setup.c. Some of it is not #ifdef CONFIG_HOTPLUG_CPU, but it should be.

So move all the cpu hotplug code into platforms/pseries/hotplug-cpu.c

Yes, much cleaner this way.

arch/powerpc/platforms/pseries/hotplug-cpu.c line 124

Quoted 1 line
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +1 -0
+	/* Isolation and deallocation are definatly done by
                                        ^^^^^^^^^

typo

arch/powerpc/platforms/pseries/hotplug-cpu.c line 129

Quoted 5 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +5 -0
+	 * drslot_chrp_cpu.  If they were not they would be
+	 * done here.  Change isolate state to Isolate and
+	 * change allocation-state to Unusable.
+	 */
+	paca[cpu].cpu_start = 0;

I can't figure out what the comment means with respect to this code ...

--linas

#11 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 8 lines · Linas Vepstas
Quoted 5 lines · Michael Ellerman

The pseries cpu hotplug code is currently spread between ./kernel/rtas.c, ./platforms/pseries/smp.c and ./platforms/pseries/setup.c. Some of it is not #ifdef CONFIG_HOTPLUG_CPU, but it should be.

So move all the cpu hotplug code into platforms/pseries/hotplug-cpu.c

Yes, much cleaner this way.

I agree :)

Quoted 4 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 124

Quoted 1 line
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +1 -0
+	/* Isolation and deallocation are definatly done by
                                        ^^^^^^^^^

typo

Quoted 8 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 129

Quoted 5 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +5 -0
+	 * drslot_chrp_cpu.  If they were not they would be
+	 * done here.  Change isolate state to Isolate and
+	 * change allocation-state to Unusable.
+	 */
+	paca[cpu].cpu_start = 0;

I can't figure out what the comment means with respect to this code ...

Me either. It's unchanged from the original merge of cpu hotplug:
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=commit;h=24c13f21f0a6abe07020a959990da2b134e6734f
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=blobdiff;h=72144b6122f9336e7fa242e50b65b099df0cd72b;hp=f671515c0676e61064ae414273fa4546d0c04938;hb=24c13f21f0a6abe07020a959990da2b134e6734f;f=arch/ppc64/kernel/smp.c

Which carries the name of one Joel Schopp :)

Grepping the tree at that point finds no other mention of drslot_chrp_cpu, so I think it's a stale comment - we should probably just rip it out - Joel?

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#12 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 3 lines
Quoted 3 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 124

Quoted 1 line
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +1 -0
+	/* Isolation and deallocation are definatly done by
                                        ^^^^^^^^^

typo

That word definitely gives me trouble. I'd imagine the typo originated with me.

Quoted 12 lines
Quoted 7 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 129

Quoted 6 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +5 -0
+	 * drslot_chrp_cpu.  If they were not they would be
+	 * done here.  Change isolate state to Isolate and
+	 * change allocation-state to Unusable.
+	 */
+	paca[cpu].cpu_start = 0;

I can't figure out what the comment means with respect to this code ...

Me either. It's unchanged from the original merge of cpu hotplug:
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=commit;h=24c13f21f0a6abe07020a959990da2b134e6734f
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=blobdiff;h=72144b6122f9336e7fa242e50b65b099df0cd72b;hp=f671515c0676e61064ae414273fa4546d0c04938;hb=24c13f21f0a6abe07020a959990da2b134e6734f;f=arch/ppc64/kernel/smp.c

Quoted 5 lines

Which carries the name of one Joel Schopp :)

Grepping the tree at that point finds no other mention of drslot_chrp_cpu, so I think it's a stale comment - we should probably just rip it out - Joel?

It might be clearer by saying "the userspace program drslot_chrp_cpu" or "isolation and deallocation occur in userspace, so just stop the cpu. If isolation and deallocation ever move from userpace to kernel space they would go here". On pseries dynamic partitioning or cpu guard (the things that initiate a cpu hotplug) all call a userspace program drmgr, which calls drslot_chrp_cpu, which then talks to the firmware to isolate and unallocate (on remove, the opposite on add) after it has hotplug removed the cpu via the /sys interface. This allows the cpu to be reassigned to another partition.

At least that's what I remember from back then when we did it. Though I don't think it's changed any. Feel free to make the comment more intelligible.

#13 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 5 lines · jschopp
Quoted 3 lines
Quoted 3 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 124

Quoted 1 line
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +1 -0
+	/* Isolation and deallocation are definatly done by
                                        ^^^^^^^^^

typo

That word definitely gives me trouble. I'd imagine the typo originated with me.

Yeah it's definitely a tricky one ;)

Quoted 27 lines
Quoted 13 lines
Quoted 8 lines

arch/powerpc/platforms/pseries/hotplug-cpu.c line 129

Quoted 6 lines
arch/powerpc/platforms/pseries/hotplug-cpu.c new file +5 -0
+	 * drslot_chrp_cpu.  If they were not they would be
+	 * done here.  Change isolate state to Isolate and
+	 * change allocation-state to Unusable.
+	 */
+	paca[cpu].cpu_start = 0;

I can't figure out what the comment means with respect to this code ...

Me either. It's unchanged from the original merge of cpu hotplug:
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=commit;h=24c13f21f0a6abe07020a959990da2b134e6734f
http://kernel.org/git/?p=linux/kernel/git/torvalds/old-2.6-bkcvs.git;a=blobdiff;h=72144b6122f9336e7fa242e50b65b099df0cd72b;hp=f671515c0676e61064ae414273fa4546d0c04938;hb=24c13f21f0a6abe07020a959990da2b134e6734f;f=arch/ppc64/kernel/smp.c

Quoted 5 lines

Which carries the name of one Joel Schopp :)

Grepping the tree at that point finds no other mention of drslot_chrp_cpu, so I think it's a stale comment - we should probably just rip it out - Joel?

It might be clearer by saying "the userspace program drslot_chrp_cpu" or "isolation and deallocation occur in userspace, so just stop the cpu. If isolation and deallocation ever move from userpace to kernel space they would go here". On pseries dynamic partitioning or cpu guard (the things that initiate a cpu hotplug) all call a userspace program drmgr, which calls drslot_chrp_cpu, which then talks to the firmware to isolate and unallocate (on remove, the opposite on add) after it has hotplug removed the cpu via the /sys interface. This allows the cpu to be reassigned to another partition.

Quoted 2 lines

At least that's what I remember from back then when we did it. Though I don't think it's changed any. Feel free to make the comment more intelligible.

OK, it didn't occur to me that it was referring to userspace. I'll change it to:

/*
* Isolation and deallocation occur in userspace, so just stop the cpu.
* If isolation and deallocation ever move from userpace to kernel space
* we would do them here.
*/

Sound OK?

cheers

Signature · 8 lines
Michael Ellerman
OzLabs, IBM Australia Development Lab

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#14 lore raw

Re: [PATCH] Reorganise and then fixup the pseries cpu hotplug code

Quoted 11 lines · Michael Ellerman

OK, it didn't occur to me that it was referring to userspace. I'll change it to:

/*
* Isolation and deallocation occur in userspace, so just stop the cpu.
* If isolation and deallocation ever move from userpace to kernel space
* we would do them here.
*/

Sound OK?

I favor just dropping the comment. Offline and isolate are distinct -- offline is a per-thread operation while isolate is per-processor. Offlining a thread does not imply that its processor will be returned to the hypervisor (e.g. going to single thread mode from SMT).