Skip to contentSkip to conversation

CPU hotplug on i386

PATCH Pavel Machek 5 participants 11 messages Mar 16, 2005 – Mar 18, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

Pavel

kernel/power/smp.c +24 -55
@@ -16,70 +16,39 @@
 #include <asm/atomic.h>
 #include <asm/tlbflush.h>
 
-static atomic_t cpu_counter, freeze;
-
-
-static void smp_pause(void * data)
-{
-	struct saved_context ctxt;
-	__save_processor_state(&ctxt);
-	printk("Sleeping in:\n");
-	dump_stack();
-	atomic_inc(&cpu_counter);
-	while (atomic_read(&freeze)) {
-		/* FIXME: restore takes place at random piece inside this.
-		   This should probably be written in assembly, and
-		   preserve general-purpose registers, too
-
-		   What about stack? We may need to move to new stack here.
-
-		   This should better be ran with interrupts disabled.
-		 */
-		cpu_relax();
-		barrier();
-	}
-	atomic_dec(&cpu_counter);
-	__restore_processor_state(&ctxt);
-}
-
-cpumask_t oldmask;
+cpumask_t frozen_cpus;
 
 void disable_nonboot_cpus(void)
 {
-	printk("Freezing CPUs (at %d)", smp_processor_id());
-	oldmask = current->cpus_allowed;
-	set_cpus_allowed(current, cpumask_of_cpu(0));
-	current->state = TASK_INTERRUPTIBLE;
-	schedule_timeout(HZ);
-	printk("...");
-	BUG_ON(smp_processor_id() != 0);
-
-	/* FIXME: for this to work, all the CPUs must be running
-	 * "idle" thread (or we deadlock). Is that guaranteed? */
-
-	atomic_set(&cpu_counter, 0);
-	atomic_set(&freeze, 1);
-	smp_call_function(smp_pause, NULL, 0, 0);
-	while (atomic_read(&cpu_counter) < (num_online_cpus() - 1)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus...\n");
+	for_each_online_cpu(cpu) {
+		if (!cpu)
+			continue;
+		cpu_set(cpu, frozen_cpus);
+		error = cpu_down(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d down: %d\n", cpu, error);
+		panic("Too many cpus");
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
 }
 
 void enable_nonboot_cpus(void)
 {
-	printk("Restarting CPUs");
-	atomic_set(&freeze, 0);
-	while (atomic_read(&cpu_counter)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;
+	printk("Thawing cpus...\n");
+	for_each_cpu_mask(cpu, frozen_cpus) {
+		if (!cpu)
+			continue;
+		error = cpu_up(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d up: %d\n", cpu, error);
+		panic("Not enough cpus");
 	}
-	printk("...");
-	set_cpus_allowed(current, oldmask);
-	schedule();
-	printk("ok\n");
-
 }
 
 
Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#1 lore raw

Hi,

Quoted 5 lines · Pavel Machek

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

Heh, that's exactly what I was thinking about. ;-)

AFAICS we don't need the full CPU hotplug to do this. For suspend, we need to enable the CPU hotplug-related code in sched.c and cpu.c, and we need to implement the functions __cpu_disable() and __cpu_die() (called from within cpu.c) on each architecture for which we want swsusp to work on SMP.

If that's acceptable, the CPU hotplug code in sched.c and cpu.c may be enabled by changing some #ifdefs there. For example, we could replace the

#idef CONFIG_CPU_HOTPLUG

with

#if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND)
	&& defined(CONFIG_SMP))

wherever necessary.

The implementation of __cpu_disable() and __cpu_die() may be a bit more tricky, however. In __cpu_disable() we need to save the settings of the local APIC of each CPU (on x86-64, at least) etc. In __cpu_die() we should give the "frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).

I've looked at the ia64 implementation of these functions, but I haven't fully understood it yet.

Greets,
Rafael

Signature · 3 lines
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"
#2 lore raw

Hi.

Quoted 26 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

Heh, that's exactly what I was thinking about. ;-)

AFAICS we don't need the full CPU hotplug to do this. For suspend, we need to enable the CPU hotplug-related code in sched.c and cpu.c, and we need to implement the functions __cpu_disable() and __cpu_die() (called from within cpu.c) on each architecture for which we want swsusp to work on SMP.

If that's acceptable, the CPU hotplug code in sched.c and cpu.c may be enabled by changing some #ifdefs there. For example, we could replace the

#idef CONFIG_CPU_HOTPLUG

with

#if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND)
	&& defined(CONFIG_SMP))

wherever necessary.

Quoted 7 lines

The implementation of __cpu_disable() and __cpu_die() may be a bit more tricky, however. In __cpu_disable() we need to save the settings of the local APIC of each CPU (on x86-64, at least) etc. In __cpu_die() we should give the "frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).

I've looked at the ia64 implementation of these functions, but I haven't fully understood it yet.

It would be good (32 bit) if you handled MTRRs as well. Putting their save/restore code in the driver model is racy (SMP calls). I've worked around that in Suspend2 by removing the driver model support and making the save/restore calls from the lowlevel code. I think this should be considered the right thing to do.

By the way, I'm keen to test this with Suspend2 as well. I would have done so before now, but I didn't realise Zwane had the support done.

Regards,

Nigel

Signature · 6 lines
Nigel Cunningham
Software Engineer, Canberra, Australia
http://www.cyclades.com
Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028;  Mob: +61 (417) 100 574

Maintainer of Suspend2 Kernel Patches http://suspend2.net
#3 lore raw
In reply to Pavel Machek
Quoted 5 lines · Pavel Machek

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.

Nathan

#4 lore raw
Quoted 8 lines · Nathan Lynch
Quoted 3 lines · Pavel Machek

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.

Well this is a good user =)

#5 lore raw
In reply to Nathan Lynch

Hi,

Quoted 9 lines · Nathan Lynch
Quoted 5 lines · Pavel Machek

Hi!

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.

Thanks a lot for this hint! ;-)

Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set automatically.

I'm going to test it together with your patch tomorrow.

Greets,
Rafael

Signature · 3 lines
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"
#6 lore raw

Hi!

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.

Quoted 8 lines

Thanks a lot for this hint! ;-)

Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set automatically.

I'm going to test it together with your patch tomorrow.

Hey, don't count on my patch. It is first shot, never tested.

Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#7 lore raw

Hi,

Quoted 11 lines · Pavel Machek

Hi!

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.

Thanks a lot for this hint! ;-)

Quoted 8 lines
Quoted 6 lines

Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set automatically.

I'm going to test it together with your patch tomorrow.

Hey, don't count on my patch. It is first shot, never tested.

It'll be the first time, then. I'm not afraid. ;-)

Rafael

Signature · 3 lines
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"
#8 lore raw
In reply to Pavel Machek

Hi again-

Quoted 25 lines · Pavel Machek
kernel/power/smp.c +14 -6
+	int cpu, error;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus...\n");
+	for_each_online_cpu(cpu) {
+		if (!cpu)
+			continue;
+		cpu_set(cpu, frozen_cpus);
+		error = cpu_down(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d down: %d\n", cpu, error);
+		panic("Too many cpus");
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
 }
 
 void enable_nonboot_cpus(void)
 {
-	printk("Restarting CPUs");
-	atomic_set(&freeze, 0);
-	while (atomic_read(&cpu_counter)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;

kernel/power/smp.c line 51

Quoted 10 lines
kernel/power/smp.c +9 -0
+	printk("Thawing cpus...\n");
+	for_each_cpu_mask(cpu, frozen_cpus) {
+		if (!cpu)
+			continue;
+		error = cpu_up(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d up: %d\n", cpu, error);
+		panic("Not enough cpus");
 	}

If you're going to depend on CONFIG_HOTPLUG_CPU for swsusp on smp, why not offline the non-boot cpus from userspace? Same goes for onlining additional cpus during resume.

Nathan

#9 lore raw

Hi!

Quoted 32 lines · Pavel Machek
Quoted 31 lines
kernel/power/smp.c +20 -6
+	int cpu, error;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus...\n");
+	for_each_online_cpu(cpu) {
+		if (!cpu)
+			continue;
+		cpu_set(cpu, frozen_cpus);
+		error = cpu_down(cpu);
+		if (!error)
+			continue;
+		printk("Error taking cpu %d down: %d\n", cpu, error);
+		panic("Too many cpus");
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
 }
 
 void enable_nonboot_cpus(void)
 {
-	printk("Restarting CPUs");
-	atomic_set(&freeze, 0);
-	while (atomic_read(&cpu_counter)) {
-		cpu_relax();
-		barrier();
+	int cpu, error;
+	printk("Thawing cpus...\n");
+	for_each_cpu_mask(cpu, frozen_cpus) {
+		if (!cpu)
+			continue;
+		error = cpu_up(cpu);
+		if (!error)
Quoted 8 lines

kernel/power/smp.c line 51

Quoted 4 lines
kernel/power/smp.c +3 -0
+			continue;
+		printk("Error taking cpu %d up: %d\n", cpu, error);
+		panic("Not enough cpus");
 	}

If you're going to depend on CONFIG_HOTPLUG_CPU for swsusp on smp, why not offline the non-boot cpus from userspace? Same goes for onlining additional cpus during resume.

I'd like to stay in control. I may want to stop drivers first, or something like that. Plus I need to do that during resume, too, when userspace is not available, yet.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#10 lore raw
In reply to Pavel Machek

Hi!

Quoted 3 lines

I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?

BTW Li Shaohua has prototype smp/S3 implementation. I'll take detailed look at that one.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!