CPU hotplug on i386
Subsystems (2)
Hi!
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that CONFIG_CPU_HOTPLUG is not available on i386. Is there way to enable CPU_HOTPLUG on i386?
Pavel
kernel/power/smp.c +24 -55
@@ -16,70 +16,39 @@#include <asm/atomic.h> #include <asm/tlbflush.h> -static atomic_t cpu_counter, freeze; - - -static void smp_pause (void * data) -{ -struct saved_context ctxt; -__save_processor_state (&ctxt); -printk ("Sleeping in:\n" ); -dump_stack (); -atomic_inc (&cpu_counter); -while (atomic_read (&freeze)) { -/* FIXME: restore takes place at random piece inside this. -This should probably be written in assembly, and -preserve general-purpose registers, too - -What about stack? We may need to move to new stack here. - -This should better be ran with interrupts disabled. -*/ -cpu_relax (); -barrier (); - } -atomic_dec (&cpu_counter); -__restore_processor_state (&ctxt); -} - -cpumask_t oldmask; +cpumask_t frozen_cpus;void disable_nonboot_cpus (void ) { -printk ("Freezing CPUs (at %d)" ,smp_processor_id ()); - oldmask = current->cpus_allowed; -set_cpus_allowed (current,cpumask_of_cpu (0 )); - current->state =TASK_INTERRUPTIBLE ; -schedule_timeout (HZ ); -printk ("..." ); -BUG_ON (smp_processor_id () !=0 ); - -/* FIXME: for this to work, all the CPUs must be running -* "idle" thread (or we deadlock). Is that guaranteed? */ - -atomic_set (&cpu_counter,0 ); -atomic_set (&freeze,1 ); -smp_call_function (smp_pause,NULL ,0 ,0 ); -while (atomic_read (&cpu_counter) < (num_online_cpus () -1 )) { -cpu_relax (); -barrier (); +int cpu, error; +cpus_clear (frozen_cpus); +printk ("Freezing cpus...\n" ); +for_each_online_cpu (cpu) { +if (!cpu) +continue ; +cpu_set (cpu, frozen_cpus); + error =cpu_down (cpu); +if (!error) +continue ; +printk ("Error taking cpu %d down: %d\n" , cpu, error); +panic ("Too many cpus" ); } -printk ("ok\n" ); +BUG_ON (smp_processor_id () !=0 ); }void enable_nonboot_cpus (void ) { -printk ("Restarting CPUs" ); -atomic_set (&freeze,0 ); -while (atomic_read (&cpu_counter)) { -cpu_relax (); -barrier (); +int cpu, error; +printk ("Thawing cpus...\n" ); +for_each_cpu_mask (cpu, frozen_cpus) { +if (!cpu) +continue ; + error =cpu_up (cpu); +if (!error) +continue ; +printk ("Error taking cpu %d up: %d\n" , cpu, error); +panic ("Not enough cpus" ); } -printk ("..." ); -set_cpus_allowed (current, oldmask); -schedule (); -printk ("ok\n" ); - }
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Hi,
Quoted 5 lines · Pavel Machek
Hi!
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?
Heh, that's exactly what I was thinking about. ;-)
AFAICS we don't need the full CPU hotplug to do this. For suspend, we need to enable the CPU hotplug-related code in sched.c and cpu.c, and we need to implement the functions __cpu_disable() and __cpu_die() (called from within cpu.c) on each architecture for which we want swsusp to work on SMP.
If that's acceptable, the CPU hotplug code in sched.c and cpu.c may be enabled by changing some #ifdefs there. For example, we could replace the
#idef CONFIG_CPU_HOTPLUG
with
#if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND) && defined(CONFIG_SMP))
wherever necessary.
The implementation of __cpu_disable() and __cpu_die() may be a bit more tricky, however. In __cpu_disable() we need to save the settings of the local APIC of each CPU (on x86-64, at least) etc. In __cpu_die() we should give the "frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).
I've looked at the ia64 implementation of these functions, but I haven't fully understood it yet.
Greets,
Rafael
Signature · 3 lines
- Would you tell me, please, which way I ought to go from here? - That depends a good deal on where you want to get to. -- Lewis Carroll "Alice's Adventures in Wonderland"
Hi.
Quoted 26 lines · Rafael J. Wysocki
Quoted 5 lines · Pavel Machek
Hi!
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?Heh, that's exactly what I was thinking about. ;-)
AFAICS we don't need the full CPU hotplug to do this. For suspend, we need to enable the CPU hotplug-related code in
sched.candcpu.c, and we need to implement the functions__cpu_disable()and__cpu_die()(called from withincpu.c) on each architecture for which we want swsusp to work on SMP.If that's acceptable, the CPU hotplug code in
sched.candcpu.cmay be enabled by changing some #ifdefs there. For example, we could replace the#idef
CONFIG_CPU_HOTPLUGwith
#if defined(CONFIG_CPU_HOTPLUG) || (defined(CONFIG_SOFTWARE_SUSPEND) && defined(CONFIG_SMP))wherever necessary.
Quoted 7 lines
The implementation of
__cpu_disable()and__cpu_die()may be a bit more tricky, however. In__cpu_disable()we need to save the settings of the local APIC of each CPU (on x86-64, at least) etc. In__cpu_die()we should give the "frozen" CPU some "neutral" code to execute, it seems (or "hlt" it?).I've looked at the ia64 implementation of these functions, but I haven't fully understood it yet.
It would be good (32 bit) if you handled MTRRs as well. Putting their save/restore code in the driver model is racy (SMP calls). I've worked around that in Suspend2 by removing the driver model support and making the save/restore calls from the lowlevel code. I think this should be considered the right thing to do.
By the way, I'm keen to test this with Suspend2 as well. I would have done so before now, but I didn't realise Zwane had the support done.
Regards,
Nigel
Signature · 6 lines
Nigel Cunningham Software Engineer, Canberra, Australia http://www.cyclades.com Bus: +61 (2) 6291 9554; Hme: +61 (2) 6292 8028; Mob: +61 (417) 100 574 Maintainer of Suspend2 Kernel Patches http://suspend2.net
Quoted 5 lines · Pavel Machek
Hi!
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?
i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.
Nathan
Quoted 8 lines · Nathan Lynch
Quoted 3 lines · Pavel Machek
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.
Well this is a good user =)
Hi,
Quoted 9 lines · Nathan Lynch
Quoted 5 lines · Pavel Machek
Hi!
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.
Thanks a lot for this hint! ;-)
Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made HOTPLUG_CPU on x86-64 depend on SMP and SOFTWARE_SUSPEND and be set automatically.
I'm going to test it together with your patch tomorrow.
Greets,
Rafael
Signature · 3 lines
- Would you tell me, please, which way I ought to go from here? - That depends a good deal on where you want to get to. -- Lewis Carroll "Alice's Adventures in Wonderland"
Hi!
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.
Quoted 8 lines
Thanks a lot for this hint! ;-)
Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made
HOTPLUG_CPUon x86-64 depend on SMP andSOFTWARE_SUSPENDand be set automatically.I'm going to test it together with your patch tomorrow.
Hey, don't count on my patch. It is first shot, never tested.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Hi,
Quoted 11 lines · Pavel Machek
Hi!
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?i386 cpu hotplug has been in -mm for a while. Don't know when (if ever) it will get merged.
Thanks a lot for this hint! ;-)
Quoted 8 lines
Quoted 6 lines
Pavel, I've ported the basic i386 CPU hotplug stuff, without the sysfs interface, to x86-64 (a cut'n'paste kind of work, mostly). For now, I've made
HOTPLUG_CPUon x86-64 depend on SMP andSOFTWARE_SUSPENDand be set automatically.I'm going to test it together with your patch tomorrow.
Hey, don't count on my patch. It is first shot, never tested.
It'll be the first time, then. I'm not afraid. ;-)
Rafael
Signature · 3 lines
- Would you tell me, please, which way I ought to go from here? - That depends a good deal on where you want to get to. -- Lewis Carroll "Alice's Adventures in Wonderland"
Hi again-
Quoted 25 lines · Pavel Machek
kernel/power/smp.c +14
-6+int cpu, error; +cpus_clear (frozen_cpus); +printk ("Freezing cpus...\n" ); +for_each_online_cpu (cpu) { +if (!cpu) +continue ; +cpu_set (cpu, frozen_cpus); + error =cpu_down (cpu); +if (!error) +continue ; +printk ("Error taking cpu %d down: %d\n" , cpu, error); +panic ("Too many cpus" ); } -printk ("ok\n" ); +BUG_ON (smp_processor_id () !=0 ); }void enable_nonboot_cpus (void ) { -printk ("Restarting CPUs" ); -atomic_set (&freeze,0 ); -while (atomic_read (&cpu_counter)) { -cpu_relax (); -barrier (); +int cpu, error;
kernel/power/smp.c line 51
Quoted 10 lines
kernel/power/smp.c +9
-0+printk ("Thawing cpus...\n" ); +for_each_cpu_mask (cpu, frozen_cpus) { +if (!cpu) +continue ; + error =cpu_up (cpu); +if (!error) +continue ; +printk ("Error taking cpu %d up: %d\n" , cpu, error); +panic ("Not enough cpus" ); }
If you're going to depend on CONFIG_HOTPLUG_CPU for swsusp on smp, why not offline the non-boot cpus from userspace? Same goes for onlining additional cpus during resume.
Nathan
Hi!
Quoted 32 lines · Pavel Machek
Quoted 31 lines
kernel/power/smp.c +20
-6+int cpu, error; +cpus_clear (frozen_cpus); +printk ("Freezing cpus...\n" ); +for_each_online_cpu (cpu) { +if (!cpu) +continue ; +cpu_set (cpu, frozen_cpus); + error =cpu_down (cpu); +if (!error) +continue ; +printk ("Error taking cpu %d down: %d\n" , cpu, error); +panic ("Too many cpus" ); } -printk ("ok\n" ); +BUG_ON (smp_processor_id () !=0 ); }void enable_nonboot_cpus (void ) { -printk ("Restarting CPUs" ); -atomic_set (&freeze,0 ); -while (atomic_read (&cpu_counter)) { -cpu_relax (); -barrier (); +int cpu, error; +printk ("Thawing cpus...\n" ); +for_each_cpu_mask (cpu, frozen_cpus) { +if (!cpu) +continue ; + error =cpu_up (cpu); +if (!error)
Quoted 8 lines
kernel/power/smp.c line 51
Quoted 4 lines
kernel/power/smp.c +3
-0+continue ; +printk ("Error taking cpu %d up: %d\n" , cpu, error); +panic ("Not enough cpus" ); }If you're going to depend on
CONFIG_HOTPLUG_CPUfor swsusp on smp, why not offline the non-boot cpus from userspace? Same goes for onlining additional cpus during resume.
I'd like to stay in control. I may want to stop drivers first, or something like that. Plus I need to do that during resume, too, when userspace is not available, yet.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Hi!
Quoted 3 lines
I tried to solve long-standing uglyness in swsusp cmp code by calling cpu hotplug... only to find out that
CONFIG_CPU_HOTPLUGis not available on i386. Is there way to enableCPU_HOTPLUGon i386?
BTW Li Shaohua has prototype smp/S3 implementation. I'll take detailed look at that one.
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!