Skip to contentSkip to conversation

smp/swsusp done right

PATCH Pavel Machek 4 participants 6 messages Mar 23, 2005 – Mar 30, 2005

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi!

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.
Pavel

drivers/pci/pci.c +2 -0
@@ -376,11 +376,13 @@
 	if (!pci_find_capability(dev, PCI_CAP_ID_PM))
 		return PCI_D0;
 
+#if 0
 	if (platform_pci_choose_state) {
 		ret = platform_pci_choose_state(dev, state);
 		if (ret >= 0)
 			state = ret;
 	}
+#endif
 	switch (state) {
 	case 0: return PCI_D0;
 	case 3: return PCI_D3hot;
kernel/power/Kconfig +1 -1
@@ -28,7 +28,7 @@
 
 config SOFTWARE_SUSPEND
 	bool "Software Suspend (EXPERIMENTAL)"
-	depends on EXPERIMENTAL && PM && SWAP
+	depends on EXPERIMENTAL && PM && SWAP && (HOTPLUG_CPU || !SMP)
 	---help---
 	  Enable the possibility of suspending the machine.
 	  It doesn't need APM.
kernel/power/smp.c +32 -58

111 lines not on this page · raw message

kernel/power/swsusp.c +3 -0
@@ -1194,8 +1194,11 @@
 		return "version";
 	if (strcmp(swsusp_info.uts.machine,system_utsname.machine))
 		return "machine";
+#if 0
+	/* We can't use number of CPUs when we use hotplug to remove them ;-))) */
 	if(swsusp_info.cpus != num_online_cpus())
 		return "number of cpus";
+#endif
 	return NULL;
 }
 
Signature · 9 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!


-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
#1 lore raw

Hi,

Quoted 16 lines · Pavel Machek

Hi!

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.
Pavel

drivers/pci/pci.c +0 -0
@@ -376,11 +376,13 @@
 	if (!pci_find_capability(dev, PCI_CAP_ID_PM))
 		return PCI_D0;
 

drivers/pci/pci.c line 388

Quoted 10 lines
drivers/pci/pci.c +2 -0
+#if 0
 	if (platform_pci_choose_state) {
 		ret = platform_pci_choose_state(dev, state);
 		if (ret >= 0)
 			state = ret;
 	}
+#endif
 	switch (state) {
 	case 0: return PCI_D0;
 	case 3: return PCI_D3hot;

You probably don't want the above change to go in the final patch?

Quoted 85 lines
kernel/power/Kconfig +1 -1
@@ -28,7 +28,7 @@
 
 config SOFTWARE_SUSPEND
 	bool "Software Suspend (EXPERIMENTAL)"
-	depends on EXPERIMENTAL && PM && SWAP
+	depends on EXPERIMENTAL && PM && SWAP && (HOTPLUG_CPU || !SMP)
 	---help---
 	  Enable the possibility of suspending the machine.
 	  It doesn't need APM.
kernel/power/smp.c +12 -45
@@ -7,79 +7,53 @@
  * This file is released under the GPLv2.
  */
 
-#undef DEBUG
-
 #include <linux/smp_lock.h>
 #include <linux/interrupt.h>
 #include <linux/suspend.h>
 #include <linux/module.h>
 #include <asm/atomic.h>
 #include <asm/tlbflush.h>
+#include <asm/cpu.h>
 
-static atomic_t cpu_counter, freeze;
-
-
-static void smp_pause(void * data)
-{
-	struct saved_context ctxt;
-	__save_processor_state(&ctxt);
-	printk("Sleeping in:\n");
-	dump_stack();
-	atomic_inc(&cpu_counter);
-	while (atomic_read(&freeze)) {
-		/* FIXME: restore takes place at random piece inside this.
-		   This should probably be written in assembly, and
-		   preserve general-purpose registers, too
-
-		   What about stack? We may need to move to new stack here.
-
-		   This should better be ran with interrupts disabled.
-		 */
-		cpu_relax();
-		barrier();
-	}
-	atomic_dec(&cpu_counter);
-	__restore_processor_state(&ctxt);
-}
-
-static cpumask_t oldmask;
+cpumask_t frozen_cpus;
 
 void disable_nonboot_cpus(void)
 {
-	printk("Freezing CPUs (at %d)", smp_processor_id());
-	oldmask = current->cpus_allowed;
-	set_cpus_allowed(current, cpumask_of_cpu(0));
-	current->state = TASK_INTERRUPTIBLE;
-	schedule_timeout(HZ);
-	printk("...");
-	BUG_ON(smp_processor_id() != 0);
-
-	/* FIXME: for this to work, all the CPUs must be running
-	 * "idle" thread (or we deadlock). Is that guaranteed? */
+	int cpu, error;
 
-	atomic_set(&cpu_counter, 0);
-	atomic_set(&freeze, 1);
-	smp_call_function(smp_pause, NULL, 0, 0);
-	while (atomic_read(&cpu_counter) < (num_online_cpus() - 1)) {
-		cpu_relax();
-		barrier();
+	error = 0;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus ...\n");
+	for_each_online_cpu(cpu) {
+		if (cpu == 0)
+			continue;
+		error = cpu_down(cpu);
+		if (!error) {
+			cpu_set(cpu, frozen_cpus);

kernel/power/smp.c line 41

Quoted 10 lines
kernel/power/smp.c +7 -1
+			printk("CPU%d is down\n", cpu);
+			continue;
+		}
+		printk("Error taking cpu %d down: %d\n", cpu, error);
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
+	if (error)
+		panic("cpus not sleeping");
 }

I'm not sure whether we should panic() here. It may be better to make suspend fail and print a "please reboot immediately" message for the user. In that case, the user may be able to reboot without loosing data ...

Quoted 45 lines
kernel/power/smp.c +13 -12
 void enable_nonboot_cpus(void)
 {
-	printk("Restarting CPUs");
-	atomic_set(&freeze, 0);
-	while (atomic_read(&cpu_counter)) {
-		cpu_relax();
-		barrier();
-	}
-	printk("...");
-	set_cpus_allowed(current, oldmask);
-	schedule();
-	printk("ok\n");
+	int cpu, error;
 
+	printk("Thawing cpus ...\n");
+	for_each_cpu_mask(cpu, frozen_cpus) {
+		if (cpu == 0)
+			continue;
+		error = cpu_up(cpu);
+		if (!error) {
+			printk("CPU%d is up\n", cpu);
+			continue;
+		}
+		printk("Error taking cpu %d up: %d\n", cpu, error);
+		panic("Not enough cpus");
+	}
 }
-
-
kernel/power/swsusp.c +3 -0
@@ -1194,8 +1194,11 @@
 		return "version";
 	if (strcmp(swsusp_info.uts.machine,system_utsname.machine))
 		return "machine";
+#if 0
+	/* We can't use number of CPUs when we use hotplug to remove them ;-))) */
 	if(swsusp_info.cpus != num_online_cpus())
 		return "number of cpus";
+#endif
 	return NULL;
 }
 
Signature · 1 line
-- 

I'll test it when I get the CPU hotplug on x86-64 done.

Greets,
Rafael

Signature · 3 lines
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"
#2 lore raw

Hi!

Quoted 14 lines
Quoted 14 lines

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.
Pavel

drivers/pci/pci.c +0 -0
@@ -376,11 +376,13 @@
 	if (!pci_find_capability(dev, PCI_CAP_ID_PM))
 		return PCI_D0;
 
Quoted 12 lines

drivers/pci/pci.c line 388

Quoted 10 lines
drivers/pci/pci.c +2 -0
+#if 0
 	if (platform_pci_choose_state) {
 		ret = platform_pci_choose_state(dev, state);
 		if (ret >= 0)
 			state = ret;
 	}
+#endif
 	switch (state) {
 	case 0: return PCI_D0;
 	case 3: return PCI_D3hot;

You probably don't want the above change to go in the final patch?

No, not in final patch.

Quoted 29 lines
Quoted 15 lines
drivers/pci/pci.c +9 -6
-	atomic_set(&cpu_counter, 0);
-	atomic_set(&freeze, 1);
-	smp_call_function(smp_pause, NULL, 0, 0);
-	while (atomic_read(&cpu_counter) < (num_online_cpus() - 1)) {
-		cpu_relax();
-		barrier();
+	error = 0;
+	cpus_clear(frozen_cpus);
+	printk("Freezing cpus ...\n");
+	for_each_online_cpu(cpu) {
+		if (cpu == 0)
+			continue;
+		error = cpu_down(cpu);
+		if (!error) {
+			cpu_set(cpu, frozen_cpus);

drivers/pci/pci.c

Quoted 10 lines
drivers/pci/pci.c +7 -1
+			printk("CPU%d is down\n", cpu);
+			continue;
+		}
+		printk("Error taking cpu %d down: %d\n", cpu, error);
 	}
-	printk("ok\n");
+	BUG_ON(smp_processor_id() != 0);
+	if (error)
+		panic("cpus not sleeping");
 }

I'm not sure whether we should panic() here. It may be better to make suspend fail and print a "please reboot immediately" message for the user. In that case, the user may be able to reboot without loosing data

Review

Quoted 1 line
(quoted diff) +0 -1
-	...

I guess I could just fail the suspend, but I want to see the messages for now. (And I do not think it will ever trigger).
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#3 lore raw
In reply to Pavel Machek
Quoted 6 lines · Pavel Machek

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.

Yay! Thanks for getting that done Pavel =)

Zwane

#4 lore raw

Hi!

Quoted 8 lines
Quoted 6 lines

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.

Yay! Thanks for getting that done Pavel =)

Well, I guess it is thank you -- I got rid of ugly FIXME that would involve arch-dependend assembly to be solved properly.

... hmm, can play_dead handle all the memory being overwritten? Also it should probably save and restore registers including MTRRs.

...so I more moved ugly FIXME to better place. Oh well, at least it uses common infrastructure now. People at Intel doing suspend-to-RAM on smp systems will need to solve it properly, anyway, because they need to go to real mode and back.

Pavel

Signature · 9 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!


-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/Info/Sentarus/hamr30
#5 lore raw
Quoted 9 lines · Pavel Machek

Hi!

Quoted 7 lines
Quoted 6 lines

This is against -mm kernel; it is smp swsusp done right, and it actually works for me. Unlike previous hacks, it uses cpu hotplug infrastructure. Disable CONFIG_MTRR before you try this...

Test this if you can, and report any problems. If not enough people scream, this is going to -mm.

Quoted 7 lines
Quoted 1 line

Yay! Thanks for getting that done Pavel =)

Well, I guess it is thank you -- I got rid of ugly FIXME that would involve arch-dependend assembly to be solved properly.

... hmm, can play_dead handle all the memory being overwritten? Also it should probably save and restore registers including MTRRs.

play_dead currently can't handle memory being overwritten, i guess you'd probably want to put the hook to kill the processor for real there. We probably only have to make sure that MTRRs are restored on resume, which should be taken care of by the current code since ACPI hotplug probably requires it too.

Quoted 4 lines

...so I more moved ugly FIXME to better place. Oh well, at least it uses common infrastructure now. People at Intel doing suspend-to-RAM on smp systems will need to solve it properly, anyway, because they need to go to real mode and back.

Sounds like a perfect candidate =)

Thanks,
Zwane