Make swsusp actually work

Hi!

There were two bugs, and linux/mm.h one took me *very* long to find... Well, those bits used for zone should have been marked. Plus I hack ide_..._suspend code not to panic, and it now seems to work. [Sorry, 2pm, have to get some sleep.]

Please apply,
Pavel

@@ -1567,7 +1567,7 @@
 		struct hwgroup_s *hwgroup = ide_hwifs[i].hwgroup;
 
 		if (!hwgroup) continue;
-		hwgroup->handler = hwgroup->handler_save;
+		hwgroup->handler = NULL; /* hwgroup->handler_save; */
 		hwgroup->handler_save = NULL;
 	}
 	driver_blocked = 0;

@@ -1584,6 +1584,7 @@
 		if (hwgroup->handler != panic_box)
 			panic("Handler was not set to panic?");
 		hwgroup->handler_save = NULL;
+		hwgroup->handler = NULL;
 	}
 	driver_blocked = 0;
 }

@@ -303,7 +303,9 @@
 #define PG_arch_1		13
 #define PG_reserved		14
 #define PG_launder		15	/* written out by VM pressure.. */
-#define PG_nosave		29
+#define PG_nosave		16
+/* Don't you dare to use high bits, they seem to be used for something else! */
+
 
 /* Make it prettier to test the above... */
 #define UnlockPage(page)	unlock_page(page)

(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

I can suspend without oopses. Yeh!

However, during the boot '2419p5a3 resume=/dev/hda6' it oopses right after saying a couple of things about not being able to determine blocksize. I'll photograph the repeatable oops and get it to you when I have access to my camera again. Probably in the next 24 hours.

Do I need APM compiled into the kernel?

The US was never a democracy. It was a constitutional republic. Be that as it may, Thomas Hobbes wrote that there are only three kinds of governments: Monarchies, Democracies, and Aristocracies. All the rest of kinds and types are just versions of those already mentioned. The US is not a Democracy by any definition so clearly I cannot choose the glass in front of me...

While the country may aspire to be a constitutional republic, it seems to operate by the golden rule mixed with a bit of thuggery, so clearly I cannot choose the glass in front of you... Are there countries that operate differently?

Brian Litzinger <[email protected]>

    Copyright (c) 2002 By Brian Litzinger, All Rights Reserved

It tries killing init on resume for me and panics. the actual suspend worked though (apparently). I attached the ksymoops output to the panic.

ksymoops 2.4.5 on i686 2.4.19-pre5-ac3. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-pre5-ac3/ (default) -m /boot/System.map-2.4.19-pre5-ac3 (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

unable to handle kernel NULL pointer dereference at virtual address 00000000 00000000
*pde = 00000000

Oops: 0000
CPU: 0

EIP: 0010:[<00000000>] Not Tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010282
eax: c0292e58 ebx: c013f000 ecx: 00000304 edx: 00000000
esi: cff6a100 edi: 00000000 ebp: 00000000 esp: c1357e98
ds: 0018 es: 00000000 ss: 0018
Process swapper (pid: 1, stackpage=c1357000)

Stack: c0196c88 c0292e58 00000000 cff6a100 00000000 c0142512 cff6a100 00000000
    00000008 00000000 00000000 00000002 c0196d2a 00000000 cff6a100 cff6a100
    00000000 c0196f3d 00000000 cff6a100 00000304 00000200 00002480 00000304
Call Trace: [<c0196c88>] [<c0142512>] [<c0196d2a>] [<c0196f3d>] [<c0140358>] [<c014604>] [<c011a776>] [<c011a866>] [<c0129c70>] [<c0129d5f>] [<c0105000>]
    [<c012a245>] [<c0105070>] [<c0105000>] [<c01071d6>] [<c0105050>]
Code: Bad EIP value

Trace; c0196c88 <generic_make_request+d8/140>
Trace; c0142512 <hash_page_buffers+102/120>
Trace; c0196d2a <submit_bh+3a/60>
Trace; c0196f3d <ll_rw_block+18d/1b0>
Trace; c0140358 <getblk+18/40>
Trace; 0c014604 Before first symbol
Trace; c011a776 <__call_console_drivers+46/60>
Trace; c011a866 <call_console_drivers+66/120>
Trace; c0129c70 <bdev_read+30/70>
Trace; c0129d5f <resume_try_to_read+af/530>
Trace; c0105000 <_stext+0/0>
Trace; c012a245 <software_resume+65/b0>
Trace; c0105070 <init+20/1b0>
Trace; c0105000 <_stext+0/0>
Trace; c01071d6 <kernel_thread+26/30>
Trace; c0105050 <init+0/1b0>

<0>Kernel Panic: Attempted to kill init!

1 warning issued. Results may not be reliable.

Hi!

I mailed two patches to the list in last two days. The first one should fix this.

I think you can have democracy and constitutional republic at same time, no?
Pavel

Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Hi!

You need my previous patch too. Here are both, combined.

Pavel

@@ -0,0 +1,168 @@
+From kernel/suspend.c:
+
+ * BIG FAT WARNING *********************************************************
+ *
+ * If you have unsupported (*) devices using DMA...
+ *				...say goodbye to your data.
+ *
+ * If you touch anything on disk between suspend and resume...
+ *				...kiss your data goodbye.
+ *
+ * If your disk driver does not support suspend... (IDE does)
+ *				...you'd better find out how to get along
+ *				   without your data.
+ *
+ * (*) pm interface support is needed to make it safe.
+
+You need to append resume=/dev/your_swap_partition to kernel command
+line. Then you suspend by echo 4 > /proc/acpi/sleep.
+
+[Notice. Rest docs is pretty outdated (see date!) It should be safe to
+use swsusp on ext3/reiserfs these days.]
+
+
+Article about goals and implementation of Software Suspend for Linux
+Author: Gábor Kuti
+Last revised: Jan 23rd, 2000
+
+Idea and goals to achieve
+
+Nowadays it is common in several laptops that they have a suspend button. It
+saves the state of the machine to a filesystem or to a partition and switches
+to standby mode. Later resuming the machine the saved state is loaded back to
+ram and the machine can continue its work. It has two real benefits. First we
+save ourselves the time machine goes down and later boots up, energy costs
+real high when running from batteries. The other gain is that we don't have to
+interrupt our programs so processes that are calculating something for a long
+time shouldn't need to be written interruptible.
+
+On desk machines the power saving function isn't as important as it is in
+laptops but we really may benefit from the second one. Nowadays the number of
+desk machines supporting suspend function in their APM is going up but there
+are (and there will still be for a long time) machines that don't even support
+APM of any kind. On the other hand it is reported that using APM's suspend
+some irqs (e.g. ATA disk irq) is lost and it is annoying for the user until
+the Linux kernel resets the device.
+
+So I started thinking about implementing Software Suspend which doesn't need
+any APM support and - since it uses pretty near only high-level routines - is
+supposed to be architecture independent code.
+
+Using the code
+
+The code is experimental right now - testers, extra eyes are welcome. To
+compile this support into the kernel, you need CONFIG_EXPERIMENTAL, 
+and then CONFIG_SOFTWARE_SUSPEND in menu General Setup to be  enabled. It
+cannot be used as a module and I don't think it will ever be needed.
+
+You have two ways to use this code. The first one is if you've compiled in
+sysrq support then you may press Sysrq-D to request suspend. The other way
+is with a patched SysVinit (my patch is against 2.76 and available at my
+home page). You might call 'swsusp' or 'shutdown -z <time>'.
+
+Either way it saves the state of the machine into active swaps and then
+reboots. By the next booting the kernel's resuming function is either triggered
+by swapon -a (which is ought to be in the very early stage of booting) or you
+may explicitly specify the swap partition/file to resume from with ``resume=''
+kernel option. If signature is found it loads and restores saved state. If the
+option ``noresume'' is specified as a boot parameter, it skips the resuming.
+Warning! Look at section ``Things to implement'' to see what isn't yet
+implemented.  Also I strongly suggest you to list all active swaps in
+/etc/fstab. Firstly because you don't have to specify anything to resume and
+secondly if you have more than one swap area you can't decide which one has the
+'root' signature. Also, it doesn't yet support ``embedded'' swapfiles, e.g. you
+have a dos partition mounted on /dos and also have a /dos/swapfile named
+swapfile..  I plan to add support to mount all necessary filesystems readonly.
+
+In the meantime while the system is suspended you should not touch any of the
+hardware!
+
+About the code
+Goals reached
+
+The code can be downloaded from
+http://falcon.sch.bme.hu/~seasons/linux/. It mainly works but there are still
+some of XXXs, TODOs, FIXMEs in the code which seem not to be too important. It
+should work all right except for the problems listed in ``Things to
+implement''. Notes about the code are really welcome.
+
+How the code works
+
+When suspending is triggered it immediately wakes up process bdflush. Bdflush
+checks whether we have anything in our run queue tq_bdflush. Since we queued up
+function do_software_suspend, it is called. Here we shrink everything including
+dcache, inodes, buffers and memory (here mainly processes are swapped out). We
+count how many pages we need to duplicate (we have to be atomical!) then we
+create an appropiate sized page directory. It will point to the original and
+the new (copied) address of the page. We get the free pages by
+__get_free_pages() but since it changes state we have to be able to track it
+later so it also flips in a bit in page's flags (a new Nosave flag). We
+duplicate pages and then mark them as used (so atomicity is ensured). After
+this we write out the image to swaps, do another sync and the machine may
+reboot. We also save registers to stack.
+
+By resuming an ``inverse'' method is executed. The image if exists is loaded,
+loadling is either triggered by ``swapon -a'' or ``resume='' kernel option.  We
+change our task to bdflush (it is needed because if we don't do this init does
+an oops when it is waken up later) and then pages are copied back to their
+original location. We restore registers, free previously allocated memory,
+activate memory context and task information. Here we should restore hardware
+state but even without this the machine is restored and processes are continued
+to work. I think hardware state should be restored by some list (using
+notify_chain) and probably by some userland program (run-parts?) for users'
+pleasure. Check out my patch at the same location for the sysvinit patch.
+
+WARNINGS!
+- Using swsusp on logging fs (e.g.reiserfs) could lead to pretty bad things:
+  mounting of reiserfs (even readonly!) means log is replayed, which 
+  means writes to filesystem. That could lead to pretty bad corruption.
+- It does not like pcmcia cards. And this is logical: pcmcia cards need cardmgr to be
+  initialized. they are not initialized during singleuser boot, but "resumed" kernel does
+  expect them to be initialized. That leads to armagedon. You should eject any pcmcia cards
+  before suspending.
+
+Things to implement
+- SMP support. I've done an SMP support but since I don't have access to a kind
+  of this one I cannot test it. Please SMP people test it.  .. Tested it,
+  doesn't work. Had no time to figure out why. There is some mess with
+  interrupts AFAIK..
+- We should only make a copy of data related to kernel segment, since any
+  process data won't be changed.
+- Should \#define ADDRESS go into linux/mm.h?
+- By copying pages back to their original position, copy_page caused General
+  Protection Fault. Why?
+- Hardware state restoring.  Now there's support for notifying via the notify
+  chain, event handlers are welcome. Some devices may have microcodes loaded
+  into them. We should have event handlers for them aswell.
+- We should support other architectures (There are really only some arch
+  related functions..)
+- We should also restore original state of swaps if the ``noresume'' kernel
+  option is specified.. Or do we need such a feature to save state for some
+  other time? Do we need some kind of ``several saved states''?  (Linux-HA
+  people?). There's been some discussion about checkpointing on linux-future.
+- Should make more sanity checks. Or are these enough?
+
+Not so important ideas for implementing
+
+- Should SIGSTOP_SUSPEND called SIGFREEZE? Or is it some kind of stealing
+  for Solaris?
+- If a real time process is running then don't suspend the machine.
+- Is there any sense in compressing the outwritten pages?
+- Support for power.conf file as in Solaris, autoshutdown, special
+  devicetypes support, maybe in sysctl.
+- Introduce timeout for SMP locking. But first locking ought to work :O
+- Pre-detect if we don't have enough swap space or free it instead of
+  calling panic.
+- Support for adding/removing hardware while suspended?
+- We should not free pages at the beginning so aggressively, most of them
+  go there anyway..
+- If X is active while suspending then by resuming calling svgatextmode
+  corrupts the virtual console of X.. (Maybe this has been fixed AFAIK).
+
+Any other idea you might have tell me!
+
+Contacting the author
+If you have any question or any patch that solves the above or detected
+problems please contact me at [email protected]. I might delay
+answering, sorry about that.
+

@@ -1567,7 +1567,7 @@
 		struct hwgroup_s *hwgroup = ide_hwifs[i].hwgroup;
 
 		if (!hwgroup) continue;
-		hwgroup->handler = hwgroup->handler_save;
+		hwgroup->handler = NULL; /* hwgroup->handler_save; */
 		hwgroup->handler_save = NULL;
 	}
 	driver_blocked = 0;

@@ -1584,6 +1584,7 @@
 		if (hwgroup->handler != panic_box)
 			panic("Handler was not set to panic?");
 		hwgroup->handler_save = NULL;
+		hwgroup->handler = NULL;
 	}
 	driver_blocked = 0;
 }

@@ -303,7 +303,9 @@
 #define PG_arch_1		13
 #define PG_reserved		14
 #define PG_launder		15	/* written out by VM pressure.. */
-#define PG_nosave		29
+#define PG_nosave		16
+/* Don't you dare to use high bits, they seem to be used for something else! */
+
 
 /* Make it prettier to test the above... */
 #define UnlockPage(page)	unlock_page(page)

@@ -522,15 +522,15 @@
 	tc_init();
 #endif
 
-	/* This has to be before mounting root, because even readonly mount of reiserfs would replay
-	   log corrupting stuff */
-	software_resume();
-
 	/* Networking initialization needs a process context */ 
 	sock_init();
 
 	start_context_thread();
 	do_initcalls();
+
+	/* This has to be before mounting root, because even readonly mount of reiserfs would replay
+	   log corrupting stuff */
+	software_resume();
 
 #ifdef CONFIG_IRDA
 	irda_proto_init();

@@ -32,20 +32,6 @@
  * More state savers are welcome. Especially for the scsi layer...
  *
  * For TODOs,FIXMEs also look in Documentation/swsusp.txt
- *
- * BIG FAT WARNING *********************************************************
- *
- * If you have unsupported (*) devices using DMA...
- *				...say goodbye to your data.
- *
- * If you touch anything on disk between suspend and resume...
- *				...kiss your data goodbye.
- *
- * If your disk driver does not support suspend... (IDE does)
- *				...you'd better find out how to get along
- *				   without your data.
- *
- * (*) pm interface support is needed to make it safe.
  */
 
 /*

(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

the documentation suggests that you do not need to specify resume= . Is this only true if you have the sysvinit patch in use? Is swapon -a simply a matter of making it the first init script executed by init?

Hi!

Then docs is wrong.
Pavel

Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Then you need to change this from your previous "all in one" patch.

from the swsusp.txt in "Using the code" 3rd paragraph
<
Either way it saves the state of the machine into active swaps and then reboots. By the next booting the kernel's resuming function is either triggered by swapon -a (which is ought to be in the very early stage of booting) or you may explicitly specify the swap partition/file to resume from with ``resume='' kernel option.

This seems to suggest that you have a choice. Which last time i checked, you dont.

from the swsusp.txt in "How the code works" 2nd paragraph Same thing as before basically. swapon -a does not trigger a resume

under warnings!
Ext3 fs seems to show no more of a risk than a non-journalling fs. perhaps that problem is reiserfs only?

Also, mention of the swap files being described in fstab is made in "Using the code" but no mention is made to how they must be loaded and must be actual raw partitions. Files of course would not work as viable swaps for resume because the fs would have to be mounted to load them.

and one more thing. What happens when you have multiple swap files all of equal priority (normal swap conditions have a striping effect (like
raid)) Will swsusp choose one ? How do we know which one it chose? Is
it just the first one in /proc/swaps all the time? That kind of behavior would be nice to document in swsusp.txt.

Thanks for the patches. it seems to work on my non X box (p4) just fine. I'll have to risk disaster and try it out on a dri X session soon since that's where the convenience would come into play.

In a technical sense the difference is when a vote is taken to
pass/not pass a law. In a republic your representative votes for
you. In a democracy every citizen in the whole nation votes.

Eric

Hi!

Fixed, thanx.

Warning about journalling filesystem was obsoleted. In now should work okay with journalled filesystems.

Killed relevant part.

Just don't use multiple swap partitions for now.

Switch to text console and do suspend there. That should work.

Pavel

Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.

Hi!

Okay, here's updated docs. Any more comments?

[Alan, either copy this over your Documentation/swsusp.txt, or just ignore the issue and I'll fixed in -ac5.]
Pavel

* BIG FAT WARNING ********************************************************* *
* If you have unsupported (*) devices using DMA...
* ...say goodbye to your data.
*
* If you touch anything on disk between suspend and resume...
* ...kiss your data goodbye.
*
* If your disk driver does not support suspend... (IDE does)
* ...you'd better find out how to get along
* without your data.
*
* (*) pm interface support is needed to make it safe.

You need to append resume=/dev/your_swap_partition to kernel command line. Then you suspend by echo 4 > /proc/acpi/sleep.

[Notice. Rest docs is pretty outdated (see date!) It should be safe to use swsusp on ext3/reiserfs these days.]

Article about goals and implementation of Software Suspend for Linux
Author: Gábor Kuti
Last revised: 2002-04-08

Idea and goals to achieve

Nowadays it is common in several laptops that they have a suspend button. It saves the state of the machine to a filesystem or to a partition and switches to standby mode. Later resuming the machine the saved state is loaded back to ram and the machine can continue its work. It has two real benefits. First we save ourselves the time machine goes down and later boots up, energy costs real high when running from batteries. The other gain is that we don't have to interrupt our programs so processes that are calculating something for a long time shouldn't need to be written interruptible.

On desk machines the power saving function isn't as important as it is in laptops but we really may benefit from the second one. Nowadays the number of desk machines supporting suspend function in their APM is going up but there are (and there will still be for a long time) machines that don't even support APM of any kind. On the other hand it is reported that using APM's suspend some irqs (e.g. ATA disk irq) is lost and it is annoying for the user until the Linux kernel resets the device.

So I started thinking about implementing Software Suspend which doesn't need any APM support and - since it uses pretty near only high-level routines - is supposed to be architecture independent code.

Using the code

The code is experimental right now - testers, extra eyes are welcome. To compile this support into the kernel, you need CONFIG_EXPERIMENTAL, and then CONFIG_SOFTWARE_SUSPEND in menu General Setup to be enabled. It cannot be used as a module and I don't think it will ever be needed.

You have two ways to use this code. The first one is if you've compiled in sysrq support then you may press Sysrq-D to request suspend. The other way is with a patched SysVinit (my patch is against 2.76 and available at my home page). You might call 'swsusp' or 'shutdown -z <time>'. Next way is to echo 4 > /proc/acpi/sleep.

Either way it saves the state of the machine into active swaps and then reboots. You must explicitly specify the swap partition to resume from with ``resume='' kernel option. If signature is found it loads and restores saved state. If the option ``noresume'' is specified as a boot parameter, it skips the resuming. Warning! Look at section ``Things to implement'' to see what isn't yet implemented. Also I strongly suggest you to list all active swaps in /etc/fstab. Firstly because you don't have to specify anything to resume and secondly if you have more than one swap area you can't decide which one has the 'root' signature.

In the meantime while the system is suspended you should not touch any of the hardware!

About the code
Goals reached

The code can be downloaded from
http://falcon.sch.bme.hu/~seasons/linux/. It mainly works but there are still some of XXXs, TODOs, FIXMEs in the code which seem not to be too important. It should work all right except for the problems listed in ``Things to implement''. Notes about the code are really welcome.

How the code works

When suspending is triggered it immediately wakes up process bdflush. Bdflush checks whether we have anything in our run queue tq_bdflush. Since we queued up function do_software_suspend, it is called. Here we shrink everything including dcache, inodes, buffers and memory (here mainly processes are swapped out). We count how many pages we need to duplicate (we have to be atomical!) then we create an appropiate sized page directory. It will point to the original and the new (copied) address of the page. We get the free pages by __get_free_pages() but since it changes state we have to be able to track it later so it also flips in a bit in page's flags (a new Nosave flag). We duplicate pages and then mark them as used (so atomicity is ensured). After this we write out the image to swaps, do another sync and the machine may reboot. We also save registers to stack.

By resuming an ``inverse'' method is executed. The image if exists is loaded, loadling is either triggered by ``resume='' kernel option. We change our task to bdflush (it is needed because if we don't do this init does an oops when it is waken up later) and then pages are copied back to their original location. We restore registers, free previously allocated memory, activate memory context and task information. Here we should restore hardware state but even without this the machine is restored and processes are continued to work. I think hardware state should be restored by some list (using notify_chain) and probably by some userland program (run-parts?) for users' pleasure. Check out my patch at the same location for the sysvinit patch.

WARNINGS!
- It does not like pcmcia cards. And this is logical: pcmcia cards need cardmgr to be initialized. they are not initialized during singleuser boot, but "resumed" kernel does expect them to be initialized. That leads to armagedon. You should eject any pcmcia cards before suspending.

Things to implement
- SMP support. I've done an SMP support but since I don't have access to a kind of this one I cannot test it. Please SMP people test it. .. Tested it, doesn't work. Had no time to figure out why. There is some mess with interrupts AFAIK..
- We should only make a copy of data related to kernel segment, since any process data won't be changed.
- By copying pages back to their original position, copy_page caused General Protection Fault. Why?
- Hardware state restoring. Now there's support for notifying via the notify chain, event handlers are welcome. Some devices may have microcodes loaded into them. We should have event handlers for them aswell.
- We should support other architectures (There are really only some arch related functions..)
- We should also restore original state of swaps if the ``noresume'' kernel option is specified.. Or do we need such a feature to save state for some other time? Do we need some kind of ``several saved states''? (Linux-HA people?). There's been some discussion about checkpointing on linux-future.
- Should make more sanity checks. Or are these enough?

Not so important ideas for implementing

- If a real time process is running then don't suspend the machine.
- Is there any sense in compressing the outwritten pages?
- Support for power.conf file as in Solaris, autoshutdown, special devicetypes support, maybe in sysctl.
- Introduce timeout for SMP locking. But first locking ought to work :O
- Pre-detect if we don't have enough swap space or free it instead of calling panic.
- Support for adding/removing hardware while suspended?
- We should not free pages at the beginning so aggressively, most of them go there anyway..
- If X is active while suspending then by resuming calling svgatextmode corrupts the virtual console of X.. (Maybe this has been fixed AFAIK).

Any other idea you might have tell me!

Contacting the author
If you have any question or any patch that solves the above or detected problems please contact me at [email protected]. I might delay answering, sorry about that.

(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

I've applied both this patch and the earlier one, and now my 2.4.19-pre5-ac3 system can suspend and it can resume. However, when it resumed, I was stuck in the kernel SysRq function.

Couldn't get out of it.

And nothing seemed to work, other than it kept displaying the help each time I touched a key.

On the other hand, the swsusp in 2.4.18-WOLK3.3 works correctly.

Brian Litzinger <[email protected]>

    Copyright (c) 2002 By Brian Litzinger, All Rights Reserved

press alt Sysrq again and then enter

Hi!

You should be able to simply press sysrq again.

There should be nothing changed in that area. Perhaps its just timing?

Pavel

Casualities in World Trade Center: ~3k dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.