Memory consumption difference between in-kernel and userspace hibernation

Hi,

Let me first introduce my question, and then give details about the context.

Question: is there any difference in terms of memory requirements for the in-kernel hibernation (echo disk > /sys/power/state) and the userspace hibernation interface (through /dev/snapshot) ? With exactly the same userspace workload and applications running, the in-kernel hibernation works, but the hibernating using the userspace hibernation interface fails because not enough memory can be freed.

Now, the context.

I'm implementing hibernation on an embedded device, which has no swap since the only storage available is NAND flash.

I started by using the in-kernel hibernation mechanism, which saved the resume image directly into an MTD partition, declared as a swap just before starting the hibernation process (swapon /dev/mtdblockX; echo disk > /sys/power/state). This worked like a charm.

But writing the resume image directly to the MTD partition is not satisfying since it doesn't handle bad erase blocks and wear leveling. Therefore, I wanted to save the resume image into a file, inside a JFFS2 or YAFFS2 filesystem.

For this, I used the /dev/snapshot
userspace interface to swsusp. With a light workload, it works perfectly (both suspend and resume). But with a similar workload than the one tested with the in-kernel hibernation, things fail at the SNAPSHOT_ATOMIC_SNAPSHOT ioctl() step, which returns ENOMEM.

To get some details about the issue, I've added a few printk()s in swsusp_shrink_memory(). Here is the patch:

==================================================================

kernel/power/swsusp.c +9 -1

@@ -226,15 +226,20 @@
                highmem_size = count_highmem_pages();
                size = count_data_pages() + PAGES_FOR_IO;
                tmp = size;
+               printk("size=%d\n", size);
                size += highmem_size;
                for_each_zone (zone)
                        if (populated_zone(zone)) {
                                if (is_highmem(zone)) {
                                        highmem_size -= zone->free_pages;
                                } else {
+                                       printk("1 tmp=%d\n", tmp);
                                        tmp -= zone->free_pages;
+                                       printk("2 tmp=%d\n", tmp);
                                        tmp += zone->lowmem_reserve[ZONE_NORMAL];
+                                       printk("3 tmp=%d\n", tmp);
                                        tmp += snapshot_additional_pages(zone);
+                                       printk("4 tmp=%d\n", tmp);
                                }
                        }
 

@@ -243,9 +248,12 @@
 
                tmp += highmem_size;
                if (tmp > 0) {
+                       printk("trying to free %d pages\n", tmp);
                        tmp = __shrink_memory(tmp);
-                       if (!tmp)
+                       if (!tmp) {
+                               printk("\bfailed, ENOMEM\n");
                                return -ENOMEM;
+                       }
                        pages += tmp;
                } else if (size > image_size / PAGE_SIZE) {
                        tmp = __shrink_memory(size - (image_size / PAGE_SIZE));

==================================================================

I get the following output:

==================================================================
Stopping tasks ... done.
Shrinking memory... size=6967
1 tmp=6967
2 tmp=6639
3 tmp=6639
4 tmp=6643
trying to free 6643 pages
-size=4036

1 tmp=4036
2 tmp=777
3 tmp=777
4 tmp=781
trying to free 781 pages
failed, ENOMEM
Restarting tasks ... done.
==================================================================

Note 1: I've already reduced PAGES_FOR_IO from 1024 to 128.

Note 2: As usual in the embedded space, I'm stuck with an old 2.6.25
kernel.

Any idea on why it works with the in-kernel solution and not the userspace one ?

The userspace interface doesn't really allow you to write to a file. You can write into the area the file occupies on the partition, but you can't use the filesystem code for the actual writing. At least you shouldn't do that.

swsusp_shrink_memory() is used by both the in-kernel code and the userspace code more-or-less in the same way, so it looks strange. How much memory is there in the system?

s2disk allocates a few buffers for itself, but I'm not sure if that matters at all.

Which version of s2disk do you use, do you have encryption enabled in s2disk?

Hello,

Thanks for your feedback.

Quoted 4 lines · Rafael J. Wysocki

The userspace interface doesn't really allow you to write to a file. You can write into the area the file occupies on the partition, but you can't use the filesystem code for the actual writing. At least you shouldn't do that.

Ah. But it seems to work fairly nicely. Why can't the filesystem code could be used to store the resume image ? Note that my file is stored in a separate partition, fully dedicated to storing the resume file and mounted only at very specific points in the system lifetime.

Quoted 12 lines · Thomas Petazzoni

Hello,

Thanks for your feedback.

Le Thu, 12 Nov 2009 21:52:02 +0100,
"Rafael J. Wysocki" <[email protected]> a ??crit :

Quoted 4 lines

The userspace interface doesn't really allow you to write to a file. You can write into the area the file occupies on the partition, but you can't use the filesystem code for the actual writing. At least you shouldn't do that.

Quoted 4 lines

Ah. But it seems to work fairly nicely. Why can't the filesystem code could be used to store the resume image ? Note that my file is stored in a separate partition, fully dedicated to storing the resume file and mounted only at very specific points in the system lifetime.

That doesn't really matter.

The problem is that the image is likely to contain filesystem data (eg. superblocks etc.) that correspond to the state before the image has been created. Now, your using the filesystem code for writing the image modifies the on-disk metadata which become inconsistent with the filesystem data in the image. This inconsistencies may very well result in an unfixable corruption of the file system after the resume (that actually happened to a number of people, so it's not just pure theory).

That really depends on the fileystem used, though.

Question: is there any difference in terms of memory requirements for the in-kernel hibernation (echo disk > /sys/power/state) and the userspace hibernation interface (through /dev/snapshot) ? With exactly the same userspace workload and applications running, the in-kernel hibernation works, but the hibernating using the userspace hibernation interface fails because not enough memory can be freed.

Hello,

Quoted 6 lines · Thomas Petazzoni

Question: is there any difference in terms of memory requirements for the in-kernel hibernation (echo disk > /sys/power/state) and the userspace hibernation interface (through /dev/snapshot) ? With exactly the same userspace workload and applications running, the in-kernel hibernation works, but the hibernating using the userspace hibernation interface fails because not enough memory can be freed.

The difference is that when using the in-kernel solution, a swap is enabled, and it seems that it allows the kernel to free some memory in order to create the snapshot.

But I don't understand why : won't the swap contents be completely erased when the resume image will be written to it, and therefore all the things migrated to the swap when freing the memory would be lost ?