Skip to contentSkip to conversation

clarify limitations of hibernation

PATCH In review Luigi Semenzato 3 participants 17 messages Dec 26, 2019 – Jan 30, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover clarify limitations of hibernation

These patches aim to make it clearer under which circumstances hibernation will function as expected. They include one documentation change, and one change which logs more information on failure to enter hibernation.

Luigi Semenzato (2):
Documentation: clarify limitations of hibernation
pm: add more logging on hibernation failure

 Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++-
 kernel/power/snapshot.c                       | 18 ++++++++++++------
 2 files changed, 29 insertions(+), 7 deletions(-)
Signature · 1 line
2.24.1.735.g03f4e72817-goog
#1 lore raw

PATCH 1/2 Documentation: clarify limitations of hibernation

Entering hibernation (suspend-to-disk) will fail if the kernel cannot allocate enough memory to create a snapshot of all pages in use; i.e., if memory in use is over 1/2 of total RAM. This patch makes this limitation clearer in the documentation. Without it, users may assume that hibernation can replace suspend-to-RAM when in fact its functionality is more limited.

---
 Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
Documentation/admin-guide/pm/sleep-states.rst +17 -1
index cd3a28cb81f4..fd0072eb8c03 100644
@@ -112,7 +112,9 @@ Hibernation
 This state (also referred to as Suspend-to-Disk or STD) offers the greatest
 energy savings and can be used even in the absence of low-level platform support
 for system suspend.  However, it requires some low-level code for resuming the
-system to be present for the underlying CPU architecture.
+system to be present for the underlying CPU architecture.  Additionally, the
+current implementation can enter the hibernation state only when memory
+pressure is low (see "Limitations" below).
 
 Hibernation is significantly different from any of the system suspend variants.
 It takes three system state changes to put it into hibernation and two system
@@ -149,6 +151,20 @@ Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel
 configuration option is set.  However, this option can only be set if support
 for the given CPU architecture includes the low-level code for system resume.
 
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.
 
 Basic ``sysfs`` Interfaces for System Suspend and Hibernation
 =============================================================
Signature · 1 line
2.24.1.735.g03f4e72817-goog
#2 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]

Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4 -0
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large

Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167

Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10 -0
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?

Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 13 lines · Michal Hocko

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]

Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10 -0
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines

Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167

Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4 -0
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?

Good question, thanks.

The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.

If that works, then it may be appropriate to describe this technique in Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.

In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?

Note that I removed the cgroup suggestion in later versions of this patch:
https://marc.info/?l=linux-pm&m=157800718520897

Also, there was some related discussion here:
https://marc.info/?l=linux-kernel&m=157177497015315

Thanks!

Quoted 3 lines

--
Michal Hocko
SUSE Labs

#4 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 23 lines · Luigi Semenzato
Quoted 13 lines · Michal Hocko

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]

Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10 -0
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines

Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167

Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4 -0
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?

Quoted 6 lines

Good question, thanks.

The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.

I do not think hibernation can choose which swap device to use. Having an additional swap device migh help though because there will be more space to swap out to.

Quoted 7 lines

If that works, then it may be appropriate to describe this technique in Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.

In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?

OK, I can see it now. I forgot about the ugly hack in the page allocator that hibernation is using. If there is no way to make a forward progress for the allocation and we enter the allocator oom path (__alloc_pages_may_oom) pm_suspended_storage() bails out early and the allocator gives up.

That being said allocator would swap out processes so it doesn't make much sense to do that pro-actively. It can still fail if the swap is depleted though and then the hibernation gives up. This makes some sense because you wouldn't like to have something killed by the oom killer while hibernating right? Graceful failure should be a preferable action and let you decide what to do IMHO.

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Thank you all for the replies and sorry for the delay (vacation + flu).

This has given me various ideas for experiments and I will try to get to them in the future. For now, the cgroup workaround (described in my first version of the patch, but removed later) will do for us.

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

From this discussion, it seems that it should be possible to set up swap and hibernation in a way that increases the probability of success when entering hibernation (or maybe make it a certainty?). It would be useful to include such setup in the documentation. I don't know how to do this (yet) but if anybody does, it would be a great contribution.

Thanks!

Quoted 58 lines · Michal Hocko
Quoted 23 lines · Luigi Semenzato
Quoted 13 lines · Michal Hocko

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]

Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10 -0
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines

Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167

Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4 -0
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?

Quoted 6 lines

Good question, thanks.

The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.

I do not think hibernation can choose which swap device to use. Having an additional swap device migh help though because there will be more space to swap out to.

Quoted 7 lines

If that works, then it may be appropriate to describe this technique in Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.

In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?

OK, I can see it now. I forgot about the ugly hack in the page allocator that hibernation is using. If there is no way to make a forward progress for the allocation and we enter the allocator oom path (__alloc_pages_may_oom) pm_suspended_storage() bails out early and the allocator gives up.

That being said allocator would swap out processes so it doesn't make much sense to do that pro-actively. It can still fail if the swap is depleted though and then the hibernation gives up. This makes some sense because you wouldn't like to have something killed by the oom killer while hibernating right? Graceful failure should be a preferable action and let you decide what to do IMHO.
--
Michal Hocko
SUSE Labs

#6 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

I was under impression that s2disk is pretty much impossible without any swap.

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 11 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap. If there is no swap, but more than 50% of RAM is free or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Quoted 3 lines

--
Michal Hocko
SUSE Labs

#8 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

#9 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

Quoted 5 lines

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

OK---I had this wrong, thanks.

Quoted 9 lines
Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Yes, that's obvious.

Quoted 11 lines

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?

Thanks!

#10 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 56 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

Quoted 5 lines

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

OK---I had this wrong, thanks.

Quoted 9 lines
Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Yes, that's obvious.

Quoted 11 lines

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

Quoted 8 lines

-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?

I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.

Can you please remind me what you do to trigger the unexpected behavior?

#11 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

Quoted 5 lines

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

OK---I had this wrong, thanks.

Quoted 9 lines
Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Yes, that's obvious.

Quoted 11 lines

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

Quoted 7 lines
Quoted 2 lines

and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?

I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.

Can you please remind me what you do to trigger the unexpected behavior?

Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:

dd if=/dev/zero bs=1G count=1 | sleep infinity

I think dd has a 2 GB limit, or around that number, so you'll need a few of those.

Thanks!

#12 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 74 lines · Luigi Semenzato
Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

Quoted 5 lines

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

OK---I had this wrong, thanks.

Quoted 9 lines
Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Yes, that's obvious.

Quoted 11 lines

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

Quoted 7 lines
Quoted 2 lines

and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?

I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.

Can you please remind me what you do to trigger the unexpected behavior?

Quoted 7 lines

Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:

dd if=/dev/zero bs=1G count=1 | sleep infinity

I think dd has a 2 GB limit, or around that number, so you'll need a few of those.

And then you get -ENOMEM from hibernate_preallocate_memory(), or from somewhere else?

#13 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

Quoted 79 lines · Rafael J. Wysocki
Quoted 77 lines · Luigi Semenzato
Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

Quoted 6 lines
Quoted 2 lines

I was under impression that s2disk is pretty much impossible without any swap.

I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.

Quoted 5 lines

That's not correct.

The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.

OK---I had this wrong, thanks.

Quoted 9 lines
Quoted 5 lines

If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.

Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.

Yes, that's obvious.

Quoted 11 lines

Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.

So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).

This all makes sense, but we do this:

-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state

Quoted 7 lines
Quoted 2 lines

and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?

I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.

Can you please remind me what you do to trigger the unexpected behavior?

Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:

Quoted 7 lines
Quoted 4 lines

dd if=/dev/zero bs=1G count=1 | sleep infinity

I think dd has a 2 GB limit, or around that number, so you'll need a few of those.

And then you get -ENOMEM from hibernate_preallocate_memory(), or from somewhere else?

That is correct. More precisely, preallocate_image_memory() doesn't get enough pages, and then preallocate_image_highmem() either gets nothing, or in any case too little.

#14 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

In reply to Michal Hocko
Quoted 11 lines · Michal Hocko

On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]

Quoted 5 lines

The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.

I was under impression that s2disk is pretty much impossible without any swap.

And you were correct. :-)

#15 lore raw

Re: [PATCH 1/2] Documentation: clarify limitations of hibernation

In reply to Michal Hocko
Quoted 13 lines · Michal Hocko

On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]

Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10 -0
+Limitations of Hibernation
+==========================
+
+When entering hibernation, the kernel tries to allocate a chunk of memory large
+enough to contain a copy of all pages in use, to use it for the system
+snapshot.  If the allocation fails, the system cannot hibernate and the
+operation fails with ENOMEM.  This will happen, for instance, when the total
+amount of anonymous pages (process data) exceeds 1/2 of total RAM.
+
+One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines

Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167

Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4 -0
+excess anonymous pages out to swap before hibernating.  This can be achieved
+with memcgroups, by lowering memory usage limits with ``echo <new limit> >
+/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``.  However, the latter
+operation is not guaranteed to succeed.

I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?

Nothing in particular AFAICS, at least in theory.

The approach taken by the hibernation code is rather straightforward:
allocate enough memory to store a copy of every page (in RAM) that needs to be saved. These allocations are made one page at a time, so in theory they should not fail as long as there is enough swap space in the system, but I'm probably missing something here.

#16 lore raw

PATCH 2/2 pm: add more logging on hibernation failure

In reply to Luigi Semenzato

Hibernation fails when the kernel cannot allocate enough memory to copy all pages in use. This patch ensures that the failure reason is clearly logged.

---
 kernel/power/snapshot.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/power/snapshot.c +12 -6
index 26b9168321e7..df498717a97e 100644
@@ -1705,16 +1705,20 @@ int hibernate_preallocate_memory(void)
 	ktime_t start, stop;
 	int error;
 
-	pr_info("Preallocating image memory... ");
+	pr_info("Preallocating hibernation image memory\n");
 	start = ktime_get();
 
 	error = memory_bm_create(&orig_bm, GFP_IMAGE, PG_ANY);
-	if (error)
+	if (error) {
+		pr_err("Cannot allocate original bitmap\n");
 		goto err_out;
+	}
 
 	error = memory_bm_create(&copy_bm, GFP_IMAGE, PG_ANY);
-	if (error)
+	if (error) {
+		pr_err("Cannot allocate copy bitmap\n");
 		goto err_out;
+	}
 
 	alloc_normal = 0;
 	alloc_highmem = 0;
@@ -1804,8 +1808,11 @@ int hibernate_preallocate_memory(void)
 		alloc -= pages;
 		pages += pages_highmem;
 		pages_highmem = preallocate_image_highmem(alloc);
-		if (pages_highmem < alloc)
+		if (pages_highmem < alloc) {
+			pr_err("Image allocation is %lu pages short\n",
+				alloc - pages_highmem);
 			goto err_out;
+		}
 		pages += pages_highmem;
 		/*
 		 * size is the desired number of saveable pages to leave in
@@ -1836,13 +1843,12 @@ int hibernate_preallocate_memory(void)
 
  out:
 	stop = ktime_get();
-	pr_cont("done (allocated %lu pages)\n", pages);
+	pr_info("Allocated %lu pages for hibernation shapshot\n", pages);
 	swsusp_show_speed(start, stop, pages, "Allocated");
 
 	return 0;
 
  err_out:
-	pr_cont("\n");
 	swsusp_free();
 	return -ENOMEM;
 }
Signature · 1 line
2.24.1.735.g03f4e72817-goog