clarify limitations of hibernation
Subsystems (4)
cover clarify limitations of hibernation
These patches aim to make it clearer under which circumstances hibernation will function as expected. They include one documentation change, and one change which logs more information on failure to enter hibernation.
Luigi Semenzato (2):
Documentation: clarify limitations of hibernation
pm: add more logging on hibernation failure
Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++- kernel/power/snapshot.c | 18 ++++++++++++------ 2 files changed, 29 insertions(+), 7 deletions(-)
Signature · 1 line
2.24.1.735.g03f4e72817-goog
PATCH 1/2 Documentation: clarify limitations of hibernation
Entering hibernation (suspend-to-disk) will fail if the kernel cannot allocate enough memory to create a snapshot of all pages in use; i.e., if memory in use is over 1/2 of total RAM. This patch makes this limitation clearer in the documentation. Without it, users may assume that hibernation can replace suspend-to-RAM when in fact its functionality is more limited.
--- Documentation/admin-guide/pm/sleep-states.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
Documentation/admin-guide/pm/sleep-states.rst +17 -1
@@ -112,7 +112,9 @@ Hibernation This state (also referred to as Suspend-to-Disk or STD) offers the greatest energy savings and can be used even in the absence of low-level platform support for system suspend. However, it requires some low-level code for resuming the -system to be present for the underlying CPU architecture. +system to be present for the underlying CPU architecture. Additionally, the +current implementation can enter the hibernation state only when memory +pressure is low (see "Limitations" below). Hibernation is significantly different from any of the system suspend variants. It takes three system state changes to put it into hibernation and two system
@@ -149,6 +151,20 @@ Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel configuration option is set. However, this option can only be set if support for the given CPU architecture includes the low-level code for system resume. +Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large +enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to force +excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed. Basic ``sysfs`` Interfaces for System Suspend and Hibernation =============================================================
Signature · 1 line
2.24.1.735.g03f4e72817-goog
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]
Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4
-0+Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large
Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167
Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10
-0+enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to force +excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed.
I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 13 lines · Michal Hocko
On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10
-0+Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large +enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines
Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167
Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4
-0+excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed.I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?
Good question, thanks.
The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.
If that works, then it may be appropriate to describe this technique in Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.
In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?
Note that I removed the cgroup suggestion in later versions of this patch:
https://marc.info/?l=linux-pm&m=157800718520897
Also, there was some related discussion here:
https://marc.info/?l=linux-kernel&m=157177497015315
Thanks!
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 23 lines · Luigi Semenzato
Quoted 13 lines · Michal Hocko
On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10
-0+Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large +enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to forceQuoted 8 lines
Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167
Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4
-0+excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed.I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?
Quoted 6 lines
Good question, thanks.
The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.
I do not think hibernation can choose which swap device to use. Having an additional swap device migh help though because there will be more space to swap out to.
Quoted 7 lines
If that works, then it may be appropriate to describe this technique in
Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?
OK, I can see it now. I forgot about the ugly hack in the page allocator that hibernation is using. If there is no way to make a forward progress for the allocation and we enter the allocator oom path (__alloc_pages_may_oom) pm_suspended_storage() bails out early and the allocator gives up.
That being said allocator would swap out processes so it doesn't make much sense to do that pro-actively. It can still fail if the swap is depleted though and then the hibernation gives up. This makes some sense because you wouldn't like to have something killed by the oom killer while hibernating right? Graceful failure should be a preferable action and let you decide what to do IMHO.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Thank you all for the replies and sorry for the delay (vacation + flu).
This has given me various ideas for experiments and I will try to get to them in the future. For now, the cgroup workaround (described in my first version of the patch, but removed later) will do for us.
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
From this discussion, it seems that it should be possible to set up swap and hibernation in a way that increases the probability of success when entering hibernation (or maybe make it a certainty?). It would be useful to include such setup in the documentation. I don't know how to do this (yet) but if anybody does, it would be a great contribution.
Thanks!
Quoted 58 lines · Michal Hocko
Quoted 23 lines · Luigi Semenzato
Quoted 13 lines · Michal Hocko
On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10
-0+Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large +enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to forceQuoted 8 lines
Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167
Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4
-0+excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed.I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?
Quoted 6 lines
Good question, thanks.
The hibernation image is stored into a swap device (or partition), so I suppose one could set up two swap devices, giving a lower priority to the hibernation device, so that it remains unused while the kernel reclaims pages for the hibernation image.
I do not think hibernation can choose which swap device to use. Having an additional swap device migh help though because there will be more space to swap out to.
Quoted 7 lines
If that works, then it may be appropriate to describe this technique in
Documentation/power/swsusp.rst. There's a brief mention of this situation in the Q/A section, but maybe this deserves more visibility.In my experience, the page allocator is prone to giving up in this kind of situation. But my experience is up to 4.X kernels. Is this guaranteed to work now?
OK, I can see it now. I forgot about the ugly hack in the page allocator that hibernation is using. If there is no way to make a forward progress for the allocation and we enter the allocator oom path (
__alloc_pages_may_oom)pm_suspended_storage()bails out early and the allocator gives up.That being said allocator would swap out processes so it doesn't make much sense to do that pro-actively. It can still fail if the swap is depleted though and then the hibernation gives up. This makes some sense because you wouldn't like to have something killed by the oom killer while hibernating right? Graceful failure should be a preferable action and let you decide what to do IMHO.
--
Michal Hocko
SUSE Labs
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]
Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
I was under impression that s2disk is pretty much impossible without any swap.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 11 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap. If there is no swap, but more than 50% of RAM is free or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Rafael J. Wysocki
maintainerRe: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.
Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
Quoted 5 lines
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
OK---I had this wrong, thanks.
Quoted 9 lines
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Yes, that's obvious.
Quoted 11 lines
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
This all makes sense, but we do this:
-- add resume=/dev/sdc to the command line -- attach a disk (/dev/sdc) with size equal to RAM -- mkswap /dev/sdc
-- swapon /dev/sdc
-- echo disk > /sys/power/state
and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?
Thanks!
Rafael J. Wysocki
maintainerRe: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 56 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
Quoted 5 lines
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
OK---I had this wrong, thanks.
Quoted 9 lines
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Yes, that's obvious.
Quoted 11 lines
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
This all makes sense, but we do this:
Quoted 8 lines
-- add resume=/dev/sdc to the command line -- attach a disk (
/dev/sdc) with size equal to RAM -- mkswap/dev/sdc
-- swapon/dev/sdc
-- echo disk >/sys/power/stateand the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?
I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.
Can you please remind me what you do to trigger the unexpected behavior?
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
Quoted 5 lines
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
OK---I had this wrong, thanks.
Quoted 9 lines
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Yes, that's obvious.
Quoted 11 lines
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
This all makes sense, but we do this:
-- add resume=/dev/sdc to the command line -- attach a disk (
/dev/sdc) with size equal to RAM -- mkswap/dev/sdc
-- swapon/dev/sdc
-- echo disk >/sys/power/state
Quoted 7 lines
Quoted 2 lines
and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?
I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.
Can you please remind me what you do to trigger the unexpected behavior?
Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:
dd if=/dev/zero bs=1G count=1 | sleep infinity
I think dd has a 2 GB limit, or around that number, so you'll need a few of those.
Thanks!
Rafael J. Wysocki
maintainerRe: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 74 lines · Luigi Semenzato
Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
Quoted 5 lines
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
OK---I had this wrong, thanks.
Quoted 9 lines
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Yes, that's obvious.
Quoted 11 lines
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
This all makes sense, but we do this:
-- add resume=/dev/sdc to the command line -- attach a disk (
/dev/sdc) with size equal to RAM -- mkswap/dev/sdc
-- swapon/dev/sdc
-- echo disk >/sys/power/stateQuoted 7 lines
Quoted 2 lines
and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?
I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.
Can you please remind me what you do to trigger the unexpected behavior?
Quoted 7 lines
Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:
dd if=/dev/zero bs=1G count=1 | sleep infinity
I think dd has a 2 GB limit, or around that number, so you'll need a few of those.
And then you get -ENOMEM from hibernate_preallocate_memory(), or from somewhere else?
Re: [PATCH 1/2] Documentation: clarify limitations of hibernation
Quoted 79 lines · Rafael J. Wysocki
Quoted 77 lines · Luigi Semenzato
Quoted 64 lines · Rafael J. Wysocki
Quoted 62 lines · Luigi Semenzato
Quoted 20 lines · Rafael J. Wysocki
Quoted 11 lines · Luigi Semenzato
Quoted 9 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
Quoted 6 lines
Quoted 2 lines
I was under impression that s2disk is pretty much impossible without any swap.
I am not sure what you mean by "swap" here. S2disk needs a swap partition for storing the image, but that partition is not used for regular swap.
Quoted 5 lines
That's not correct.
The swap partition (or file) used by s2disk needs to be made active before it can use it and the mm subsystem is also able to use it for regular swap then.
OK---I had this wrong, thanks.
Quoted 9 lines
Quoted 5 lines
If there is no swap, but more than 50% of RAM is free
or reclaimable, s2disk works fine. If anonymous is more than 50%, hibernation can still work, but swap needs to be set up (in addition to the space for the hibernation image). The setup is not obvious and I don't think that the documentation is clear on this.Well, the entire contents of RAM must be preserved, this way or another, during hibernation. That should be totally obvious to anyone using it really.
Yes, that's obvious.
Quoted 11 lines
Some of the RAM contents is copies of data already there in the filesystems on persistent storage and that does not need to be saved again. Everything else must be saved and s2disk (and Linux hibernation in general) uses active swap space to save these things. This implies that in order to hibernate the system, you generally need the amount of swap space equal to the size of RAM minus the size of files mapped into memory.
So, to be on the safe side, the total amount of swap space to be used for hibernation needs to match the size of RAM (even though realistically it may be smaller than that in the majority of cases).
This all makes sense, but we do this:
-- add resume=/dev/sdc to the command line -- attach a disk (
/dev/sdc) with size equal to RAM -- mkswap/dev/sdc
-- swapon/dev/sdc
-- echo disk >/sys/power/stateQuoted 7 lines
Quoted 2 lines
and the last operation fails with ENOMEM. Are we doing something wrong? Are we hitting some other mm bug?
I would expect this to work, so the fact that it doesn't work for you indicates a bug somewhere or at least an assumption that doesn't hold.
Can you please remind me what you do to trigger the unexpected behavior?
Yes, I create processes that use a large amount of anon memory, more than 50% of RAM, like this:
Quoted 7 lines
Quoted 4 lines
dd if=/dev/zero bs=1G count=1 | sleep infinity
I think dd has a 2 GB limit, or around that number, so you'll need a few of those.
And then you get
-ENOMEMfromhibernate_preallocate_memory(), or from somewhere else?
That is correct. More precisely, preallocate_image_memory() doesn't get enough pages, and then preallocate_image_highmem() either gets nothing, or in any case too little.
Rafael J. Wysocki
maintainerRe: [PATCH 1/2] Documentation: clarify limitations of hibernation
In reply to Michal HockoQuoted 11 lines · Michal Hocko
On Fri 24-01-20 08:37:12, Luigi Semenzato wrote:
[...]Quoted 5 lines
The purpose of my documentation patch was to make it clearer that hibernation may fail in situations in which suspend-to-RAM works; for instance, when there is no swap, and anonymous pages are over 50% of total RAM. I will send a new version of the patch which hopefully makes this clearer.
I was under impression that s2disk is pretty much impossible without any swap.
And you were correct. :-)
Rafael J. Wysocki
maintainerRe: [PATCH 1/2] Documentation: clarify limitations of hibernation
In reply to Michal HockoQuoted 13 lines · Michal Hocko
On Thu 26-12-19 14:02:04, Luigi Semenzato wrote:
[...]Quoted 10 lines
Documentation/admin-guide/pm/sleep-states.rst +10
-0+Limitations of Hibernation +========================== + +When entering hibernation, the kernel tries to allocate a chunk of memory large +enough to contain a copy of all pages in use, to use it for the system +snapshot. If the allocation fails, the system cannot hibernate and the +operation fails with ENOMEM. This will happen, for instance, when the total +amount of anonymous pages (process data) exceeds 1/2 of total RAM. + +One possible workaround (besides terminating enough processes) is to force
Quoted 8 lines
Documentation/admin-guide/pm/sleep-states.rst Hibernation is supported if the :c:macro:`CONFIG_HIBERNATION` kernel · line 167
Quoted 4 lines
Documentation/admin-guide/pm/sleep-states.rst +4
-0+excess anonymous pages out to swap before hibernating. This can be achieved +with memcgroups, by lowering memory usage limits with ``echo <new limit> > +/dev/cgroup/memory/<group>/memory.mem.usage_in_bytes``. However, the latter +operation is not guaranteed to succeed.I am not familiar with the hibernation process much. But what prevents those allocations to reclaim memory and push out the anonymous memory to the swap on demand during the hibernation's allocations?
Nothing in particular AFAICS, at least in theory.
The approach taken by the hibernation code is rather straightforward:
allocate enough memory to store a copy of every page (in RAM) that needs to be saved. These allocations are made one page at a time, so in theory they should not fail as long as there is enough swap space in the system, but I'm probably missing something here.
PATCH 2/2 pm: add more logging on hibernation failure
In reply to Luigi SemenzatoHibernation fails when the kernel cannot allocate enough memory to copy all pages in use. This patch ensures that the failure reason is clearly logged.
--- kernel/power/snapshot.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/power/snapshot.c +12 -6
@@ -1705,16 +1705,20 @@ int hibernate_preallocate_memory(void)ktime_t start, stop;int error; -pr_info ("Preallocating image memory... " ); +pr_info ("Preallocating hibernation image memory\n" ); start =ktime_get (); error =memory_bm_create (&orig_bm,GFP_IMAGE ,PG_ANY ); -if (error) +if (error) { +pr_err ("Cannot allocate original bitmap\n" ); goto err_out; + } error =memory_bm_create (©_bm,GFP_IMAGE ,PG_ANY ); -if (error) +if (error) { +pr_err ("Cannot allocate copy bitmap\n" ); goto err_out; + } alloc_normal =0 ; alloc_highmem =0 ;
@@ -1804,8 +1808,11 @@ int hibernate_preallocate_memory(void) alloc -= pages; pages += pages_highmem; pages_highmem =preallocate_image_highmem (alloc); -if (pages_highmem < alloc) +if (pages_highmem < alloc) { +pr_err ("Image allocation is %lu pages short\n" , + alloc - pages_highmem); goto err_out; + } pages += pages_highmem;/* * size is the desired number of saveable pages to leave in
@@ -1836,13 +1843,12 @@ int hibernate_preallocate_memory(void) out: stop =ktime_get (); -pr_cont ("done (allocated %lu pages)\n" , pages); +pr_info ("Allocated %lu pages for hibernation shapshot\n" , pages);swsusp_show_speed (start, stop, pages,"Allocated" );return 0 ; err_out: -pr_cont ("\n" );swsusp_free ();return -ENOMEM ; }
Signature · 1 line
2.24.1.735.g03f4e72817-goog