Skip to contentSkip to conversation

Software Suspend Functionality in 2.5

DISCUSSION Nigel Cunningham 3 participants 16 messages Feb 26, 2003 – Mar 3, 2003

lore
Share & tools
lore
#0 lore raw

Hi Linus.

Pavel and I have been discussing what functionality should be in software suspend, and I wanted to ask your opinion since you're the final decision maker anyway.

Under 2.4, I've added a lot functionality which has recently been merged by Florent (swsusp beta 18). This functionality brings us as close as I can get to the point where what you have in memory when you initiate the suspend is very close to what you have when you finish resuming. Settings on the proc interface and the amount of swap available do mean the degree to which this is true varies - you can set a soft limit on the image size, for example - but that's the gist of the changes. I implemented all the changes because as a user, I wanted the system to be as responsive as possible upon resume - I didn't want a ton of thrashing and an unresponsive system while processes tried to get back whatever had been discarded or forced to swap. Performance with the changes does not seem to be degraded, even though pages are currently write synchronously (batching of requests is in the pipeline).

Where the rubber hits the road, I guess, is that the process is a bit more complicated than the one in the kernel at the moment:
- Rather than freeing memory until no more can be freed, memory is freed until the image fits in available swap, will be able to be saved and reloaded and meets the user's size limit. (The limit can be disabled by setting it to zero. If we can't reduce the image size to the requested amount (eg the user requests a 1MB image), we continue anyway - hence the description of 'soft limit' above.
- Pages to be saved are put into two categories - those we definitely won't need during suspend and those that might/will be needed (simple algorithm). The two sets of pages are saved separately - those not needed are saved directly to disk, those needed are later copied and then the copy is saved (as per current algorithm).
- Since we're saving more pages, we need more pages for the pagedir (index). Unlike now, we probably won't be able to allocate (say) a hundred contiguous set of pages, so we need to be able to have a scattered (discontiguous) pagedir.
- We also need to be careful to free buffers & swap cache generated during the save/resume process, so as to not corrupt the image.

Naturally there are other changes, but perhaps the simplest comparison is to say that the 2.5.62 suspend.c is 34549 bytes (suspend.o 25132), whereas beta18 is 77945 bytes (suspend.o 53756 with all the debugging code turned off).

So can we please get your thoughts? Would you be willing to accept these additions when they're ready?

Regards,

Nigel

#1 lore raw

Nigel,

When I noticed some of your discussions on swsusp mailing list earlier, a question that crossed my mind was whether you'd thought about the possibility of compression of data at the time of copy.

Would that have been another way to helped achieve the objective you have in mind ? Do any issues come to mind ?

Regards
Suparna

Quoted 58 lines · Nigel Cunningham

Hi Linus.

Pavel and I have been discussing what functionality should be in software suspend, and I wanted to ask your opinion since you're the final decision maker anyway.

Under 2.4, I've added a lot functionality which has recently been merged by Florent (swsusp beta 18). This functionality brings us as close as I can get to the point where what you have in memory when you initiate the suspend is very close to what you have when you finish resuming. Settings on the proc interface and the amount of swap available do mean the degree to which this is true varies - you can set a soft limit on the image size, for example - but that's the gist of the changes. I implemented all the changes because as a user, I wanted the system to be as responsive as possible upon resume - I didn't want a ton of thrashing and an unresponsive system while processes tried to get back whatever had been discarded or forced to swap. Performance with the changes does not seem to be degraded, even though pages are currently write synchronously (batching of requests is in the pipeline).

Where the rubber hits the road, I guess, is that the process is a bit more complicated than the one in the kernel at the moment:
- Rather than freeing memory until no more can be freed, memory is freed until the image fits in available swap, will be able to be saved and reloaded and meets the user's size limit. (The limit can be disabled by setting it to zero. If we can't reduce the image size to the requested amount (eg the user requests a 1MB image), we continue anyway - hence the description of 'soft limit' above.
- Pages to be saved are put into two categories - those we definitely won't need during suspend and those that might/will be needed (simple algorithm). The two sets of pages are saved separately - those not needed are saved directly to disk, those needed are later copied and then the copy is saved (as per current algorithm).
- Since we're saving more pages, we need more pages for the pagedir (index). Unlike now, we probably won't be able to allocate (say) a hundred contiguous set of pages, so we need to be able to have a scattered (discontiguous) pagedir.
- We also need to be careful to free buffers & swap cache generated during the save/resume process, so as to not corrupt the image.

Naturally there are other changes, but perhaps the simplest comparison is to say that the 2.5.62 suspend.c is 34549 bytes (suspend.o 25132), whereas beta18 is 77945 bytes (suspend.o 53756 with all the debugging code turned off).

So can we please get your thoughts? Would you be willing to accept these additions when they're ready?

Regards,

Nigel

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#2 lore raw

Hi.

I've looked at LKCD and seen that they have a provision for compressing the pages which are written. Ideally, I'd like to see us sharing code because I reckon there's a fair bit in common between the two projects. For the moment, though, I haven't seriously considered implementing compression. I've just been concentrating on getting things as stable as possible under 2.4 (given that there's no driver model there), and beginning to seek to port the changes to 2.5. Perhaps compression could be added later, but I am worrying about the basics (getting the pages saved in any form!) first.

Quoted 13 lines · Suparna Bhattacharya

Nigel,

When I noticed some of your discussions on swsusp mailing list earlier, a question that crossed my mind was whether you'd thought about the possibility of compression of data at the time of copy.

Would that have been another way to helped achieve the objective you have in mind ? Do any issues come to mind ?

Regards
Suparna

#3 lore raw
Quoted 11 lines · Nigel Cunningham

Hi.

I've looked at LKCD and seen that they have a provision for compressing the pages which are written. Ideally, I'd like to see us sharing code because I reckon there's a fair bit in common between the two projects. For the moment, though, I haven't seriously considered implementing compression. I've just been concentrating on getting things as stable as possible under 2.4 (given that there's no driver model there), and beginning to seek to port the changes to 2.5. Perhaps compression could be added later, but I am worrying about the basics (getting the pages saved in any form!) first.

The thought was that with compression you can copy more pages into the same free memory space; meaning that you need to free up far less pages. If you are lucky enough, you might get to resume from very close to the state you suspended and hence avoid the initial sluggishness on resume.

But yes, you've rightly guessed that my question did stem from what we are doing on LKCD. People have asked me if we could reuse swsusp logic, and even I'd thought about that way back when swsusp started for 2.5. I remember talking to Pavel about it last year at OLS - hoping it could save some work for us :)

The problem was that we have to deal with far more stringent conditions in a crash dump situation, and the very things that seemed like interesting ideas or neat twists in swsusp (like the trick of freeing-up enough pages to be able to an atomic suspend-to-ram, or maybe even the refridgerator concept), just couldn't work or apply in that situation.

Aside of the fact that we can't swapout at that point of time, we do want a near-exact snapshot of memory. And we are ambitious enough to want to be able to snapshot full memory or as close as possible to that, if so required.

Since we've had to work on a solution that can be used for accurate non-disruptive dumps as well as crash dumps (the latter using kexec), I was wondering whether it was worth exploring possibilities of commonality with swsusp down the line ... I know its not probably not something very immediate, but just an indication on whether we should keep applicability for swsusp (probably reuse and share ideas/code back and forth between the two efforts) in mind as we move forward. Because we have to support a more restrictive situation when it comes to dumping, it just may be usable by swsusp too if we can get it right.

Regards
Suparna

Quoted 17 lines · Suparna Bhattacharya
Quoted 13 lines

Nigel,

When I noticed some of your discussions on swsusp mailing list earlier, a question that crossed my mind was whether you'd thought about the possibility of compression of data at the time of copy.

Would that have been another way to helped achieve the objective you have in mind ? Do any issues come to mind ?

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#4 lore raw

Hi!

Quoted 4 lines

Since we've had to work on a solution that can be used for accurate non-disruptive dumps as well as crash dumps (the latter using kexec), I was wondering whether it was worth exploring possibilities of commonality with

Quoted 8 lines

swsusp down the line ... I know its not probably not something very immediate, but just an indication on whether we should keep applicability for swsusp (probably reuse and share ideas/code back and forth between the two efforts) in mind as we move forward. Because we have to support a more restrictive situation when it comes to dumping, it just may be usable by swsusp too if we can get it right.

Well, less code duplication is always welcome. But notice we need *atomic* snapshots in swsusp, else we might corrupt data.

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.
#5 lore raw
Quoted 15 lines · Pavel Machek

Hi!

Quoted 4 lines

Since we've had to work on a solution that can be used for accurate non-disruptive dumps as well as crash dumps (the latter using kexec), I was wondering whether it was worth exploring possibilities of commonality with

Quoted 8 lines

swsusp down the line ... I know its not probably not something very immediate, but just an indication on whether we should keep applicability for swsusp (probably reuse and share ideas/code back and forth between the two efforts) in mind as we move forward. Because we have to support a more restrictive situation when it comes to dumping, it just may be usable by swsusp too if we can get it right.

Quoted 2 lines

Well, less code duplication is always welcome. But notice we need *atomic* snapshots in swsusp, else we might corrupt data.

Atomic snapshots are what we'd like for dump too, since we desire accurate dumps (minimum drift), so its not a conflicting requirement. The difference is that while you could do i/o (e.g to flush pages to free up memory) before initiating an atomic snapshot, we can't.

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#6 lore raw

Hi!

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

Since we've had to work on a solution that can be used for accurate non-disruptive dumps as well as crash dumps (the latter using kexec), I was wondering whether it was worth exploring possibilities of commonality with swsusp down the line ... I know its not probably not something very immediate, but just an indication on whether we should keep applicability for swsusp (probably reuse and share ideas/code back and forth between the two efforts) in mind as we move forward. Because we have to support a more restrictive situation when it comes to dumping, it just may be usable by swsusp too if we can get it right.

Quoted 7 lines
Quoted 2 lines

Well, less code duplication is always welcome. But notice we need *atomic* snapshots in swsusp, else we might corrupt data.

Atomic snapshots are what we'd like for dump too, since we desire accurate dumps (minimum drift), so its not a conflicting requirement. The difference is that while you could do i/o (e.g to flush pages to free up memory) before initiating an atomic snapshot, we can't.

OTOH "best-effort-atomic" is probably okay for you, while it is not acceptable for swsusp. Hopefully the code is not going to get too complicated by "must be atomic" and "must work with crashed system" requirements...

Pavel

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.
#7 lore raw
Quoted 10 lines · Pavel Machek
Quoted 4 lines

Atomic snapshots are what we'd like for dump too, since we desire accurate dumps (minimum drift), so its not a conflicting requirement. The difference is that while you could do i/o (e.g to flush pages to free up memory) before initiating an atomic snapshot, we can't.

OTOH "best-effort-atomic" is probably okay for you, while it is not acceptable for swsusp. Hopefully the code is not going to get too complicated by "must be atomic" and "must work with crashed system" requirements...

For the kind of atomicity you need there probably are two steps:
1) Quiesce the system - get to a point of consistency (when you can take a resumable snapshot)
2) Perform an atomic copy / snapshot

Step (1) would be different for swsusp and crash dump (not intended to be common ). But for Step (2), do you think what you need/do is complicated by crashed system requirements ?

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#8 lore raw

Hi!

Quoted 16 lines
Quoted 10 lines
Quoted 4 lines

Atomic snapshots are what we'd like for dump too, since we desire accurate dumps (minimum drift), so its not a conflicting requirement. The difference is that while you could do i/o (e.g to flush pages to free up memory) before initiating an atomic snapshot, we can't.

OTOH "best-effort-atomic" is probably okay for you, while it is not acceptable for swsusp. Hopefully the code is not going to get too complicated by "must be atomic" and "must work with crashed system" requirements...

For the kind of atomicity you need there probably are two steps:
1) Quiesce the system - get to a point of consistency (when you can take a resumable snapshot)
2) Perform an atomic copy / snapshot

Quoted 3 lines

Step (1) would be different for swsusp and crash dump (not intended to be common ). But for Step (2), do you think what you need/do is complicated by crashed system requirements ?

Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is really small piece of code. Also do you think you can free half of memory in crashed system? Thats what swsusp currently does...

[I need really little about LKCD... But you are going to need modified disk drivers etc, right? I'd like to get away without that in swsusp, at least in 2.6.X.]

Pavel

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.
#9 lore raw
Quoted 10 lines · Pavel Machek
Quoted 9 lines

For the kind of atomicity you need there probably are two steps:
1) Quiesce the system - get to a point of consistency (when you can take a resumable snapshot)
2) Perform an atomic copy / snapshot

Step (1) would be different for swsusp and crash dump (not intended to be common ). But for Step (2), do you think what you need/do is complicated by crashed system requirements ?

Quoted 8 lines

Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is really small piece of code. Also do you think you can free half of memory in crashed system? Thats what swsusp currently does...

[I need really little about LKCD... But you are going to need modified disk drivers etc, right? I'd like to get away without that in swsusp, at least in 2.6.X.]

With the changes I've made, which I'm starting to merge with Pavel, I think the two are a lot closer to each other.

With regard to quiescing the system, we need the same things stopped that you need. We can of course use drivers_suspend when you can't, but we could probably also use the SMP code you have.

I've got swsusp so that freeing memory is not necessary - the whole image can be written to disk. There is still an option for the user to aim for a smaller image (a soft limit can be set), and if there's not enough swap available, that will also cause some memory to be freed, but LKCD would run into that problem writing to swap too.

Regards,

Nigel

#10 lore raw
Quoted 15 lines · Nigel Cunningham
Quoted 14 lines · Pavel Machek
Quoted 9 lines

For the kind of atomicity you need there probably are two steps:
1) Quiesce the system - get to a point of consistency (when you can take a resumable snapshot)
2) Perform an atomic copy / snapshot

Step (1) would be different for swsusp and crash dump (not intended to be common ). But for Step (2), do you think what you need/do is complicated by crashed system requirements ?

Well, I guess count_and_copy_data_pages() is easy to share, OTOH it is really small piece of code. Also do you think you can free half of memory in crashed system? Thats what swsusp currently does...

Quoted 7 lines
Quoted 4 lines

[I need really little about LKCD... But you are going to need modified disk drivers etc, right? I'd like to get away without that in swsusp, at least in 2.6.X.]

With the changes I've made, which I'm starting to merge with Pavel, I think the two are a lot closer to each other.

Yes, I've noticed that, this is why it was in the context of your changes that I brought up the question.

Quoted 9 lines

With regard to quiescing the system, we need the same things stopped that you need. We can of course use drivers_suspend when you can't, but we could probably also use the SMP code you have.

I've got swsusp so that freeing memory is not necessary - the whole image can be written to disk. There is still an option for the user to aim for a smaller image (a soft limit can be set), and if there's not enough swap available, that will also cause some memory to be freed, but

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#11 lore raw

Hi.

Quoted 2 lines · Suparna Bhattacharya

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression we're going to get ahead of time, so we don't know how many extra pages we can save. The compression/decompression also takes extra time and puts more drain on a potentially low battery.

I suppose it could be included as an option if it was going to already be there for LKCD anyway, but I'm not sure it would be helpful for the usual use of swsusp.

Regards,

Nigel

#12 lore raw
Quoted 8 lines · Nigel Cunningham

Hi.

Quoted 2 lines · Suparna Bhattacharya

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression we're going to get ahead of time, so we don't know how many extra pages

The algorithm could be adjusted to deal with that, however ...

Quoted 2 lines

we can save. The compression/decompression also takes extra time and puts more drain on a potentially low battery.

.. I didn't think about the battery drain - valid point !

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India
#13 lore raw
Quoted 11 lines · Suparna Bhattacharya
Quoted 8 lines · Nigel Cunningham

Hi.

Quoted 2 lines · Suparna Bhattacharya

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression we're going to get ahead of time, so we don't know how many extra pages

The algorithm could be adjusted to deal with that, however ...

Ah I see. You're thinking of compressing the pages as we eat memory? I guess I need to look at LKCD more closely. I think it can wait until we get the basics of the expanded functionality sorted.

Quoted 5 lines
Quoted 3 lines

we can save. The compression/decompression also takes extra time and puts more drain on a potentially low battery.

.. I didn't think about the battery drain - valid point !

Of course that can be overcome too. The user simply starts swsusp earlier or turns off compression if necessary.

Thanks for the thought provoking :>

Regards,

Nigel

#14 lore raw
In reply to Suparna Bhattacharya

Hi!

Quoted 7 lines
Quoted 6 lines · Suparna Bhattacharya
Quoted 2 lines

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression we're going to get ahead of time, so we don't know how many extra pages

Quoted 6 lines

The algorithm could be adjusted to deal with that, however ...

Quoted 2 lines

we can save. The compression/decompression also takes extra time and puts more drain on a potentially low battery.

.. I didn't think about the battery drain - valid point !

Actually I don't quiet think so. gzip compression is pretty cheap, and if it makes you suspend faster and with less disk writes...

But I think it adds unneccessary complexity.

Pavel

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.
#15 lore raw
Quoted 14 lines · Pavel Machek

Hi!

Quoted 12 lines
Quoted 6 lines · Suparna Bhattacharya
Quoted 2 lines

If you add to that the possibility of being able to save more in less space if you have compression, would it be useful ?

I'm not sure that it would because we don't know how much compression we're going to get ahead of time, so we don't know how many extra pages

The algorithm could be adjusted to deal with that, however ...

Quoted 2 lines

we can save. The compression/decompression also takes extra time and puts more drain on a potentially low battery.

Quoted 6 lines
Quoted 1 line

.. I didn't think about the battery drain - valid point !

Actually I don't quiet think so. gzip compression is pretty cheap, and if it makes you suspend faster and with less disk writes...

But I think it adds unneccessary complexity.

If that's the only concern left, I guess its time for us to go back and complete what we have (we still have a few issues to think over and concentrate on fixing), look at Nigel's patches more closely, and then come back and discuss the algo sometime later .. You could make up your mind about the actual complexity then (both for suspend and resume paths).

Quoted 1 line

From what I can gather from this whole discussion, it seems

worth at least a detailed look on our part.

Thanks for the inputs and insights.

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Labs, India