kexec based hibernation -v3: kexec jump

This patch implements the functionality of jumping between the kexeced kernel and the original kernel.

A new reboot command named LINUX_REBOOT_CMD_KJUMP is defined to trigger the jumping to (executing) the new kernel and jumping back to the original kernel.

To support jumping between two kernels, before jumping to (executing) the new kernel and jumping back to the original kernel, the devices are put into quiescent state (to be fully implemented), and the state of devices and CPU is saved. After jumping back from kexeced kernel and jumping to the new kernel, the state of devices and CPU are restored accordingly. The devices/CPU state save/restore code of software suspend is called to implement corresponding function.

To support jumping without preserving memory. One shadow backup page is allocated for each page used by new (kexeced) kernel. When do kexec_load, the image of new kernel is loaded into shadow pages, and before executing, the original pages and the shadow pages are swapped, so the contents of original pages are backuped. Before jumping to the new (kexeced) kernel and after jumping back to the original kernel, the original pages and the shadow pages are swapped too.

A jump back protocol is defined and documented.

Signed-off-by Huang Ying <[email protected]>

This patch implements the functionality of jumping between the kexeced kernel and the original kernel.

A new reboot command named LINUX_REBOOT_CMD_KJUMP is defined to trigger the jumping to (executing) the new kernel and jumping back to the original kernel.

To support jumping between two kernels, before jumping to (executing) the new kernel and jumping back to the original kernel, the devices are put into quiescent state (to be fully implemented), and the state of devices and CPU is saved. After jumping back from kexeced kernel and jumping to the new kernel, the state of devices and CPU are restored accordingly. The devices/CPU state save/restore code of software suspend is called to implement corresponding function.

To support jumping without preserving memory. One shadow backup page is allocated for each page used by new (kexeced) kernel. When do kexec_load, the image of new kernel is loaded into shadow pages, and before executing, the original pages and the shadow pages are swapped, so the contents of original pages are backuped. Before jumping to the new (kexeced) kernel and after jumping back to the original kernel, the original pages and the shadow pages are swapped too.

A jump back protocol is defined and documented.

Signed-off-by Huang Ying <[email protected]>

Seems like good enough for -mm to me.

Pavel

Seems like good enough for -mm to me.

Pavel

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going to consider merging this kexec code, will you also please consider merging TuxOnIce?

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 4 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

Quoted 5 lines

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

Quoted 2 lines

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

See, it's different.

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

Quoted 2 lines

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

See, it's different.

Hi.

Quoted 4 lines · On Friday 21 September 2007 11:41:06 Andrew Morton wrote:

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

want

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

going

Quoted 2 lines

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

merging

Quoted 15 lines

Quoted 11 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

Quoted 1 line

See, it's different.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support. And it won't result in the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I wouldn't

be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device. On top of

that, there are all the issues related to device reinitialisation and so on,

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise you'd

Quoted 1 line

up with people screaming about no hibernation support. And it won't result

Quoted 1 line

the complete removal of the existing hibernation code from the kernel. At

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

Quoted 3 lines

hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to

Quoted 1 line

if it wants to try to save the image to a network storage device. On top

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

Quoted 1 line

that, there are all the issues related to device reinitialisation and so

Yes. Device reinitialisation is needed. But all in all, kexec based hibernation can be much simpler on the kernel side.

Quoted 3 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Configuration is a problem, we will work on it.

But, because it is based on kexec/kdump instead of starting from scratch, the duplicated part between hibernation and kexec/kdump can be eliminated.

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise you'd

Quoted 1 line

up with people screaming about no hibernation support. And it won't result

Quoted 1 line

the complete removal of the existing hibernation code from the kernel. At

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

Quoted 3 lines

hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to

Quoted 1 line

if it wants to try to save the image to a network storage device. On top

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

Quoted 1 line

that, there are all the issues related to device reinitialisation and so

Yes. Device reinitialisation is needed. But all in all, kexec based hibernation can be much simpler on the kernel side.

Quoted 3 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Configuration is a problem, we will work on it.

But, because it is based on kexec/kdump instead of starting from scratch, the duplicated part between hibernation and kexec/kdump can be eliminated.

Hi.

Quoted 1 line · Ying

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise you'd

end

Quoted 1 line

Quoted 1 line

up with people screaming about no hibernation support. And it won't result

in

Quoted 1 line

Quoted 1 line

the complete removal of the existing hibernation code from the kernel. At

the

Quoted 2 lines

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

wouldn't

Quoted 1 line

Quoted 3 lines

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

You've already said that you are currently saving all pages. How are you going to avoid saving free pages if you don't get the information from the kernel being saved? This will require more than just code reuse.

Quoted 1 line

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

being

Quoted 3 lines

Quoted 3 lines

hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to

use

Quoted 1 line

Quoted 1 line

if it wants to try to save the image to a network storage device. On top

of

Quoted 1 line

Quoted 2 lines

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

That only complicates things more. Now you need to get the information on where to save the image from the kernel being saved, then transfer it to userspace after switching to the kexec kernel. That's more kernel code, not less.

Quoted 1 line

Quoted 1 line

that, there are all the issues related to device reinitialisation and so

on,

Quoted 3 lines

Yes. Device reinitialisation is needed. But all in all, kexec based hibernation can be much simpler on the kernel side.

Sorry, but I'm yet to be convinced. I'm not unwilling, I'm just not there yet.

Quoted 9 lines

Quoted 3 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Configuration is a problem, we will work on it.

But, because it is based on kexec/kdump instead of starting from scratch, the duplicated part between hibernation and kexec/kdump can be eliminated.

Quoted 4 lines · Nigel Cunningham

Hi.

Quoted 1 line · Ying

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise

Quoted 2 lines

end

Quoted 1 line

Quoted 1 line

up with people screaming about no hibernation support. And it won't

Quoted 2 lines

in

Quoted 1 line

Quoted 1 line

the complete removal of the existing hibernation code from the kernel.

Quoted 10 lines

the

Quoted 2 lines

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

wouldn't

Quoted 4 lines

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

You've already said that you are currently saving all pages. How are you

Quoted 1 line

to avoid saving free pages if you don't get the information from the

Quoted 1 line

being saved? This will require more than just code reuse.

I have not tried "makedumpfile". The "makedumpfile" avoids saving free pages through checking the "mem_map" of the original kernel. I think there is nothing prevent it been used for kexec based hibernation image writing.

This is an example of duplicated effort between kexec/kdump and original hibernation implementation. Both kexec/kdump and hibernation need to save memory image without saving the free pages. This can be done once instead of twice.

Quoted 3 lines

Quoted 1 line

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

being

Quoted 1 line

Quoted 1 line

hibernated tells it what bdev/sectors in which to save the image as

Quoted 1 line

Quoted 1 line

Quoted 1 line

(otherwise you're going to need a dedicated, otherwise untouched

Quoted 1 line

Quoted 1 line

Quoted 1 line

exclusively for the kexec'd kernel to use), or what network settings

Quoted 2 lines

use

Quoted 1 line

Quoted 1 line

if it wants to try to save the image to a network storage device. On

Quoted 8 lines

of

Quoted 3 lines

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

That only complicates things more. Now you need to get the information on where to save the image from the kernel being saved, then transfer it to userspace after switching to the kexec kernel. That's more kernel code,

Quoted 1 line

less.

This is fairly simple in fact. For example, you can specify the bdev/sectors in kernel command line when do kexec load "kexec -l <...> --append='...'", then the image writing system can get it through "cat /proc/cmdline".

Quoted 4 lines · Nigel Cunningham

Hi.

Quoted 1 line · Ying

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise

Quoted 2 lines

end

Quoted 1 line

Quoted 1 line

up with people screaming about no hibernation support. And it won't

Quoted 2 lines

in

Quoted 1 line

Quoted 1 line

the complete removal of the existing hibernation code from the kernel.

Quoted 10 lines

the

Quoted 2 lines

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

wouldn't

Quoted 4 lines

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

You've already said that you are currently saving all pages. How are you

Quoted 1 line

to avoid saving free pages if you don't get the information from the

Quoted 1 line

being saved? This will require more than just code reuse.

I have not tried "makedumpfile". The "makedumpfile" avoids saving free pages through checking the "mem_map" of the original kernel. I think there is nothing prevent it been used for kexec based hibernation image writing.

This is an example of duplicated effort between kexec/kdump and original hibernation implementation. Both kexec/kdump and hibernation need to save memory image without saving the free pages. This can be done once instead of twice.

Quoted 3 lines

Quoted 1 line

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

being

Quoted 1 line

Quoted 1 line

hibernated tells it what bdev/sectors in which to save the image as

Quoted 1 line

Quoted 1 line

Quoted 1 line

(otherwise you're going to need a dedicated, otherwise untouched

Quoted 1 line

Quoted 1 line

Quoted 1 line

exclusively for the kexec'd kernel to use), or what network settings

Quoted 2 lines

use

Quoted 1 line

Quoted 1 line

if it wants to try to save the image to a network storage device. On

Quoted 8 lines

of

Quoted 3 lines

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

That only complicates things more. Now you need to get the information on where to save the image from the kernel being saved, then transfer it to userspace after switching to the kexec kernel. That's more kernel code,

Quoted 1 line

less.

This is fairly simple in fact. For example, you can specify the bdev/sectors in kernel command line when do kexec load "kexec -l <...> --append='...'", then the image writing system can get it through "cat /proc/cmdline".

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Andrew, you're seeing that it really doesn't mean the removal of all hibernation code from the kernel being suspended, aren't you? (And if the kexec'd kernel is the same binary, then there's more code again).

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Andrew, you're seeing that it really doesn't mean the removal of all hibernation code from the kernel being suspended, aren't you? (And if the kexec'd kernel is the same binary, then there's more code again).

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

Hi!

Quoted 9 lines

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Quoted 2 lines

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

I'm not sure how you'd write it to a separate file. Notice that kjump kernel may not mount journalling filesystems, not even read-only. (Ext3 replays journal in that case). You could pass block numbers from the original kernel...

Hi!

Quoted 9 lines

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Quoted 2 lines

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

I'm not sure how you'd write it to a separate file. Notice that kjump kernel may not mount journalling filesystems, not even read-only. (Ext3 replays journal in that case). You could pass block numbers from the original kernel...

Quoted 10 lines · Pavel Machek

Hi!

Quoted 9 lines

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Quoted 7 lines

Quoted 2 lines

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

I'm not sure how you'd write it to a separate file. Notice that kjump kernel may not mount journalling filesystems, not even read-only. (Ext3 replays journal in that case). You could pass block numbers from the original kernel...

The ext3 thing is a bug, the case for which I don't think has been
adequately explained to the ext[34] folks. There should be at least a no_replay mount flag available, or something. It has ramifications for more than just hibernation.

And yeah, I'm gonna bring up the swap files thing again. If you
can hibernate to a swap file, you can hibernate to a dedicated hibernation file, and vice versa.

If you can't hibernate to a swap file, then swap files are
effectively unsupported for any system you might want to hibernate. <handwave> I wonder what embedded folks would think about that </handwave>.

But, in my ignorance, I'm not sure even fixing the ext3 bug will
guarantee you consistent metadata so that you can handle a swap/hibernate file. You can do a sync(), but how do you make that not race against running processes without the freezer, or blkdev snapshots?

I guess uswsusp and the-patch-previously-known-as-suspend2 handle
this somehow, though.

(It's that same ignorance that has me waiting for someone with
established credit with kernel people to make that argument for the ext3 bug, so I can hang my own reasons for thinking that it's bad off of theirs).

Quoted 10 lines · Pavel Machek

Hi!

Quoted 9 lines

Quoted 4 lines

Sounds doable, as long as you can cope with long command lines (which shouldn't be a biggie). (If you've got a swapfile or parts of a swap partition already in use, it can be quite fragmented).

Hmm. This is an interesting problem. Sharing a swap file or a swap partition with the actual swap of user space pages does seem to be a limitation of this approach.

Quoted 7 lines

Quoted 2 lines

Although the fact that it is simple to write to a separate file may be a reasonable compensation.

I'm not sure how you'd write it to a separate file. Notice that kjump kernel may not mount journalling filesystems, not even read-only. (Ext3 replays journal in that case). You could pass block numbers from the original kernel...

The ext3 thing is a bug, the case for which I don't think has been
adequately explained to the ext[34] folks. There should be at least a no_replay mount flag available, or something. It has ramifications for more than just hibernation.

And yeah, I'm gonna bring up the swap files thing again. If you
can hibernate to a swap file, you can hibernate to a dedicated hibernation file, and vice versa.

If you can't hibernate to a swap file, then swap files are
effectively unsupported for any system you might want to hibernate. <handwave> I wonder what embedded folks would think about that </handwave>.

But, in my ignorance, I'm not sure even fixing the ext3 bug will
guarantee you consistent metadata so that you can handle a swap/hibernate file. You can do a sync(), but how do you make that not race against running processes without the freezer, or blkdev snapshots?

I guess uswsusp and the-patch-previously-known-as-suspend2 handle
this somehow, though.

(It's that same ignorance that has me waiting for someone with
established credit with kernel people to make that argument for the ext3 bug, so I can hang my own reasons for thinking that it's bad off of theirs).

But, in my ignorance, I'm not sure even fixing the ext3 bug will
guarantee you consistent metadata so that you can handle a swap/hibernate file. You can do a sync(), but how do you make that not race against running processes without the freezer, or blkdev snapshots?

I guess uswsusp and the-patch-previously-known-as-suspend2 handle
this somehow, though.

Quoted 6 lines · Joseph Fannin

But, in my ignorance, I'm not sure even fixing the ext3 bug will
guarantee you consistent metadata so that you can handle a swap/hibernate file. You can do a sync(), but how do you make that not race against running processes without the freezer, or blkdev snapshots?

Quoted 2 lines

I guess uswsusp and the-patch-previously-known-as-suspend2 handle
this somehow, though.

The image-writing kernel of kexec based hibernation run in a controlled way. It is not used by normal user, so only really necessary process need to be run. For example, it is possible that there is only one user process -- the image-writing process running in image-writing kernel. So, no freezer or blkdev snapshot is needed.

But, in my ignorance, I'm not sure even fixing the ext3 bug will
guarantee you consistent metadata so that you can handle a swap/hibernate file. You can do a sync(), but how do you make that not race against running processes without the freezer, or blkdev snapshots?

I guess uswsusp and the-patch-previously-known-as-suspend2 handle
this somehow, though.

This is fairly simple in fact. For example, you can specify the bdev/sectors in kernel command line when do kexec load "kexec -l <...> --append='...'", then the image writing system can get it through "cat /proc/cmdline".

This is fairly simple in fact. For example, you can specify the bdev/sectors in kernel command line when do kexec load "kexec -l <...> --append='...'", then the image writing system can get it through "cat /proc/cmdline".

Hi.

Quoted 1 line · Ying

Quoted 1 line

That's not true. Kexec will itself be an implementation, otherwise you'd

end

Quoted 1 line

Quoted 1 line

up with people screaming about no hibernation support. And it won't result

in

Quoted 1 line

Quoted 1 line

the complete removal of the existing hibernation code from the kernel. At

the

Quoted 2 lines

Quoted 2 lines

very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I

wouldn't

Quoted 1 line

Quoted 3 lines

This has been done by kexec/kdump guys. There is a makedumpfile utility and vmcoreinfo kernel mechanism to implement this. We can just reuse the work of kexec/kdump.

You've already said that you are currently saving all pages. How are you going to avoid saving free pages if you don't get the information from the kernel being saved? This will require more than just code reuse.

Quoted 1 line

Quoted 1 line

be surprised if it also ends up with an interface in which the kernel

being

Quoted 3 lines

Quoted 3 lines

hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to

use

Quoted 1 line

Quoted 1 line

if it wants to try to save the image to a network storage device. On top

of

Quoted 1 line

Quoted 2 lines

These can be done in user space. The image writing will be done in user space for kexec base hibernation.

That only complicates things more. Now you need to get the information on where to save the image from the kernel being saved, then transfer it to userspace after switching to the kexec kernel. That's more kernel code, not less.

Quoted 1 line

Quoted 1 line

that, there are all the issues related to device reinitialisation and so

on,

Quoted 3 lines

Yes. Device reinitialisation is needed. But all in all, kexec based hibernation can be much simpler on the kernel side.

Sorry, but I'm yet to be convinced. I'm not unwilling, I'm just not there yet.

Quoted 9 lines

Quoted 3 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Configuration is a problem, we will work on it.

But, because it is based on kexec/kdump instead of starting from scratch, the duplicated part between hibernation and kexec/kdump can be eliminated.

Hi.

Quoted 4 lines · On Friday 21 September 2007 11:41:06 Andrew Morton wrote:

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

want

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

going

Quoted 2 lines

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

merging

Quoted 15 lines

Quoted 11 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

Quoted 1 line

See, it's different.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support. And it won't result in the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I wouldn't

be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device. On top of

that, there are all the issues related to device reinitialisation and so on,

and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

On top of
that, there are all the issues related to device reinitialisation and so on,

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Kexec is by no means proven to be the panacea for all the issues.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

On top of
that, there are all the issues related to device reinitialisation and so on,

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Kexec is by no means proven to be the panacea for all the issues.

Quoted 3 lines · Nigel Cunningham

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

There needs to be an implementation of hibernation based on kexec with return yes.

Quoted 4 lines

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

That interface should be running kernel -> user space -> target kernel. Not direct kernel to kernel.

Quoted 6 lines

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

initramfs. We already seem to have that interface. And distros seems to do a pretty decent job of using it to configure systems.

Quoted 2 lines

On top of
that, there are all the issues related to device reinitialisation and so on,

Yes.

Quoted 2 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Not to be callous but that really is a user space and distro issue.

Quoted 1 line

Kexec is by no means proven to be the panacea for all the issues.

I agree. I'm still not quite convinced it will do a satisfactory job. But I think it does make sense to implement a general kexec with return and see if that can reasonably be used for handling hibernation issues. If done cleanly and with care the implementation won't be hibernation specific.

Frankly this looks like the best way I can see to implement a general mechanism for calling silly firmware/BIOS/EFI services after we have a kernel up and running. It's a little bit like allowing X to call iopl(3) and do inb/outb directly.

The configuration issues you raise pretty much exist for kexec on panic, and they seem to be being resolved for that case in a reasonable way. I do agree that the current kexec+return effort seems to be one of those unfortunate cases where we give every mechanism in the kernel to do something in user space and then no one actually implements the user space. That doesn't do any one any good.

For hibernation we don't have the absolute need to step outside of the current kernel that we do in the kexec on panic approach. However we have this practical fight about mechanism and policy, and kexec with return has this seductive allure that it appears to be the minimal necessary mechanism in the kernel.

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 36 lines · Eric W. Biederman

Quoted 3 lines · Nigel Cunningham

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

There needs to be an implementation of hibernation based on kexec with return yes.

Quoted 4 lines

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

That interface should be running kernel -> user space -> target kernel. Not direct kernel to kernel.

Quoted 6 lines

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

initramfs. We already seem to have that interface. And distros seems to do a pretty decent job of using it to configure systems.

Quoted 2 lines

On top of
that, there are all the issues related to device reinitialisation and so on,

Yes.

Quoted 2 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Not to be callous but that really is a user space and distro issue.

Quoted 7 lines

Quoted 1 line

Kexec is by no means proven to be the panacea for all the issues.

I agree. I'm still not quite convinced it will do a satisfactory job. But I think it does make sense to implement a general kexec with return and see if that can reasonably be used for handling hibernation issues. If done cleanly and with care the implementation won't be hibernation specific.

Yes, and that's worth doing anyway, IMO.

Quoted 12 lines

Frankly this looks like the best way I can see to implement a general mechanism for calling silly firmware/BIOS/EFI services after we have a kernel up and running. It's a little bit like allowing X to call iopl(3) and do inb/outb directly.

The configuration issues you raise pretty much exist for kexec on panic, and they seem to be being resolved for that case in a reasonable way. I do agree that the current kexec+return effort seems to be one of those unfortunate cases where we give every mechanism in the kernel to do something in user space and then no one actually implements the user space. That doesn't do any one any good.

Quoted 8 lines

For hibernation we don't have the absolute need to step outside of the current kernel that we do in the kexec on panic approach. However we have this practical fight about mechanism and policy, and kexec with return has this seductive allure that it appears to be the minimal necessary mechanism in the kernel.

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

Quoted 36 lines · Eric W. Biederman

Quoted 3 lines · Nigel Cunningham

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

There needs to be an implementation of hibernation based on kexec with return yes.

Quoted 4 lines

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

That interface should be running kernel -> user space -> target kernel. Not direct kernel to kernel.

Quoted 6 lines

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

initramfs. We already seem to have that interface. And distros seems to do a pretty decent job of using it to configure systems.

Quoted 2 lines

On top of
that, there are all the issues related to device reinitialisation and so on,

Yes.

Quoted 2 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Not to be callous but that really is a user space and distro issue.

Quoted 7 lines

Quoted 1 line

Kexec is by no means proven to be the panacea for all the issues.

I agree. I'm still not quite convinced it will do a satisfactory job. But I think it does make sense to implement a general kexec with return and see if that can reasonably be used for handling hibernation issues. If done cleanly and with care the implementation won't be hibernation specific.

Yes, and that's worth doing anyway, IMO.

Quoted 12 lines

Frankly this looks like the best way I can see to implement a general mechanism for calling silly firmware/BIOS/EFI services after we have a kernel up and running. It's a little bit like allowing X to call iopl(3) and do inb/outb directly.

The configuration issues you raise pretty much exist for kexec on panic, and they seem to be being resolved for that case in a reasonable way. I do agree that the current kexec+return effort seems to be one of those unfortunate cases where we give every mechanism in the kernel to do something in user space and then no one actually implements the user space. That doesn't do any one any good.

Quoted 8 lines

For hibernation we don't have the absolute need to step outside of the current kernel that we do in the kexec on panic approach. However we have this practical fight about mechanism and policy, and kexec with return has this seductive allure that it appears to be the minimal necessary mechanism in the kernel.

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

[CC: list trimmed]

Quoted 9 lines · Rafael J. Wysocki

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 4 lines

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

I'm confused. You seem to be saying that for hibernation the required sequence of steps is:

1. Put devices into low-power states
2. Execute _PTS method
3. Create and write out the image
4. Execute acpi_enter_sleep_state()

Am I missing something -- a step to put devices back in their full-power states before writing out the image? After all, you can't write an image if the disk drive isn't at full power.

Also, how exactly does this conflict with the requirements of the kexec-based approach?

At what point in the above sequence would the kexec call be made?

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 8 lines

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 8 lines

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Quoted 27 lines · Rafael J. Wysocki

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 8 lines

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 4 lines

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Then, is it possible to separate device quiesce from device suspend.

Perhaps not for swsusp, but for kexec based hibernation?

Quoted 27 lines · Rafael J. Wysocki

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 8 lines

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 4 lines

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Then, is it possible to separate device quiesce from device suspend.

Perhaps not for swsusp, but for kexec based hibernation?

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Well, I've been playing a bit with that for some time, but it's not easy by

any

Quoted 3 lines

Quoted 3 lines

means.

In short, I'm seeing some problems related to the handling of ACPI that seem

to

Quoted 1 line

Quoted 1 line

shatter the entire idea of having separate hibernate methods, at least as

far

Quoted 3 lines

Quoted 1 line

as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Well, I've been playing a bit with that for some time, but it's not easy by

any

Quoted 3 lines

Quoted 3 lines

means.

In short, I'm seeing some problems related to the handling of ACPI that seem

to

Quoted 1 line

Quoted 1 line

shatter the entire idea of having separate hibernate methods, at least as

far

Quoted 3 lines

Quoted 1 line

as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

Quoted 21 lines · Rafael J. Wysocki

Quoted 3 lines · huang ying

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

[--snip--]

Quoted 5 lines

Quoted 5 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Well, I've been playing a bit with that for some time, but it's not easy by

any

Quoted 3 lines

Quoted 3 lines

means.

In short, I'm seeing some problems related to the handling of ACPI that seem

to

Quoted 1 line

Quoted 1 line

shatter the entire idea of having separate hibernate methods, at least as

far

Quoted 3 lines

Quoted 1 line

as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Quoted 8 lines

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Suppose that instead of using ACPI S4 state at all, you instead just power off. Yes, you'll lose wakeup event functionality, and flashy LEDs, but doesn't this take care of the problem?

The firmware shouldn't see the hibernate as anything other than a shutdown and reboot.

ACPI should be initialized normally when resuming, which should take care of getting AC power status reported properly.

This should be the behavior, anyway, on the many systems that do not support S4.

Quoted 4 lines

Now, the ACPI specification requires us to put devices into low power states before executing _PTS and that's exactly what we're doing before a suspend to RAM. Thus, it seems that in general we need to do the same for hibernation on ACPI systems.

It seems that if ACPI S4 is going to be used, Switching to low power state is something that should be done only immediately before entering that state (i.e. after the image has already been saved).

In particular, it should not be done just before the atomic copy. It is true that (during resume) after the atomic copy snapshot is restored, drivers will need to be prepared (i.e. have saved whatever information is necessary) to _resume_ devices from the low power state, but that does not mean they have to actually be put into that low power state before the copy is made.

I agree that for the kexec implementation there may be additional issues. For swsusp, uswsusp, and tuxonice, though, I don't see why there should be a problem. I think that, as was recognized before, all of the issues are resolved by properly considering exactly what each callback should do and when it should be called. The problems stem from ambiguous specifications, or trying to use the same callback for two different purposes or in two different cases.

Let me know if I'm mistaken.

Quoted 9 lines

Quoted 8 lines

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 3 lines

Suppose that instead of using ACPI S4 state at all, you instead just power off. Yes, you'll lose wakeup event functionality, and flashy LEDs, but doesn't this take care of the problem?

Nope.

Quoted 2 lines

The firmware shouldn't see the hibernate as anything other than a shutdown and reboot.

Actually, this assumption is apparently wrong.

Quoted 9 lines

Quoted 8 lines

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 3 lines

Suppose that instead of using ACPI S4 state at all, you instead just power off. Yes, you'll lose wakeup event functionality, and flashy LEDs, but doesn't this take care of the problem?

Nope.

Quoted 2 lines

The firmware shouldn't see the hibernate as anything other than a shutdown and reboot.

Actually, this assumption is apparently wrong.

Quoted 13 lines · Rafael J. Wysocki

Quoted 9 lines

Quoted 8 lines

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 3 lines

Suppose that instead of using ACPI S4 state at all, you instead just power off. Yes, you'll lose wakeup event functionality, and flashy LEDs, but doesn't this take care of the problem?

Quoted 6 lines

Nope.

Quoted 2 lines

The firmware shouldn't see the hibernate as anything other than a shutdown and reboot.

Actually, this assumption is apparently wrong.

One gets the impression that the hibernation image includes a memory area used by the firmware. That could explain why devices need to be in a low-power state when the image is created -- so that when the image is restored, the firmware doesn't get confused about the device states.

It would also explain why the firmware sees resume-from-power-off-hibernation as different from a regular reboot:
because its data area gets overwritten as part of the resume.

In reality it's probably more complicated than this, with weird interactions between the firmware and the various ACPI methods. Nevertheless, the main idea seems valid.

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the above happens.

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the above happens.

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 3 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the above happens.

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 5 lines

Quoted 1 line

happens.

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 5 lines

Quoted 1 line

happens.

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Quoted 10 lines · Rafael J. Wysocki

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 2 lines

Quoted 1 line

happens.

Quoted 5 lines

Quoted 3 lines

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Well, it seems that there just must be some other bug. I would define anything that differs between the post-resume initialization of ACPI

from the normal boot initialization of ACPI as a bug. If the interaction with the hardware is the same, then the behavior will be the same.

Quoted 12 lines · Jeremy Maitin-Shepard

Quoted 10 lines · Rafael J. Wysocki

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 2 lines

Quoted 1 line

happens.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Well, it seems that there just must be some other bug. I would define anything that differs between the post-resume initialization of ACPI

I'm not sure what you mean.

Quoted 2 lines

from the normal boot initialization of ACPI as a bug. If the interaction with the hardware is the same, then the behavior will be the same.

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

Quoted 12 lines · Jeremy Maitin-Shepard

Quoted 10 lines · Rafael J. Wysocki

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 2 lines

Quoted 1 line

happens.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Well, it seems that there just must be some other bug. I would define anything that differs between the post-resume initialization of ACPI

I'm not sure what you mean.

Quoted 2 lines

from the normal boot initialization of ACPI as a bug. If the interaction with the hardware is the same, then the behavior will be the same.

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

Quoted 1 line · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

Quoted 1 line · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment. I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Since we currently throw away most of that in-kernel ACPI interpreter state data when we load the to-be-resumed image and replace it with the state from the previous boot it looks to the ACPI code and firmware like our system's hardware magically changed behind its back. The result is that the ACPI and firmware code is justifiably confused (although probably it should be more idempotent to begin with). There's 2 potential solutions:
1) Formalize and copy a *lot* of ACPI state from the resume-ing
kernel to the resume-ed kernel.
2) Properly call the ACPI S4 methods in the proper order

Neither one is particularly easy or particularly pleasant, especially given all the vendor bugs in this general area. Theoretically we should be able to do both, since one will be more reliable than the other on different systems depending on what kinds of firmware bugs they have.

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment. I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Since we currently throw away most of that in-kernel ACPI interpreter state data when we load the to-be-resumed image and replace it with the state from the previous boot it looks to the ACPI code and firmware like our system's hardware magically changed behind its back. The result is that the ACPI and firmware code is justifiably confused (although probably it should be more idempotent to begin with). There's 2 potential solutions:
1) Formalize and copy a *lot* of ACPI state from the resume-ing
kernel to the resume-ed kernel.
2) Properly call the ACPI S4 methods in the proper order

Neither one is particularly easy or particularly pleasant, especially given all the vendor bugs in this general area. Theoretically we should be able to do both, since one will be more reliable than the other on different systems depending on what kinds of firmware bugs they have.

Hi.

Quoted 7 lines · On Saturday 22 September 2007 09:19:18 Kyle Moffett wrote:

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment. I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

I could conceive of a scheme wherein this was extended for driver data. Since the memory needed would depend on the drivers loaded, it would probably require that the space be allocated when hibernating, and the locations of structures be stored in the image header and then drivers notified of the locations to use when preparing to resume, but it could work...

Quoted 10 lines

Since we currently throw away most of that in-kernel ACPI interpreter state data when we load the to-be-resumed image and replace it with the state from the previous boot it looks to the ACPI code and firmware like our system's hardware magically changed behind its back. The result is that the ACPI and firmware code is justifiably confused (although probably it should be more idempotent to begin with). There's 2 potential solutions:
1) Formalize and copy a *lot* of ACPI state from the resume-ing
kernel to the resume-ed kernel.
2) Properly call the ACPI S4 methods in the proper order

... that said, I don't think the above should be necessary in most cases. I believe we're already calling the ACPI S4 methods in the proper order. If I understood correctly, Rafael put a lot of effort into learning what that was, and into ensuring it does get done.

Quoted 3 lines

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

Well, if the boot and image kernels are different, which is now possible on x86_64 with some recent patches (currently in -mm), the nosave trick won't work.

Quoted 3 lines

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

Well, if the boot and image kernels are different, which is now possible on x86_64 with some recent patches (currently in -mm), the nosave trick won't work.

Hi!

Quoted 4 lines

Quoted 3 lines

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

Quoted 3 lines

Well, if the boot and image kernels are different, which is now possible on x86_64 with some recent patches (currently in -mm), the nosave trick won't work.

I guess we should remove the nosave.... at least from x86-64. If someone tries to use it, he'll get a nasty surprise.

Hi!

Quoted 4 lines

Quoted 3 lines

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

Quoted 3 lines

Well, if the boot and image kernels are different, which is now possible on x86_64 with some recent patches (currently in -mm), the nosave trick won't work.

I guess we should remove the nosave.... at least from x86-64. If someone tries to use it, he'll get a nasty surprise.

Hi.

Quoted 7 lines · On Saturday 22 September 2007 09:19:18 Kyle Moffett wrote:

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment. I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

That's certainly possible. We already pass a very small amount of data between the boot and resuming kernels at the moment, and it's done quite simply - by putting the variables we want to 'transfer' in a nosave page/section.

I could conceive of a scheme wherein this was extended for driver data. Since the memory needed would depend on the drivers loaded, it would probably require that the space be allocated when hibernating, and the locations of structures be stored in the image header and then drivers notified of the locations to use when preparing to resume, but it could work...

Quoted 10 lines

Since we currently throw away most of that in-kernel ACPI interpreter state data when we load the to-be-resumed image and replace it with the state from the previous boot it looks to the ACPI code and firmware like our system's hardware magically changed behind its back. The result is that the ACPI and firmware code is justifiably confused (although probably it should be more idempotent to begin with). There's 2 potential solutions:
1) Formalize and copy a *lot* of ACPI state from the resume-ing
kernel to the resume-ed kernel.
2) Properly call the ACPI S4 methods in the proper order

... that said, I don't think the above should be necessary in most cases. I believe we're already calling the ACPI S4 methods in the proper order. If I understood correctly, Rafael put a lot of effort into learning what that was, and into ensuring it does get done.

Quoted 9 lines · Jeremy Maitin-Shepard

Quoted 2 lines · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Quoted 4 lines · Kyle Moffett

Quoted 3 lines · Jeremy Maitin-Shepard

Quoted 1 line · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

Quoted 3 lines

Quoted 1 line

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

In fact we don't need to do this.

The solution is not to touch ACPI in the boot kernel (ie. the one that loads the image) and pass control to the image kernel. This is how it's supposed to work according to the spec, more or less (well, there are some ugly details that need handling, like the restoration of the NVS area).

Quoted 1 line

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Rather the ACPI state data that the platform firmware stores may be different, depending on whether you enter S4 or S5 during "power off" and that determines the interactions between the kernel and the firmware after the next boot.

Quoted 11 lines · Kyle Moffett

Quoted 9 lines · Jeremy Maitin-Shepard

Quoted 2 lines · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

Quoted 4 lines

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

In fact we don't need to do this.

The solution is not to touch ACPI in the boot kernel (ie. the one that loads the image) and pass control to the image kernel. This is how it's supposed to work according to the spec, more or less (well, there are some ugly details that need handling, like the restoration of the NVS area).

Quoted 3 lines

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Rather the ACPI state data that the platform firmware stores may be different, depending on whether you enter S4 or S5 during "power off" and that determines the interactions between the kernel and the firmware after the next boot.

Quoted 17 lines · Rafael J. Wysocki

Quoted 11 lines · Kyle Moffett

Quoted 9 lines · Jeremy Maitin-Shepard

Quoted 2 lines · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

Quoted 4 lines

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

Quoted 7 lines

In fact we don't need to do this.

The solution is not to touch ACPI in the boot kernel (ie. the one that loads the image) and pass control to the image kernel. This is how it's supposed to work according to the spec, more or less (well, there are some ugly details that need handling, like the restoration of the NVS area).

First of all, we will need to make the resumed kernel throw away *ALL* of its ACPI state on S5 and completely reinitialize ACPI as though it was booting for the first time on resume.

From what I can tell, we "throw away" all the ACPI state in the boot kernel and reinitialize it there, but then the reinitialized state is overwritten with the resumed kernel's state and the two don't always happen to be the same. (Like if a battery got replaced or AC status changed).

Umm, I don't see how that can possibly work properly. For a laptop, for example, the restore kernel will need to access the disk, the LCD display, and possibly the AC/battery and current CPU frequency. From what I understand of ACPI, both of the former may need ACPI code to operate properly (Isn't there an ATA taskfile object of some kind?) and the latter two almost definitely need ACPI.

Ergo the boot kernel may need to initialize and use ACPI just to run an ATA taskfile so it can read from the HDD efficiently.

Quoted 8 lines

Quoted 3 lines

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Rather the ACPI state data that the platform firmware stores may be different, depending on whether you enter S4 or S5 during "power off" and that determines the interactions between the kernel and the firmware after the next boot.

That's not what he was talking about. The problem discussed was:
(A) You hibernate your box, entering S5 (IE: power off)
(B) You resume the box and the boot kernel inits all the ACPI stuff.
(C) The boot kernel's ACPI state is completely replaced by the
resumed kernel's state.
(D) Hardware stops working mysteriously because of ACPI problems.

The only possible conclusion is that the state between the boot kernel and the resume kernel was *different* and so the device failed because the ACPI state in the resume kernel doesn't match the actual state of the hardware.

Quoted 17 lines · Rafael J. Wysocki

Quoted 11 lines · Kyle Moffett

Quoted 9 lines · Jeremy Maitin-Shepard

Quoted 2 lines · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

Quoted 4 lines

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

Quoted 7 lines

In fact we don't need to do this.

The solution is not to touch ACPI in the boot kernel (ie. the one that loads the image) and pass control to the image kernel. This is how it's supposed to work according to the spec, more or less (well, there are some ugly details that need handling, like the restoration of the NVS area).

First of all, we will need to make the resumed kernel throw away *ALL* of its ACPI state on S5 and completely reinitialize ACPI as though it was booting for the first time on resume.

From what I can tell, we "throw away" all the ACPI state in the boot kernel and reinitialize it there, but then the reinitialized state is overwritten with the resumed kernel's state and the two don't always happen to be the same. (Like if a battery got replaced or AC status changed).

Umm, I don't see how that can possibly work properly. For a laptop, for example, the restore kernel will need to access the disk, the LCD display, and possibly the AC/battery and current CPU frequency. From what I understand of ACPI, both of the former may need ACPI code to operate properly (Isn't there an ATA taskfile object of some kind?) and the latter two almost definitely need ACPI.

Ergo the boot kernel may need to initialize and use ACPI just to run an ATA taskfile so it can read from the HDD efficiently.

Quoted 8 lines

Quoted 3 lines

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Rather the ACPI state data that the platform firmware stores may be different, depending on whether you enter S4 or S5 during "power off" and that determines the interactions between the kernel and the firmware after the next boot.

That's not what he was talking about. The problem discussed was:
(A) You hibernate your box, entering S5 (IE: power off)
(B) You resume the box and the boot kernel inits all the ACPI stuff.
(C) The boot kernel's ACPI state is completely replaced by the
resumed kernel's state.
(D) Hardware stops working mysteriously because of ACPI problems.

The only possible conclusion is that the state between the boot kernel and the resume kernel was *different* and so the device failed because the ACPI state in the resume kernel doesn't match the actual state of the hardware.

Quoted 9 lines · Jeremy Maitin-Shepard

Quoted 2 lines · Rafael J. Wysocki

The ACPI platform firmware is allowed to preserve information accross the hibernation-resume cycle, so this need not be the same.

All of my comments related to the case where S4 is not being used (instead the system is just powered off normally), and a boot kernel that does not initialize ACPI is used. In that case, the ACPI platform firmware should not be able to distinguish a normal boot from a resume from hibernation.

I think that in order for this to work, there would need to be some ABI whereby the resume-ing kernel can pass its entire ACPI state and a bunch of other ACPI-related device details to the resume-ed kernel, which I believe it does not do at the moment.

I believe that what causes problems is the ACPI state data that the kernel stores is *different* between identical sequential boots, especially when you add/remove/replace batteries, AC, etc.

Quoted 10 lines · Rafael J. Wysocki

Quoted 6 lines · Jeremy Maitin-Shepard

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 2 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the

above

Quoted 2 lines

Quoted 1 line

happens.

Quoted 5 lines

Quoted 3 lines

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

No, it's not. I have tested that too with an ACPI-less boot kernel.

Well, it seems that there just must be some other bug. I would define anything that differs between the post-resume initialization of ACPI

from the normal boot initialization of ACPI as a bug. If the interaction with the hardware is the same, then the behavior will be the same.

"Rafael J. Wysocki" <[email protected]> writes:

[snip]

Quoted 3 lines

The ACPI NVS area is explicitly marked as reserved and we don't save it. On x86_64 we don't save any memory areas marked as reserved and yet the above happens.

I think you have mentioned before, though, that ACPI is first initialized by the boot kernel, before it is later initialized by resuming kernel. This could well be the source of the problem.

Quoted 13 lines · Rafael J. Wysocki

Quoted 9 lines

Quoted 8 lines

Well, the problem is that apparently some systems (eg. my HP nx6325) expect us to execute the _PTS ACPI global control method before creating the image _and_ to execute acpi_enter_sleep_state(ACPI_STATE_S4) in order to finally put the system into the sleep state. In particular, on nx6325, if we don't do that, then after the restore the status of the AC power will not be reported correctly (and if you replace the battery while in the sleep state, the battery status will not be updated correctly after the restore). Similar issues have been reported for other machines.

Quoted 3 lines

Suppose that instead of using ACPI S4 state at all, you instead just power off. Yes, you'll lose wakeup event functionality, and flashy LEDs, but doesn't this take care of the problem?

Quoted 6 lines

Nope.

Quoted 2 lines

The firmware shouldn't see the hibernate as anything other than a shutdown and reboot.

Actually, this assumption is apparently wrong.

One gets the impression that the hibernation image includes a memory area used by the firmware. That could explain why devices need to be in a low-power state when the image is created -- so that when the image is restored, the firmware doesn't get confused about the device states.

It would also explain why the firmware sees resume-from-power-off-hibernation as different from a regular reboot:
because its data area gets overwritten as part of the resume.

In reality it's probably more complicated than this, with weird interactions between the firmware and the various ACPI methods. Nevertheless, the main idea seems valid.

Quoted 2 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Nigel Cunningham <[email protected]> writes:

Quoted 4 lines

Quoted 3 lines

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Quoted 6 lines

Well, I've been playing a bit with that for some time, but it's not easy by any means.

In short, I'm seeing some problems related to the handling of ACPI that seem to shatter the entire idea of having separate hibernate methods, at least as far as ACPI systems are concerned.

So sadly to hear this. Can you details it a little? Or a link?

Quoted 3 lines · Nigel Cunningham

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support.

There needs to be an implementation of hibernation based on kexec with return yes.

Quoted 4 lines

And it won't result in
the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved.

That interface should be running kernel -> user space -> target kernel. Not direct kernel to kernel.

Quoted 6 lines

I wouldn't
be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device.

initramfs. We already seem to have that interface. And distros seems to do a pretty decent job of using it to configure systems.

Quoted 2 lines

On top of
that, there are all the issues related to device reinitialisation and so on,

Yes.

Quoted 2 lines

and it looks like there's greatly increased pain for users wanting to configure this new implementation.

Not to be callous but that really is a user space and distro issue.

Quoted 1 line

Kexec is by no means proven to be the panacea for all the issues.

I agree. I'm still not quite convinced it will do a satisfactory job. But I think it does make sense to implement a general kexec with return and see if that can reasonably be used for handling hibernation issues. If done cleanly and with care the implementation won't be hibernation specific.

Frankly this looks like the best way I can see to implement a general mechanism for calling silly firmware/BIOS/EFI services after we have a kernel up and running. It's a little bit like allowing X to call iopl(3) and do inb/outb directly.

The configuration issues you raise pretty much exist for kexec on panic, and they seem to be being resolved for that case in a reasonable way. I do agree that the current kexec+return effort seems to be one of those unfortunate cases where we give every mechanism in the kernel to do something in user space and then no one actually implements the user space. That doesn't do any one any good.

For hibernation we don't have the absolute need to step outside of the current kernel that we do in the kexec on panic approach. However we have this practical fight about mechanism and policy, and kexec with return has this seductive allure that it appears to be the minimal necessary mechanism in the kernel.

No one has yet attacked the hard problem of coming up with separate hibernate methods for drivers.

Hi.

Quoted 4 lines · On Friday 21 September 2007 11:41:06 Andrew Morton wrote:

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

want

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

going

Quoted 2 lines

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

merging

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

Quoted 7 lines

Quoted 2 lines

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

See, it's different.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support. And it won't result in the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I wouldn't be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device. On top of that, there are all the issues related to device reinitialisation and so on, and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Hi.

Quoted 4 lines · On Friday 21 September 2007 11:41:06 Andrew Morton wrote:

Quoted 4 lines · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

Quoted 1 line

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 1 line · <[email protected]> wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't

want

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

another hibernation implementation in the vanilla kernel. If you're

going

Quoted 2 lines

Quoted 2 lines

to

Quoted 1 line

Quoted 1 line

consider merging this kexec code, will you also please consider

merging

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

Quoted 7 lines

Quoted 2 lines

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

See, it's different.

That's not true. Kexec will itself be an implementation, otherwise you'd end up with people screaming about no hibernation support. And it won't result in the complete removal of the existing hibernation code from the kernel. At the very least, it's going to want the kernel being hibernated to have an interface by which it can find out which pages need to be saved. I wouldn't be surprised if it also ends up with an interface in which the kernel being hibernated tells it what bdev/sectors in which to save the image as well (otherwise you're going to need a dedicated, otherwise untouched partition exclusively for the kexec'd kernel to use), or what network settings to use if it wants to try to save the image to a network storage device. On top of that, there are all the issues related to device reinitialisation and so on, and it looks like there's greatly increased pain for users wanting to configure this new implementation. Kexec is by no means proven to be the panacea for all the issues.

Quoted 24 lines · On Fri, 21 Sep 2007 11:19:59 +1000 Nigel Cunningham <[email protected]>…

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 8 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

Quoted 4 lines

Quoted 1 line

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

Hi.

Quoted 4 lines · On Friday 21 September 2007 21:56:29 Rafael J. Wysocki wrote:

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

Could it be fudged by giving userland a null image and having (say) the first ioctl be one that triggers all the real work (with other ioctls being noops or such like, as appropriate)?

Quoted 4 lines · Nigel Cunningham

Hi.

Quoted 1 line · On Friday 21 September 2007 21:56:29 Rafael J. Wysocki wrote:

[Besides, the current hibernation userland interface is used by default

Quoted 2 lines

Quoted 2 lines

openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of

Quoted 3 lines

Quoted 1 line

kexec-based solution.]

Could it be fudged by giving userland a null image and having (say) the

Quoted 1 line

ioctl be one that triggers all the real work (with other ioctls being

Quoted 1 line

or such like, as appropriate)?

Well, the "suspend" part is probably doable, but I'm afraid of the "resume" one.

Quoted 4 lines · Nigel Cunningham

Hi.

Quoted 1 line · On Friday 21 September 2007 21:56:29 Rafael J. Wysocki wrote:

[Besides, the current hibernation userland interface is used by default

Quoted 2 lines

Quoted 2 lines

openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of

Quoted 3 lines

Quoted 1 line

kexec-based solution.]

Could it be fudged by giving userland a null image and having (say) the

Quoted 1 line

ioctl be one that triggers all the real work (with other ioctls being

Quoted 1 line

or such like, as appropriate)?

Well, the "suspend" part is probably doable, but I'm afraid of the "resume" one.

Hi.

Quoted 4 lines · On Friday 21 September 2007 21:56:29 Rafael J. Wysocki wrote:

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

Could it be fudged by giving userland a null image and having (say) the first ioctl be one that triggers all the real work (with other ioctls being noops or such like, as appropriate)?

Hi Andrew,

Quoted 26 lines · Andrew Morton

Quoted 24 lines · On Fri, 21 Sep 2007 11:19:59 +1000 Nigel Cunningham <[email protected]>…

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 8 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

Well, I don't quite agree.

For now, the kexec-based approach is missing the handling of devices, AFAICS. Namely, it's quite easy to snapshot memory with the help of kexec, but the state of devices gets trashed in the process, so you need some additional code saving the state of devices for you, executed before the kexec.

Moreover, on ACPI systems the transition to the S4 sleep state and back to S0 (working state) is more complicated than a system checkpointing, because we are supposed to take the platform firmware into consideration in that case. The more I think about this, the more it seems to me that it just can't be done on top of kexec in a reasonable fashion. Of course, we could avoid handling the ACPI S4, but that would leave some people (including me ;-)) with semi-working hardware after the "restore". I don't think that's generally acceptable in the long run.

IMHO, for ACPI systems the way to go is to harden suspend to RAM (with s2ram in place and the graphics adapters specifications from Intel and AMD released we are in a good position to do that) and build the S4 transition mechanism on top of that. It can be done easlily by adapting the current hibernation code, but not on top of kexec (I'm afraid).

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

Hi Andrew,

Quoted 26 lines · Andrew Morton

Quoted 24 lines · On Fri, 21 Sep 2007 11:19:59 +1000 Nigel Cunningham <[email protected]>…

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 8 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

Well, I don't quite agree.

For now, the kexec-based approach is missing the handling of devices, AFAICS. Namely, it's quite easy to snapshot memory with the help of kexec, but the state of devices gets trashed in the process, so you need some additional code saving the state of devices for you, executed before the kexec.

Moreover, on ACPI systems the transition to the S4 sleep state and back to S0 (working state) is more complicated than a system checkpointing, because we are supposed to take the platform firmware into consideration in that case. The more I think about this, the more it seems to me that it just can't be done on top of kexec in a reasonable fashion. Of course, we could avoid handling the ACPI S4, but that would leave some people (including me ;-)) with semi-working hardware after the "restore". I don't think that's generally acceptable in the long run.

IMHO, for ACPI systems the way to go is to harden suspend to RAM (with s2ram in place and the graphics adapters specifications from Intel and AMD released we are in a good position to do that) and build the S4 transition mechanism on top of that. It can be done easlily by adapting the current hibernation code, but not on top of kexec (I'm afraid).

[Besides, the current hibernation userland interface is used by default by openSUSE and it's also used by quite some Debian users, so we can't drop it overnight and it can't be implemented in a compatible way on top of the kexec-based solution.]

Quoted 24 lines · On Fri, 21 Sep 2007 11:19:59 +1000 Nigel Cunningham <[email protected]>…

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 8 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

Quoted 4 lines

Quoted 1 line

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Right. So we end up with two implementations in-tree. Whereas kexec-based-hibernation leads us to having zero implementations in-tree.

Hi.

Quoted 1 line · On Friday 21 September 2007 11:06:23 Andrew Morton wrote:

On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham

Quoted 10 lines · <[email protected]> wrote:

Quoted 10 lines

Hi Andrew.

Quoted 3 lines · On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going

to

Quoted 4 lines

Quoted 3 lines

consider merging this kexec code, will you also please consider merging TuxOnIce?

Quoted 5 lines

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it.

TuxOnIce doesn't remove the existing implementation either. It can transparently replace it, but you can enable/disable that at compile time.

Quoted 4 lines · On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham <[email protected]>…

Hi Andrew.

On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Quoted 10 lines

Quoted 4 lines

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going to consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it

Quoted 4 lines · On Fri, 21 Sep 2007 10:24:34 +1000 Nigel Cunningham <[email protected]>…

Hi Andrew.

On Thursday 20 September 2007 20:09:41 Pavel Machek wrote:

Quoted 10 lines

Quoted 4 lines

Seems like good enough for -mm to me.

Pavel

Andrew, if I recall correctly, you said a while ago that you didn't want another hibernation implementation in the vanilla kernel. If you're going to consider merging this kexec code, will you also please consider merging TuxOnIce?

The theory is that kexec-based hibernation will mainly use preexisting kexec code and will permit us to delete the existing hibernation implementation.

That's different from replacing it

Quoted 1 line

Seems like good enough for -mm to me.

Quoted 1 line

Seems like good enough for -mm to me.

Hi!

Quoted 1 line

Quoted 1 line

Seems like good enough for -mm to me.

(For the record, I do not think this is going to be hibernation-replacement any time soon. But it is functionality useful for other stuff -- dump memory and continue -- and yes it may be able to do hibernation in the long term.

It really comes from the other side of reliability:

* swsusp is "if your kernel is perfectly healthy, it will work"

while this, coming from kdump is

* "if your kernel is not completely trashed, it should work"

...which is why can't use swsusp to do dump memory and continue -- you want to do dumps on "slightly broken" systems. And yes, as a sideeffect it may be able to do hibernation... why not, lets see how it works out).

Hi!

Quoted 1 line

Quoted 1 line

Seems like good enough for -mm to me.

(For the record, I do not think this is going to be hibernation-replacement any time soon. But it is functionality useful for other stuff -- dump memory and continue -- and yes it may be able to do hibernation in the long term.

It really comes from the other side of reliability:

* swsusp is "if your kernel is perfectly healthy, it will work"

while this, coming from kdump is

* "if your kernel is not completely trashed, it should work"

...which is why can't use swsusp to do dump memory and continue -- you want to do dumps on "slightly broken" systems. And yes, as a sideeffect it may be able to do hibernation... why not, lets see how it works out).

This patch implements the functionality of jumping between the kexeced kernel and the original kernel.

A new reboot command named LINUX_REBOOT_CMD_KJUMP is defined to trigger the jumping to (executing) the new kernel and jumping back to the original kernel.

To support jumping between two kernels, before jumping to (executing) the new kernel and jumping back to the original kernel, the devices are put into quiescent state (to be fully implemented),

and the state of devices and CPU is saved.

After jumping back from kexeced kernel
and jumping to the new kernel, the state of devices and CPU are restored accordingly. The devices/CPU state save/restore code of software suspend is called to implement corresponding function.

To support jumping without preserving memory. One shadow backup page is allocated for each page used by new (kexeced) kernel.

When do
kexec_load, the image of new kernel is loaded into shadow pages,

and
before executing, the original pages and the shadow pages are swapped, so the contents of original pages are backuped.

Before jumping to the
new (kexeced) kernel and after jumping back to the original kernel, the original pages and the shadow pages are swapped too.

A jump back protocol is defined and documented.

Known issues

- A field is added to Linux kernel real-mode header. This is temporary, and should be replaced after the 32-bit boot protocol and setup data patches are accepted.

- The suspend method of device is used to put device in quiescent state. But if the ACPI is enabled this will also put devices into low power state, which prevent the new kernel from booting. So, the ACPI must be disabled both in original kernel and kexeced kernel. This is planed to be resolved after the suspend method and hibernate method is separated for device as proposed earlier in the LKML.

- The NX (none executable) bit should be turned off for the control page if available.

Signed-off-by Huang Ying <[email protected]>

---

 Documentation/i386/jump_back_protocol.txt |   81 ++++++++++++
 arch/i386/Kconfig                         |    7 +
 arch/i386/boot/header.S                   |    2 
 arch/i386/kernel/machine_kexec.c          |   77 +++++++++---
 arch/i386/kernel/relocate_kernel.S | 187 ++++++++++++++++++++++++++----
 arch/i386/kernel/setup.c                  |    3 
 include/asm-i386/bootparam.h              |    3 
 include/asm-i386/kexec.h                  |   48 ++++++-
 include/linux/kexec.h                     |    9 +
 include/linux/reboot.h                    |    2 
 kernel/kexec.c                            |   59 +++++++++
 kernel/ksysfs.c                           |   17 ++
 kernel/power/Kconfig                      |    2 
 kernel/sys.c                              |    8 +
 14 files changed, 463 insertions(+), 42 deletions(-)

Index: linux-2.6.23-rc6/arch/i386/kernel/machine_kexec.c
===================================================================
--- linux-2.6.23-rc6.orig/arch/i386/kernel/machine_kexec.c 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/arch/i386/kernel/machine_kexec.c 2007-09-20 11:24:31.000000000 +0800

arch/i386/kernel/machine_kexec.c +0 -1

 #define PAGE_ALIGNED __attribute__ ((__aligned__(PAGE_SIZE)))
 static u32 kexec_pgd[1024] PAGE_ALIGNED;

@@ -98,23 +99,23 @@
 {
 }
 
-/*

arch/i386/kernel/machine_kexec.c +33 -10

 	unsigned long page_list[PAGES_NR];
-	void *control_page;
-
-	/* Interrupts aren't acceptable while we reboot */
-	local_irq_disable();
-
-	control_page = page_address(image->control_code_page);
-	memcpy(control_page, relocate_kernel, PAGE_SIZE);
+	asmlinkage NORET_TYPE void
+		(*relocate_kernel_ptr)(unsigned long indirection_page,
+				       unsigned long control_page,
+				       unsigned long start_address,
+				       unsigned int has_pae) ATTRIB_NORET;
 
+	relocate_kernel_ptr = control_page +
+		((void *)relocate_kernel - (void *)relocate_page);
 	page_list[PA_CONTROL_PAGE] = __pa(control_page);
-	page_list[VA_CONTROL_PAGE] = (unsigned long)relocate_kernel;
+	page_list[VA_CONTROL_PAGE] = (unsigned long)control_page;
 	page_list[PA_PGD] = __pa(kexec_pgd);
 	page_list[VA_PGD] = (unsigned long)kexec_pgd;
 #ifdef CONFIG_X86_PAE

@@ -127,6 +128,7 @@
 	page_list[VA_PTE_0] = (unsigned long)kexec_pte0;
 	page_list[PA_PTE_1] = __pa(kexec_pte1);
 	page_list[VA_PTE_1] = (unsigned long)kexec_pte1;
+ page_list[PA_SWAP_PAGE] = (page_to_pfn(image->swap_page) << PAGE_SHIFT);
 
 	/* The segment registers are funny things, they have both a
 	 * visible and an invisible part.  Whenever the visible part is

@@ -145,8 +147,26 @@
 	set_idt(phys_to_virt(0),0);
 
 	/* now call it */
-	relocate_kernel((unsigned long)image->head, (unsigned long)page_list,
-			image->start, cpu_has_pae);
+	relocate_kernel_ptr((unsigned long)image->head,
+			    (unsigned long)page_list,
+			    image->start, cpu_has_pae);
+}
+
+/*
+ * Do not allocate memory (or fail in any way) in machine_kexec().
+ * We are past the point of no return, committed to rebooting now.
+ */
+NORET_TYPE void machine_kexec(struct kimage *image)
+{
+	void *control_page;
+
+	/* Interrupts aren't acceptable while we reboot */
+	local_irq_disable();
+
+	control_page = page_address(image->control_code_page);
+	memcpy(control_page, relocate_page, PAGE_SIZE);
+
+	__machine_kexec(image, control_page);
 }
 
 /* crashkernel=size@addr specifies the location to reserve for

@@ -182,3 +202,30 @@
 #endif
 }
 
+#ifdef CONFIG_KEXEC_JUMP
+int machine_kexec_jump(struct kimage *image)
+{
+	void *control_page;

arch/i386/kernel/machine_kexec.c +14 -0

+
+	if (!kexec_jump_save_cpu(control_page))
+		__machine_kexec(image, control_page);
+
+	kexec_jump_back_entry = KJUMP_ENTRY(control_page);
+	image->start = kexec_jump_back_entry;
+	return 0;
+}
+
+void __init parse_kexec_jump_back_entry(void)
+{
+	kexec_jump_back_entry = boot_params.hdr.jump_back_entry;
+}
+#endif /* CONFIG_KEXEC_JUMP */

Index: linux-2.6.23-rc6/include/asm-i386/kexec.h
===================================================================
--- linux-2.6.23-rc6.orig/include/asm-i386/kexec.h 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/include/asm-i386/kexec.h 2007-09-20 11:24:31.000000000 +0800

include/asm-i386/kexec.h +42 -6

@@ -9,16 +9,42 @@
 #define VA_PTE_0         5
 #define PA_PTE_1         6
 #define VA_PTE_1         7
+#define PA_SWAP_PAGE     8
 #ifdef CONFIG_X86_PAE
-#define PA_PMD_0         8
-#define VA_PMD_0         9
-#define PA_PMD_1         10
-#define VA_PMD_1         11
-#define PAGES_NR         12
+#define PA_PMD_0         9
+#define VA_PMD_0         10
+#define PA_PMD_1         11
+#define VA_PMD_1         12
+#define PAGES_NR         13
 #else
-#define PAGES_NR         8
+#define PAGES_NR         9
 #endif
 
+#define KJUMP_DATA_BASE		0x800
+
+#define KJUMP_MAGIC_NUMBER	0x626a
+#define KJUMP_VERSION_NUMBER	0x0100
+
+#define KJUMP_DATA(buf) ((unsigned char *)(buf)+KJUMP_DATA_BASE)
+#define KJUMP_OFF(off)		(KJUMP_DATA_BASE+(off))
+
+#define KJUMP_MAGIC_OFF		KJUMP_OFF(0x0)
+#define KJUMP_MAGIC(buf)	(*(unsigned short *)(KJUMP_DATA(buf)+0x0))
+#define KJUMP_VERSION(buf)	(*(unsigned short *)(KJUMP_DATA(buf)+0x2))
+#define KJUMP_BACKUP_PAGES_MAP_OFF \
+				KJUMP_OFF(0x4)
+#define KJUMP_BACKUP_PAGES_MAP(buf) \
+				(*(unsigned long *)(KJUMP_DATA(buf)+0x4))
+
+/*
+ * The following are not a part of jump back protocol, for internal
+ * use only
+ */
+#define KJUMP_ENTRY_OFF		KJUMP_OFF(0x20)
+#define KJUMP_ENTRY(buf)	(*(unsigned long *)(KJUMP_DATA(buf)+0x20))
+/* Other internal data fields base */
+#define KJUMP_OTHER_OFF		KJUMP_OFF(0x24)
+
 #ifndef __ASSEMBLY__
 
 #include <asm/ptrace.h>

@@ -94,6 +120,16 @@
 		unsigned long start_address,
 		unsigned int has_pae) ATTRIB_NORET;
 
+extern char relocate_page[PAGE_SIZE];
+
+extern asmlinkage int kexec_jump_save_cpu(void *buf);
+
+#ifdef CONFIG_KEXEC_JUMP
+void parse_kexec_jump_back_entry(void);
+#else
+static inline void parse_kexec_jump_back_entry(void) { }
+#endif
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _I386_KEXEC_H */

Index: linux-2.6.23-rc6/include/linux/kexec.h
===================================================================
--- linux-2.6.23-rc6.orig/include/linux/kexec.h 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/include/linux/kexec.h 2007-09-20 11:26:03.000000000 +0800

include/linux/kexec.h +8 -0

 	unsigned long nr_segments;
 	struct kexec_segment segment[KEXEC_SEGMENT_MAX];

@@ -194,4 +195,12 @@
 static inline void crash_kexec(struct pt_regs *regs) { }
 static inline int kexec_should_crash(struct task_struct *p) { return 0; }
 #endif /* CONFIG_KEXEC */
+
+#ifdef CONFIG_KEXEC_JUMP
+extern int machine_kexec_jump(struct kimage *image);
+extern unsigned long kexec_jump_back_entry;
+extern int kexec_jump(void);
+#else /* !CONFIG_KEXEC_JUMP */
+static inline int kexec_jump(void) { return 0; }
+#endif /* CONFIG_KEXEC_JUMP */
 #endif /* LINUX_KEXEC_H */

Index: linux-2.6.23-rc6/kernel/kexec.c
===================================================================

kernel/kexec.c renamed from kernel/kexec.c 2007-09-20 11:24:25.000000000 +0800 +59 -0

@@ -24,6 +24,10 @@
 #include <linux/utsrelease.h>
 #include <linux/utsname.h>
 #include <linux/numa.h>
+#include <linux/suspend.h>
+#include <linux/pm.h>
+#include <linux/cpu.h>
+#include <linux/console.h>
 
 #include <asm/page.h>
 #include <asm/uaccess.h>

@@ -243,6 +247,12 @@
 		goto out;
 	}
 
+	image->swap_page = kimage_alloc_control_pages(image, 0);
+	if (!image->swap_page) {
+		printk(KERN_ERR "Could not allocate swap buffer\n");
+		goto out;
+	}
+
 	result = 0;
  out:
 	if (result == 0)

@@ -1246,3 +1256,52 @@
 }
 
 module_init(crash_save_vmcoreinfo_init)
+
+#ifdef CONFIG_KEXEC_JUMP
+unsigned long kexec_jump_back_entry;
+
+int kexec_jump(void)
+{
+	int error;
+
+	if (!kexec_image)
+		return -EINVAL;
+
+	pm_prepare_console();
+	suspend_console();
+	error = device_suspend(PMSG_FREEZE);
+	if (error)
+		goto Resume_console;
+	error = disable_nonboot_cpus();
+	if (error)
+		goto Resume_devices;
+	local_irq_disable();
+	/* At this point, device_suspend() has been called, but *not*
+	 * device_power_down(). We *must* device_power_down() now.
+	 * Otherwise, drivers for some devices (e.g. interrupt controllers)
+	 * become desynchronized with the actual state of the hardware
+	 * at resume time, and evil weirdness ensues.
+	 */
+	error = device_power_down(PMSG_FREEZE);
+	if (error)
+		goto Enable_irqs;
+
+	save_processor_state();
+	error = machine_kexec_jump(kexec_image);
+	restore_processor_state();
+
+	/* NOTE:  device_power_up() is just a resume() for devices
+	 * that suspended with irqs off ... no overall powerup.
+	 */
+	device_power_up();
+ Enable_irqs:
+	local_irq_enable();
+	enable_nonboot_cpus();
+ Resume_devices:
+	device_resume();
+ Resume_console:
+	resume_console();
+	pm_restore_console();
+	return error;
+}
+#endif /* CONFIG_KEXEC_JUMP */

Index: linux-2.6.23-rc6/kernel/ksysfs.c
===================================================================

kernel/ksysfs.c renamed from kernel/ksysfs.c 2007-09-20 11:24:25.000000000 +0800 +14 -0

@@ -69,6 +69,20 @@
 }
 KERNEL_ATTR_RO(vmcoreinfo);
 
+#ifdef CONFIG_KEXEC_JUMP
+static ssize_t kexec_jump_back_entry_show(struct kset *kset, char *page)
+{
+	return sprintf(page, "0x%lx\n", kexec_jump_back_entry);
+}
+static ssize_t kexec_jump_back_entry_store(struct kset *kset, const char *page,
+					   size_t count)
+{
+	kexec_jump_back_entry = simple_strtoul(page, NULL, 0);
+	return count;
+}
+
+KERNEL_ATTR_RW(kexec_jump_back_entry);
+#endif /* CONFIG_KEXEC_JUMP */
 #endif /* CONFIG_KEXEC */
 
 /*

kernel/ksysfs.c renamed from kernel/ksysfs.c 2007-09-20 11:24:25.000000000 +0800 +0 -0

 #endif
 	NULL
 };

Index: linux-2.6.23-rc6/kernel/sys.c
===================================================================

kernel/sys.c +8 -0

@@ -424,6 +424,14 @@
 		unlock_kernel();
 		return -EINVAL;
 
+	case LINUX_REBOOT_CMD_KEXEC_JUMP:
+		{
+			int ret;
+			ret = kexec_jump();
+			unlock_kernel();
+			return ret;
+		}
+
 #ifdef CONFIG_HIBERNATION
 	case LINUX_REBOOT_CMD_SW_SUSPEND:
 		{

Index: linux-2.6.23-rc6/include/linux/reboot.h
===================================================================
--- linux-2.6.23-rc6.orig/include/linux/reboot.h 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/include/linux/reboot.h 2007-09-20 11:24:31.000000000 +0800

kernel/sys.c +1 -0

@@ -23,6 +23,7 @@
  * RESTART2    Restart system using given command string.
  * SW_SUSPEND  Suspend system using software suspend if compiled in.
  * KEXEC       Restart system using a previously loaded Linux kernel
+ * KEXEC_JUMP  Jump between original kernel and kexeced kernel.
  */
 
 #define	LINUX_REBOOT_CMD_RESTART	0x01234567

kernel/sys.c +0 -0

 #ifdef __KERNEL__

Index: linux-2.6.23-rc6/arch/i386/Kconfig
===================================================================

arch/i386/Kconfig renamed from arch/i386/Kconfig 2007-09-20 11:24:25.000000000 +0800 +7 -0

@@ -830,6 +830,13 @@
 	  (CONFIG_RELOCATABLE=y).
 	  For more details see Documentation/kdump/kdump.txt
 
+config KEXEC_JUMP
+	bool "kexec jump (EXPERIMENTAL)"
+	depends on EXPERIMENTAL
+	depends on PM && X86_32 && KEXEC
+	---help---
+	  Jump between the kexeced kernel and the orignal kernel.
+
 config PHYSICAL_START
 	hex "Physical address where the kernel is loaded" if (EMBEDDED ||
 CRASH_DUMP)

default "0x1000000" if X86_NUMAQ
Index: linux-2.6.23-rc6/kernel/power/Kconfig
===================================================================
--- linux-2.6.23-rc6.orig/kernel/power/Kconfig 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/kernel/power/Kconfig 2007-09-20 11:24:31.000000000 +0800

arch/i386/Kconfig renamed from arch/i386/Kconfig 2007-09-20 11:24:25.000000000 +0800 +1 -1

@@ -70,7 +70,7 @@
 
 config PM_SLEEP
 	bool
-	depends on SUSPEND || HIBERNATION
+	depends on SUSPEND || HIBERNATION || KEXEC_JUMP
 	default y
 
 config SUSPEND_UP_POSSIBLE

Index: linux-2.6.23-rc6/arch/i386/kernel/relocate_kernel.S
===================================================================
--- linux-2.6.23-rc6.orig/arch/i386/kernel/relocate_kernel.S 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/arch/i386/kernel/relocate_kernel.S 2007-09-20 11:24:31.000000000 +0800

arch/i386/Kconfig renamed from arch/i386/Kconfig 2007-09-20 11:24:25.000000000 +0800 +167 -20

@@ -19,8 +19,87 @@
 #define PAGE_ATTR 0x63 /* _PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_DIRTY */
 #define PAE_PGD_ATTR 0x01 /* _PAGE_PRESENT */
 
+#define STACK_TOP		0x1000
+
+#define DATA(offset)		(KJUMP_OTHER_OFF+(offset))
+
+/* Minimal CPU stat */
+#define EBX			DATA(0x0)
+#define ESI			DATA(0x4)
+#define EDI			DATA(0x8)
+#define EBP			DATA(0xc)
+#define ESP			DATA(0x10)
+#define CR0			DATA(0x14)
+#define CR3			DATA(0x18)
+#define CR4			DATA(0x1c)
+#define FLAG			DATA(0x20)
+#define RET			DATA(0x24)
+
+/* some information saved in control page (CP) for jumping back */
+#define CP_VA_CONTROL_PAGE	DATA(0x30)
+#define CP_PA_PGD		DATA(0x34)
+#define CP_PA_SWAP_PAGE		DATA(0x38)
+
 	.text
 	.align PAGE_ALIGNED
+	.globl relocate_page
+relocate_page:
+
+/*
+ * Entry point for jumping back from kexeced kernel, the paging is
+ * turned off, the information needed is at relocate_page +
+ * PAGE_SIZE/2
+ */
+kexec_jump_back_entry:
+	movl	$relocate_page, %ebx
+	movl	%edi, KJUMP_ENTRY_OFF(%ebx)
+	movl	CP_VA_CONTROL_PAGE(%ebx), %edi
+
+	lea	STACK_TOP(%ebx), %esp
+
+	movl	CP_PA_SWAP_PAGE(%ebx), %eax
+	movl	KJUMP_BACKUP_PAGES_MAP_OFF(%ebx), %edx
+	pushl	%eax
+	pushl	%edx
+	call	swap_pages
+	addl	$8, %esp
+
+	movl	CP_PA_PGD(%ebx), %eax
+	movl	%eax, %cr3
+
+	movl	%cr0, %eax
+	orl	$(1<<31), %eax
+	movl	%eax, %cr0
+
+	movl	%edi, %esp
+	addl	$STACK_TOP, %esp
+
+	movl	%edi, %eax
+	addl	$(virtual_mapped - relocate_page), %eax
+	pushl	%eax
+	ret
+
+virtual_mapped:
+	movl	%edi, %edx
+	movl	EBX(%edx), %ebx
+	movl	ESI(%edx), %esi
+	movl	EDI(%edx), %edi
+	movl	EBP(%edx), %ebp
+	movl	FLAG(%edx), %eax
+	pushl	%eax
+	popf
+	movl	ESP(%edx), %esp
+	movl	CR4(%edx), %eax
+	movl	%eax, %cr4
+	movl	CR3(%edx), %eax
+	movl	%eax, %cr3
+	movl	CR0(%edx), %eax
+	movl	%eax, %cr0
+	movl	RET(%edx), %eax
+	movl	%eax, (%esp)
+	mov	$1, %eax
+	ret
+
 	.globl relocate_kernel
 relocate_kernel:
 	movl	8(%esp), %ebp /* list of pages */

@@ -146,6 +225,15 @@
 	pushl $0
 	popfl
 
+	/* save some information for jumping back */
+	movl	PTR(VA_CONTROL_PAGE)(%ebp), %edi
+	movl	%edi, CP_VA_CONTROL_PAGE(%edi)
+	movl	PTR(PA_PGD)(%ebp), %eax
+	movl	%eax, CP_PA_PGD(%edi)
+	movl	PTR(PA_SWAP_PAGE)(%ebp), %eax
+	movl	%eax, CP_PA_SWAP_PAGE(%edi)
+	movl	%ebx, KJUMP_BACKUP_PAGES_MAP_OFF(%edi)
+
 	/* get physical address of control page now */
 	/* this is impossible after page table switch */
 	movl	PTR(PA_CONTROL_PAGE)(%ebp), %edi

@@ -155,11 +243,11 @@
 	movl	%eax, %cr3
 
 	/* setup a new stack at the end of the physical control page */
-	lea	4096(%edi), %esp
+	lea	STACK_TOP(%edi), %esp
 
 	/* jump to identity mapped page */
 	movl    %edi, %eax
-	addl    $(identity_mapped - relocate_kernel), %eax
+	addl    $(identity_mapped - relocate_page), %eax
 	pushl   %eax
 	ret
 

@@ -197,8 +285,44 @@
 	xorl	%eax, %eax
 	movl	%eax, %cr3
 
+	movl	CP_PA_SWAP_PAGE(%edi), %eax
+	pushl	%eax
+	pushl	%ebx
+	call	swap_pages
+	addl	$8, %esp
+
+	/* To be certain of avoiding problems with self-modifying code
+	 * I need to execute a serializing instruction here.
+	 * So I flush the TLB, it's handy, and not processor dependent.
+	 */
+	xorl	%eax, %eax
+	movl	%eax, %cr3
+
+	/* set all of the registers to known values */
+	/* leave %esp alone */
+
+	movw	KJUMP_MAGIC_OFF(%edi), %ax
+	cmpw	$KJUMP_MAGIC_NUMBER, %ax
+	jz 1f
+	xorl	%edi, %edi
+1:
+	xorl	%eax, %eax
+	xorl	%ebx, %ebx
+	xorl    %ecx, %ecx
+	xorl    %edx, %edx
+	xorl    %esi, %esi
+	xorl    %ebp, %ebp
+	ret
+
 	/* Do the copies */
-	movl	%ebx, %ecx
+swap_pages:
+	movl	8(%esp), %edx
+	movl	4(%esp), %ecx
+	pushl	%ebp
+	pushl	%ebx
+	pushl	%edi
+	pushl	%esi
+	movl	%ecx, %ebx
 	jmp	1f
 
 0:	/* top, read another word from the indirection page */

@@ -226,27 +350,50 @@
 	movl    %ecx,   %esi /* For every source page do a copy */
 	andl    $0xfffff000, %esi
 
+	movl	%edi, %eax
+	movl	%esi, %ebp
+
+	movl	%edx, %edi
 	movl    $1024, %ecx
 	rep ; movsl
-	jmp     0b
 
-3:
+	movl	%ebp, %edi
+	movl	%eax, %esi
+	movl	$1024, %ecx
+	rep ; movsl
 
-	/* To be certain of avoiding problems with self-modifying code
-	 * I need to execute a serializing instruction here.
-	 * So I flush the TLB, it's handy, and not processor dependent.
-	 */
-	xorl	%eax, %eax
-	movl	%eax, %cr3
+	movl	%eax, %edi
+	movl	%edx, %esi
+	movl	$1024, %ecx
+	rep ; movsl
 
-	/* set all of the registers to known values */
-	/* leave %esp alone */
+	lea	4096(%ebp), %esi
+	jmp     0b
+3:
+	popl	%esi
+	popl	%edi
+	popl	%ebx
+	popl	%ebp
+	ret
 
-	xorl	%eax, %eax
-	xorl	%ebx, %ebx
-	xorl    %ecx, %ecx
-	xorl    %edx, %edx
-	xorl    %esi, %esi
-	xorl    %edi, %edi
-	xorl    %ebp, %ebp
+	.globl kexec_jump_save_cpu
+kexec_jump_save_cpu:
+	movl	4(%esp), %edx
+	movl	%ebx, EBX(%edx)
+	movl	%esi, ESI(%edx)
+	movl	%edi, EDI(%edx)
+	movl	%ebp, EBP(%edx)
+	movl	%esp, ESP(%edx)
+	movl	%cr0, %eax
+	movl	%eax, CR0(%edx)
+	movl	%cr3, %eax
+	movl	%eax, CR3(%edx)
+	movl	%cr4, %eax
+	movl	%eax, CR4(%edx)
+	pushf
+	popl	%eax
+	movl	%eax, FLAG(%edx)
+	movl	(%esp), %eax
+	movl	%eax, RET(%edx)
+	mov	$0, %eax
 	ret

Index: linux-2.6.23-rc6/include/asm-i386/bootparam.h
===================================================================
--- linux-2.6.23-rc6.orig/include/asm-i386/bootparam.h 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/include/asm-i386/bootparam.h 2007-09-20 11:24:31.000000000 +0800

arch/i386/Kconfig renamed from arch/i386/Kconfig 2007-09-20 11:24:25.000000000 +0800 +3 -0

@@ -41,6 +41,9 @@
 	u32	initrd_addr_max;
 	u32	kernel_alignment;
 	u8	relocatable_kernel;
+	u8	_pad2[3];
+	u32	cmdline_size;
+	u32	jump_back_entry;
 } __attribute__((packed));
 
 struct sys_desc_table {

Index: linux-2.6.23-rc6/arch/i386/boot/header.S
===================================================================
--- linux-2.6.23-rc6.orig/arch/i386/boot/header.S 2007-09-20 11:24:25.000000000 +0800
+++ linux-2.6.23-rc6/arch/i386/boot/header.S 2007-09-20 11:24:31.000000000 +0800