kexec jump -v9

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

3. Boot kernel compiled in step 1 (kernel A).

4. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

5. Boot the kernel B with following shell command line:

/sbin/kexec -e

6. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

3. Boot kernel compiled in step 1 (kernel A).

4. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

5. Boot the kernel B with following shell command line:

/sbin/kexec -e

6. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

While this might be feasible on machines with larger amounts of memory (you might validly be able to argue that a user won't miss 10MB of RAM), it does make hibernation less viable or unviable for systems with less memory (embedded!). It also means that there are 10MB of RAM (or whatever amount) that the user has paid good money for, but which are probably only used for 30s at a time a couple of times a day.

Any attempt to start to use storage available to the hibernating kernel is also going to have these race issues.

2. Lack of ACPI support.

At the moment, noone is going to want to use kexec based hibernation if they have an ACPI system. This needs to be addressed before it can be considered a serious contender.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

While this might be feasible on machines with larger amounts of memory (you might validly be able to argue that a user won't miss 10MB of RAM), it does make hibernation less viable or unviable for systems with less memory (embedded!). It also means that there are 10MB of RAM (or whatever amount) that the user has paid good money for, but which are probably only used for 30s at a time a couple of times a day.

Any attempt to start to use storage available to the hibernating kernel is also going to have these race issues.

2. Lack of ACPI support.

At the moment, noone is going to want to use kexec based hibernation if they have an ACPI system. This needs to be addressed before it can be considered a serious contender.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any

hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

While this might be feasible on machines with larger amounts of memory (you might validly be able to argue that a user won't miss 10MB of RAM), it does make hibernation less viable or unviable for systems with less memory (embedded!). It also means that there are 10MB of RAM (or whatever amount) that the user has paid good money for, but which are probably only used for 30s at a time a couple of times a day.

Any attempt to start to use storage available to the hibernating kernel is also going to have these race issues.

2. Lack of ACPI support.

At the moment, noone is going to want to use kexec based hibernation if they have an ACPI system. This needs to be addressed before it can be considered a serious contender.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

Quoted 17 lines · Nigel Cunningham

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any

Quoted 8 lines

hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

As Eric said kexec need only to allocate memory during loading, not executing.

Quoted 5 lines

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

No, the newest implementation need not to boot a different kernel or different bootloader entry. You just use one bootloader entry, it will resume if there's an image, booting normally if there's not. You can look at the newest hibernation example description.

Quoted 17 lines · Nigel Cunningham

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any

Quoted 8 lines

hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

As Eric said kexec need only to allocate memory during loading, not executing.

Quoted 5 lines

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

No, the newest implementation need not to boot a different kernel or different bootloader entry. You just use one bootloader entry, it will resume if there's an image, booting normally if there's not. You can look at the newest hibernation example description.

Yes. But this memory gets reserved at loading time and then this memory remains unused for the whole duration (except hibernation).

In the example you gave, looks like you are reserving 15MB of memory for second kernel. In practice, we we finding it difficult to boot a regular kernel in 16MB of memory in kdump. We are now reserving 128MB of memory for kdump kernel on x86 arch, otheriwse OOM kill kicks in during init or while core is being copied.

Kexec based hibernation does not look any different than kdump in terms of memory requirements. The only difference seems to be that kdump does the contiguous memory reservation at boot time and kexec based hibernation does the memory reservation at kernel loading time.

The only difference I can think of is, kdump will generally run on servers and hibernation will be required on desktops/laptops and run time memory requirements might be little different. I don't have numbers though.

At the same time carrying a separate kernel binary just for hibernation purposes does not sound very good.

[..]

Quoted 11 lines

Quoted 5 lines

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

No, the newest implementation need not to boot a different kernel or different bootloader entry. You just use one bootloader entry, it will resume if there's an image, booting normally if there's not. You can look at the newest hibernation example description.

Following is the step from new method you have given.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

This mentions that use rootfs.gz as initrd. Without modifying the boot loader entry, how would I switch the initrd dynamically.

Looks like it might be a typo. So basically we can just boot back into normal kernel and then a user can load the resumable core file and kexec to it?

I think all this functionality can be packed into normal initrd itself to make user interface better.

A user can configure the destination for hibernated image at system installation time and initrd will be modified accordingly to save the hibernated image as well to check that user specfied location to find out if a hibernation image is available and needs to be resumed.

Yes. But this memory gets reserved at loading time and then this memory remains unused for the whole duration (except hibernation).

In the example you gave, looks like you are reserving 15MB of memory for second kernel. In practice, we we finding it difficult to boot a regular kernel in 16MB of memory in kdump. We are now reserving 128MB of memory for kdump kernel on x86 arch, otheriwse OOM kill kicks in during init or while core is being copied.

Kexec based hibernation does not look any different than kdump in terms of memory requirements. The only difference seems to be that kdump does the contiguous memory reservation at boot time and kexec based hibernation does the memory reservation at kernel loading time.

The only difference I can think of is, kdump will generally run on servers and hibernation will be required on desktops/laptops and run time memory requirements might be little different. I don't have numbers though.

At the same time carrying a separate kernel binary just for hibernation purposes does not sound very good.

[..]

Quoted 11 lines

Quoted 5 lines

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

No, the newest implementation need not to boot a different kernel or different bootloader entry. You just use one bootloader entry, it will resume if there's an image, booting normally if there's not. You can look at the newest hibernation example description.

Following is the step from new method you have given.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

This mentions that use rootfs.gz as initrd. Without modifying the boot loader entry, how would I switch the initrd dynamically.

Looks like it might be a typo. So basically we can just boot back into normal kernel and then a user can load the resumable core file and kexec to it?

I think all this functionality can be packed into normal initrd itself to make user interface better.

A user can configure the destination for hibernated image at system installation time and initrd will be modified accordingly to save the hibernated image as well to check that user specfied location to find out if a hibernation image is available and needs to be resumed.

Hi all.

I hope kexec turns out to be a good, usable solution. Unfortunately, however, I still have some areas where I'm not convinced that kexec is going to work or work well:

1. Reliability.

It's being sold as a replacement for freezing processes, yet AFAICS it's still going to require the freezer in order to be reliable. In the normal case, there isn't much of an issue with freeing memory or allocating swap, and so these steps can be expected to progress without pain. Imagine, however, the situation where another process or processes are trying to allocate large amounts of memory at the same time, or the system is swapping heavily. Although such situations will not be common, they are entirely conceivable, and any implementation ought to be able to handle such a situation efficiently. If the freezer is removed, any

hibernation implementation - not just kexec - is going to have a much harder job of being reliable in all circumstances. AFAICS, the only way a kexec based solution is going to be able to get around this will be to not have to allocate memory, but that will require permanent allocation of memory for the kexec kernel and it's work area as well as the permanent, exclusive allocation of storage for the kexec hibernation implementation that's currently in place (making the LCA complaint about not being able to hibernate to swap on NTFS on fuse equally relevant).

While this might be feasible on machines with larger amounts of memory (you might validly be able to argue that a user won't miss 10MB of RAM), it does make hibernation less viable or unviable for systems with less memory (embedded!). It also means that there are 10MB of RAM (or whatever amount) that the user has paid good money for, but which are probably only used for 30s at a time a couple of times a day.

Any attempt to start to use storage available to the hibernating kernel is also going to have these race issues.

2. Lack of ACPI support.

At the moment, noone is going to want to use kexec based hibernation if they have an ACPI system. This needs to be addressed before it can be considered a serious contender.

3. Usability.

Right now, kexec based hibernation looks quite complicated to configure, and the user is apparently going to have to remember to boot a different kernel or at least a different bootloader entry in order to resume. Not

a plus. It would be good if you could find a way to use one bootloader entry, resuming if there's an image, booting normally if there's not.

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 11 lines

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 11 lines

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Well, what can I say?

I haven't been a big fan of doing hibernation this way since the very beginning and I still have the same reservations. Namely, my opinion is that the hibernation-related problems we have are not just solvable this way. For one example, in order to stop using the freezer for suspend/hibernation we first need to revamp the suspending/resuming of devices (uder way) and the kexec-based approach doesn't help us here. I wouldn't like to start another discussion about it though.

That said, I can imagine some applications of the $subject functionality not directly related to hibernation. For example, one can use it for kernel debgging (jump to a new kernel, change something in the old kernel's data, jump back and see what happens etc.). Also, in principle it may be used for such things as live migration of VMs.

Well, what can I say?

I haven't been a big fan of doing hibernation this way since the very beginning and I still have the same reservations. Namely, my opinion is that the hibernation-related problems we have are not just solvable this way. For one example, in order to stop using the freezer for suspend/hibernation we first need to revamp the suspending/resuming of devices (uder way) and the kexec-based approach doesn't help us here. I wouldn't like to start another discussion about it though.

That said, I can imagine some applications of the $subject functionality not directly related to hibernation. For example, one can use it for kernel debgging (jump to a new kernel, change something in the old kernel's data, jump back and see what happens etc.). Also, in principle it may be used for such things as live migration of VMs.

On Tuesday, 11 of March 2008, Vivek Goyal wrote:

Quoted 2 lines

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Well, what can I say?

I haven't been a big fan of doing hibernation this way since the very beginning and I still have the same reservations. Namely, my opinion is that the hibernation-related problems we have are not just solvable this way. For one example, in order to stop using the freezer for suspend/hibernation we first need to revamp the suspending/resuming of devices (uder way) and the kexec-based approach doesn't help us here. I wouldn't like to start another discussion about it though.

On Tuesday, 11 of March 2008, Vivek Goyal wrote:

Quoted 2 lines

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Well, what can I say?

I haven't been a big fan of doing hibernation this way since the very beginning and I still have the same reservations. Namely, my opinion is that the hibernation-related problems we have are not just solvable this way. For one example, in order to stop using the freezer for suspend/hibernation we first need to revamp the suspending/resuming of devices (uder way) and the kexec-based approach doesn't help us here. I wouldn't like to start another discussion about it though.

Quoted 4 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Hi!

Quoted 11 lines

Quoted 4 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

...

Quoted 8 lines

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 5 lines

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

Yes, I'd like these patches to go in, being able to switch kernels seems like useful tool.

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

...OTOH this is probably only chance to eliminate freezer from swsusp...

Hi!

Quoted 11 lines

Quoted 4 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

...

Quoted 8 lines

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 5 lines

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

Yes, I'd like these patches to go in, being able to switch kernels seems like useful tool.

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

...OTOH this is probably only chance to eliminate freezer from swsusp...

On Wednesday, 12 of March 2008, Pavel Machek wrote:

Quoted 3 lines

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

What exactly are you referring to?

Quoted 2 lines

Yes, I'd like these patches to go in, being able to switch kernels seems like useful tool.

No objection from me.

Quoted 6 lines

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

...OTOH this is probably only chance to eliminate freezer from swsusp...

Some facts:

* In order to be able to do suspend (STR) without the freezer, we need to make device drivers block access to devices from applications during suspend.
* There's no reason to think that we can't use this same mechanism for hibernation (the only difficulty seems to be the handling of devices used for saving the image).

* We need the drivers to quiesce devices to be able to do the kexec jump in the first place (and to avoid races, we'll need them to block applications' access to devices just like for STR, which is the sufficient condition for removing the freezer).

So, I don't really think that the "freezer removal" argument is valid here.

Moreover, if this had been the _only_ argument for the $subject functionality, I'd have been against it.

I think "kexec based hibernation" is the only currently available possible method to write out image without freezer (after driver works are done). If other process is running, how to prevent them from writing to disk without freezing them in current implementation?

I think "kexec based hibernation" is the only currently available possible method to write out image without freezer (after driver works are done). If other process is running, how to prevent them from writing to disk without freezing them in current implementation?

Quoted 4 lines · Ying

I think "kexec based hibernation" is the only currently available possible method to write out image without freezer (after driver works are done). If other process is running, how to prevent them from writing to disk without freezing them in current implementation?

This is a very good question.

It's a matter of managing the block layer's request queues. Somehow the existing I/O requests must remain blocked while the requests needed for writing the image must be allowed to proceed.

I don't know what would be needed to make this work, but it ought to be possible somehow...

Quoted 4 lines · Ying

I think "kexec based hibernation" is the only currently available possible method to write out image without freezer (after driver works are done). If other process is running, how to prevent them from writing to disk without freezing them in current implementation?

This is a very good question.

It's a matter of managing the block layer's request queues. Somehow the existing I/O requests must remain blocked while the requests needed for writing the image must be allowed to proceed.

I don't know what would be needed to make this work, but it ought to be possible somehow...

Yes, it ought to be possible.

Ultimately, IMHO, we should put all devices unnecessary for saving the image (and doing some eye-candy work) into low power states before the image is created and keep them in low power states until the system is eventually powered off.

If this is done, the remaining problem is the handling of the devices that we need to save the image. I believe that will be achievable without using the freezer.

Yes, it ought to be possible.

Ultimately, IMHO, we should put all devices unnecessary for saving the image (and doing some eye-candy work) into low power states before the image is created and keep them in low power states until the system is eventually powered off.

If this is done, the remaining problem is the handling of the devices that we need to save the image. I believe that will be achievable without using the freezer.

Quoted 6 lines · Rafael J. Wysocki

Yes, it ought to be possible.

Ultimately, IMHO, we should put all devices unnecessary for saving the image (and doing some eye-candy work) into low power states before the image is created and keep them in low power states until the system is eventually powered off.

Why? I guess I don't see why we care what power state the devices are in. Especially since we should be able to quickly save the image.

We need to disconnect the drivers from the hardware yes. So filesystems still work and applications that do direct hardware access still work and don't need to reopen their connections.

I'm leery of low power states as they don't always work, and bringing low power states seems to confuse hibernation to disk with suspend to ram.

From the ACPI compliance point of view it's better to do it this way. We need

Quoted 6 lines · Rafael J. Wysocki

Yes, it ought to be possible.

Ultimately, IMHO, we should put all devices unnecessary for saving the image (and doing some eye-candy work) into low power states before the image is created and keep them in low power states until the system is eventually powered off.

Why? I guess I don't see why we care what power state the devices are in. Especially since we should be able to quickly save the image.

We need to disconnect the drivers from the hardware yes. So filesystems still work and applications that do direct hardware access still work and don't need to reopen their connections.

I'm leery of low power states as they don't always work, and bringing low power states seems to confuse hibernation to disk with suspend to ram.

From the ACPI compliance point of view it's better to do it this way. We need

From the ACPI compliance point of view it's better to do it this way. We need to put the devices into low power states anyway before "powering off" the system and we won't need to touch them for the second time if we do that in advance.

Still, it would be sufficient if we disconnected the drivers from the hardware and thus prevented applications from accessing that hardware.

From the ACPI compliance point of view it's better to do it this way. We need to put the devices into low power states anyway before "powering off" the system and we won't need to touch them for the second time if we do that in advance.

Still, it would be sufficient if we disconnected the drivers from the hardware and thus prevented applications from accessing that hardware.

Quoted 4 lines · Rafael J. Wysocki

From the ACPI compliance point of view it's better to do it this way. We need to put the devices into low power states anyway before "powering off" the system and we won't need to touch them for the second time if we do that in advance.

Interesting. From a kexec jump where we exit the kernel and then return to it seem all that is required.

I will have to look at the ACPI case. That seems to be the lynch pin of a couple of arguments for doing strange things: ACPI requires it.

Quoted 2 lines

Still, it would be sufficient if we disconnected the drivers from the hardware and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

Quoted 4 lines · Rafael J. Wysocki

From the ACPI compliance point of view it's better to do it this way. We need to put the devices into low power states anyway before "powering off" the system and we won't need to touch them for the second time if we do that in advance.

Interesting. From a kexec jump where we exit the kernel and then return to it seem all that is required.

I will have to look at the ACPI case. That seems to be the lynch pin of a couple of arguments for doing strange things: ACPI requires it.

Quoted 2 lines

Still, it would be sufficient if we disconnected the drivers from the hardware and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

Does using the S4 state have advantages that I currently do not see?

Does using the S4 state have advantages that I currently do not see?

Quoted 20 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Quoted 8 lines

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

If we do use the ACPI S4 state I completely agree we should be at least spec compliant in how we use it.

I took a quick skim through my copy of the ACPI spec so I could get a feel for this issue. Hibernation maps to the ACPI S4 state. The only thing we appear to gain from S4 is the ability to tell the BIOS (so it can tell a bootloader) that this was a hibernation power off instead of simply a software power off.

It looks like entering the ACPI S4 state has a few advantages with respect to how the system wakes up. In general using the ACPI S5 state (soft off) appears simpler, and potentially more reliable.

The sequence we appear to want is:
- Disconnecting drivers from devices.
- Saving the image.
- Placing the system in a low power or off state.

- Coming out of the low power state.
- Restoring the image.
- Reconnecting drivers to devices.
(We must assume the device state could have changed here
no matter what we do)

It is mostly a matter of where we place the code.

Right now I don't see a limitation either with a kexec based approach or without one. Especially since the common case would be using the same kernel with the same drivers both before and after the hibernation event.

The low power states for S4 seem to be just so that we can decide which devices have enough life that they can wake up the system. If we handle all of that as a second pass after we have the system in a state where we have saved it we should be in good shape.

My inclination is to just use S5 (soft off).

One of the cool things about hibernation to disk was that we were supposed to get the BIOS totally out of that path so we could get something that was rock solid and reliable. I don't see why we should use ACPI S4 when the BIOS doesn't seem to give us anything useful, and causes us headaches we should even consider using S4.

Does using the S4 state have advantages that I currently do not see?

Len? Rafael? Anyone?

Quoted 20 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Quoted 8 lines

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

If we do use the ACPI S4 state I completely agree we should be at least spec compliant in how we use it.

I took a quick skim through my copy of the ACPI spec so I could get a feel for this issue. Hibernation maps to the ACPI S4 state. The only thing we appear to gain from S4 is the ability to tell the BIOS (so it can tell a bootloader) that this was a hibernation power off instead of simply a software power off.

It looks like entering the ACPI S4 state has a few advantages with respect to how the system wakes up. In general using the ACPI S5 state (soft off) appears simpler, and potentially more reliable.

The sequence we appear to want is:
- Disconnecting drivers from devices.
- Saving the image.
- Placing the system in a low power or off state.

- Coming out of the low power state.
- Restoring the image.
- Reconnecting drivers to devices.
(We must assume the device state could have changed here
no matter what we do)

It is mostly a matter of where we place the code.

Right now I don't see a limitation either with a kexec based approach or without one. Especially since the common case would be using the same kernel with the same drivers both before and after the hibernation event.

The low power states for S4 seem to be just so that we can decide which devices have enough life that they can wake up the system. If we handle all of that as a second pass after we have the system in a state where we have saved it we should be in good shape.

My inclination is to just use S5 (soft off).

One of the cool things about hibernation to disk was that we were supposed to get the BIOS totally out of that path so we could get something that was rock solid and reliable. I don't see why we should use ACPI S4 when the BIOS doesn't seem to give us anything useful, and causes us headaches we should even consider using S4.

Does using the S4 state have advantages that I currently do not see?

Len? Rafael? Anyone?

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

Quoted 6 lines · Rafael J. Wysocki

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

Quoted 4 lines · Alan Stern

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Quoted 4 lines · Alan Stern

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Quoted 10 lines · Eric W. Biederman

Quoted 4 lines · Alan Stern

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

? Doesn't this happen every time Rafael turns the computer off and then turns it back on?

Do you mean that Rafael should do an S5-type hibernate, but then reboot in such a way that the image isn't loaded and resumed?

Quoted 10 lines · Eric W. Biederman

Quoted 4 lines · Alan Stern

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

? Doesn't this happen every time Rafael turns the computer off and then turns it back on?

Do you mean that Rafael should do an S5-type hibernate, but then reboot in such a way that the image isn't loaded and resumed?

Quoted 12 lines · Alan Stern

Quoted 6 lines · Rafael J. Wysocki

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Quoted 12 lines · Alan Stern

Quoted 6 lines · Rafael J. Wysocki

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Quoted 14 lines · Eric W. Biederman

Quoted 12 lines · Alan Stern

Quoted 6 lines · Rafael J. Wysocki

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 14 lines · Eric W. Biederman

Quoted 12 lines · Alan Stern

Quoted 6 lines · Rafael J. Wysocki

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 21 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

Quoted 3 lines

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

Quoted 21 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

Quoted 3 lines

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

Quoted 21 lines · Rafael J. Wysocki

Quoted 21 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

Quoted 7 lines

Quoted 3 lines

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot. All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

Quoted 27 lines · Alan Stern

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

How can we pass interpretter state? I do not think we do this kind of passing.

If it was enough to pass some static area, we could just mark it nosave...

Len: Is ACPI AML permitted to allocate memory (like in ACPI_ALLOC or something)? Could we easily identify BIOS data so we could mark them nosave?

Quoted 27 lines · Alan Stern

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

How can we pass interpretter state? I do not think we do this kind of passing.

If it was enough to pass some static area, we could just mark it nosave...

Len: Is ACPI AML permitted to allocate memory (like in ACPI_ALLOC or something)? Could we easily identify BIOS data so we could mark them nosave?

Quoted 28 lines · Pavel Machek

Quoted 27 lines · Alan Stern

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

Quoted 6 lines

If it was enough to pass some static area, we could just mark it nosave...

Len: Is ACPI AML permitted to allocate memory (like in ACPI_ALLOC or something)? Could we easily identify BIOS data so we could mark them nosave?

This wouldn't work even if we could (at least on x86-64).

Quoted 28 lines · Pavel Machek

Quoted 27 lines · Alan Stern

Quoted 25 lines · Rafael J. Wysocki

Quoted 24 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

Quoted 6 lines

If it was enough to pass some static area, we could just mark it nosave...

Len: Is ACPI AML permitted to allocate memory (like in ACPI_ALLOC or something)? Could we easily identify BIOS data so we could mark them nosave?

This wouldn't work even if we could (at least on x86-64).

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

Quoted 10 lines

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 7 lines

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

It's supposed to do that when resuming from an S4 hibernation, not when resuming from an S5 hibernation.

Quoted 4 lines

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

Quoted 10 lines

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 7 lines

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

It's supposed to do that when resuming from an S4 hibernation, not when resuming from an S5 hibernation.

Quoted 4 lines

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

The interpreter state is passed withing the image. The platform state is not.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

Quoted 25 lines · Rafael J. Wysocki

Quoted 10 lines

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 7 lines

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

It's supposed to do that when resuming from an S4 hibernation, not when resuming from an S5 hibernation.

Quoted 3 lines

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

Quoted 5 lines

Quoted 1 line

The interpreter state is passed withing the image. The platform state is not.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

yes, nosave pages could be used to do this passing -- if we can put interpretter state into pre-allocated memory block.

Quoted 25 lines · Rafael J. Wysocki

Quoted 10 lines

Quoted 9 lines

Quoted 9 lines

Quoted 4 lines

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot.

Quoted 7 lines

I agree here.

Quoted 2 lines

Quoted 2 lines

All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

However, the image kernel is supposed to restore the NVS area (from the image) before executing _WAK.

It's supposed to do that when resuming from an S4 hibernation, not when resuming from an S5 hibernation.

Quoted 3 lines

Quoted 2 lines

How can we pass interpretter state? I do not think we do this kind of passing.

Quoted 5 lines

Quoted 1 line

The interpreter state is passed withing the image. The platform state is not.

For an S5 hibernation, the interpreter state within the image is wrong. The image kernel needs to have the interpreter state from the boot kernel -- I don't know if this is possible.

yes, nosave pages could be used to do this passing -- if we can put interpretter state into pre-allocated memory block.

However, as far as the ACPI NVS area is concerned, this is probably not necessary, because the spec wants us to restore the ACPI NVS before calling _WAK, which is just after the image kernel gets the control back. So, in theory, the ACPI NVS data could be stored in the image and restored by the image kernel from a location known to it (the procedure may be to copy the ACPI NVS data into a region of regular RAM before creating the image and copy them back into the ACPI NVS area in platform->leave(), for example), but I suspect that for this to work we'll have to switch ACPI off in the boot kernel, just prior to passing control back to the image kernel.

However, as far as the ACPI NVS area is concerned, this is probably not necessary, because the spec wants us to restore the ACPI NVS before calling _WAK, which is just after the image kernel gets the control back. So, in theory, the ACPI NVS data could be stored in the image and restored by the image kernel from a location known to it (the procedure may be to copy the ACPI NVS data into a region of regular RAM before creating the image and copy them back into the ACPI NVS area in platform->leave(), for example), but I suspect that for this to work we'll have to switch ACPI off in the boot kernel, just prior to passing control back to the image kernel.

Quoted 9 lines · Rafael J. Wysocki

However, as far as the ACPI NVS area is concerned, this is probably not necessary, because the spec wants us to restore the ACPI NVS before calling _WAK, which is just after the image kernel gets the control back. So, in theory, the ACPI NVS data could be stored in the image and restored by the image kernel from a location known to it (the procedure may be to copy the ACPI NVS data into a region of regular RAM before creating the image and copy them back into the ACPI NVS area in platform->leave(), for example), but I suspect that for this to work we'll have to switch ACPI off in the boot kernel, just prior to passing control back to the image kernel.

That sounds by far the simplest solution. If the boot kernel can tell (by looking at some header field in the image or any other way) that the hibernation used S5 instead of S4, then it should just turn off ACPI before passing control to the image kernel. Then the image kernel can turn ACPI back on and all should be well. If you do this, does the NVS region still need to be preserved?

Quoted 9 lines · Rafael J. Wysocki

However, as far as the ACPI NVS area is concerned, this is probably not necessary, because the spec wants us to restore the ACPI NVS before calling _WAK, which is just after the image kernel gets the control back. So, in theory, the ACPI NVS data could be stored in the image and restored by the image kernel from a location known to it (the procedure may be to copy the ACPI NVS data into a region of regular RAM before creating the image and copy them back into the ACPI NVS area in platform->leave(), for example), but I suspect that for this to work we'll have to switch ACPI off in the boot kernel, just prior to passing control back to the image kernel.

That sounds by far the simplest solution. If the boot kernel can tell (by looking at some header field in the image or any other way) that the hibernation used S5 instead of S4, then it should just turn off ACPI before passing control to the image kernel. Then the image kernel can turn ACPI back on and all should be well. If you do this, does the NVS region still need to be preserved?

On Saturday, 22 of March 2008, Alan Stern wrote:

The spec doesn't say much about that, so we'll need to carry out some experiments.

Still, as far as I can figure out what the spec authors _might_ mean, I think that it would be inappropriate to restore the ACPI NVS area if S5 was entered on "power off". The idea seems to be that the restoration of the ACPI NVS area should complement whatever has been preserved by the platform over the hibernation/resume cycle.

IMO, if S5 was entered on "powe off", there are two possible ways to go. Either ACPI is initialized by the boot kernel, in which case the image kernel should not touch things like _WAK and similar, just throw away whatever ACPI-related state it got from the image and try to rebuild the ACPI-related data from scratch. Or the boot kernel doesn't touch ACPI and the image kernel initializes it in the same way as during a fresh boot (that might be difficult, though).

On Saturday, 22 of March 2008, Alan Stern wrote:

The spec doesn't say much about that, so we'll need to carry out some experiments.

Still, as far as I can figure out what the spec authors _might_ mean, I think that it would be inappropriate to restore the ACPI NVS area if S5 was entered on "power off". The idea seems to be that the restoration of the ACPI NVS area should complement whatever has been preserved by the platform over the hibernation/resume cycle.

IMO, if S5 was entered on "powe off", there are two possible ways to go. Either ACPI is initialized by the boot kernel, in which case the image kernel should not touch things like _WAK and similar, just throw away whatever ACPI-related state it got from the image and try to rebuild the ACPI-related data from scratch. Or the boot kernel doesn't touch ACPI and the image kernel initializes it in the same way as during a fresh boot (that might be difficult, though).

Quoted 11 lines · Rafael J. Wysocki

On Saturday, 22 of March 2008, Alan Stern wrote:

The spec doesn't say much about that, so we'll need to carry out some experiments.

Still, as far as I can figure out what the spec authors _might_ mean, I think that it would be inappropriate to restore the ACPI NVS area if S5 was entered on "power off". The idea seems to be that the restoration of the ACPI NVS area should complement whatever has been preserved by the platform over the hibernation/resume cycle.

Quoted 7 lines

IMO, if S5 was entered on "powe off", there are two possible ways to go. Either ACPI is initialized by the boot kernel, in which case the image kernel should not touch things like _WAK and similar, just throw away whatever ACPI-related state it got from the image and try to rebuild the ACPI-related data from scratch. Or the boot kernel doesn't touch ACPI and the image kernel initializes it in the same way as during a fresh boot (that might be difficult, though).

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

So I expect shutting down ACPI and restarting it should work reliably and that is easy to test as that is already implemented with kexec.

Quoted 4 lines · Eric W. Biederman

"Rafael J. Wysocki" <[email protected]> writes:

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

You don't remove power from devices while doing that.

Quoted 4 lines · Eric W. Biederman

"Rafael J. Wysocki" <[email protected]> writes:

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

You don't remove power from devices while doing that.

Quoted 3 lines · Rafael J. Wysocki

Quoted 2 lines · Eric W. Biederman

"Rafael J. Wysocki" <[email protected]> writes:

Quoted 4 lines

Quoted 2 lines

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

You don't remove power from devices while doing that.

No. It is the second half of S5. When we go from the boot kernel to the restored kernel I am talking about.

Quoted 3 lines · Rafael J. Wysocki

Quoted 2 lines · Eric W. Biederman

"Rafael J. Wysocki" <[email protected]> writes:

Quoted 4 lines

Quoted 2 lines

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

You don't remove power from devices while doing that.

No. It is the second half of S5. When we go from the boot kernel to the restored kernel I am talking about.

Quoted 11 lines · Rafael J. Wysocki

On Saturday, 22 of March 2008, Alan Stern wrote:

The spec doesn't say much about that, so we'll need to carry out some experiments.

Still, as far as I can figure out what the spec authors _might_ mean, I think that it would be inappropriate to restore the ACPI NVS area if S5 was entered on "power off". The idea seems to be that the restoration of the ACPI NVS area should complement whatever has been preserved by the platform over the hibernation/resume cycle.

Quoted 7 lines

IMO, if S5 was entered on "powe off", there are two possible ways to go. Either ACPI is initialized by the boot kernel, in which case the image kernel should not touch things like _WAK and similar, just throw away whatever ACPI-related state it got from the image and try to rebuild the ACPI-related data from scratch. Or the boot kernel doesn't touch ACPI and the image kernel initializes it in the same way as during a fresh boot (that might be difficult, though).

Just an added data partial point. In the kexec case I have had not heard anyone screaming to me that ACPI doesn't work after we switch kernels.

So I expect shutting down ACPI and restarting it should work reliably and that is easy to test as that is already implemented with kexec.

Quoted 21 lines · Rafael J. Wysocki

Quoted 21 lines

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

Well, I've been saying that for I-don't-remember-how-long: on my box, if you use S5 instead of entering S4, the fan doesn't work correctly after the resume. Plain and simple.

Perhaps there's a problem with our ACPI drivers that causes this to happen, but I have no idea what that can be at the moment.

IMO it would be worthwhile to track this down. It's a clear indication that something is wrong somewhere.

Quoted 8 lines

Quoted 4 lines

Could it be connected with the way the boot kernel hands control over to the image kernel? Presumably ACPI isn't prepared to deal with that sort of thing during a boot from S5. It would have to be fooled into thinking the two kernels were one and the same.

It should be easy to test if it is a hand over problem, by turning off the laptop by placing it in S5 (shutdown -h now) and then booting same kernel again.

Feel free to help with testing.

Quoted 7 lines

Quoted 3 lines

I believe ACPI is simply getting confused by us overwriting memory with that from old image. I don't see how you can emulate it with shutdown.

Well, in fact ACPI has something called the NVS memory, which we're supposed to restore during the resume and which we're not doing. The problem may be related to this.

No, it can't be. ACPI won't expect the NVS memory to be restored following an S5-shutdown. In fact, as far as ACPI is concerned, resuming from an S5-type hibernation should not be considered a resume at all but just an ordinary reboot. All ACPI-related memory areas in the boot kernel should be passed directly through to the image kernel.

Quoted 20 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Quoted 8 lines

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

If we do use the ACPI S4 state I completely agree we should be at least spec compliant in how we use it.

I took a quick skim through my copy of the ACPI spec so I could get a feel for this issue. Hibernation maps to the ACPI S4 state. The only thing we appear to gain from S4 is the ability to tell the BIOS (so it can tell a bootloader) that this was a hibernation power off instead of simply a software power off.

It looks like entering the ACPI S4 state has a few advantages with respect to how the system wakes up. In general using the ACPI S5 state (soft off) appears simpler, and potentially more reliable.

The sequence we appear to want is:
- Disconnecting drivers from devices.
- Saving the image.
- Placing the system in a low power or off state.

- Coming out of the low power state.
- Restoring the image.
- Reconnecting drivers to devices.
(We must assume the device state could have changed here
no matter what we do)

It is mostly a matter of where we place the code.

Right now I don't see a limitation either with a kexec based approach or without one. Especially since the common case would be using the same kernel with the same drivers both before and after the hibernation event.

The low power states for S4 seem to be just so that we can decide which devices have enough life that they can wake up the system. If we handle all of that as a second pass after we have the system in a state where we have saved it we should be in good shape.

My inclination is to just use S5 (soft off).

One of the cool things about hibernation to disk was that we were supposed to get the BIOS totally out of that path so we could get something that was rock solid and reliable. I don't see why we should use ACPI S4 when the BIOS doesn't seem to give us anything useful, and causes us headaches we should even consider using S4.

Does using the S4 state have advantages that I currently do not see?

Len? Rafael? Anyone?

Eric
_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

Quoted 20 lines · Rafael J. Wysocki

Quoted 1 line · Eric W. Biederman

Quoted 1 line

Still, it would be sufficient if we disconnected the drivers from the

hardware

Quoted 9 lines

Quoted 1 line

and thus prevented applications from accessing that hardware.

My gut feeling is that except for a handful of drivers we could even get away with simply implementing hot unplug and hot replug. Disks are the big exception here.

Which suggests to me that it is at least possible that the methods we want for a kexec jump hibernation may be different from an in-kernel hibernation and quite possibly are easier to implement.

I'm not sure about the "easier" part, quite frankly. Also, with our current ordering of code the in-kernel hibernation will need the same callbacks as the kexec-based thing. However, with the in-kernel approach we can attempt (in the future) to be more ACPI compliant, so to speak, but with the kexec-based approach that won't be possible.

Quoted 8 lines

Whether it's a good idea to follow ACPI, as far as hibernation is concerned, is a separate question, but IMO we won't be able to answer it without _lots_ of testing on vaious BIOS/firmware configurations. Our experience so far indicates that at least some BIOSes expect us to follow ACPI and misbehave otherwise, so for those systems there should be an "ACPI way" available. [Others just don't work well if we try to follow ACPI and those may be handled using the kexec-based approach, but that doesn't mean that we can just ignore the ACPI compliance issue, at least for now.]

If we do use the ACPI S4 state I completely agree we should be at least spec compliant in how we use it.

I took a quick skim through my copy of the ACPI spec so I could get a feel for this issue. Hibernation maps to the ACPI S4 state. The only thing we appear to gain from S4 is the ability to tell the BIOS (so it can tell a bootloader) that this was a hibernation power off instead of simply a software power off.

It looks like entering the ACPI S4 state has a few advantages with respect to how the system wakes up. In general using the ACPI S5 state (soft off) appears simpler, and potentially more reliable.

The sequence we appear to want is:
- Disconnecting drivers from devices.
- Saving the image.
- Placing the system in a low power or off state.

- Coming out of the low power state.
- Restoring the image.
- Reconnecting drivers to devices.
(We must assume the device state could have changed here
no matter what we do)

It is mostly a matter of where we place the code.

Right now I don't see a limitation either with a kexec based approach or without one. Especially since the common case would be using the same kernel with the same drivers both before and after the hibernation event.

The low power states for S4 seem to be just so that we can decide which devices have enough life that they can wake up the system. If we handle all of that as a second pass after we have the system in a state where we have saved it we should be in good shape.

My inclination is to just use S5 (soft off).

One of the cool things about hibernation to disk was that we were supposed to get the BIOS totally out of that path so we could get something that was rock solid and reliable. I don't see why we should use ACPI S4 when the BIOS doesn't seem to give us anything useful, and causes us headaches we should even consider using S4.

Does using the S4 state have advantages that I currently do not see?

Len? Rafael? Anyone?

Eric
_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

First of all S4 ACPI code turns some leds on some systems, cosmetic thing, but still nice.

Secondary, what about wakeup devices?
Hardware can disable some devices in S5 while leave them running in S4 on my system for example network card will do WOL in S4, but to make it WOL in S5 I have to turn a specific option in BIOS.

While my system doesn't have this, it isn't uncommon for system to leave USB ports
running so one can turn the PC with keyboard/mouse even in S4. in S5 those ports will probably be disabled.
My system on have this for S3 only.

On laptops we can expect even more ACPI functionality, so some more differences between
S4 and S5 can happen.

Last thing that I want to say is that, when linux puts PC in S? state, on top of executing
_PTS, _GTS acpi functions, it writes the destination S state to a fixed register, thus the hardware
can (and does) behave differently.

First of all S4 ACPI code turns some leds on some systems, cosmetic thing, but still nice.

Secondary, what about wakeup devices?
Hardware can disable some devices in S5 while leave them running in S4 on my system for example network card will do WOL in S4, but to make it WOL in S5 I have to turn a specific option in BIOS.

While my system doesn't have this, it isn't uncommon for system to leave USB ports
running so one can turn the PC with keyboard/mouse even in S4. in S5 those ports will probably be disabled.
My system on have this for S3 only.

On laptops we can expect even more ACPI functionality, so some more differences between
S4 and S5 can happen.

Last thing that I want to say is that, when linux puts PC in S? state, on top of executing
_PTS, _GTS acpi functions, it writes the destination S state to a fixed register, thus the hardware
can (and does) behave differently.

On Wednesday, 12 of March 2008, Pavel Machek wrote:

Quoted 3 lines

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

What exactly are you referring to?

Quoted 2 lines

Yes, I'd like these patches to go in, being able to switch kernels seems like useful tool.

No objection from me.

Quoted 6 lines

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

...OTOH this is probably only chance to eliminate freezer from swsusp...

Some facts:

* In order to be able to do suspend (STR) without the freezer, we need to make device drivers block access to devices from applications during suspend.
* There's no reason to think that we can't use this same mechanism for hibernation (the only difficulty seems to be the handling of devices used for saving the image).

* We need the drivers to quiesce devices to be able to do the kexec jump in the first place (and to avoid races, we'll need them to block applications' access to devices just like for STR, which is the sufficient condition for removing the freezer).

So, I don't really think that the "freezer removal" argument is valid here.

Moreover, if this had been the _only_ argument for the $subject functionality, I'd have been against it.

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Quoted 5 lines

Quoted 2 lines

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

What exactly are you referring to?

Quoted 4 lines

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

Quoted 2 lines

...OTOH this is probably only chance to eliminate freezer from swsusp...

Some facts:

* There's no reason to think that we can't use this same mechanism for hibernation (the only difficulty seems to be the handling of devices used for saving the image).

Moreover, if this had been the _only_ argument for the $subject functionality, I'd have been against it.

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Quoted 5 lines

Quoted 2 lines

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Its certainly "more traditional" method of doing hibernation than tricks swsusp currently plays.

What exactly are you referring to?

Quoted 4 lines

Now, I guess they are some difficulties, like ACPI integration, and some basic drawbacks, like few seconds needed to boot second kernel during suspend.

Quoted 2 lines

...OTOH this is probably only chance to eliminate freezer from swsusp...

Some facts:

* There's no reason to think that we can't use this same mechanism for hibernation (the only difficulty seems to be the handling of devices used for saving the image).

Moreover, if this had been the _only_ argument for the $subject functionality, I'd have been against it.

...OTOH this is probably only chance to eliminate freezer from swsusp...

...OTOH this is probably only chance to eliminate freezer from swsusp...

Quoted 4 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 11 lines

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

[..]

Quoted 29 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

3. Boot kernel compiled in step 1 (kernel A).

4. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

5. Boot the kernel B with following shell command line:

/sbin/kexec -e

Quoted 8 lines

6. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Why not store the entry point in dump.elf itself, instead of storing it in a separate file?

I think this is more like a resumable core file. Something similar to functionality what qemu does for resuming an already booted kernel image. So we might have to introduce an ELF_NOTE to mark an image as resumable core.

Quoted 5 lines

Then you can shutdown the machine as normal.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

Quoted 4 lines

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

How the memory segments of dump.elf loaded? Normal kexec way? Memory segments of dump.elf are first stored somewhere and then moved to destination at "kexec -e" time?

Does this really work? If we have 4G RAM, what will be the size of dump.elf? And when we load it back for resuming, do we have sufficient memory left?

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

Resumes an hibernated image. Load a ELF64 hibernated image.

Context of first kernel/boot-loader will not be preserved.

First kernel will not save cpu states. Will put devices into suspended state though so that these can be resumed by resumable core

This option can be used by kboot or kernel C to resume an hibernated image.

- kexec --load-resume-entry <entry-point>

Image is already loaded. Just prepare the entry point so that one can enter back to previous image. cpu states will be saved and devices will be put to suspended states.

will be used for A --> B and B ---> A transitions. Both A and B are booted. This is just for switching back and forth between A and B.

- kexec -e
Transition into the new kernel

This patch looks in pretty decent shape. Once there is some sort of understanding that this approach is promising for hibernation and we have consensus on high level interface, then we can get into line by line review of the patch set.

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

Quoted 13 lines

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Quoted 3 lines

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

In original kexec-tools, this can be done through:
kexec -l --args-none <resumable-core>

Do you need to define an alias for it?

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

Quoted 13 lines

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Quoted 3 lines

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

In original kexec-tools, this can be done through:
kexec -l --args-none <resumable-core>

Do you need to define an alias for it?

Quoted 9 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

Quoted 2 lines

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

Purgatory is for all intents and purposes user space. Because the return address falls on the trampoline page we won't know it's address before we call kexec. But a return address and a stack on that page should be a perfectly good way to communicate.

Quoted 7 lines

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

We need a fixed protocol so we do not make assumptions about how things will be implemented, allowing kernels to diverge and kinds of other good things.

For communicating extra information from the kernel being shut down we have elf notes.

Direct kernel to kernel communication is forbidden. We must have a well defined protocol. Allowing the implementations to change at their different speeds, and still work together.

Quoted 4 lines

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

Quoted 4 lines

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

My gut feel is we look at the image and detect what kind it is, and simply not enable image processing after we have read the note that says it is a resumable core or whatever.

Quoted 9 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

Quoted 2 lines

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

Purgatory is for all intents and purposes user space. Because the return address falls on the trampoline page we won't know it's address before we call kexec. But a return address and a stack on that page should be a perfectly good way to communicate.

Quoted 7 lines

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

We need a fixed protocol so we do not make assumptions about how things will be implemented, allowing kernels to diverge and kinds of other good things.

For communicating extra information from the kernel being shut down we have elf notes.

Direct kernel to kernel communication is forbidden. We must have a well defined protocol. Allowing the implementations to change at their different speeds, and still work together.

Quoted 4 lines

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

Quoted 4 lines

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

My gut feel is we look at the image and detect what kind it is, and simply not enable image processing after we have read the note that says it is a resumable core or whatever.

Quoted 7 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

Quoted 2 lines

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

Quoted 2 lines

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

Purgatory is for all intents and purposes user space. Because the return address falls on the trampoline page we won't know it's address before we call kexec. But a return address and a stack on that page should be a perfectly good way to communicate.

Quoted 7 lines

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

We need a fixed protocol so we do not make assumptions about how things will be implemented, allowing kernels to diverge and kinds of other good things.

For communicating extra information from the kernel being shut down we have elf notes.

Direct kernel to kernel communication is forbidden. We must have a well defined protocol. Allowing the implementations to change at their different speeds, and still work together.

Quoted 4 lines

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

Quoted 4 lines

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

My gut feel is we look at the image and detect what kind it is, and simply not enable image processing after we have read the note that says it is a resumable core or whatever.

Quoted 7 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

Quoted 2 lines

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

Quoted 2 lines

- Eric's opinion is to communicate between the user space in kernel A and user space in kernel B.

Purgatory is for all intents and purposes user space. Because the return address falls on the trampoline page we won't know it's address before we call kexec. But a return address and a stack on that page should be a perfectly good way to communicate.

Quoted 7 lines

- My opinion is to communicate between two kernel directly.

I think as a minimal infrastructure patch, we can communicate minimal information between user space of two kernels. When we have consensus on this topic, we can use makedumpfile for both excluding free pages and saving the entry point. Now, we can save the entry point in a separate file or I can write a simple tool to do this.

We need a fixed protocol so we do not make assumptions about how things will be implemented, allowing kernels to diverge and kinds of other good things.

For communicating extra information from the kernel being shut down we have elf notes.

Direct kernel to kernel communication is forbidden. We must have a well defined protocol. Allowing the implementations to change at their different speeds, and still work together.

Quoted 4 lines

Quoted 3 lines

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

Quoted 4 lines

There is already similar flag in original kexec-tools implementation:
"--args-none". If it is specified, kexec-tools does not prepare command line and zero page/setup page etc. I think we can just re-use this flag. And If it is desired an alias is good for me too.

My gut feel is we look at the image and detect what kind it is, and simply not enable image processing after we have read the note that says it is a resumable core or whatever.

Quoted 9 lines · Eric W. Biederman

Quoted 7 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

Quoted 5 lines

Quoted 2 lines

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

I think he needs to pass on much more data than just return address.

IIUC, he needs to pass backup pages map to new kernel, so that any user space tool can use backup pages map to reconstruct/rearrange the first kernel's memory core and tools like makedumpfile can do filtering before hibernated images is saved.

This brings me to a random thought. Can we break the process of loading a hibernation kernel in two steps.

- In first step just do the memory reservation for running second kernel. (kexec -l <dummpy-file-for-reserving-memory>)

- This memory map of reserved pages is exported to user space.

- Use this memory map and regenerate the hibernation kernel initrd (rootfs.gz) and put the memory map there. This memory map can be used by makedumpfile in second kernel for filtering.

This way it will user space to user space communication of information which gets fixed at kernel loading time.

Quoted 9 lines · Eric W. Biederman

Quoted 7 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

Quoted 5 lines

Quoted 2 lines

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

I think he needs to pass on much more data than just return address.

IIUC, he needs to pass backup pages map to new kernel, so that any user space tool can use backup pages map to reconstruct/rearrange the first kernel's memory core and tools like makedumpfile can do filtering before hibernated images is saved.

This brings me to a random thought. Can we break the process of loading a hibernation kernel in two steps.

- In first step just do the memory reservation for running second kernel. (kexec -l <dummpy-file-for-reserving-memory>)

- This memory map of reserved pages is exported to user space.

- Use this memory map and regenerate the hibernation kernel initrd (rootfs.gz) and put the memory map there. This memory map can be used by makedumpfile in second kernel for filtering.

This way it will user space to user space communication of information which gets fixed at kernel loading time.

Quoted 29 lines · Vivek Goyal

Quoted 9 lines · Eric W. Biederman

Quoted 7 lines · Ying

Yes. The entry point should be saved in dump.elf itself, this can be done via a user-space tool such as "makedumpfile". Because "makedumpfile" is also used to exclude free pages from disk image, it needs a communication method between two kernels (to get backup pages map or something like that from kernel A). We have talked about this before.

Quoted 5 lines

Quoted 2 lines

- Your opinion is to communicate via the purgatory. (But I don't know how to communicate between kernel A and purgatory).

How about the return address on the stack?

I think he needs to pass on much more data than just return address.

IIUC, he needs to pass backup pages map to new kernel, so that any user space tool can use backup pages map to reconstruct/rearrange the first kernel's memory core and tools like makedumpfile can do filtering before hibernated images is saved.

This brings me to a random thought. Can we break the process of loading a hibernation kernel in two steps.

- In first step just do the memory reservation for running second kernel. (kexec -l <dummpy-file-for-reserving-memory>)

Quoted 8 lines

- This memory map of reserved pages is exported to user space.

- Use this memory map and regenerate the hibernation kernel initrd (rootfs.gz) and put the memory map there. This memory map can be used by makedumpfile in second kernel for filtering.

This way it will user space to user space communication of information which gets fixed at kernel loading time.

Doing kexec load in two steps is a possible solution. Although this is a little complex, we can wrap the two steps into one /sbin/kexec invoking. That is, When do /sbin/kexec --load-preserve-context <kernel-image>, /sbin/kexec first call sys_kexec_load() to load the kernel image and reserving memory, then amend the memory image of loaded kernel (B) according to the new information available such as return address and backup pages map. For this solution, something still need to be solved is how to pass some information back from kernel B (hibernating kernel) to kernel A (original kernel) and how to pass some information from kernel C (resuming kernel) to kernel A (original kernel).

Hi Huang,

I am kind of ok with both the methods.

- Communicate information between two kernels using an ELF NOTE prepared by kernel.

- Communicate information between user space tools using initrd.

But which method to use will depend on what information we want to exchange between two kernels.

For example, re-entry points can be on stack or in ELF NOTE.

Backup page map probably can be communicated using initrd as only user space need to access that (ELF Core headers can be put in a memory area which is not swapped during transition from kernel A to B. This way kernel B never needs to know that kernel A had done some swapping of pages?).

So far I have understood only following.

1. We need to pass around entry/re-entry points between kernels.

2. We need to pass backup pages map from kernel A to kernel B, so that user space tool can do filtering.

3. We need to pass address of ELF core headers from kernel A to kernel B so that a valid vmcore of kernel A can be exported.

- For first time boot of kernel B, address of ELF core header is passed through command line.

- For re-entry into B, ELF core header address can be passed using some register, or on stack or using kernel ELF NOTE.

What else? What information do we need to communicate from kernel B to kernel A or from kernel C to kernel A?

I am sure that you have told it in the past. Just that I don't recollect it.

Hi Huang,

I am kind of ok with both the methods.

- Communicate information between two kernels using an ELF NOTE prepared by kernel.

- Communicate information between user space tools using initrd.

But which method to use will depend on what information we want to exchange between two kernels.

For example, re-entry points can be on stack or in ELF NOTE.

Backup page map probably can be communicated using initrd as only user space need to access that (ELF Core headers can be put in a memory area which is not swapped during transition from kernel A to B. This way kernel B never needs to know that kernel A had done some swapping of pages?).

So far I have understood only following.

1. We need to pass around entry/re-entry points between kernels.

2. We need to pass backup pages map from kernel A to kernel B, so that user space tool can do filtering.

3. We need to pass address of ELF core headers from kernel A to kernel B so that a valid vmcore of kernel A can be exported.

- For first time boot of kernel B, address of ELF core header is passed through command line.

- For re-entry into B, ELF core header address can be passed using some register, or on stack or using kernel ELF NOTE.

What else? What information do we need to communicate from kernel B to kernel A or from kernel C to kernel A?

I am sure that you have told it in the past. Just that I don't recollect it.

Quoted 13 lines

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Quoted 3 lines

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

In original kexec-tools, this can be done through:
kexec -l --args-none <resumable-core>

Do you need to define an alias for it?

Quoted 13 lines

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Quoted 3 lines

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

In original kexec-tools, this can be done through:
kexec -l --args-none <resumable-core>

Do you need to define an alias for it?

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

This patchset is slowly getting better. True that first we need to come up with minimal infrastructure patch and then think of building more functionality on top of it.

Quoted 11 lines

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

Quoted 7 lines

The features of this patch can be used for as follow:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

The main usage of this functionality is for hibernation. I am not sure what has been the conclusion of previous discussions.

Rafael/Pavel, does the approach of doing hibernation using a separate kernel holds promise?

[..]

Quoted 29 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

3. Boot kernel compiled in step 1 (kernel A).

4. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

5. Boot the kernel B with following shell command line:

/sbin/kexec -e

Quoted 8 lines

6. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Why not store the entry point in dump.elf itself, instead of storing it in a separate file?

I think this is more like a resumable core file. Something similar to functionality what qemu does for resuming an already booted kernel image. So we might have to introduce an ELF_NOTE to mark an image as resumable core.

Quoted 5 lines

Then you can shutdown the machine as normal.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

Quoted 4 lines

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

How the memory segments of dump.elf loaded? Normal kexec way? Memory segments of dump.elf are first stored somewhere and then moved to destination at "kexec -e" time?

Does this really work? If we have 4G RAM, what will be the size of dump.elf? And when we load it back for resuming, do we have sufficient memory left?

May be we can have a separate load flag (--load-resume-image) to mark that we are resuming an hibernated image and kexec does not have to prepare commandline, does not have to prepare zero page/setup page etc.

I have thought through it again and try to put together some of the new kexec options we can introduce to make the whole thing work. I am considering a simple case where a user boots the kernel A and then launches kernel B using "kexec --load-preseve-context". Now a user might save the hibernated image or might want to come back to A.

- kexec -l <kernel-image>
Normal kexec functionality. Boot a new kernel, without preserving existing kernel's context.

- kexec --load-preserve-context <kernel-image>
Boot a new kernel while preserving existing kernel's context.

Will be used for booting kernel B for the first time.

- kexec --load-resume-image <resumable-core>

Resumes an hibernated image. Load a ELF64 hibernated image.

Context of first kernel/boot-loader will not be preserved.

First kernel will not save cpu states. Will put devices into suspended state though so that these can be resumed by resumable core

This option can be used by kboot or kernel C to resume an hibernated image.

- kexec --load-resume-entry <entry-point>

Image is already loaded. Just prepare the entry point so that one can enter back to previous image. cpu states will be saved and devices will be put to suspended states.

will be used for A --> B and B ---> A transitions. Both A and B are booted. This is just for switching back and forth between A and B.

- kexec -e
Transition into the new kernel

This patch looks in pretty decent shape. Once there is some sort of understanding that this approach is promising for hibernation and we have consensus on high level interface, then we can get into line by line review of the patch set.

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Now, only the i386 architecture is supported. The patchset is based on Linux kernel 2.6.25-rc3-mm1, and has been tested on IBM T42 with ACPI on and off.

Signed-off-by Huang Ying <[email protected]>

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Now, only the i386 architecture is supported. The patchset is based on Linux kernel 2.6.25-rc3-mm1, and has been tested on IBM T42 with ACPI on and off.

Signed-off-by Huang Ying <[email protected]>

Hi!

Quoted 3 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Eric, can we get some reviewing/merging going on? Patches seem pretty clean to me, and I do not think holding them outside mainline will help them... they will only get bigger and harder to merge :-(.

Hi!

Quoted 3 lines

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Eric, can we get some reviewing/merging going on? Patches seem pretty clean to me, and I do not think holding them outside mainline will help them... they will only get bigger and harder to merge :-(.

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

Ok, after a long time, I am back to testing and reviewing this patch.

[..]

Quoted 7 lines

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

How do I got back to original kernel without loading dump.elf. I mean, original kernel is already in memory and I don't have to first save it to disk and then reload back. Is there a way to do it? If not, then we need to modify kexec-tools to support that.

Something like

kexec --entry=<entry point>, should tell kexec that kernel is already loaded. Just do the bit to set the entry point properly.

Quoted 4 lines · Ying

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Hi Huang,

Ok, after a long time, I am back to testing and reviewing this patch.

[..]

Quoted 7 lines

7. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

8. In kernel C, load the memory image of kernel A as follow:

   /sbin/kexec -l --args-none --entry=`cat kexec_jump_back_entry` dump.elf

How do I got back to original kernel without loading dump.elf. I mean, original kernel is already in memory and I don't have to first save it to disk and then reload back. Is there a way to do it? If not, then we need to modify kexec-tools to support that.

Something like

kexec --entry=<entry point>, should tell kexec that kernel is already loaded. Just do the bit to set the entry point properly.

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Best Regards,
Huang Ying

------------------------------------>

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

arch/x86/kernel/relocate_kernel_32.S +21 -0

 	.text
 	.align PAGE_ALIGNED
+	.global kexec_relocate_page
+kexec_relocate_page:
+
+/*
+ * Entry point for jumping back from kexeced kernel, the paging is
+ * turned off.
+ */
+kexec_jump_back_entry:
+	call	1f
+1:
+	popl	%ebx
+	subl	$(1b - kexec_relocate_page), %ebx
+	movl	%edi, KJUMP_ENTRY_OFF(%ebx)
+	movl	CP_VA_CONTROL_PAGE(%ebx), %edi
+	lea	STACK_TOP(%ebx), %esp
+	movl	CP_PA_SWAP_PAGE(%ebx), %eax
+	movl	CP_PA_BACKUP_PAGES_MAP(%ebx), %edx
+	pushl	%eax
+	pushl	%edx
+	call	swap_pages
+	addl	$8, %esp

arch/x86/kernel/relocate_kernel_32.S +30 -0

@@ -197,8 +282,54 @@ identity_mapped:
 	xorl	%eax, %eax
 	movl	%eax, %cr3
 
+	movl	CP_PA_SWAP_PAGE(%edi), %eax
+	pushl	%eax
+	pushl	%ebx
+	call	swap_pages
+	addl	$8, %esp
+
+	/* To be certain of avoiding problems with self-modifying code
+	 * I need to execute a serializing instruction here.
+	 * So I flush the TLB, it's handy, and not processor dependent.
+	 */
+	xorl	%eax, %eax
+	movl	%eax, %cr3
+
+	/* set all of the registers to known values */
+	/* leave %esp alone */
+
+	movl	KJUMP_MAGIC_OFF(%edi), %eax
+	cmpl	$KJUMP_MAGIC_NUMBER, %eax
+	jz 1f
+	xorl	%edi, %edi
+	xorl	%eax, %eax
+	xorl	%ebx, %ebx
+	xorl    %ecx, %ecx
+	xorl    %edx, %edx
+	xorl    %esi, %esi
+	xorl    %ebp, %ebp
+	ret
+1:
+	popl	%edx
+	movl	CP_PA_SWAP_PAGE(%edi), %esp

Documentation/i386/jump_back_protocol.txt new file +18 -0

@@ -0,0 +1,66 @@
+		THE LINUX/I386 JUMP BACK PROTOCOL
+		---------------------------------
+
+		Huang Ying <[email protected]>
+		    Last update 2007-12-19
+
+Currently, the following versions of the jump back protocol exist.
+
+Protocol 1.00:	Jumping between original kernel and kexeced kernel
+		support. Calling ordinary C function support.
+
+
+*** JUMP BACK ENTRY
+
+At jump back entry of callee, the CPU must be in 32-bit protected mode
+with paging disabled; the CS, DS, ES and SS must be 4G flat segments;
+CS must have execute/read permission, and DS, ES and SS must have
+read/write permission; interrupt must be disabled; the contents of

This is a minimal patch with only the essential features. All additional features are split out and can be discussed later. I think it may be easier to get consensus on this minimal patch.

Best Regards,
Huang Ying

------------------------------------>

This patch provides an enhancement to kexec/kdump. It implements the following features:

- Jumping between the original kernel and the kexeced kernel.

- Backup/restore memory used by both the original kernel and the kexeced kernel.

- Save/restore CPU and devices state before after kexec.

arch/x86/kernel/relocate_kernel_32.S +21 -0

 	.text
 	.align PAGE_ALIGNED
+	.global kexec_relocate_page
+kexec_relocate_page:
+
+/*
+ * Entry point for jumping back from kexeced kernel, the paging is
+ * turned off.
+ */
+kexec_jump_back_entry:
+	call	1f
+1:
+	popl	%ebx
+	subl	$(1b - kexec_relocate_page), %ebx
+	movl	%edi, KJUMP_ENTRY_OFF(%ebx)
+	movl	CP_VA_CONTROL_PAGE(%ebx), %edi
+	lea	STACK_TOP(%ebx), %esp
+	movl	CP_PA_SWAP_PAGE(%ebx), %eax
+	movl	CP_PA_BACKUP_PAGES_MAP(%ebx), %edx
+	pushl	%eax
+	pushl	%edx
+	call	swap_pages
+	addl	$8, %esp

arch/x86/kernel/relocate_kernel_32.S +30 -0

@@ -197,8 +282,54 @@ identity_mapped:
 	xorl	%eax, %eax
 	movl	%eax, %cr3
 
+	movl	CP_PA_SWAP_PAGE(%edi), %eax
+	pushl	%eax
+	pushl	%ebx
+	call	swap_pages
+	addl	$8, %esp
+
+	/* To be certain of avoiding problems with self-modifying code
+	 * I need to execute a serializing instruction here.
+	 * So I flush the TLB, it's handy, and not processor dependent.
+	 */
+	xorl	%eax, %eax
+	movl	%eax, %cr3
+
+	/* set all of the registers to known values */
+	/* leave %esp alone */
+
+	movl	KJUMP_MAGIC_OFF(%edi), %eax
+	cmpl	$KJUMP_MAGIC_NUMBER, %eax
+	jz 1f
+	xorl	%edi, %edi
+	xorl	%eax, %eax
+	xorl	%ebx, %ebx
+	xorl    %ecx, %ecx
+	xorl    %edx, %edx
+	xorl    %esi, %esi
+	xorl    %ebp, %ebp
+	ret
+1:
+	popl	%edx
+	movl	CP_PA_SWAP_PAGE(%edi), %esp

Documentation/i386/jump_back_protocol.txt new file +18 -0

@@ -0,0 +1,66 @@
+		THE LINUX/I386 JUMP BACK PROTOCOL
+		---------------------------------
+
+		Huang Ying <[email protected]>
+		    Last update 2007-12-19
+
+Currently, the following versions of the jump back protocol exist.
+
+Protocol 1.00:	Jumping between original kernel and kexeced kernel
+		support. Calling ordinary C function support.
+
+
+*** JUMP BACK ENTRY
+
+At jump back entry of callee, the CPU must be in 32-bit protected mode
+with paging disabled; the CS, DS, ES and SS must be 4G flat segments;
+CS must have execute/read permission, and DS, ES and SS must have
+read/write permission; interrupt must be disabled; the contents of

Ok, I have done some testing on this patch. Currently I have just tested switching back and forth between two kernels and it is working for me.

Quoted 24 lines

arch/x86/kernel/relocate_kernel_32.S +24 -0

+/*
+ * Entry point for jumping back from kexeced kernel, the paging is
+ * turned off.
+ */
+kexec_jump_back_entry:
+	call	1f
+1:
+	popl	%ebx
+	subl	$(1b - kexec_relocate_page), %ebx
+	movl	%edi, KJUMP_ENTRY_OFF(%ebx)
+	movl	CP_VA_CONTROL_PAGE(%ebx), %edi
+	lea	STACK_TOP(%ebx), %esp
+	movl	CP_PA_SWAP_PAGE(%ebx), %eax
+	movl	CP_PA_BACKUP_PAGES_MAP(%ebx), %edx
+	pushl	%eax
+	pushl	%edx
+	call	swap_pages
+	addl	$8, %esp
+	movl	CP_PA_PGD(%ebx), %eax
+	movl	%eax, %cr3
+	movl	%cr0, %eax
+	orl	$(1<<31), %eax
+	movl	%eax, %cr0
+	lea	STACK_TOP(%edi), %esp

arch/x86/kernel/relocate_kernel_32.S line 77

Quoted 4 lines

arch/x86/kernel/relocate_kernel_32.S +4 -0

+	movl	%edi, %eax
+	addl	$(virtual_mapped - kexec_relocate_page), %eax
+	pushl	%eax
+	ret

Upon re-entering the kernel, what happens to GDT table? So gdtr will be pointing to GDT of other kernel (which is not there as pages have been swapped)? Do we need to reload the gdtr upon re-entering the kernel.

[..]

Quoted 38 lines

arch/x86/kernel/relocate_kernel_32.S +34 -0

@@ -197,8 +282,54 @@ identity_mapped:
 	xorl	%eax, %eax
 	movl	%eax, %cr3
 
+	movl	CP_PA_SWAP_PAGE(%edi), %eax
+	pushl	%eax
+	pushl	%ebx
+	call	swap_pages
+	addl	$8, %esp
+
+	/* To be certain of avoiding problems with self-modifying code
+	 * I need to execute a serializing instruction here.
+	 * So I flush the TLB, it's handy, and not processor dependent.
+	 */
+	xorl	%eax, %eax
+	movl	%eax, %cr3
+
+	/* set all of the registers to known values */
+	/* leave %esp alone */
+
+	movl	KJUMP_MAGIC_OFF(%edi), %eax
+	cmpl	$KJUMP_MAGIC_NUMBER, %eax
+	jz 1f
+	xorl	%edi, %edi
+	xorl	%eax, %eax
+	xorl	%ebx, %ebx
+	xorl    %ecx, %ecx
+	xorl    %edx, %edx
+	xorl    %esi, %esi
+	xorl    %ebp, %ebp
+	ret
+1:
+	popl	%edx
+	movl	CP_PA_SWAP_PAGE(%edi), %esp
+	addl	$PAGE_SIZE_asm, %esp
+	pushl	%edx
+2:
+	call	*%edx

arch/x86/kernel/relocate_kernel_32.S identity_mapped: · line 323

Quoted 5 lines

arch/x86/kernel/relocate_kernel_32.S +5 -0

+	movl	%edi, %edx
+	popl	%edi
+	pushl	%edx
+	jmp	2b
+

What does above piece of code do? Looks like redundant for switching between the kernels? After call *%edx, we never return here. Instead we come back to "kexec_jump_back_entry"?

[..]

Quoted 27 lines

Documentation/i386/jump_back_protocol.txt new file +24 -0

@@ -0,0 +1,66 @@
+		THE LINUX/I386 JUMP BACK PROTOCOL
+		---------------------------------
+
+		Huang Ying <[email protected]>
+		    Last update 2007-12-19
+
+Currently, the following versions of the jump back protocol exist.
+
+Protocol 1.00:	Jumping between original kernel and kexeced kernel
+		support. Calling ordinary C function support.
+
+
+*** JUMP BACK ENTRY
+
+At jump back entry of callee, the CPU must be in 32-bit protected mode
+with paging disabled; the CS, DS, ES and SS must be 4G flat segments;
+CS must have execute/read permission, and DS, ES and SS must have
+read/write permission; interrupt must be disabled; the contents of
+registers and corresponding memory must be as follow:
+
+Offset/Size	Meaning
+
+%edi		Real jump back entry of caller if supported,
+		otherwise 0.

Documentation/i386/jump_back_protocol.txt line 28

Quoted 4 lines

Documentation/i386/jump_back_protocol.txt new file +4 -0

+%esp		Stack top pointer, the size of stack is about 4k bytes.
+(%esp)/4	Helper jump back entry of caller if %edi != 0,
+		otherwise undefined.
+

I am not sure what is helper jump back entry? I understand that you are using %edi to pass around entry point between two kernels. Can you please shed some more light on this?

Quoted 12 lines

arch/x86/kernel/relocate_kernel_32.S identity_mapped: · line 323

Quoted 8 lines

+2:

arch/x86/kernel/relocate_kernel_32.S +6 -0

+	call	*%edx
 
+	movl	%edi, %edx
+	popl	%edi
+	pushl	%edx
+	jmp	2b
+

What does above piece of code do? Looks like redundant for switching between the kernels? After call *%edx, we never return here. Instead we come back to "kexec_jump_back_entry"?

For switching between the kernels, this is redundant. Originally another feature of kexec jump is to call some code in physical mode. This is used to provide a C ABI to called code.

Quoted 12 lines

arch/x86/kernel/relocate_kernel_32.S identity_mapped: · line 323

Quoted 8 lines

+2:

arch/x86/kernel/relocate_kernel_32.S +6 -0

+	call	*%edx
 
+	movl	%edi, %edx
+	popl	%edi
+	pushl	%edx
+	jmp	2b
+

What does above piece of code do? Looks like redundant for switching between the kernels? After call *%edx, we never return here. Instead we come back to "kexec_jump_back_entry"?

For switching between the kernels, this is redundant. Originally another feature of kexec jump is to call some code in physical mode. This is used to provide a C ABI to called code.

Ok, You want to make BIOS calls. We already do that using vm86 mode and use bios real mode interrupts. So why do we need this interface? Or, IOW, how is this interface better?

On Thu, 2008-05-15 at 16:09 -0400, Vivek Goyal wrote:
[...]

Quoted 3 lines

Ok, You want to make BIOS calls. We already do that using vm86 mode and use bios real mode interrupts. So why do we need this interface? Or, IOW, how is this interface better?

It can call code in 32-bit physical mode in addition to real mode. So It can be used to call EFI runtime service, especially call EFI 64 runtime service under 32-bit kernel or vice versa.

The main purpose of kexec jump is for hibernation. But I think if the effort is small, why not support general 32-bit physical mode code call at same time.

On Thu, 2008-05-15 at 16:09 -0400, Vivek Goyal wrote:
[...]

Quoted 3 lines

Ok, You want to make BIOS calls. We already do that using vm86 mode and use bios real mode interrupts. So why do we need this interface? Or, IOW, how is this interface better?

It can call code in 32-bit physical mode in addition to real mode. So It can be used to call EFI runtime service, especially call EFI 64 runtime service under 32-bit kernel or vice versa.

The main purpose of kexec jump is for hibernation. But I think if the effort is small, why not support general 32-bit physical mode code call at same time.

Quoted 6 lines · Ying

On Thu, 2008-05-15 at 16:09 -0400, Vivek Goyal wrote:
[...]

Quoted 3 lines

Ok, You want to make BIOS calls. We already do that using vm86 mode and use bios real mode interrupts. So why do we need this interface? Or, IOW, how is this interface better?

Quoted 8 lines

It can call code in 32-bit physical mode in addition to real mode. So It can be used to call EFI runtime service, especially call EFI 64 runtime service under 32-bit kernel or vice versa.

The main purpose of kexec jump is for hibernation. But I think if the effort is small, why not support general 32-bit physical mode code call at same time.