kexec jump -v11: save/restore device state

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

Quoted 14 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 4 lines

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Quoted 14 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 4 lines

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Quoted 20 lines · Ying

Quoted 18 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

Quoted 6 lines

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Doesn't that resume the devices and their drivers into an unknown state?

Quoted 20 lines · Ying

Quoted 18 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

Quoted 6 lines

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Doesn't that resume the devices and their drivers into an unknown state?

Quoted 14 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 4 lines

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Quoted 14 lines · Len Brown

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

Quoted 4 lines

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

ACPI S5 is used instead of S4 for this simple hibernation implementation. That is, before creating the hibernation image, the ACPI _PTS is not executed, devices are not put into low power state and wake devices are not enabled. After creating the hibernation image, the image is saved to disk and system is shutdown (go to S5). When resuming from hibernated image, ACPI _BFS and _WAK are not executed too.

Can't we implement ACPI S5 state as an option in current hibernation framework? Or kexec jump is a requirement for that?

Can't we implement ACPI S5 state as an option in current hibernation framework? Or kexec jump is a requirement for that?

Quoted 10 lines · Ying

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

What part of ACPI's role in hibernation are you trying to avoid
1. enabling wake devices
2. removing power from the system
3. something else?

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

- Kernel/system debug through making system snapshot. You can make system snapshot, jump back, do some thing and make another system snapshot.

- Cooperative multi-kernel/system. With kexec jump, you can switch between several kernels/systems quickly without boot process except the first time. This appears like swap a whole kernel/system out/in.

- A general method to call program in physical mode (paging turning off). This can be used to invoke BIOS code under Linux.

The following user-space tools can be used with kexec jump:

- kexec-tools needs to be patched to support kexec jump. The patches
and the precompiled kexec can be download from the following URL:
source: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec-tools-src_git_kh10.tar.bz2
patches: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec-tools-patches_git_kh10.tar.bz2
binary: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec_git_kh10

- makedumpfile with patches are used as memory image saving tool, it
can exclude free pages from original kernel memory image file. The
patches and the precompiled makedumpfile can be download from the
following URL:
source: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile-src_cvs_kh10.tar.bz2
patches: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile-patches_cvs_kh10.tar.bz2
binary: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile_cvs_kh10

- An initramfs image can be used as the root file system of kexeced
kernel. An initramfs image built with "BuildRoot" can be downloaded
from the following URL:
initramfs image: http://khibernation.sourceforge.net/download/release_v10/initramfs/rootfs_cvs_kh10.gz
All user space tools above are included in the initramfs image.

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

This patch implements devices state save/restore before after kexec.

This patch together with features in kexec_jump patch can be used for following:

- A simple hibernation implementation without ACPI support. You can kexec a hibernating kernel, save the memory image of original system and shutdown the system. When resuming, you restore the memory image of original system via ordinary kexec load then jump back.

- Kernel/system debug through making system snapshot. You can make system snapshot, jump back, do some thing and make another system snapshot.

- Cooperative multi-kernel/system. With kexec jump, you can switch between several kernels/systems quickly without boot process except the first time. This appears like swap a whole kernel/system out/in.

- A general method to call program in physical mode (paging turning off). This can be used to invoke BIOS code under Linux.

The following user-space tools can be used with kexec jump:

- kexec-tools needs to be patched to support kexec jump. The patches
and the precompiled kexec can be download from the following URL:
source: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec-tools-src_git_kh10.tar.bz2
patches: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec-tools-patches_git_kh10.tar.bz2
binary: http://khibernation.sourceforge.net/download/release_v10/kexec-tools/kexec_git_kh10

- makedumpfile with patches are used as memory image saving tool, it
can exclude free pages from original kernel memory image file. The
patches and the precompiled makedumpfile can be download from the
following URL:
source: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile-src_cvs_kh10.tar.bz2
patches: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile-patches_cvs_kh10.tar.bz2
binary: http://khibernation.sourceforge.net/download/release_v10/makedumpfile/makedumpfile_cvs_kh10

- An initramfs image can be used as the root file system of kexeced
kernel. An initramfs image built with "BuildRoot" can be downloaded
from the following URL:
initramfs image: http://khibernation.sourceforge.net/download/release_v10/initramfs/rootfs_cvs_kh10.gz
All user space tools above are included in the initramfs image.

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

Quoted 33 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

Quoted 9 lines

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

One of the concerns raised by hibernation people in the past was to use single boot loader entry to boot normally as well while resuming a kernel.

So in this case a user either needs to maintain two boot-loader entries or modify it on the fly. I wished there was a better way to handle that.

I am more interested in ability to have multiple kernel loaded in RAM and capability to switch between them. Allows me to take non-disruptive core dumps and somebody wanted to snapshots the kernels. That should still work.

[..]

Quoted 6 lines

arch/x86/kernel/machine_kexec_32.c +0 -0

@@ -125,6 +125,12 @@ void machine_kexec(struct kimage *image)
 	/* Interrupts aren't acceptable while we reboot */
 	local_irq_disable();
 

arch/x86/kernel/machine_kexec_32.c void machine_kexec(struct kimage *image) · line 131

Quoted 4 lines

arch/x86/kernel/machine_kexec_32.c +4 -0

+	if (image->preserve_context) {
+#ifdef CONFIG_X86_IO_APIC
+		disable_IO_APIC();
+#endif

I think it would be a good idea to put some kind of comment here. We need to put APICs in legacy mode so that we can get timer interrupts in second kernel. kexec/kdump paths already have calls to disable_IO_APIC() in one form or other. kexec jump path also needs one.

On Wed, 2008-06-11 at 12:30 -0400, Vivek Goyal wrote:
[...]

Quoted 46 lines

Quoted 33 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

Quoted 9 lines

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

One of the concerns raised by hibernation people in the past was to use single boot loader entry to boot normally as well while resuming a kernel.

Quoted 2 lines

So in this case a user either needs to maintain two boot-loader entries or modify it on the fly. I wished there was a better way to handle that.

Now it is not needed to have two boot-loader entries, just one is enough. Step 4 and step 8 can share the same boot-loader entries. The rootfs.gz can be the normal initramfs or initrd when deployment. In rootfs.gz, if there is a valid hibernation image, the hibernated system will be restored, otherwise, normal boot process follows.

On Wed, 2008-06-11 at 12:30 -0400, Vivek Goyal wrote:
[...]

Quoted 46 lines

Quoted 33 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

Quoted 9 lines

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

One of the concerns raised by hibernation people in the past was to use single boot loader entry to boot normally as well while resuming a kernel.

Quoted 2 lines

So in this case a user either needs to maintain two boot-loader entries or modify it on the fly. I wished there was a better way to handle that.

Now it is not needed to have two boot-loader entries, just one is enough. Step 4 and step 8 can share the same boot-loader entries. The rootfs.gz can be the normal initramfs or initrd when deployment. In rootfs.gz, if there is a valid hibernation image, the hibernated system will be restored, otherwise, normal boot process follows.

Few things I don't understand.

- Are you saying that hibernated image will be saved in initrd (rootfs.gz)? But that saving is only in RAM, we never write back it to disk?

- I thought we probably have to dedicate a raw partition kind of thing for saving image and then modify boot loader command line to something similar to, "resume=partition". Then initrd can go hunting for image in respective partition (as specified by command line parameter) and if image is not available then continue with normal boot.

Few things I don't understand.

- Are you saying that hibernated image will be saved in initrd (rootfs.gz)? But that saving is only in RAM, we never write back it to disk?

- I thought we probably have to dedicate a raw partition kind of thing for saving image and then modify boot loader command line to something similar to, "resume=partition". Then initrd can go hunting for image in respective partition (as specified by command line parameter) and if image is not available then continue with normal boot.

Quoted 33 lines

Usage example of simple hibernation:

1. Compile and install patched kernel with following options selected:

CONFIG_X86_32=y
CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PM=y

2. Build an initramfs image contains kexec-tool and makedumpfile, or download the pre-built initramfs image, called rootfs.gz in following text.

3. Prepare a partition to save memory image of original kernel, called hibernating partition in following text.

4. Boot kernel compiled in step 1 (kernel A).

5. In the kernel A, load kernel compiled in step 1 (kernel B) with /sbin/kexec. The shell command line can be as follow:

   /sbin/kexec --load-preserve-context /boot/bzImage --mem-min=0x100000
     --mem-max=0xffffff --initrd=rootfs.gz

6. Boot the kernel B with following shell command line:

/sbin/kexec -e

7. The kernel B will boot as normal kexec. In kernel B the memory image of kernel A can be saved into hibernating partition as follow:

Quoted 9 lines

jump_back_entry=cat /proc/cmdline | tr ' ' '\n' | grep kexec_jump_back_entry | cut -d '=' echo $jump_back_entry > kexec_jump_back_entry
cp /proc/vmcore dump.elf

Then you can shutdown the machine as normal.

8. Boot kernel compiled in step 1 (kernel C). Use the rootfs.gz as root file system.

One of the concerns raised by hibernation people in the past was to use single boot loader entry to boot normally as well while resuming a kernel.

So in this case a user either needs to maintain two boot-loader entries or modify it on the fly. I wished there was a better way to handle that.

I am more interested in ability to have multiple kernel loaded in RAM and capability to switch between them. Allows me to take non-disruptive core dumps and somebody wanted to snapshots the kernels. That should still work.

[..]

Quoted 6 lines

arch/x86/kernel/machine_kexec_32.c +0 -0

@@ -125,6 +125,12 @@ void machine_kexec(struct kimage *image)
 	/* Interrupts aren't acceptable while we reboot */
 	local_irq_disable();
 

arch/x86/kernel/machine_kexec_32.c void machine_kexec(struct kimage *image) · line 131

Quoted 4 lines

arch/x86/kernel/machine_kexec_32.c +4 -0

+	if (image->preserve_context) {
+#ifdef CONFIG_X86_IO_APIC
+		disable_IO_APIC();
+#endif

I think it would be a good idea to put some kind of comment here. We need to put APICs in legacy mode so that we can get timer interrupts in second kernel. kexec/kdump paths already have calls to disable_IO_APIC() in one form or other. kexec jump path also needs one.