Firmware assisted dump support in dracut

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

2. Make dracut tool fadump aware and it will build fadump aware initrd (default OS initrd) during kernel install. Once built, this initrd should never be rebuilt again. Which means the default initramfs will contain vmcore capture steps. This approach would pack all possible dracut modules (nfs, ssh, bonding etc.) so that the default initrd will be capable of supporting all possible dump target. Hence this approach will bloat the initramfs. On my Power test system, the size of the default initrd generated without any changes is 16M. With the above approach this size jumps to 27M.

Since both the above approaches would require changes to dracut package, I would like to know your views on above approaches.

Thanks,
-Mahesh.

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Of course such a service could be created and create the initramfs on shutdown, if any of the files, which should be watched, have changed.

This of course is a dangerous automatism, which might as well lead to a unbootable system. Further steps (with grubby) would have to make sure, the last bootable entry is still there, along with the last kernel version.

I was already thinking about creating such a monster, but this would involve to reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has to be done anyway in the next years. The first step was to create a common configuration file format for all the bootloaders, which they parse and display.

http://harald.fedorapeople.org/downloads/boot-unification.pdf
A simple boot manager parsing the config layout as a reference implementation:
http://freedesktop.org/wiki/Software/gummiboot

Next step is to patch grub2 to parse those config files and display the menu entries.

Next step is to enhance /sbin/new-kernel-package and obsolete grubby.

Then, I think we are ready to create such a service.

Quoted 10 lines

2. Make dracut tool fadump aware and it will build fadump aware initrd (default OS initrd) during kernel install. Once built, this initrd should never be rebuilt again. Which means the default initramfs will contain vmcore capture steps. This approach would pack all possible dracut modules (nfs, ssh, bonding etc.) so that the default initrd will be capable of supporting all possible dump target. Hence this approach will bloat the initramfs. On my Power test system, the size of the default initrd generated without any changes is 16M. With the above approach this size jumps to 27M.

Yeah, this is crazy.

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:

Quoted 38 lines

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Of course such a service could be created and create the initramfs on shutdown, if any of the files, which should be watched, have changed.

This of course is a dangerous automatism, which might as well lead to a unbootable system. Further steps (with grubby) would have to make sure, the last bootable entry is still there, along with the last kernel version.

I was already thinking about creating such a monster, but this would involve to reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has to be done anyway in the next years. The first step was to create a common configuration file format for all the bootloaders, which they parse and display.

http://harald.fedorapeople.org/downloads/boot-unification.pdf
A simple boot manager parsing the config layout as a reference implementation:
http://freedesktop.org/wiki/Software/gummiboot

Next step is to patch grub2 to parse those config files and display the menu entries.

Next step is to enhance /sbin/new-kernel-package and obsolete grubby.

Then, I think we are ready to create such a service.

Quoted 9 lines

2. Make dracut tool fadump aware and it will build fadump aware initrd (default OS initrd) during kernel install. Once built, this initrd should never be rebuilt again. Which means the default initramfs will contain vmcore capture steps. This approach would pack all possible dracut modules (nfs, ssh, bonding etc.) so that the default initrd will be capable of supporting all possible dump target. Hence this approach will bloat the initramfs. On my Power test system, the size of the default initrd generated without any changes is 16M. With the above approach this size jumps to 27M.

Yeah, this is crazy.

Quoted 7 lines

Since both the above approaches would require changes to dracut package, I would like to know your views on above approaches.

Thanks,
-Mahesh.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 39 lines · Harald Hoyer

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:

Quoted 38 lines

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Quoted 8 lines

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Normally user would restart kdump service after changing kdump config file. Idea is to invoke dracut through kdump service, as it already watches and detect changes in kdump config files.

I was wondering if dracut allow us to insert/install additional dracut modules to already existing initramfs? OR does it rebuild initramfs all over again?

Quoted 13 lines

Of course such a service could be created and create the initramfs on shutdown, if any of the files, which should be watched, have changed.

This of course is a dangerous automatism, which might as well lead to a unbootable system. Further steps (with grubby) would have to make sure, the last bootable entry is still there, along with the last kernel version.

I was already thinking about creating such a monster, but this would involve to reinvent the current /sbin/new-kernel-package / grubby infrastructure, which has to be done anyway in the next years. The first step was to create a common configuration file format for all the bootloaders, which they parse and display.

Quoted 6 lines

http://harald.fedorapeople.org/downloads/boot-unification.pdf
A simple boot manager parsing the config layout as a reference implementation:
http://freedesktop.org/wiki/Software/gummiboot

Next step is to patch grub2 to parse those config files and display the menu entries.

I think once such a infrastructure exists, things would be much easier. Also, would it be possible to make such a infrastructure OS crash aware, so that it can load different initramfs in crash scenario and default initramfs in a normal boot process. I am not sure how practical this idea would be to implement.

Quoted 21 lines

Next step is to enhance /sbin/new-kernel-package and obsolete grubby.

Then, I think we are ready to create such a service.

Quoted 9 lines

2. Make dracut tool fadump aware and it will build fadump aware initrd (default OS initrd) during kernel install. Once built, this initrd should never be rebuilt again. Which means the default initramfs will contain vmcore capture steps. This approach would pack all possible dracut modules (nfs, ssh, bonding etc.) so that the default initrd will be capable of supporting all possible dump target. Hence this approach will bloat the initramfs. On my Power test system, the size of the default initrd generated without any changes is 16M. With the above approach this size jumps to 27M.

Yeah, this is crazy.

Quoted 4 lines

Since both the above approaches would require changes to dracut package, I would like to know your views on above approaches.

Quoted 9 lines

Quoted 3 lines

Thanks,
-Mahesh.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 49 lines · Mahesh Jagannath Salgaonkar

Quoted 39 lines · Harald Hoyer

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:

Quoted 38 lines

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Quoted 8 lines

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Quoted 7 lines

Normally user would restart kdump service after changing kdump config file. Idea is to invoke dracut through kdump service, as it already watches and detect changes in kdump config files.

I was wondering if dracut allow us to insert/install additional dracut modules to already existing initramfs? OR does it rebuild initramfs all over again?

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

OR

b) Add additional modules to an existing initramfs instead of regenerating again. Is this possible at all?

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

Thanks
Vivek

Quoted 24 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Quoted 4 lines

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Quoted 24 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Quoted 4 lines

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Quoted 30 lines · Mahesh Jagannath Salgaonkar

Quoted 28 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

Quoted 4 lines

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Also what about kernel command line? Kernel command line can be very specific to kdump environment. It is atleast in x86. I am hoping that's not the case with firmware assisted dump.

Quoted 36 lines · Vivek Goyal

Quoted 34 lines · Mahesh Jagannath Salgaonkar

Quoted 24 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Quoted 4 lines

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Quoted 3 lines

Also what about kernel command line? Kernel command line can be very specific to kdump environment. It is atleast in x86. I am hoping that's not the case with firmware assisted dump.

If first kernel and kdump kernel command line are same in fadump and boot environment is same, I think it is worth exploring the option of saving kdump from kdump service when system boots back. That will do away with any need to tweak initramfs.

It just requires more memory to be reserved so that system can boot , mount root, run services and then save dump. I am hoping that firmware can set aside enough memory to (1G?), to copy the memory contents and boot the kernel in that 1G.

Also does fadump work well with new mmap() interface /proc/vmcore is going to support. Patches are in -mm now. s390 zfcpdump is going to face some issues and michael is working on resolving these. I suspect that fadump will run into similar issues if firmware is copying memory contents somewhere else.

Thanks
Vivek

Quoted 37 lines · Vivek Goyal

Quoted 36 lines · Vivek Goyal

Quoted 34 lines · Mahesh Jagannath Salgaonkar

Quoted 24 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Quoted 4 lines

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Quoted 8 lines

Quoted 3 lines

Also what about kernel command line? Kernel command line can be very specific to kdump environment. It is atleast in x86. I am hoping that's not the case with firmware assisted dump.

If first kernel and kdump kernel command line are same in fadump and boot environment is same, I think it is worth exploring the option of saving kdump from kdump service when system boots back. That will do away with any need to tweak initramfs.

This needs larger memory to be reserved. We may need to make sure that kdump service is invoked at very early stage before other services kicks in which may start requesting for memory and run into OOM issue. If we can make sure that no other services are invoked when kdump service is saving dump then it will help to fix the size of reserved memory. The systemd infrastructure provides aggressive parallelization capabilities. I am not expert on systemd but will need to explore whether with systemd is this possible.

Quoted 5 lines

Quoted 4 lines

If first kernel and kdump kernel command line are same in fadump and boot environment is same, I think it is worth exploring the option of saving kdump from kdump service when system boots back. That will do away with any need to tweak initramfs.

Quoted 8 lines

This needs larger memory to be reserved. We may need to make sure that kdump service is invoked at very early stage before other services kicks in which may start requesting for memory and run into OOM issue. If we can make sure that no other services are invoked when kdump service is saving dump then it will help to fix the size of reserved memory. The systemd infrastructure provides aggressive parallelization capabilities. I am not expert on systemd but will need to explore whether with systemd is this possible.

I think we can create a new service for saving dump and make some of the systemd services dependent on start of this service.

Can't integrate with existing kdump service as that one can start later and can take time in regenerating initramfs and holding back other services for that duration will not make sense.

By creating a seprate service, delay will be introduced only when there s dump to be saved. Before this we will have bring up all networking and storage in the system so that we can save dump to intended target.

I think same service can be used to save dump from inside initramfs too. (As systemd is now started inside initramfs too).

Other appraoch is dumping through initramfs and if you do things conditionally (if dump is available), then any dracut parameter used by kdump will have to have a conditional version too. Something like

mount=
mount.kdump_only=

mount.kdump_only will kick in only dump is available. Also we need to make sure that effects of all kdump_only options are undone before root is mounted or we create nice handover mechanism to systemd. I think there handover mechanism already between initramfs and systemd for complex storage targets like multipathd.

I will let harald comment on what does he think of kdump_only variant options.

On Wed, Jun 05, 2013 at 10:55:04AM -0400, Vivek Goyal wrote:

[..]

Quoted 35 lines

Quoted 5 lines

Quoted 4 lines

If first kernel and kdump kernel command line are same in fadump and boot environment is same, I think it is worth exploring the option of saving kdump from kdump service when system boots back. That will do away with any need to tweak initramfs.

Quoted 8 lines

This needs larger memory to be reserved. We may need to make sure that kdump service is invoked at very early stage before other services kicks in which may start requesting for memory and run into OOM issue. If we can make sure that no other services are invoked when kdump service is saving dump then it will help to fix the size of reserved memory. The systemd infrastructure provides aggressive parallelization capabilities. I am not expert on systemd but will need to explore whether with systemd is this possible.

I think we can create a new service for saving dump and make some of the systemd services dependent on start of this service.

Can't integrate with existing kdump service as that one can start later and can take time in regenerating initramfs and holding back other services for that duration will not make sense.

By creating a seprate service, delay will be introduced only when there s dump to be saved. Before this we will have bring up all networking and storage in the system so that we can save dump to intended target.

I think same service can be used to save dump from inside initramfs too. (As systemd is now started inside initramfs too).

Other appraoch is dumping through initramfs and if you do things conditionally (if dump is available), then any dracut parameter used by kdump will have to have a conditional version too. Something like

mount=
mount.kdump_only=

Quoted 4 lines

mount.kdump_only will kick in only dump is available. Also we need to make sure that effects of all kdump_only options are undone before root is mounted or we create nice handover mechanism to systemd. I think

Quoted 5 lines

there handover mechanism already between initramfs and systemd for complex storage targets like multipathd.

I will let harald comment on what does he think of kdump_only variant options.

Thinking more about it, I think from kdump point of view, it is better to be able to dump from initrafs. We don't want different mechanism for different architecture which will result in more confusion and possibly code duplication.

Quoted 31 lines · Mahesh Jagannath Salgaonkar

Quoted 25 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 10 lines

Quoted 9 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Quoted 4 lines

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

Quoted 4 lines

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path.

Also what about kernel command line? Kernel command line can be very specific to kdump environment. It is atleast in x86. I am hoping that's not the case with firmware assisted dump.

Also can you point me to some writeup of fadump again. I want to refresh what fadump is doing and how firmware is assisting and how does it make better than regular kdump.

I suspect it is very similar to s390 firmware dump (zfcpdump). Does s390 kdump or s390 zfcpdump reboot after saving dump or they have mechanism to continue to boot and hotplug memory after saving dump. I am wondering how do they do it.

Quoted 30 lines · Mahesh Jagannath Salgaonkar

Quoted 28 lines · Vivek Goyal

On Thu, May 30, 2013 at 08:49:07AM +0200, Harald Hoyer wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

Quoted 6 lines

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

I am not sure how well it will work in the context of kdump. kdump options and original options might conflict. kdump might drop some of its own cmdline options in initramfs which might not make any sense in regular boot.

Kdump will specify additional mount points. IIUC, then we will try to bring up those targets in regular boot too and mount at user specified mount point. And later init services might get confused when respective daemon tries to bring up same targets again.

Quoted 4 lines

I think we should be able differentiate between regular boot and boot after crash by checking existence of '/proc/vmcore' file, and do the kdump specific configs/mount if this file exists. In regular boot we can mute the kdump code path

Also what about kernel command line? Kernel command line can be very specific to kdump environment. It is atleast in x86. I am hoping that's not the case with firmware assisted dump.

Also can you point me to some writeup of fadump again. I want to refresh what fadump is doing and how firmware is assisting and how does it make better than regular kdump.

I suspect it is very similar to s390 firmware dump (zfcpdump). Does s390 kdump or s390 zfcpdump reboot after saving dump or they have mechanism to continue to boot and hotplug memory after saving dump. I am wondering how do they do it.

CCing michael.

Vivek

Quoted 58 lines · Mahesh Jagannath Salgaonkar

Quoted 49 lines · Mahesh Jagannath Salgaonkar

Quoted 39 lines · Harald Hoyer

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:

Quoted 38 lines

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Quoted 8 lines

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Quoted 7 lines

Normally user would restart kdump service after changing kdump config file. Idea is to invoke dracut through kdump service, as it already watches and detect changes in kdump config files.

I was wondering if dracut allow us to insert/install additional dracut modules to already existing initramfs? OR does it rebuild initramfs all over again?

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

Quoted 1 line

Quoted 4 lines

OR

b) Add additional modules to an existing initramfs instead of regenerating again. Is this possible at all?

You can always generate a second cpio image with additional files and either concatenate both images or specify those two images in the bootloader.

OR

unpack the initramfs, add the file, (maybe run depmod, if kernel driver added), and repack.

Quoted 77 lines · Mahesh Jagannath Salgaonkar

Quoted 58 lines · Mahesh Jagannath Salgaonkar

Quoted 49 lines · Harald Hoyer

Quoted 39 lines

Am 28.11.2012 17:12, schrieb Mahesh J Salgaonkar:

Quoted 38 lines

Hi Harald,

Last year I worked on adding Firmware assisted dump (fadump) support in PowerLinux (ppc64). This feature is now accepted upstream Linux kernel and the documentation is available at http://lwn.net/Articles/488132/

Like kdump, fadump also exports the memory dump through /proc/vmcore in ELF format. This enables us to reuse the existing kdump infrastructure for dump capture and filtering. However, unlike kdump, fadump does not use kexec-based approach, instead it depends on Power firmware to preserve the memory dump and reboot into new kernel. This is what happens in fadump after crash:

1. At the crash, kernel informs power firmware that kernel has crashed.
2. Firmware takes the control and reboots the entire system preserving only the memory (resets all other devices).
3. The reboot follows the normal booting process (non-kexec).
4. The boot loader loads the default kernel and initrd from /boot

I am working on integrating fadump with existing kdump infrastructure. The current kdump infrastructure builds a separate initrd (whenever there is a change detected in kdump config file /etc/kdump.conf) which then gets loaded into memory by kexec tool for use by kdump kernel. But, in the fadump approach, the second kernel (after crash) always use the default (OS built) initramfs. Hence, to support fadump, change is required to introduce dump capturing steps in default initramfs itself. Hence the possible approaches I am looking into are:

1. Rebuild the default initramfs every time when there is a change detected in kdump config file (/etc/kdump.conf)
This approach would modify existing initramfs in place. I did work on this approach by enhancing mkdumprd (tool from kexec-tools package) to extract and rebuild default initramfs with dump capturing steps. After discussing with Vivek Goyal, he suggested that better approach to add code to dracut for rebuilding boot initramfs instead of enhancing mkdumprd. This means introducing a new dracut module that will be responsible for introducing dump capture steps inside the rebuilt initramfs by pulling required modules e.g. ssh, nfs etc. depending on /etc/kdump.conf

Quoted 8 lines

Quoted 3 lines

Now the question is whether it is possible for dracut to rebuild boot initramfs in place? if Yes, is there any issues in rebuilding of boot initramfs everytime when there is a change in /etc/kdump.conf?

dracut is called by /sbin/new-kernel-package. dracut is not a service, which is run on every boot. So, no, dracut can't watch files and build an initramfs on itsself.

Quoted 7 lines

Normally user would restart kdump service after changing kdump config file. Idea is to invoke dracut through kdump service, as it already watches and detect changes in kdump config files.

I was wondering if dracut allow us to insert/install additional dracut modules to already existing initramfs? OR does it rebuild initramfs all over again?

Quoted 8 lines

Sorry for restarting this discussion very late. I would like to know how safe is to rebuild kernel's default (boot) initramfs for an already installed kernel ?

Also, Does dracut provides any of following mechanism?
a) Mechanism where dracut can detect what options were used during first build for a given (exsisting) initramfs. (This mechanism may help one to regenerate similar initramfs with additional dracut modules.)

currently dracut only stores which modules were used to generate the image in usr/lib/dracut/modules.txt

But yes, you are right. Would be nice to save all the options and have a mechanism to regenerate it with those.

Quoted 3 lines

OR

Quoted 5 lines

Quoted 2 lines

b) Add additional modules to an existing initramfs instead of regenerating again. Is this possible at all?

You can always generate a second cpio image with additional files and either concatenate both images or specify those two images in the bootloader.

Is it possible to specify two images to bootloader for a single kernel entry? Does grub2 support it?

Quoted 6 lines

OR

unpack the initramfs, add the file, (maybe run depmod, if kernel driver added), and repack.

--
To unsubscribe from this list: send the line "unsubscribe initramfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html