kdump: Patch series for s390 support

This patch series adds kdump support for the s390 architecture (64 bit). There are a few common code changes necessary because the s390 implementation is different to other architectures in some points. Especially these common code patches (1-7) should be reviewed. Patch 8 "s390: kdump backend code" contains the s390 specific part. Patch 9 includes the necessary changes for the kexec tool.

In the following I describe the main differences of the s390 implementation:

The s390 kernel is not relocatable therefore the crashkernel memory is swapped with the area [0 - crashkernel memory] before the kdump kernel is started. Architectures other than s390 run the kdump kernel at a memory location that is disjunct to the standard location for the kernel image and to all memory that might be in use for I/O by the production system. The main reason for this seems to be that these architectures do not have a means to clear all ongoing I/O. If active memory of the production system is reused by the kdump kernel they run into memory corruption issues. On s390 with diagnose call 308 or boot (IPL) there is the possibility to stop all ongoing I/O. Therefore we can safely run the kdump kernel at the old location.

On s390 we do not create page tables for the crashkernel memory and use a memcpy_real() function to load the kdump kernel and ramdisk in kexec_load() system call.

On s390 we have external kdump triggers. For example stand-alone dump tools. The address range information of crashkernel memory is stored at a well defined storage location that can be used by the external dump triggers to find the kdump entry point. To export the address range for the crashkernel memory we introduce a new mechanism that we call meminfo. This allows to define checksum secured information in memory that is accessible via an s390 ABI defined storage address. The following information is currently stored via meminfo:
* Crashkernel memory range
* kexec segments for kdump
* Pointer to vmcoreinfo note

On Mon, Jul 04, 2011 at 07:09:22PM +0200, Michael Holzheu wrote:

I don't understand what is stand-alone dump tools and

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

Hello Vivec,

Quoted 1 line · Vivek Goyal

On Mon, Jul 04, 2011 at 07:09:22PM +0200, Michael Holzheu wrote:

[snip]

Quoted 1 line

I don't understand what is stand-alone dump tools and

S390 stand-alone dump tools are independent mini operating systems that are installed on disks or tapes. When a dump should be created, these stand-alone dump tools are booted. All that they do is to write the dump (current memory plus the CPU registers) to the disk/tape device.

The advantage compared to kdump is that since they are freshly loaded into memory they can't be overwritten in memory.

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

Also the boot process ensures that the hardware is in a initialized state.

And last but not least, with the stand-alone dump tools you can dump early kernel problems which is not possible using kdump, because you can't dump before the kdump kernel has been loaded with kexec.

That were more or less the arguments, why we did not support kdump in the past.

In order to increase dump reliability with kdump, we now implemented a two stage approach. The stand-alone dump tools first check via meminfo, if kdump is valid using checksums. If kdump is loaded and healthy it is started. Otherwise the stand-alone dump tools create a full-blown stand-alone dump.

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 6 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

* You do not have to preallocate CPU crash notes etc.

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Quoted 14 lines · Michael Holzheu

Hello Vivec,

Quoted 1 line · Vivek Goyal

On Mon, Jul 04, 2011 at 07:09:22PM +0200, Michael Holzheu wrote:

[snip]

Quoted 1 line

I don't understand what is stand-alone dump tools and

S390 stand-alone dump tools are independent mini operating systems that are installed on disks or tapes. When a dump should be created, these stand-alone dump tools are booted. All that they do is to write the dump (current memory plus the CPU registers) to the disk/tape device.

Quoted 6 lines

The advantage compared to kdump is that since they are freshly loaded into memory they can't be overwritten in memory.

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

Quoted 2 lines

Also the boot process ensures that the hardware is in a initialized state.

Who makes sure that hardware is in initiliazed state? Kdump kernel, stand alone kernel or BIOS.

Quoted 4 lines

And last but not least, with the stand-alone dump tools you can dump early kernel problems which is not possible using kdump, because you can't dump before the kdump kernel has been loaded with kexec.

That is one limitation but again if your kernel can't even boot, it is not ready to ship and it is more of a development issue and there are other ways to debug problems. So I would not worry too much about it.

On a side note, few months back there were folks who were trying to enhance bootloaders to be able to prepare basic environment so that a kdump kernel can boot even in the event of early first kernel boot.

Quoted 8 lines

That were more or less the arguments, why we did not support kdump in the past.

In order to increase dump reliability with kdump, we now implemented a two stage approach. The stand-alone dump tools first check via meminfo, if kdump is valid using checksums. If kdump is loaded and healthy it is started. Otherwise the stand-alone dump tools create a full-blown stand-alone dump.

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel. So this additional meminfo strucutres and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Quoted 12 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 6 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

Quoted 3 lines

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Quoted 4 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

Quoted 1 line

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

When do you load kdump kernel and who does it?

Who gets the control first after crash?

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

Do you load kdump kenrel from some tape/storage after system crash. Where does bootloader lies and how do you make sure it is not corrupted and associated device is in good condition.

To me we should not create a arch specific way of passing information between kernels. Stand alone kernel should be able to parse the ELF headers which contains all the relevant info. They have already been checksum verified.

Quoted 7 lines

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel. So this additional meminfo strucutres and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Yes, but what do you do if the checksum tells you that the kexec kernel has been compromised? If the independent stand-alone dumper does the check it can fall back to the "dump-all" case.

Quoted 5 lines

Quoted 5 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

Quoted 4 lines

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Again, what do you do if the verification fails? Fail to dump the borked system? Imho not a good option.

Quoted 6 lines

Quoted 5 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Quoted 3 lines

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

We do not move away from the ELF interface, we just create the ELF headers at a different time, no?

Quoted 5 lines

Quoted 2 lines

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Probably doesn't matter.

Quoted 6 lines

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Quoted 3 lines

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

Before we started working on kdump the only way to get a dump is to boot a stand-alone dumper. That is a small piece of assembler code that is loaded into the first 64KB of memory (which is reserved for these kind of things). This assembler code will then write everything to the dump device. This works very reliable (which is of utmost importance to us) but has the problem that it will be awfully slow for large memory sizes.

Quoted 2 lines

When do you load kdump kernel and who does it?

If the crashed kernel is still operational enough to call panic it can cause an IPL to the stand-alone dump tool (or do a reset of the I/O subsystem and directly call kdump with the new code if the checksums turn out ok).
If the crashed kernel is totally bust then the administrator has to do a manual IPL from the disk where the stand-alone dumper has been installed.

Quoted 1 line

Who gets the control first after crash?

Depends. If the kernel can recognize the crash as such it can proceed to execute the configured "on_panic" shutdown action. If the kernel is bust the code loaded by the next IPL gets control. This can be a "normal" boot or a stand-alone dumper.

Quoted 3 lines

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

It is the other way round. We load the standalone dumper, then check if the kdump kernel looks good. Only if all the checksums turn out ok we jump to the purgatory code from the standalone dump code.

Quoted 4 lines

Do you load kdump kenrel from some tape/storage after system crash. Where does bootloader lies and how do you make sure it is not corrupted and associated device is in good condition.

The bootloader sits on the boot disk / tape. If you are able to boot from that device then it is reasonable to assume that the device is in good condition. To get a corrupted bootloader you'd need a stray I/O to that device. The stand-alone dumper sits on its own disk / tape which is not in use for normal operation. Very unlikely that this device will get hit.

Quoted 4 lines

To me we should not create a arch specific way of passing information between kernels. Stand alone kernel should be able to parse the ELF headers which contains all the relevant info. They have already been checksum verified.

Ok, so this seems to be the main point of discussion. When to create the ELF headers and how to pass all the required information from the crashed system to the kdump kernel.

On Fri, Jul 08, 2011 at 11:01:21AM +0200, Martin Schwidefsky wrote:

[..]

Quoted 8 lines

Quoted 7 lines

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel. So this additional meminfo strucutres and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Quoted 3 lines

Yes, but what do you do if the checksum tells you that the kexec kernel has been compromised? If the independent stand-alone dumper does the check it can fall back to the "dump-all" case.

So this independent dump (which takes the decision whether to continue to boot kdump kernel or stand alone dumper) is loaded where? On x86, every thing is loaded in crashkernel memory and at run time we update purgatory with entry point of kernel.

I guess you could write s390 specific purgatory code where you do the checksum on loaded kdump kernel and if it corrupted, then you can continue to jump to boot stand alone kernel.

BTW, you seem to have capability of doing IPL of stand alone kernel from disk/tape after kernel crash. If yes, then why not IPL the regular linux kernel in case its copy in memory is corrupted.

What happens if kdump kernel is not corrupted and later it fails to boot due to some platform issue or device driver issue etc? I am assuming that dump capture will fail. If yes, then backup mechanism is designed only to protect against kdump kernel's corruption while loaded in memory?

In Michael's doc, I noticed he talked about unmapping the crashkernel memory so that kernel. That should protect against kernel but he mentioned about the possibility of device being able to DMA to said memory reason. I am wondering that is it possible to program IOMMU in such a way that any DMA attempt to said memory reason fails. If yes, then I guess corruption problem will be solved without one being worried about crating a backup plan for stand alone kernel and one can just focus on making kdump kernel work.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

So reliability only comes from the fact that stand alone kernel is booted from the disk? So as long as kdump kernel is not corrupted, it is as realiable as stand alone kernel?

How many a time in practice we have run into kdump kernel corruption issues? Will unmapping from kernel page tables and doing something at IOMMU level not take care of that issue?

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

kexec-tools purgatory code already has the checksum logic. So you don't have to redo that in stand alone tools. I think you probably need to s390 specic purgatory and jump to IPLing stand alone kernel if kdump kernel is corrupted instead of rebooting back or spinning infinitely in the loop/

Quoted 7 lines

Quoted 4 lines

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Again, what do you do if the verification fails? Fail to dump the borked system? Imho not a good option.

On regular systems we did not have any backup plan so IIRC, we spin in infinite loop.

If one can do something about it, fine. But this again takes me back to original question, then instead of creating backup plan, why not IPL the kdump kernel from disk/tape the way you do for stand alone kernels.

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Quoted 6 lines

Quoted 3 lines

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

We do not move away from the ELF interface, we just create the ELF headers at a different time, no?

Existing kernel already provides a way to communicate relevant information to new kernel/binary about the first kernel and that is through ELF. You are moving away from that and creating one more interface, meminfo to get all the info about first kernel. What's wrong with continue parsing ELF to get all the needed info. Is there any piece of information missing which you require?

Quoted 14 lines

Quoted 5 lines

Quoted 2 lines

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Probably doesn't matter.

Quoted 6 lines

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

So looks like you are loading two kernels at a time. One primary kernel and other kernel in crashkernel memory area. But that would solve only early crash dump problem and not the corruption problem?

I think we are trying to solve multiple problems at one go. We want the regular capability to boot a kdump kernel and also solve the problem of eary boot crash.

Why not solve the bigger problem in first step (and that is capturing filtered dump of big RAM systems fast) and do the integration with regular kexec-tools (create ELF headers etc) and s390 specific purgatory code.

Once all this is done, then you can look at how to capture early kernel crashes (if it turns out to be a real problem).

Quoted 10 lines

Quoted 3 lines

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

Before we started working on kdump the only way to get a dump is to boot a stand-alone dumper. That is a small piece of assembler code that is loaded into the first 64KB of memory (which is reserved for these kind of things). This assembler code will then write everything to the dump device. This works very reliable (which is of utmost importance to us) but has the problem that it will be awfully slow for large memory sizes.

When and who loads this assembler code into memory and how do we make sure this code is not corrupted.

I got the part about being slow because you have to write specific drivers for saving dump and you don't have filtering capabilty. In today's big memory systems it makes sense to reuse kdump's capability to use first kernel's drivers and filtering in user space.

Quoted 17 lines

Quoted 2 lines

When do you load kdump kernel and who does it?

If the crashed kernel is still operational enough to call panic it can cause an IPL to the stand-alone dump tool (or do a reset of the I/O subsystem and directly call kdump with the new code if the checksums turn out ok).
If the crashed kernel is totally bust then the administrator has to do a manual IPL from the disk where the stand-alone dumper has been installed.

Quoted 1 line

Who gets the control first after crash?

Depends. If the kernel can recognize the crash as such it can proceed to execute the configured "on_panic" shutdown action. If the kernel is bust the code loaded by the next IPL gets control. This can be a "normal" boot or a stand-alone dumper.

Quoted 7 lines

Quoted 3 lines

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

It is the other way round. We load the standalone dumper, then check if the kdump kernel looks good. Only if all the checksums turn out ok we jump to the purgatory code from the standalone dump code.

Ok. So again why not reuse the checksump capability of kexec-tools and instead of infinite looping you can jump to stand alone tools + IPL etc. I understand this will require a tighter integration with kexec-tools and using ELF header mechanism and will not cover the early kernel crashes.

Quoted 11 lines

Quoted 4 lines

Do you load kdump kenrel from some tape/storage after system crash. Where does bootloader lies and how do you make sure it is not corrupted and associated device is in good condition.

The bootloader sits on the boot disk / tape. If you are able to boot from that device then it is reasonable to assume that the device is in good condition. To get a corrupted bootloader you'd need a stray I/O to that device. The stand-alone dumper sits on its own disk / tape which is not in use for normal operation. Very unlikely that this device will get hit.

Quoted 8 lines

Quoted 4 lines

To me we should not create a arch specific way of passing information between kernels. Stand alone kernel should be able to parse the ELF headers which contains all the relevant info. They have already been checksum verified.

Ok, so this seems to be the main point of discussion. When to create the ELF headers and how to pass all the required information from the crashed system to the kdump kernel.

To me we seem to be diverging a lot from existing kdump+kexec-tools mechanism just to solve the case of early crash dumping. If we break down the problem in two parts and do thing kexec-tools way (with a backup path of booting stand alone kernel if kdump kenrel is corrupted), things might be better.

Quoted 5 lines

kexec-tools purgatory code already has the checksum logic. So you don't have to redo that in stand alone tools. I think you probably need to s390 specic purgatory and jump to IPLing stand alone kernel if kdump kernel is corrupted instead of rebooting back or spinning infinitely in the loop/

I can not quite follow you here. The purgatory code is part of the kdump kernel, no? When we trigger a dump with the stand-alone tools we will start executing code in the assembler function of that stand-alone tools. We can not trust the kdump kernel yet, not without doing the checksums first.

Quoted 5 lines

Ok. So again why not reuse the checksump capability of kexec-tools and instead of infinite looping you can jump to stand alone tools + IPL etc. I understand this will require a tighter integration with kexec-tools and using ELF header mechanism and will not cover the early kernel crashes.

Imho the checksum of kexec-tools is in the wrong place.

Quoted 5 lines

To me we seem to be diverging a lot from existing kdump+kexec-tools mechanism just to solve the case of early crash dumping. If we break down the problem in two parts and do thing kexec-tools way (with a backup path of booting stand alone kernel if kdump kenrel is corrupted), things might be better.

The "backup path of booting stand alone kernel" would result in passing the control twice, once from the stand-alone dumper to the kexec purgatory (after the purgatory checksum has been verified), then doing more checks in the kdump kernel, only to return to the stand-alone dumper if some check fails. Does not really sound enticing to me.

On Mon, Jul 11, 2011 at 05:56:26PM +0200, Martin Schwidefsky wrote:

[..]

Quoted 6 lines

Quoted 5 lines

kexec-tools purgatory code already has the checksum logic. So you don't have to redo that in stand alone tools. I think you probably need to s390 specic purgatory and jump to IPLing stand alone kernel if kdump kernel is corrupted instead of rebooting back or spinning infinitely in the loop/

Quoted 4 lines

I can not quite follow you here. The purgatory code is part of the kdump kernel, no? When we trigger a dump with the stand-alone tools we will start executing code in the assembler function of that stand-alone tools. We can not trust the kdump kernel yet, not without doing the checksums first.

Purgatory is another piece of binary code which is loaded along with kdump kernel in reserved memory area. So yes, there is a chance that this code itself get corrupted.

So in case of stand alone dump, you save the calculated checksum of kdump kernel at disk and not in memory? And then calculate the checksum of memory image of kdump kernel and decide whether kdump kenrel is corrupted or not?

If yes, this sounds more reliable as checksum of kernel is stored on some disk/tape.

[..]

Quoted 7 lines

Quoted 5 lines

Ok. So again why not reuse the checksump capability of kexec-tools and instead of infinite looping you can jump to stand alone tools + IPL etc. I understand this will require a tighter integration with kexec-tools and using ELF header mechanism and will not cover the early kernel crashes.

Imho the checksum of kexec-tools is in the wrong place.

Because you think that stored checksum can get corrupted?

[..]

Quoted 11 lines

Quoted 5 lines

To me we seem to be diverging a lot from existing kdump+kexec-tools mechanism just to solve the case of early crash dumping. If we break down the problem in two parts and do thing kexec-tools way (with a backup path of booting stand alone kernel if kdump kenrel is corrupted), things might be better.

The "backup path of booting stand alone kernel" would result in passing the control twice, once from the stand-alone dumper to the kexec purgatory (after the purgatory checksum has been verified), then doing more checks in the kdump kernel, only to return to the stand-alone dumper if some check fails. Does not really sound enticing to me.

What I am suggesting is that stand alone dumper gets control only if kdump kernel is corrupted.

So following sequence.

Kernel Crash ---> purgatory --> either kdump kenrel/IPL stand alone tools

Here only drawback seems to be that we assume that purgatory code and pre-calculated checksum has not been corrupted. The big advantage is that s390 kdump support looks very similar to other arches and understaning and supporting kdump across architectures becomes easy.

Quoted 5 lines · On Wed, 13 Jul 2011 12:02:39 -0400 Vivek Goyal <[email protected]> wrote:

So in case of stand alone dump, you save the calculated checksum of kdump kernel at disk and not in memory? And then calculate the checksum of memory image of kdump kernel and decide whether kdump kenrel is corrupted or not?

Quoted 2 lines

If yes, this sounds more reliable as checksum of kernel is stored on some disk/tape.

No, the checksum for the purgatory code is stored in memory. If the purgatory code is corrupted you would have to corrupt the checksum in a very specific way as well to make it fail.

Quoted 7 lines · Martin Schwidefsky

Quoted 5 lines · On Wed, 13 Jul 2011 12:02:39 -0400 Vivek Goyal <[email protected]> wrote:

So in case of stand alone dump, you save the calculated checksum of kdump kernel at disk and not in memory? And then calculate the checksum of memory image of kdump kernel and decide whether kdump kenrel is corrupted or not?

Quoted 6 lines

Quoted 2 lines

If yes, this sounds more reliable as checksum of kernel is stored on some disk/tape.

No, the checksum for the purgatory code is stored in memory. If the purgatory code is corrupted you would have to corrupt the checksum in a very specific way as well to make it fail.

Currently we store the checksums for the loaded *kexec segments* in memory at the end of kexec_load(). The stand-alone dump tools also calculate the checksums for all segments and compare them with the stored checksums. The dump tools can do that because we have meminfos for all segments. A meminfo element contains:
* address of memory chunk
* size of memory chunk
* checksum of memory chunk (calculated at the end of kexec_load()

Quoted 11 lines

What I am suggesting is that stand alone dumper gets control only if kdump kernel is corrupted.

So following sequence.

Kernel Crash ---> purgatory --> either kdump kenrel/IPL stand alone tools

Here only drawback seems to be that we assume that purgatory code and pre-calculated checksum has not been corrupted. The big advantage is that s390 kdump support looks very similar to other arches and understaning and supporting kdump across architectures becomes easy.

My problem with that is the following: how do we get from the "Kernel Crash" step to the purgatory code? It does work for "normal" panics, but it fails miserably for a hard crash that does not even get as far as panic. That is why we insist on a possible second order of things:

On Wed, Jul 13, 2011 at 06:46:11PM +0200, Martin Schwidefsky wrote:

[..]

Quoted 12 lines

Quoted 11 lines

What I am suggesting is that stand alone dumper gets control only if kdump kernel is corrupted.

So following sequence.

Kernel Crash ---> purgatory --> either kdump kenrel/IPL stand alone tools

Here only drawback seems to be that we assume that purgatory code and pre-calculated checksum has not been corrupted. The big advantage is that s390 kdump support looks very similar to other arches and understaning and supporting kdump across architectures becomes easy.

Quoted 4 lines

My problem with that is the following: how do we get from the "Kernel Crash" step to the purgatory code? It does work for "normal" panics, but it fails miserably for a hard crash that does not even get as far as panic. That is why we insist on a possible second order of things:

What is hard crash? How does that happen and what does x86 and s390 do in that case?

Though I don't have details but your argument seems to be that in s390 we are always guranteed that we will jump to IPLing the stand alone tools code irresepective of the system state hence it is relatively safer to do checks in stand alone tools instead of purgatory where code is in memory.

If due to hard hang, code can not even make to purgatory, where would it go? Can't we do IPLing of stand alone tool then.

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

This will also get rid of requirement passing all the segment and cheksum info to stand alone tool with the help of meminfo (That's another sore point).

Bottom line, even if you can't make to purgatory reliably, you always have the option of capturing dump manually using stand alone tools. We don't have to mix up kdump and stand alone mechanism. If kdump fails, we just need to have capability to still capture the dump using stand alone tools manually. I think that will make things simpler even for stand alone tools.

Quoted 20 lines · On Wed, 13 Jul 2011 16:00:04 -0400 Vivek Goyal <[email protected]> wrote:

On Wed, Jul 13, 2011 at 06:46:11PM +0200, Martin Schwidefsky wrote:

[..]

Quoted 16 lines

Quoted 11 lines

What I am suggesting is that stand alone dumper gets control only if kdump kernel is corrupted.

So following sequence.

Kernel Crash ---> purgatory --> either kdump kenrel/IPL stand alone tools

Here only drawback seems to be that we assume that purgatory code and pre-calculated checksum has not been corrupted. The big advantage is that s390 kdump support looks very similar to other arches and understaning and supporting kdump across architectures becomes easy.

My problem with that is the following: how do we get from the "Kernel Crash" step to the purgatory code? It does work for "normal" panics, but it fails miserably for a hard crash that does not even get as far as panic. That is why we insist on a possible second order of things:

Quoted 2 lines

What is hard crash? How does that happen and what does x86 and s390 do in that case?

E.g. an endless loop with interrupts disabled. To get out of this situation we will IPL/boot a new system. That is either the production system itself or the stand-alone dump tool.

Quoted 6 lines

Though I don't have details but your argument seems to be that in s390 we are always guranteed that we will jump to IPLing the stand alone tools code irresepective of the system state hence it is relatively safer to do checks in stand alone tools instead of purgatory where code is in memory.

Now you got it. That is the crux of the argument.

Quoted 2 lines

If due to hard hang, code can not even make to purgatory, where would it go? Can't we do IPLing of stand alone tool then.

It doesn't go anywhere. Basically the system is manually stopped and restarted. But on s390 we can still get to all the required information to generated a dump. That is one of the major differences to x86, if you have to do a restart the registers on x86 will be gone, no?

Quoted 4 lines

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

How are we suddenly on the purgatory path again? The code that gets control in case of a hard crash + IPL is the stand-alone dump tool, not the purgatory code.

The first thing we want to do is to check if
the purgatory is still fine, that is do a checksum. If we have the infrastructure in place to do one checksum then we can easily do the other checksums as well.

Quoted 4 lines

This will also get rid of requirement passing all the segment and cheksum info to stand alone tool with the help of meminfo (That's another sore point).

No, it doesn't. We will still need to do the checksum for the purgatory code and we already have the re-ipl information which won't go away.

Quoted 7 lines

Bottom line, even if you can't make to purgatory reliably, you always have the option of capturing dump manually using stand alone tools. We don't have to mix up kdump and stand alone mechanism. If kdump fails, we just need to have capability to still capture the dump using stand alone tools manually. I think that will make things simpler even for stand alone tools.

If we decide not to mix kdump and stand-alone dump then we loose something. Consider a hard crash where the kdump segments are still intact. What our customers do in that case is to start the stand-alone dump utility. Without a way to find and verify the kdump setup we would have to do a full dump. Which will take its time if the memory size is big. See?

Quoted 8 lines

Quoted 4 lines

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

How are we suddenly on the purgatory path again? The code that gets control in case of a hard crash + IPL is the stand-alone dump tool, not the purgatory code.

I think that's the biggest contetion point. From the start of discussion you have this hardcoded requirement that the moment panic() happens you are jumping to some IPL code and that's what I am questioning. Why can't you execute some more code after panic() (purgatory), before you jump to IPL code (only if you have to).

On Thu, Jul 14, 2011 at 01:55:32PM -0400, Vivek Goyal wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

How are we suddenly on the purgatory path again? The code that gets control in case of a hard crash + IPL is the stand-alone dump tool, not the purgatory code.

Quoted 6 lines

I think that's the biggest contetion point. From the start of discussion you have this hardcoded requirement that the moment panic() happens you are jumping to some IPL code and that's what I am questioning. Why can't you execute some more code after panic() (purgatory), before you jump to IPL code (only if you have to).

In your parlance of shutdown actions, I think it is equivalnet to saying that "kdump" is a shutdown action and that that means is that for specific trigger points we will execute "crash_kexec()" function which will try to capture the dump.

In user space I think one can modify the kexec-tools infrastrucuture a bit so that one is able to define an entry point in case checksum of loaded segment failes. Once you are loding kdump kernel, you can define that entry point. (And this would be jump to IPL etc.).

Quoted 24 lines · Vivek Goyal

On Thu, Jul 14, 2011 at 01:55:32PM -0400, Vivek Goyal wrote:

[..]

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

How are we suddenly on the purgatory path again? The code that gets control in case of a hard crash + IPL is the stand-alone dump tool, not the purgatory code.

Quoted 6 lines

I think that's the biggest contetion point. From the start of discussion you have this hardcoded requirement that the moment panic() happens you are jumping to some IPL code and that's what I am questioning. Why can't you execute some more code after panic() (purgatory), before you jump to IPL code (only if you have to).

In your parlance of shutdown actions, I think it is equivalnet to saying that "kdump" is a shutdown action and that that means is that for specific trigger points we will execute "crash_kexec()" function which will try to capture the dump.

Quoted 4 lines

In user space I think one can modify the kexec-tools infrastrucuture a bit so that one is able to define an entry point in case checksum of loaded segment failes. Once you are loding kdump kernel, you can define that entry point. (And this would be jump to IPL etc.).

You mean to jump back into the crashed kernel code in case the kdump checksum failed?

In the meantime I was looking a bit more into the kexec code to find out, what we would have to do, if we use the preallocated ELF header as you want us to do. With our actual solution, we do not have to reserve any special areas for the kdump kernel. Now we have to reserve the ELF header. So what are the options?

The x86 implementation uses a kernel parameter "memmap=exactmap" to do that.

On ia64 - if I understood the code correctly - they seem to pass a kdump segment "EFI_memmap" to the kdump kernel that contains information about all loaded kexec segments. With this segment they can find out the size of the ELF header segment in the kdump kernel and then do the memory reservation at boot time. Is that correct?

Quoted 5 lines

Quoted 4 lines

In user space I think one can modify the kexec-tools infrastrucuture a bit so that one is able to define an entry point in case checksum of loaded segment failes. Once you are loding kdump kernel, you can define that entry point. (And this would be jump to IPL etc.).

Quoted 2 lines

You mean to jump back into the crashed kernel code in case the kdump checksum failed?

No. I meant jump to entry point so that one can IPL the dump tools. I am not sure how do initiate the IPL after panic. Similar thing needs to be done here. If it is as simple as jumping to some location in low memory, then purgatory should be able to do that. I think we shall have to figure out the details here.

Basically I am saying that purgatory detected that kdump kernel is corrupted. In x86_64 we spin in inifinite loop as we don't have a backup plan. But s390 has a backup plan of being able to IPL dump tools.

Or in first step we can keep it even simpler. We can spin in infinite loop

and wait for either hypervisor watchdog to kick in for automatic IPL or wait for operator intervention.
That would simplify it even
further.

Quoted 1 line

Quoted 5 lines

In the meantime I was looking a bit more into the kexec code to find out, what we would have to do, if we use the preallocated ELF header as you want us to do. With our actual solution, we do not have to reserve any special areas for the kdump kernel. Now we have to reserve the ELF header. So what are the options?

ELF headers go into same memory area as kdump kenrel.

Anyway you are
doing to reserve memory for kdump kernel and ELF headers will go right there.
Once you swap the kernel I think ELF headers continue to remain in original location. Or may be you can move ELF headers too depending on what turns out to be easier.

Quoted 1 line

Quoted 2 lines

The x86 implementation uses a kernel parameter "memmap=exactmap" to do that.

It tells second kernel to use a memory map defined on command line. Kexec-tools prepares this memory map with the help of memap= options. This is to limit the memory second kernel use to boot into so that it does not overwrite in any piece of memory used by first kernel.

In your case I think you shall have to do little more so that second kernel also seems some of the lower memory areas so that later swapping of kernel can be done.

Quoted 7 lines

On ia64 - if I understood the code correctly - they seem to pass a kdump segment "EFI_memmap" to the kdump kernel that contains information about all loaded kexec segments. With this segment they can find out the size of the ELF header segment in the kdump kernel and then do the memory reservation at boot time. Is that correct?

Sorry, I don't know the details of IA64. May be somebody else on the list can pitch in with some clarifications here.

Hello Vivek,

Quoted 5 lines · Vivek Goyal

Quoted 5 lines

Quoted 4 lines

In user space I think one can modify the kexec-tools infrastrucuture a bit so that one is able to define an entry point in case checksum of loaded segment failes. Once you are loding kdump kernel, you can define that entry point. (And this would be jump to IPL etc.).

Quoted 8 lines

Quoted 2 lines

You mean to jump back into the crashed kernel code in case the kdump checksum failed?

No. I meant jump to entry point so that one can IPL the dump tools. I am not sure how do initiate the IPL after panic. Similar thing needs to be done here. If it is as simple as jumping to some location in low memory, then purgatory should be able to do that. I think we shall have to figure out the details here.

We have a machine instruction to IPL a dump tool from a device. The parameters (e.g. device number, or WWPN/LUN for SCSI devices) are currently configured via a s390 sysfs interface and an etc config file. In theory we could read the sysfs files or the config file from the kexec tool and patch the parameters into the purgatory code. The user would then have to restart kexec each time when the configuration is changed.

Quoted 8 lines

Basically I am saying that purgatory detected that kdump kernel is corrupted. In x86_64 we spin in inifinite loop as we don't have a backup plan. But s390 has a backup plan of being able to IPL dump tools.

Or in first step we can keep it even simpler. We can spin in infinite loop

Looping is probably not a good option in a hypervisor environment like we have it on s390. At least we should load a disabled wait PSW.

Quoted 6 lines

and wait for either hypervisor watchdog to kick in for automatic IPL or wait for operator intervention.
That would simplify it even
further.

Quoted 1 line

Quoted 7 lines

Quoted 5 lines

In the meantime I was looking a bit more into the kexec code to find out, what we would have to do, if we use the preallocated ELF header as you want us to do. With our actual solution, we do not have to reserve any special areas for the kdump kernel. Now we have to reserve the ELF header. So what are the options?

ELF headers go into same memory area as kdump kenrel.

sure

Quoted 11 lines

Anyway you are
doing to reserve memory for kdump kernel and ELF headers will go right there.
Once you swap the kernel I think ELF headers continue to remain in original location. Or may be you can move ELF headers too depending on what turns out to be easier.

Quoted 3 lines

The x86 implementation uses a kernel parameter "memmap=exactmap" to do that.

Quoted 4 lines

It tells second kernel to use a memory map defined on command line. Kexec-tools prepares this memory map with the help of memap= options. This is to limit the memory second kernel use to boot into so that it does not overwrite in any piece of memory used by first kernel.

And to reserve the ELF header that is prepared by kexec tools, no?

Quoted 4 lines

In your case I think you shall have to do little more so that second kernel also seems some of the lower memory areas so that later swapping of kernel can be done.

After the swap the ELF header is contained in the same memory than the kdump kernel. When the kdump kernel starts, the ELF header has to be saved from being overwritten (as kernel and ramdisk). I get the address from the "elfcorehdr=" kernel parameter. How will I get the size?

Looking at the ia64 and x86 implementations I have the feeling there are different mechanism available to do that.

Quoted 8 lines

Quoted 7 lines

On ia64 - if I understood the code correctly - they seem to pass a kdump segment "EFI_memmap" to the kdump kernel that contains information about all loaded kexec segments. With this segment they can find out the size of the ELF header segment in the kdump kernel and then do the memory reservation at boot time. Is that correct?

Quoted 2 lines

Sorry, I don't know the details of IA64. May be somebody else on the list can pitch in with some clarifications here.

For me it looks like a mechanism where a block of information is prepared by kexec tools and a pointer to that block is passed somehow to the second kernel. I would assume that the definition of this block is ia64 kernel ABI.

Quoted 5 lines · Michael Holzheu

Quoted 2 lines

Or in first step we can keep it even simpler. We can spin in infinite loop

Looping is probably not a good option in a hypervisor environment like we have it on s390. At least we should load a disabled wait PSW.

What is "disabled wait PSW"?

Quoted 8 lines

Quoted 3 lines

In your case I think you shall have to do little more so that second kernel also seems some of the lower memory areas so that later swapping of kernel can be done.

After the swap the ELF header is contained in the same memory than the kdump kernel. When the kdump kernel starts, the ELF header has to be saved from being overwritten (as kernel and ramdisk). I get the address from the "elfcorehdr=" kernel parameter. How will I get the size?

By parsing the ELF header. It will give you information about how many program headers and notes are there, their sizes and locations etc.

When kexec-tools loads ELF headers, it knows what's the total size of ELF headers and it removes that chunk of memory from the memory map passed to second kernel with memmap= options. IOW, some memory out of reserved region is not usable by second kernel because we have stored information in that memory. Kdump kernel maps that memory and gets to read the ELF headers.

So you shall have to do something similar where you need to tell second kernel what memory areas it can use for boot and remove ELF header memory area from the map.

Quoted 11 lines

Looking at the ia64 and x86 implementations I have the feeling there are different mechanism available to do that.

Quoted 8 lines

Quoted 7 lines

On ia64 - if I understood the code correctly - they seem to pass a kdump segment "EFI_memmap" to the kdump kernel that contains information about all loaded kexec segments. With this segment they can find out the size of the ELF header segment in the kdump kernel and then do the memory reservation at boot time. Is that correct?

Quoted 7 lines

Quoted 2 lines

Sorry, I don't know the details of IA64. May be somebody else on the list can pitch in with some clarifications here.

For me it looks like a mechanism where a block of information is prepared by kexec tools and a pointer to that block is passed somehow to the second kernel. I would assume that the definition of this block is ia64 kernel ABI.

It is possible. Even in x86, we prepare a block of information, one 4K page and fill lots of x86 boot protocol information.

Look at.

kexec-tools/include/x86/x86-linux.h
kexec-tools/kexec/arch/i386/x86-linux-setup.c

Above header information contains information about e820 memory map also and we fill that map info for normal kexec (fastboot, not kdump) also and that's how second kernel comes to know about memory map of system.

I think one could possibly truncate the same map for kdump kernel to tell second kernel about the memory to use. But IIRC, original memory map is also used to determine max_pfn present in first kernel so that in second kernel we don't try to map a memory beyond that and access it, etc. Hence it was decided to leave it that way and pass the memory map for second kernel on command line.

So its possible that IA64 is doing preparing boot protocal specific block and passing all the releavant information in that block instead of making use of commnad line.

Hello Vivek,

Quoted 8 lines · Vivek Goyal

Quoted 5 lines · Michael Holzheu

Quoted 2 lines

Or in first step we can keep it even simpler. We can spin in infinite loop

Looping is probably not a good option in a hypervisor environment like we have it on s390. At least we should load a disabled wait PSW.

What is "disabled wait PSW"?

This is a PSW where interrupts are disabled and the wait bit is on. This ensures that the virtual CPU is stopped and does not consume any CPU time.

Quoted 9 lines

Quoted 8 lines

Quoted 3 lines

In your case I think you shall have to do little more so that second kernel also seems some of the lower memory areas so that later swapping of kernel can be done.

After the swap the ELF header is contained in the same memory than the kdump kernel. When the kdump kernel starts, the ELF header has to be saved from being overwritten (as kernel and ramdisk). I get the address from the "elfcorehdr=" kernel parameter. How will I get the size?

Quoted 2 lines

By parsing the ELF header. It will give you information about how many program headers and notes are there, their sizes and locations etc.

The only thing we need is the size of the preallocated header that is in kdump memory. All other architectures seem to pass this information somehow with different mechanisms to the kdump kernel (memmap kernel parameter, boot parameters, etc.). Why should *we* parse the ELF header?

Quoted 8 lines

When kexec-tools loads ELF headers, it knows what's the total size of ELF headers and it removes that chunk of memory from the memory map passed to second kernel with memmap= options. IOW, some memory out of reserved region is not usable by second kernel because we have stored information in that memory. Kdump kernel maps that memory and gets to read the ELF headers.

Quoted 3 lines

So you shall have to do something similar where you need to tell second kernel what memory areas it can use for boot and remove ELF header memory area from the map.

So if we do that, why should we parse the ELF header?

Quoted 19 lines

It is possible. Even in x86, we prepare a block of information, one 4K page and fill lots of x86 boot protocol information.

Look at.

kexec-tools/include/x86/x86-linux.h
kexec-tools/kexec/arch/i386/x86-linux-setup.c

Above header information contains information about e820 memory map also and we fill that map info for normal kexec (fastboot, not kdump) also and that's how second kernel comes to know about memory map of system.

I think one could possibly truncate the same map for kdump kernel to tell second kernel about the memory to use. But IIRC, original memory map is also used to determine max_pfn present in first kernel so that in second kernel we don't try to map a memory beyond that and access it, etc. Hence it was decided to leave it that way and pass the memory map for second kernel on command line.

Quoted 3 lines

So its possible that IA64 is doing preparing boot protocal specific block and passing all the releavant information in that block instead of making use of commnad line.

Just to come back to your initial argumentation against our meminfo
approach: It looks like that there are already other mechanisms besides of ELF-header and kernel parameters to pass information to the kdump kernel. Where is the conceptional difference to our meminfo interface?

Quoted 7 lines

Quoted 2 lines

By parsing the ELF header. It will give you information about how many program headers and notes are there, their sizes and locations etc.

The only thing we need is the size of the preallocated header that is in kdump memory. All other architectures seem to pass this information somehow with different mechanisms to the kdump kernel (memmap kernel parameter, boot parameters, etc.). Why should *we* parse the ELF header?

ELF headers and memmap parameters are communicating two different pieces of information to second kenrel.

- memap tells what memory second kernel can use to boot.
- ELF headers tell what memory areas first kernel was using and using that information how to construct ELF headers for /proc/vmcore interface in second kernel. On x86, ELF headers also communicate where the saved cpu state is for the first kernel.

Arch independent code in kdump kenrel (fs/proc/vmcore.c) is parsing those ELF headers to export /proc/vmcore. So if you set up the headers right you get that arch independent code for free without any changes to generic code.

Quoted 12 lines

Quoted 11 lines

When kexec-tools loads ELF headers, it knows what's the total size of ELF headers and it removes that chunk of memory from the memory map passed to second kernel with memmap= options. IOW, some memory out of reserved region is not usable by second kernel because we have stored information in that memory. Kdump kernel maps that memory and gets to read the ELF headers.

So you shall have to do something similar where you need to tell second kernel what memory areas it can use for boot and remove ELF header memory area from the map.

Quoted 1 line

So if we do that, why should we parse the ELF header?

To know three things.

- Memory areas being used by first kernel.
- Cpu states at the time of crash of first kernel.
- Some config options exported by first kernel with the help of ELF notes.

fs/proc/vmcore.c already does it for you. You just need to make sure that you tell it following.

- Where to find the headers in memory (elfcorehdr=)
- A way to map that memory and access contents.

- Make sure these headers are not overwritten by newly booted kernel.

Quoted 17 lines · Vivek Goyal

Quoted 7 lines

Quoted 2 lines

By parsing the ELF header. It will give you information about how many program headers and notes are there, their sizes and locations etc.

The only thing we need is the size of the preallocated header that is in kdump memory. All other architectures seem to pass this information somehow with different mechanisms to the kdump kernel (memmap kernel parameter, boot parameters, etc.). Why should *we* parse the ELF header?

ELF headers and memmap parameters are communicating two different pieces of information to second kenrel.

- memap tells what memory second kernel can use to boot.
- ELF headers tell what memory areas first kernel was using and using that information how to construct ELF headers for /proc/vmcore interface in second kernel. On x86, ELF headers also communicate where the saved cpu state is for the first kernel.

Quoted 4 lines

Arch independent code in kdump kenrel (fs/proc/vmcore.c) is parsing those ELF headers to export /proc/vmcore. So if you set up the headers right you get that arch independent code for free without any changes to generic code.

Vivek, I know all these things. So, we (s390) do *not* have to parse the ELF header. We only have to ensure the kexec prepared ELF header is reserved until the /proc/vmcore parses it. All the ELF notes for CPUs, etc. should automatically be reserved, because they are allocated in oldmem by the old crashed kernel.

All what I was asking is how we pass best the information "size of the preallocated ELF header" to the 2nd kernel for reserving the header. We currently do not have the memmap kernel parameter.

Quoted 13 lines

Quoted 13 lines

Quoted 12 lines

When kexec-tools loads ELF headers, it knows what's the total size of ELF headers and it removes that chunk of memory from the memory map passed to second kernel with memmap= options. IOW, some memory out of reserved region is not usable by second kernel because we have stored information in that memory. Kdump kernel maps that memory and gets to read the ELF headers.

So you shall have to do something similar where you need to tell second kernel what memory areas it can use for boot and remove ELF header memory area from the map.

Quoted 7 lines

Quoted 1 line

So if we do that, why should we parse the ELF header?

To know three things.

- Memory areas being used by first kernel.
- Cpu states at the time of crash of first kernel.
- Some config options exported by first kernel with the help of ELF notes.

sure

Quoted 5 lines

fs/proc/vmcore.c already does it for you. You just need to make sure that you tell it following.

- Where to find the headers in memory (elfcorehdr=)
- A way to map that memory and access contents.

sure

Quoted 1 line

- Make sure these headers are not overwritten by newly booted kernel.

And that was my question: What is the best way to do that. E.g. we could pass a 2nd kernel parameter "elfcorehdr_size", implement s390 boot parameter or implement the memmap kernel parameter.

Quoted 6 lines · Michael Holzheu

Quoted 1 line · Vivek Goyal

- Make sure these headers are not overwritten by newly booted kernel.

And that was my question: What is the best way to do that. E.g. we could pass a 2nd kernel parameter "elfcorehdr_size", implement s390 boot parameter or implement the memmap kernel parameter.

You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Hello Vivek,

Quoted 8 lines · Vivek Goyal

Quoted 6 lines · Michael Holzheu

Quoted 1 line · Vivek Goyal

- Make sure these headers are not overwritten by newly booted kernel.

And that was my question: What is the best way to do that. E.g. we could pass a 2nd kernel parameter "elfcorehdr_size", implement s390 boot parameter or implement the memmap kernel parameter.

Quoted 6 lines

You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Ok, understood. Thanks for the information.

We still have discussions here, if we could somehow implement our original idea of triggering kdump by the stand-alone dump tools. Sorry for being so stubborn :-(

So here comes the modified suggestion:

As requested by you we can pre-allocate the ELF header and use purgatory as done on other architectures.

To allow the stand-alone dump tools as kdump triggers, we then only would have to provide an s390 specific way to tell the stand-alone dump tools:
1. Entry point address into purgatory
2. Address, size and checksum for purgatory

We could store address, size and checksum of the purgatory to a fixed offset in the kdump kernel image. This can be done in the kexec tools code.

Then the dump tools only would need the crashkernel memory offset to find all information. Then dump tools will verify purgatory and afterwards jump to the purgatory code. Then purgatory verifies all kexec segments. For s390, if this check fails, we return to caller (stand-alone tools). If the check is ok, then purgatory code on s390 saves all registers to the preallocated ELF notes and starts kdump.

I think, this is all s390 specific and IMHO will not affect other architectures at all.

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

Quoted 18 lines · Michael Holzheu

Hello Vivek,

Quoted 14 lines · Vivek Goyal

Quoted 6 lines · Michael Holzheu

Quoted 1 line · Vivek Goyal

- Make sure these headers are not overwritten by newly booted kernel.

And that was my question: What is the best way to do that. E.g. we could pass a 2nd kernel parameter "elfcorehdr_size", implement s390 boot parameter or implement the memmap kernel parameter.

You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Quoted 5 lines

Ok, understood. Thanks for the information.

We still have discussions here, if we could somehow implement our original idea of triggering kdump by the stand-alone dump tools. Sorry for being so stubborn :-(

What's the advantage of that. Why are we so stuborn about first passing the control to dump tools after panic()?

The case of purgatory corruption is no different then panic() code and associated hook code corruption.

It is a corner case and even if it gets corrupted you have other mechanisms to IPL dump tools and capture dump.

Why do you want to mix two mechanisms. What's the advantage of making even dump tools complicated and make it aware of a kernel binary object purgatory?

To me the simple interface is that there is no coupling between dump tools and kdump. If there is no coupling, then there is no need to exchange any information and no need to make any assumption about hard coded location where purgatory entry point, size and checksums are stored.

Quoted 12 lines

So here comes the modified suggestion:

As requested by you we can pre-allocate the ELF header and use purgatory as done on other architectures.

To allow the stand-alone dump tools as kdump triggers, we then only would have to provide an s390 specific way to tell the stand-alone dump tools:
1. Entry point address into purgatory
2. Address, size and checksum for purgatory

Quoted 3 lines

We could store address, size and checksum of the purgatory to a fixed offset in the kdump kernel image. This can be done in the kexec tools code.

I think this will require kernel changes also? Otherwise how would you store variables in kernel address space.

Secondly, if the goal is to just be able to checksum purgatory also, then it probably should be done in a generic mannner so that kernel could checksum purgatory before jumping to it.

Quoted 6 lines

Then the dump tools only would need the crashkernel memory offset to find all information. Then dump tools will verify purgatory and afterwards jump to the purgatory code. Then purgatory verifies all kexec segments. For s390, if this check fails, we return to caller (stand-alone tools). If the check is ok, then purgatory code on s390 saves all registers to the preallocated ELF notes and starts kdump.

So far I really don't think that there is any need of involving dump tools here. By making it a requirement we are just making the design complex with no gains.

Quoted 10 lines

I think, this is all s390 specific and IMHO will not affect other architectures at all.

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

when would you save vmcoreinfo? I guess I shall have to look at the patches.

On Mon, Jul 18, 2011 at 08:03:08PM +0200, Michael Holzheu wrote:

Quoted 6 lines

To allow the stand-alone dump tools as kdump triggers, we then only would have to provide an s390 specific way to tell the stand-alone dump tools:
1. Entry point address into purgatory
2. Address, size and checksum for purgatory

Quoted 3 lines

We could store address, size and checksum of the purgatory to a fixed offset in the kdump kernel image. This can be done in the kexec tools code.

I think this will require kernel changes also? Otherwise how would you store variables in kernel address space.

Quoted 6 lines

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

when would you save vmcoreinfo? I guess I shall have to look at the patches.

Quoted 7 lines

Quoted 6 lines

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

Quoted 2 lines

when would you save vmcoreinfo? I guess I shall have to look at the patches.

patch #4: We have to save vmcoreinfo at startup time of the production kernel.

Perhaps it would be best to send you a new patch series with our latest proposal.

Hello Vivek,

Quoted 8 lines · Vivek Goyal

Quoted 6 lines · Michael Holzheu

Quoted 1 line · Vivek Goyal

- Make sure these headers are not overwritten by newly booted kernel.

And that was my question: What is the best way to do that. E.g. we could pass a 2nd kernel parameter "elfcorehdr_size", implement s390 boot parameter or implement the memmap kernel parameter.

Quoted 6 lines

You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Ok, understood. Thanks for the information.

We still have discussions here, if we could somehow implement our original idea of triggering kdump by the stand-alone dump tools. Sorry for being so stubborn :-(

So here comes the modified suggestion:

As requested by you we can pre-allocate the ELF header and use purgatory as done on other architectures.

To allow the stand-alone dump tools as kdump triggers, we then only would have to provide an s390 specific way to tell the stand-alone dump tools:
1. Entry point address into purgatory
2. Address, size and checksum for purgatory

We could store address, size and checksum of the purgatory to a fixed offset in the kdump kernel image. This can be done in the kexec tools code. Then the dump tools only would need the crashkernel memory offset to find all information. Then dump tools will verify purgatory and afterwards jump to the purgatory code. Then purgatory verifies all kexec segments. For s390, if this check fails, we return to caller (stand-alone tools). If the check is ok, then purgatory code on s390 saves all registers to the preallocated ELF notes and starts kdump.

I think, this is all s390 specific and IMHO will not affect other architectures at all.

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

On Mon, Jul 18, 2011 at 08:03:08PM +0200, Michael Holzheu wrote:

Quoted 6 lines

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

Hi Michael,

Instead of introdcuing a new entry point for second kernel, why not jump to crash_kexec() from stand alone tools? That should be functionally equivalent to what you described above without any need to pass the purgatory details to stand alone tools.

Only thing which needs to be figured out is how to pass the address of crash_kexec() to stand alone tools and set registers/parameters appropriately.

Hello Vivek,

Quoted 1 line · Vivek Goyal

On Mon, Jul 18, 2011 at 08:03:08PM +0200, Michael Holzheu wrote:

[snip]

Quoted 7 lines

Quoted 6 lines

What you as kdump framework maintainer would have to accept with this solution is that it is allowed now to start kdump directly via purgatory without using code from the old kernel (e.g. crash_kexec). This has as implication that all things that the old kernel has to initialize for kdump has to be done before the system crashes. Currently this is only the initialization of vmcoreinfo.

Quoted 6 lines

Hi Michael,

Instead of introdcuing a new entry point for second kernel, why not jump to crash_kexec() from stand alone tools? That should be functionally equivalent to what you described above without any need to pass the purgatory details to stand alone tools.

That has the drawback that we still execute unchecked code from the crashed kernel. But ...

... I discussed that with Martin and we had an idea how to deal with this problem. On s390 when an invalid opcode is executed or invalid parameters are used, we get a program check interrupt. When the crash_kexec() code path or data is corrupted, it is almost sure that we get a program check. The stand-alone dump tools could establish a program check interrupt handler that jumps back to the dump tools code and then create full-blown dump.

So I think with this mechanism we could use an entry point (name it s390_kdump_entry) in the old kernel that calls crash_kexec().

We would change the purgatory code that for s390 it returns to the caller, if the checksum test fails. This *requires* that s390_kdump_entry()->crash_kexec()->machine_kexec() is allowed to return. Currently this is the case.

Quoted 4 lines

Only thing which needs to be figured out is how to pass the address of crash_kexec() to stand alone tools and set registers/parameters appropriately.

We could do this s390 specific (e.g. using meminfo). In this case this would only be used for kernel/dump tools communication and not for kernel/kernel communication. So I hope this should not be a problem for you.

Then the design would look like the following:
* Define s390_kdump_entry in old kernel that calls crash_kexec()
* Use preallocated ELF core header
* s390_kdump_entry code path stores registers to ELF notes, ...

* ... and finally jumps to purgatory code
* For s390 the purgatory code returns to caller in case of checksum failure
* dump tools call s390_kdump_entry with program check handler for error handling

I think, if we do it that way, we do not affect the current kdump framework at all.

Quoted 4 lines · Michael Holzheu

We would change the purgatory code that for s390 it returns to the caller, if the checksum test fails. This *requires* that s390_kdump_entry()->crash_kexec()->machine_kexec() is allowed to return. Currently this is the case.

Can we directly jump to entry point of stand alone dump tools from purgaotry if checksum fails? We should be able to set this entry point in user space while loading kdump kernel.

Quoted 8 lines

Quoted 3 lines

Only thing which needs to be figured out is how to pass the address of crash_kexec() to stand alone tools and set registers/parameters appropriately.

We could do this s390 specific (e.g. using meminfo). In this case this would only be used for kernel/dump tools communication and not for kernel/kernel communication. So I hope this should not be a problem for you.

So you will be preparing a block/segment of data (called meminfo, though this name does not make much sense anymore), and pass it to second kernel? All done in user space and no first kernel involvement?

I am trying to remember the details that how do you tell second kernel where this this data block is. I recall that last time you said something about setting this in kernel in kexec-tools but I did not understand it.

Quoted 5 lines

Then the design would look like the following:
* Define s390_kdump_entry in old kernel that calls crash_kexec()
* Use preallocated ELF core header
* s390_kdump_entry code path stores registers to ELF notes, ...

crash_kexec() -> crash_setup_regs() already does that. We just need to define an s390 specific crash_setup_regs().

Quoted 6 lines

* ... and finally jumps to purgatory code
* For s390 the purgatory code returns to caller in case of checksum failure
* dump tools call s390_kdump_entry with program check handler for error handling

I thought that program check handler will call something else and not s390_kdump_entry()? Because program check handler is supposed to hit when any of the code we are executing is corrupted and we can not jump to kdump tool any more. Otherwise we will be nesting.

Quoted 3 lines

I think, if we do it that way, we do not affect the current kdump framework at all.

Can you give some more details about various code flows and entry points. Like panic() path, hard hang path. From your mail it sounds that even with program check handler, after panic() you would like to jump to stand alone tools first and then call s390_kdump_entry(). I think that should not be required any more as you are not doing any checksumming in dump tools anymore?

Quoted 6 lines

Quoted 5 lines

Then the design would look like the following:
* Define s390_kdump_entry in old kernel that calls crash_kexec()
* Use preallocated ELF core header
* s390_kdump_entry code path stores registers to ELF notes, ...

Quoted 2 lines

crash_kexec() -> crash_setup_regs() already does that. We just need to define an s390 specific crash_setup_regs().

I looked at the code. x86 seems to store only registers for current CPU. Where are all other CPUs stored? ia64 has an empty implementation. Where are registers stored there?

Quoted 8 lines

Quoted 7 lines

* ... and finally jumps to purgatory code
* For s390 the purgatory code returns to caller in case of checksum failure
* dump tools call s390_kdump_entry with program check handler for error handling

Quoted 4 lines

I thought that program check handler will call something else and not s390_kdump_entry()? Because program check handler is supposed to hit when any of the code we are executing is corrupted and we can not jump to kdump tool any more. Otherwise we will be nesting.

Looks like the sentence was misleading. What I wanted to say is:
* First dump tools setup program check handler that jumps back to dump tool in case kdump fails
* Then dump tools call s390_dump_entry

Quoted 10 lines

Quoted 3 lines

I think, if we do it that way, we do not affect the current kdump framework at all.

Can you give some more details about various code flows and entry points. Like panic() path, hard hang path. From your mail it sounds that even with program check handler, after panic() you would like to jump to stand alone tools first and then call s390_kdump_entry(). I think that should not be required any more as you are not doing any checksumming in dump tools anymore?

Ok some code flows:

Generally we have the flow:
* crash_kexec -> machine_kexec -> purgatory -> kdump

crash_kexec can be entered by e.g.:
* panic -> kdump shutdown action -> crash_kexec
* panic -> s390 dump shutdown action -> auto IPL dump tool -> s390_kdump_entry -> crash_kexec

* hard hang -> manual IPL dump tool -> s390_kdump_entry -> crash_kexec

Handling for corrupted kdump:

New idea for returning to dump tools in case of program check:
We could force a program check for s390, if purgatory checksum fails. Then we would automatically return to stand-alone dump tools.

The flow would look like the following in this case:

IPL dump tool -> s390_kdump_entry -> crash_kexec +--> purgatory -+->[checksum ok]---> kdump

      ^                                          |               |
      |                                          |        [checksum fail]
      |                                          |               |
      |                                          |     [forced program check]
      +------[program check]---------------------+               |
      |                                                          |
      +----------------------------------------------------------+

Then of course also the kernel code would have to install a special program check handler before calling purgatory.

On Fri, Jul 22, 2011 at 11:33:11AM +0200, Michael Holzheu wrote:

Quoted 3 lines

crash_kexec can be entered by e.g.:
* panic -> kdump shutdown action -> crash_kexec
* panic -> s390 dump shutdown action -> auto IPL dump tool -> s390_kdump_entry -> crash_kexec

So after panic() You will still jump to dump tools?

The only thing you
need to do there is installing program check handler and could have been easily done in kernel too.

Quoted 1 line

* hard hang -> manual IPL dump tool -> s390_kdump_entry -> crash_kexec

This one makes sense as kernel is hard hung and dump tools need to force crash_kexec() now. It is more like x86 NMI handler.

Quoted 19 lines

Handling for corrupted kdump:

New idea for returning to dump tools in case of program check:
We could force a program check for s390, if purgatory checksum fails. Then we would automatically return to stand-alone dump tools.

The flow would look like the following in this case:

IPL dump tool -> s390_kdump_entry -> crash_kexec +--> purgatory -+->[checksum ok]---> kdump

      ^                                          |               |
      |                                          |        [checksum fail]
      |                                          |               |
      |                                          |     [forced program check]
      +------[program check]---------------------+               |
      |                                                          |
      +----------------------------------------------------------+

Quoted 2 lines

Then of course also the kernel code would have to install a special program check handler before calling purgatory.

If kernel code is going to install the program check handler before calling purgatory, then we don't need to jump to dump tools at all after panic()?

On Mon, Jul 18, 2011 at 04:44:13PM +0200, Michael Holzheu wrote:
You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Hello Vivek,

Still thinking how we best get elfcorehdr size...

Quoted 7 lines · Vivek Goyal

On Mon, Jul 18, 2011 at 04:44:13PM +0200, Michael Holzheu wrote:
You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Wouldn't it be most natural to pass the length along with the address of the ELF core header? What about extending the kernel elfcorehdr kernel parameter and adding optional size:

elfcorehdr=<addr>[KMG],<size>[KMG]

Quoted 17 lines · Michael Holzheu

Hello Vivek,

Still thinking how we best get elfcorehdr size...

Quoted 7 lines · Vivek Goyal

On Mon, Jul 18, 2011 at 04:44:13PM +0200, Michael Holzheu wrote:
You could do that but I think a more generic parameter will make more sense.

- Either something along the lines of memmap=
- Or excludemem=x@y
- Or modify memory map in s390 specific bootloading protocol block etc.

Wouldn't it be most natural to pass the length along with the address of the ELF core header? What about extending the kernel elfcorehdr kernel parameter and adding optional size:

Quoted 1 line

elfcorehdr=<addr>[KMG],<size>[KMG]

I think it could be memap= style. elfcorehdr=X[KMG]@Y[KMG]. Though to support backward compatibility we will have to support old format of plain elfcorehdr=X too.

Quoted 4 lines

The first thing we want to do is to check if
the purgatory is still fine, that is do a checksum. If we have the infrastructure in place to do one checksum then we can easily do the other checksums as well.

Some piece of code you have to assume is fine. Are you not already assuming that IPL code you have in first 64K bytes is fine and no body has overwritten it.

Are you not assuming that hook in panic()
(I think you are calling it shutdown trigger) is fine so that it can help you jump to right place.

Hello Vivec,

On Thu, 2011-07-14 at 13:55 -0400, Vivek Goyal wrote:

[snip]

Quoted 8 lines

Quoted 4 lines

The first thing we want to do is to check if
the purgatory is still fine, that is do a checksum. If we have the infrastructure in place to do one checksum then we can easily do the other checksums as well.

Some piece of code you have to assume is fine. Are you not already assuming that IPL code you have in first 64K bytes is fine and no body has overwritten it.

We can assume that the IPL dump code is fine, because it is freshly loaded into memory. Only when the disk is somehow corrupted we have a problem.

Quoted 3 lines

Are you not assuming that hook in panic()
(I think you are calling it shutdown trigger) is fine so that it can help you jump to right place.

Yes, that is correct for automatic dump in case of panic(). The panic() path can fail.

But there are two other options where really *no* code that was in memory, when the system crashed, is used for the dump process or verification of kdump:
1) Manual IPL/boot of stand-alone dump by the operator via the virtual guest console
2) Automatic IPL/boot of stand-alone dump by our z/VM hypervisor watchdog

Quoted 26 lines · Martin Schwidefsky

Quoted 22 lines · On Wed, 13 Jul 2011 16:00:04 -0400 Vivek Goyal <[email protected]> wrote:

On Wed, Jul 13, 2011 at 06:46:11PM +0200, Martin Schwidefsky wrote:

[..]

Quoted 12 lines

Quoted 11 lines

What I am suggesting is that stand alone dumper gets control only if kdump kernel is corrupted.

So following sequence.

Kernel Crash ---> purgatory --> either kdump kenrel/IPL stand alone tools

Here only drawback seems to be that we assume that purgatory code and pre-calculated checksum has not been corrupted. The big advantage is that s390 kdump support looks very similar to other arches and understaning and supporting kdump across architectures becomes easy.

Quoted 4 lines

My problem with that is the following: how do we get from the "Kernel Crash" step to the purgatory code? It does work for "normal" panics, but it fails miserably for a hard crash that does not even get as far as panic. That is why we insist on a possible second order of things:

What is hard crash? How does that happen and what does x86 and s390 do in that case?

Quoted 3 lines

E.g. an endless loop with interrupts disabled. To get out of this situation we will IPL/boot a new system. That is either the production system itself or the stand-alone dump tool.

NMI hardware lockup detection will work in this situation and will lead to kdump trigger.

Quoted 17 lines

Quoted 6 lines

Though I don't have details but your argument seems to be that in s390 we are always guranteed that we will jump to IPLing the stand alone tools code irresepective of the system state hence it is relatively safer to do checks in stand alone tools instead of purgatory where code is in memory.

Now you got it. That is the crux of the argument.

Quoted 2 lines

If due to hard hang, code can not even make to purgatory, where would it go? Can't we do IPLing of stand alone tool then.

It doesn't go anywhere. Basically the system is manually stopped and restarted. But on s390 we can still get to all the required information to generated a dump. That is one of the major differences to x86, if you have to do a restart the registers on x86 will be gone, no?

Quoted 8 lines

Quoted 4 lines

So we first try to take purgatory path which does the checksum and is consistent with other architectures. If that does not work in case of hard hang, you always have the option of IPLing the stand alone tool later manually.

How are we suddenly on the purgatory path again? The code that gets control in case of a hard crash + IPL is the stand-alone dump tool, not the purgatory code.

I think that's the biggest contetion point. From the start of discussion you have this hardcoded requirement that the moment panic() happens you are jumping to some IPL code and that's what I am questioning. Why can't you execute some more code after panic() (purgatory), before you jump to IPL code (only if you have to).

Quoted 4 lines

The first thing we want to do is to check if
the purgatory is still fine, that is do a checksum. If we have the infrastructure in place to do one checksum then we can easily do the other checksums as well.

Some piece of code you have to assume is fine. Are you not already assuming that IPL code you have in first 64K bytes is fine and no body has overwritten it. Are you not assuming that hook in panic() (I think you are calling it shutdown trigger) is fine so that it can help you jump to right place.

Quoted 7 lines

Quoted 4 lines

This will also get rid of requirement passing all the segment and cheksum info to stand alone tool with the help of meminfo (That's another sore point).

No, it doesn't. We will still need to do the checksum for the purgatory code and we already have the re-ipl information which won't go away.

It is a very small piece of code. The way you assume that your 8KB of IPL code is fine, I think we shall have to have this assumption here also.

Quoted 8 lines

Quoted 7 lines

Bottom line, even if you can't make to purgatory reliably, you always have the option of capturing dump manually using stand alone tools. We don't have to mix up kdump and stand alone mechanism. If kdump fails, we just need to have capability to still capture the dump using stand alone tools manually. I think that will make things simpler even for stand alone tools.

Quoted 5 lines

If we decide not to mix kdump and stand-alone dump then we loose something. Consider a hard crash where the kdump segments are still intact. What our customers do in that case is to start the stand-alone dump utility. Without a way to find and verify the kdump setup we would have to do a full dump. Which will take its time if the memory size is big. See?

This is a really-2 corner case where purgatory went bad. And even in corner case you capture the dump just that it is not filtered.

I really don't understand that to address the corner case why would you complicate the general kexec infrastructure and introduce new interfaces like meminfo.

Quoted 3 lines

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

Quoted 4 lines

And last but not least, with the stand-alone dump tools you can dump early kernel problems which is not possible using kdump, because you can't dump before the kdump kernel has been loaded with kexec.

That is one limitation but again if your kernel can't even boot, it is not ready to ship and it is more of a development issue and there are other ways to debug problems. So I would not worry too much about it.

On a side note, few months back there were folks who were trying to enhance bootloaders to be able to prepare basic environment so that a kdump kernel can boot even in the event of early first kernel boot.

Quoted 8 lines

That were more or less the arguments, why we did not support kdump in the past.

In order to increase dump reliability with kdump, we now implemented a two stage approach. The stand-alone dump tools first check via meminfo, if kdump is valid using checksums. If kdump is loaded and healthy it is started. Otherwise the stand-alone dump tools create a full-blown stand-alone dump.

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel.

So this additional meminfo strucutres
and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Quoted 12 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 6 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

Quoted 3 lines

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Quoted 4 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

Quoted 1 line

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

When do you load kdump kernel and who does it?

Who gets the control first after crash?

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

Do you load kdump kenrel from some tape/storage after system crash. Where does bootloader lies and how do you make sure it is not corrupted and associated device is in good condition.

To me we should not create a arch specific way of passing information between kernels.

Hello Vivek,

Quoted 9 lines · Vivek Goyal

Quoted 3 lines

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

I don't want to argue, about probabilities. Even if we gain only a little more reliability this is important for us. Don't forget that we write software for mainframes. We accept that the last 0.1 percent of reliability can be very expensive compared to the first 99.9 percent.

[snip]

Quoted 9 lines

Quoted 4 lines

And last but not least, with the stand-alone dump tools you can dump early kernel problems which is not possible using kdump, because you can't dump before the kdump kernel has been loaded with kexec.

That is one limitation but again if your kernel can't even boot, it is not ready to ship and it is more of a development issue and there are other ways to debug problems. So I would not worry too much about it.

We worry about that. See the comment above regarding the 100 percent.

Quoted 4 lines

On a side note, few months back there were folks who were trying to enhance bootloaders to be able to prepare basic environment so that a kdump kernel can boot even in the event of early first kernel boot.

This is one more argument to create the ELF header in the 2nd kernel. With our approach loading the kdump kernel at boot time is almost trivial.

Example (e.g. crashkernel=xxxM@256M):

1. The boot loader loads standard kernel and kdump kernel into memory. The kdump kernel is loaded into crashkernel memory to 256M. No more setup (e.g. creating ELF headers) is necessary.
2. We could add a kernel parameter "kexec_load=<segm addr>,<segm size>, ..." that does an internal kexec_load(). After this kernel parameter is processed, kdump is armed.

What do you think?

Quoted 9 lines

Quoted 8 lines

That were more or less the arguments, why we did not support kdump in the past.

In order to increase dump reliability with kdump, we now implemented a two stage approach. The stand-alone dump tools first check via meminfo, if kdump is valid using checksums. If kdump is loaded and healthy it is started. Otherwise the stand-alone dump tools create a full-blown stand-alone dump.

Quoted 3 lines

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel.

Can you point me to the code where this is done and from where in the kernel that code is called? Currently with our implementation we do not use any purgatory code from kexec tools.

Quoted 19 lines

and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Quoted 15 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 6 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

Quoted 3 lines

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Ok, where is this done and when?

Quoted 22 lines

Quoted 5 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

Quoted 1 line

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Quoted 2 lines

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

Maybe I confused you here. What I wanted to describe is the following idea:
1. The running production kernel starts with "crashkernel=" and reserves memory for kdump. No kdump is loaded with kexec.
2. The system crashes
3. To create the dump, a prepared dump disk is booted. The boot loader loads the kdump kernel into crashkernel memory.
4. The boot loader starts kdump kernel on s390 with entry point <crashkernel base> + 0x10008
5. The kdump kernel creates ELF header etc...

So this is simple for the boot loader code because no preparation steps like creating the ELF header are required. This is similar to scenario of pre-loading the kdump kernel together with the standard kernel at startup that I described above.

Quoted 1 line

Quoted 1 line

When do you load kdump kernel and who does it?

Currently we load the kdump kernel with kexec like it is done on all other architectures. The other options I described above are currently just ideas that we have for the future.

Quoted 6 lines

Who gets the control first after crash?

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

As Martin already said: It can be the other way round. The stand-alone dump tool gets first control. We trust this code because it is freshly loaded and has a different code base.

This code verifies the kdump setup
and jumps into the pre-loaded kdump (crashkernel base + 0x10008) if everything is ok. Otherwise it creates a traditional s390 dump.

Quoted 8 lines

Quoted 4 lines

On a side note, few months back there were folks who were trying to enhance bootloaders to be able to prepare basic environment so that a kdump kernel can boot even in the event of early first kernel boot.

This is one more argument to create the ELF header in the 2nd kernel. With our approach loading the kdump kernel at boot time is almost trivial.

I think ELF header is just the way of passing some required information from first kernel to second kernel. In second kernel, we anyway prepare fresh headers for /proc/vmcore.

So in your mechanism if you don't need any info from second kernel it is fine to not use ELF. But if you do need, then it makes sense to use existing mechanism instead of creating a new one (seems to be meminfo in your case).

Quoted 14 lines · Michael Holzheu

Hello Vivec,

Quoted 1 line · Vivek Goyal

On Mon, Jul 04, 2011 at 07:09:22PM +0200, Michael Holzheu wrote:

[snip]

Quoted 1 line

I don't understand what is stand-alone dump tools and

S390 stand-alone dump tools are independent mini operating systems that are installed on disks or tapes. When a dump should be created, these stand-alone dump tools are booted. All that they do is to write the dump (current memory plus the CPU registers) to the disk/tape device.

Quoted 6 lines

The advantage compared to kdump is that since they are freshly loaded into memory they can't be overwritten in memory.

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

Quoted 2 lines

Also the boot process ensures that the hardware is in a initialized state.

Who makes sure that hardware is in initiliazed state? Kdump kernel, stand alone kernel or BIOS.

Quoted 4 lines

And last but not least, with the stand-alone dump tools you can dump early kernel problems which is not possible using kdump, because you can't dump before the kdump kernel has been loaded with kexec.

That is one limitation but again if your kernel can't even boot, it is not ready to ship and it is more of a development issue and there are other ways to debug problems. So I would not worry too much about it.

On a side note, few months back there were folks who were trying to enhance bootloaders to be able to prepare basic environment so that a kdump kernel can boot even in the event of early first kernel boot.

Quoted 8 lines

That were more or less the arguments, why we did not support kdump in the past.

In order to increase dump reliability with kdump, we now implemented a two stage approach. The stand-alone dump tools first check via meminfo, if kdump is valid using checksums. If kdump is loaded and healthy it is started. Otherwise the stand-alone dump tools create a full-blown stand-alone dump.

kexec-tools purgatory code also checks the checksum of loaded kernel and other information and next kernel boot starts only if nothing has been corrupted in first kernel. So this additional meminfo strucutres and need of checksums sounds unnecessary. I think what you do need is that somehow invoking second hook (s390 specific stand alone kernel) in case primary kernel is corrupted.

Quoted 12 lines

With this approach we still keep our s390 dump reliability and gain the great kdump features, e.g. distributor installer support, dump filtering with makedumpfile, etc.

Quoted 6 lines

why the existing
mechanism of preparing ELF headers to describe all the above info and just passing the address of header on kernel commnad line (crashkernel=) will not work for s390. Introducing an entirely new infrastructure for communicating the same information does not sound too exciting.

Quoted 3 lines

We need the meminfo interface anyway for the two stage approach. The stand-alone dump tools have to find and verify the kdump kernel in order to start it.

kexec-tools does this verification already. We verify the checksum of all the loaded information in reserved area. So why introduce this meminfo interface.

Quoted 4 lines

Therefore the interface is there and can be used. Also creating the ELF header in the 2nd kernel is more flexible and easier IMHO:
* You do not have to care about memory or CPU hotplug.

Reloading the kernel upon memory or cpu hotplug should be trivial. This does not justify to move away from standard ELF interface and creation of a new one.

Quoted 1 line

* You do not have to preallocate CPU crash notes etc.

Its a small per cpu area. Looks like otherwise you will create meminfo areas otherwise.

Quoted 6 lines

* It works independently from the tool/mechanism that loads the kdump kernel into memory. E.g. we have the idea to load the kdump kernel at boot time into the crashkernel memory (not via the kexec_load system call). That would solve the main kdump problems: The kdump kernel can't be overwritten by I/O and also early kernel problems could then be dumped using kdump.

Can you give more details how exactly it works. I know very little about s390 dump mechanism.

When do you load kdump kernel and who does it?

Who gets the control first after crash?

To me it looked like that you regularly load kdump kernel and if that is corrupted then somehow you boot standalone kernel. So corruption of kdump kernel should not be a issue for you.

Do you load kdump kenrel from some tape/storage after system crash. Where does bootloader lies and how do you make sure it is not corrupted and associated device is in good condition.

To me we should not create a arch specific way of passing information between kernels. Stand alone kernel should be able to parse the ELF headers which contains all the relevant info. They have already been checksum verified.

Thanks
Vivek

Hello Vivek,

I attached a document where the s390 port is described in more detail. Perhaps this helps you to understand what want and what we are doing. If not - just delete it :-)

Quoted 6 lines · Michael Holzheu

Hello Vivek,

I attached a document where the s390 port is described in more detail. Perhaps this helps you to understand what want and what we are doing. If not - just delete it :-)

Michael,

Thanks for the documentation. I have gone through it quickly and there are some parts I am still missing.

On x86, after the kernel crash we jump to purgatory code which does the checksum verification of all the loaded segments and if everything is fine, it jumps to kdump kernel's entry point and second kernel boots.

On s390, looks like after purgatory, control is going to some other piece of code (which does IPL?), and that code decides whether to start kdump kernel or launch stand alone kernel as backup plan?

If yes, is that code whic does the IPL, also loaded in crashkernel memory as part of kdump kernel? If not, how does kexec-tools come to know where to jump after doing checksum on loaded kernel?

Quoted 11 lines · Michael Holzheu

S390 stand-alone dump tools are independent mini operating systems that are installed on disks or tapes. When a dump should be created, these stand-alone dump tools are booted. All that they do is to write the dump (current memory plus the CPU registers) to the disk/tape device.

The advantage compared to kdump is that since they are freshly loaded into memory they can't be overwritten in memory.

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

Quoted 14 lines · Vivek Goyal

Quoted 11 lines · Michael Holzheu

S390 stand-alone dump tools are independent mini operating systems that are installed on disks or tapes. When a dump should be created, these stand-alone dump tools are booted. All that they do is to write the dump (current memory plus the CPU registers) to the disk/tape device.

The advantage compared to kdump is that since they are freshly loaded into memory they can't be overwritten in memory.

Another advantage is
that since it is different code, it is much less likely that the dump tool will run into the same problem than the previously crashed kernel.

Quoted 5 lines

I think in practice this is not really a problem. If your kernel is not stable enough to even boot and copy a file, then most likely it has not even been deployed. The very fact that a kernel has been up and running verifies that it is a stable kernel for that machine and is capable of capturing the dump.

Vivek: I used to do VM/XA on S/390 boxes for a living, and that's *not* where Michael is coming from.

What the standalone dump code does is take a system that may have the moral equivalent of 256 separate PCI buses, several hundred disks all visible in multipath configurations, dozens of other devices, and as long as you can find *one* console and *one* tape/disk drive that works, you can capture a dump.

More than once in my career, I got into a situation where the production system would hang - and booting off another disk that contained an older copy with maybe a few less patches would *also* hang. VM/XA would simply *not run*. Booting the standalone dump utility (which shared zero code with VM/XA, and did *much* less initialization of I/O devices not needed for the actual dump) would work just fine. This would get me a dump that would show that we had a (usually) hardware issue - either we were tripping over an errata that *no* released version of VM/XA had a workaround for, or outright defective hardware.

For the same efficiency reasons that Linux doesn't do a lot of checking for "can never happen" cases, VM/XA doesn't check some things. So when busted hardware would present logically impossible combinations of status bits (for instance, "device still connected" but "I/O bus disconnected"), Bad Things would happen. Booting a tiny dump program that never even *tried* to look at the bad bits posted by the miscreant hardware would allow you to get the info you needed to debug it.