kexec on ppc64

Hello All,
Sometime back, I started looking into ppc and working towards getting a
port of kexec to ppc64. As a recap (this is because I am cross-posting this mailto ppc64 and other lists as well), the idea behind kexec was to bypass the kernel - firmware (bios/OF) interaction during bootup.
More kexec information can be obtained from Randy Dunlap's site on osdl:
http://developer.osdl.org/rddunlap/kexec

Some Background:

Towards this, I looked at the ppc kernel boot code (primarily the prom
code that interacts with the openfirmware) and also asked a lot of questions to
Benh and Paulus regarding some of the ppc boot stuff. Ben and Paulus provided
lots of very useful information and suggestions for a clean implementation.
One major requirement that came up for a clean implementation was to
re-order the prom_init code in ppc64. The prom_init (prom.c) code interacts withthe openfirmware to perform a lot of kernel initializations. Since a lot of initializations were being done using firmware interaction, it
meant that for a kexec case, we would need to be able to save all of that from
the previous boot cleanly.
Talking to Ben and Paulus, I found out that all that initialization
was not necessary at that point in boot.
Hence, we got the idea to work on the re-order of the prom_init code.
The motivation to do this for kexec would be that it would facilitate branching
into a point after prom_init in a more cleaner setup and do away with having to
maintain a lot of kernel initialization information from the earlier boot.

To Dos:

To start with, I put together the following preliminary list for the cleanup:
- only basic minimal initialization to be done in prom_init code. This
would include, instantiating the rtas, initializing client services, tce_table
initialization, cpu initializations, etc.
- some other initializations, like initalization of the naca structure,
systemcfg data structures, to be moved to a early_init call, after prom_init,
but before memory management was setup, since some of the fields in systemcfg
and naca are required prior to htab/stab initializations.
- still other initializations to be moved to a later call, maybe called
from setup_system() after kernel was relocated and memory management setup. Thiscould include setting up kernel globals like cpumask data structures (they are currently setup in prom_init)
- also identify relevant data (that might be required for the kexec
kernel) that could be pushed into the device_tree so that it can be passed to
the new kexec kernel

My current focus:

I am currently working on moving some of the cpumask data structures out into setup_system and debugging some problems related to getting it working.

Other issues:

With respect to kexec, a few points came up in the discussions with Ben:
Memory accessibility in real mode in HV environment (only 256mb is accessible), which has to be taken into account when we want to copy the new kernel. Ben suggested that we could setup a mmu mapping making use of the HV calls, or we have a real mode routine to copy pages.
Another point that came up was the unregistration and re-registration of the vpa_areas in HV environment after a kexec reboot.
All this still needs to be thought through a lot more.
There are most likely to be more issues, points that have to be discussed, sorted out and resolved to get kexec on ppc64 working.

Request for comments

I would like to solicit inputs, feedback, opinions, changes on this preliminary idea and planned list of 'things to do'. I would also like to know the interest to participate in this effort or anything else related to 'kexec on ppc64'.

Thanks and Regards,
Sharada

taking silly IBM list off the cc...

On Wed, 2004-07-14 at 07:45, R Sharada

We actually had a similar problem on x86. The e820 table is presented by the BIOS, and must be saved or reconstructed when you boot into the new kernel.

It would be really cool to have a way of passing the memory layout information to the new kernel that is relatively cross-platform. That way, we don't get stuck rewriting it for each new arch.

For instance, instead of passing the BIOS/firmware structures like LMBs in the device tree or e820 tables to the kexec kernel, you'd pass the Linux concepts like zone_start_pfn and so forth.

While you're ripping apart prom.c, have you thought about getting rid of all of the RELOC() stuff? I think Ben H. had an evil plan for that, too.

BTW, have you seen any opportunities to replace the arch-specific things like lmb_alloc() with normal bootmem calls?

-- Dave

I wouldn't even try to go that way at first. What we need is to pass an OF device-tree around on ppc, with all the necessary informations stuffed in it as additional properties.

I'd prefer letting the new kernel decide those based on the low level firmware info, or you introduce a higher level dependency between the "old" and "new" kernels (what if those concepts change between the 2 kernels ?)

Paulus already removed a bunch, once we manage to completely isolate prom_init from the rest of the kernel, it will be easier to "finish" it by having prom_init be a totally separate link entity (could even be 32 bits relocatable code).

As far as lmb_alloc() is concerned, there are some constraints like beeing in RMO for some things, or beeing at the top of the memory for others, that aren't quite addressed by bootmem. Again, we are in the firmware env at this point, I wouldn't rely on too much linux-specific data structures.

What we need, as we already discussed for a while with Rusty, is a specific allocator for use within prom_init() which properly deals with both OF-side allocations (claim the memory) to avoid walking over things like initrd, deals with the various constraints we have, and produce a map of allocated area as a property for use by the rest of the kernel.

As far as the post-prom-init stage is concerned, we can just turn that map into lmb's, or try to get rid of lmb's and turn that into bootmem provided we can deal with the various RMO constraints.

Ben.