IA64 kexec

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zou Nan hai
Sent: 2005Äê11ÔÂ8ÈÕ 7:28
To: [email protected]
Cc: [email protected]
Subject: [Patch]IA64 kexec

Here is my patches of kexec on IA64.

The kernel patch is against 2.6.14
The kexec-tools patch is against kexec-tools-1.101

I have tested it on a Tiger-4 and a ZX1 machine.

Please test and review it.

Thanks.
Zou Nan hai

Here is my patches of kexec on IA64.

The kernel patch is against 2.6.14
The kexec-tools patch is against kexec-tools-1.101

I have tested it on a Tiger-4 and a ZX1 machine.

Please test and review it.

Quoted 4 lines · On 2005-11-07 at 23:27:49 Zou Nan Hai wrote:

Here is my patches of kexec on IA64.

The kernel patch is against 2.6.14
The kexec-tools patch is against kexec-tools-1.101

Am i right in assuming that this is still the latest version of kexec for ia64? If not, could someone point me to a newer one.

Quoted 1 line

I have tested it on a Tiger-4 and a ZX1 machine.

Is it possible to get the config that you used on Tiger-4?

Quoted 1 line

Please test and review it.

Here is an as-yet untested forward port of the kexec-ia64 patch to today's Linus git tree (~2.6.16-rc3).

I haven't looked into what other features have been added to other arches kexec. Nor if the features above are applicable - seems that they probably are, exept that ia64 doesn't have NMI (right?) so the cpu shutdown would need to be done another way.

The ia64 equivalent of NMI (large brick through the window) is INIT. Some systems have a button on the front panel to generate INIT, or have a maintenance processor that can send INIT. So a good kdump solution should eventually make use of INIT.

Quoted 2 lines

Here is an as-yet untested forward port of the kexec-ia64 patch to today's Linus git tree (~2.6.16-rc3).

Thanks for taking a look at this ... I'm glad to see that there is still interest in kexec.

Khalid Aziz at HP is woking on merging the good parts of that patch from Nan Hai with the kexec patch that he had produced earlier). We should see the results of that merge next week, & I hope to see lots more commentary and testing this time around.

Quoted 4 lines

I haven't looked into what other features have been added to other arches kexec. Nor if the features above are applicable - seems that they probably are, exept that ia64 doesn't have NMI (right?) so the cpu shutdown would need to be done another way.

Nan Hai makes use of HOTPLUG_CPU to offline the other cpus ... which in many ways is a very elegant solution (as it puts the cpus neatly back into SAL ready for the new OS to bring it back online again). But there are a couple of downsides:
1) Requires CONFIG_HOTPLUG_CPU (perhaps this isn't really a big issue)

2) May run into trouble for kdump case where we'd like to rely on less known state/code to get a good dump when the Linux kernel is known to be in some unstable state.

The ia64 equivalent of NMI (large brick through the window) is INIT. Some systems have a button on the front panel to generate INIT, or have a maintenance processor that can send INIT. So a good kdump solution should eventually make use of INIT.

-Tony

Quoted 4 lines · "Luck, Tony" (on Mon, 13 Feb 2006 09:26:58 -0800) wrote:

The ia64 equivalent of NMI (large brick through the window) is INIT. Some systems have a button on the front panel to generate INIT, or have a maintenance processor that can send INIT. So a good kdump solution should eventually make use of INIT.

Which raises a small problem. As of about 2.6.15, INIT is a recoverable event. INIT _must_ be recoverable, because it can be sent when an MCA occurs and one or more cpus was running with interrupts disabled. For example, when the cpu that takes the MCA owns a disabled spinlock that other cpus are waiting on. If INIT is not recoverable then some MCAs that could be recovered also become unrecoverable, at random.

Since INIT is recoverable, pressing NMI gives you a stack trace for each cpu, then the system resumes. This allows a user to see if the system is making progress, albeit slowly, or if it really is stuck. The downside of a recoverable INIT is that you cannot use it to take a dump, or at least not the first time that NMI is issued. Unfortunately there is no way to distinguish between an NMI where the user wants to see what the system is doing and an NMI to take a dump. Nobody has implemented the "Read Programmer's Mind" instruction yet.

Quoted 2 lines

Here is an as-yet untested forward port of the kexec-ia64 patch to today's Linus git tree (~2.6.16-rc3).

Quoted 8 lines · Keith Owens

Which raises a small problem. As of about 2.6.15, INIT is a recoverable event. INIT _must_ be recoverable, because it can be sent when an MCA occurs and one or more cpus was running with interrupts disabled. For example, when the cpu that takes the MCA owns a disabled spinlock that other cpus are waiting on. If INIT is not recoverable then some MCAs that could be recovered also become unrecoverable, at random.

Quoted 8 lines

Since INIT is recoverable, pressing NMI gives you a stack trace for each cpu, then the system resumes. This allows a user to see if the system is making progress, albeit slowly, or if it really is stuck. The downside of a recoverable INIT is that you cannot use it to take a dump, or at least not the first time that NMI is issued. Unfortunately there is no way to distinguish between an NMI where the user wants to see what the system is doing and an NMI to take a dump. Nobody has implemented the "Read Programmer's Mind" instruction yet.

I sense pain. Looking over the code - very naievely - would it be possible to register an alternate INIT handler when kexecing.

Quoted 5 lines · Tony

Quoted 2 lines

Here is an as-yet untested forward port of the kexec-ia64 patch to today's Linus git tree (~2.6.16-rc3).

Thanks for taking a look at this ... I'm glad to see that there is still interest in kexec.

Likewise.

In case anyone cares, my interest in kexec is twofold. Firstly the ia64 box I have takes a really long time to reboot, and it would be nice if kexec could trim that down to speed up my crash-and-burn development cycle.

But more importantly, I'm interested in using it for kdump functionality, hopefully in conjunction with Xen - though as you can see, I haven't got that far yet.

Quoted 4 lines

Khalid Aziz at HP is woking on merging the good parts of that patch from Nan Hai with the kexec patch that he had produced earlier). We should see the results of that merge next week, & I hope to see lots more commentary and testing this time around.

Awsome, I look forward to seeing it. Would I be right in thinking that it will show up on this list?

Quoted 11 lines

Quoted 5 lines

I haven't looked into what other features have been added to other arches kexec. Nor if the features above are applicable - seems that they probably are, exept that ia64 doesn't have NMI (right?) so the cpu shutdown would need to be done another way.

Nan Hai makes use of HOTPLUG_CPU to offline the other cpus ... which in many ways is a very elegant solution (as it puts the cpus neatly back into SAL ready for the new OS to bring it back online again). But there are a couple of downsides:
1) Requires CONFIG_HOTPLUG_CPU (perhaps this isn't really a big issue)

That isn't a particular concern to me.

Quoted 10 lines

2) May run into trouble for kdump case where we'd like to rely on less known state/code to get a good dump when the Linux kernel is known to be in some unstable state.

The ia64 equivalent of NMI (large brick through the window) is INIT. Some systems have a button on the front panel to generate INIT, or have a maintenance processor that can send INIT. So a good kdump solution should eventually make use of INIT.

-Tony

Quoted 14 lines · Keith Owens

Quoted 4 lines · "Luck, Tony" (on Mon, 13 Feb 2006 09:26:58 -0800) wrote:

The ia64 equivalent of NMI (large brick through the window) is INIT. Some systems have a button on the front panel to generate INIT, or have a maintenance processor that can send INIT. So a good kdump solution should eventually make use of INIT.

Which raises a small problem. As of about 2.6.15, INIT is a recoverable event. INIT _must_ be recoverable, because it can be sent when an MCA occurs and one or more cpus was running with interrupts disabled. For example, when the cpu that takes the MCA owns a disabled spinlock that other cpus are waiting on. If INIT is not recoverable then some MCAs that could be recovered also become unrecoverable, at random.

Quoted 8 lines

Since INIT is recoverable, pressing NMI gives you a stack trace for each cpu, then the system resumes. This allows a user to see if the system is making progress, albeit slowly, or if it really is stuck. The downside of a recoverable INIT is that you cannot use it to take a dump, or at least not the first time that NMI is issued. Unfortunately there is no way to distinguish between an NMI where the user wants to see what the system is doing and an NMI to take a dump. Nobody has implemented the "Read Programmer's Mind" instruction yet.

I sense pain. Looking over the code - very naievely - would it be possible to register an alternate INIT handler when kexecing.

What I'm getting at is ia64_os_init_dispatch_monarch and ia64_os_init_dispatch_slave are basically the same, but r19 is set so the code knows which variant is running for the core that cares. I wonder if an aditional bit in r19 could be used by alternate handlers that are registered when kexec wants to shut down the cpus.

Off course, this assume that reregistering handlers is possible, which is where the "naieve" bit comes in.

Quoted 3 lines · Horms (on Tue, 14 Feb 2006 13:06:44 +0900) wrote:

I sense pain. Looking over the code - very naievely - would it be possible to register an alternate INIT handler when kexecing.

Not a good idea, the INIT handler code is very closely tied to the SAL/OS interface. But what kexec can do is to register itself on the notify_die() chain, it will get called for multiple events including DIE_INIT_SLAVE_ENTER, DIE_INIT_SLAVE_PROCESS, DIE_INIT_SLAVE_LEAVE, DIE_INIT_MONARCH_ENTER, DIE_INIT_MONARCH_PROCESS and DIE_INIT_MONARCH_LEAVE. That chain and the associated events is meant for debuggers, crash dumpers and assorted RAS tools. See also the DIE_MCA_* events on the same chain.

Quoted 2 lines · Keith Owens

But what kexec can do is to register itself on the notify_die() chain ...

Thanks, that looks quite promising indeed. However, after poking round a bit more I'm a little confused about what the intent of using INIT is.

Is the idea to intercept an INIT, produced by the front panel, a maintenence processor, (or perhaps an internal error), and then start kexecing? Or is the idea for kexec to use INIT internally to halt the processors.

Lastly, if INIT is being used to shut off the processors by kexec, is it reasonable to assume that an INIT will hit all processors, and thus the slave processors can halt themselves in the callback (using cpu_die()?).

Quoted 11 lines · Horms (on Wed, 15 Feb 2006 11:10:57 +0900) wrote:

Quoted 2 lines · Keith Owens

But what kexec can do is to register itself on the notify_die() chain ...

Thanks, that looks quite promising indeed. However, after poking round a bit more I'm a little confused about what the intent of using INIT is.

Is the idea to intercept an INIT, produced by the front panel, a maintenence processor, (or perhaps an internal error), and then start kexecing? Or is the idea for kexec to use INIT internally to halt the processors.

kexec (or any other RAS tool) should avoid using INIT itself. The ia64 INIT handlers are coded on the assumption that INIT is sent to all cpus at the same time, or that INIT is issued as part of the MCA rendezvous. In either case, the code assumes that the entire system is first brought to a dead stop, with all cpus under MCA or INIT control, before processing with the RAS code. IOW, the user invokes INIT via a button or BMC command, all cpus stop, then you start the debug process.

But there is still the problem of working out what the user means when they send INIT. Do they want a debugger or kexec to run, followed by reboot? Or do they just want a stack trace followed by resumption of normal processing. Some people want one option, some want another, and they are mutually exclusive.

Quoted 3 lines

Lastly, if INIT is being used to shut off the processors by kexec, is it reasonable to assume that an INIT will hit all processors, and thus the slave processors can halt themselves in the callback (using cpu_die()?).

The combination of MCA and INIT will hit all processors. Both the MCA and INIT handlers call ia64_wait_for_slaves(), so the monarch event will not proceed until all slaves have been stopped, or we decide that they are never going to stop and proceed anyway. So kexec should run off the monarch notifier.

Have you read linux/Documentation/ia64/mca.txt?