ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
From: Srinivas Kandagatla <[email protected]>
Use-case is simple kexec on smp system.
Lets say there are two cores core-0 and core-1.
sequence here is:
As part of kexec call, the secondary cores are stopped by calling machine_shutdown(), which in turn issues IPIs to off-line the other CPUs and then call smp_ops.cpu_kill.
machine_shutdown invokes smp_send_stop from core-0 which then sends IPI_CPU_STOP ipi message for core-1 and then smp_kill_cpus call smp_ops.cpu_kill from core-0 with cpu=1.
The expectation here is that cpu_kill will relocate secondary-core to safe location from primary-core(core-0).
This is a catch 22 situation for SOCs, which can't change PC's of other cores. At the point when cpu_kill is invoked, it cant execute any function calls for core-1, as it is off-line and is spinning in cpu_relax.
So the only way is to put the core into safe place by executing some SOC specific relocation code.
Having a callback from ipi handler when cpu is stopped would be very usefull for SOC's which want to relocate the secondary-core to safe location.
---
Hi All,
I just tried kexec on an CA9 dual core board, and it looks like it is broken
on smp system.
It looks like its a very basic issue, Am not not sure if I missed something here?
It is almost impossible to get kexec working, For cases where SOC's can't change PC of secondary-core from a primary-core.
As cpu_kill is the only callback which is invoked from primary core to stop an secondary-core during machine_shutdown it to relocate the secondary-core to safe place, as it is offine and it is spinning in cpu_relax.
Having a callback from ipi handler when cpu is stopped would be very usefull for SOC's which want to relocate the secondary-core to safe location.
Thanks,
srini
arch/arm/include/asm/smp.h | 7 +++++++ arch/arm/kernel/smp.c | 3 +++ 2 files changed, 10 insertions(+), 0 deletions(-)
arch/arm/include/asm/smp.h +7 -0
@@ -100,6 +100,13 @@ struct smp_operations {* This also gives us the initial stack to use for this CPU. */ int (*smp_boot_secondary)(unsigned int cpu,struct task_struct *idle); + +/* Notify interested parties before parking core in cpu_relax. +* SOC's can park cores in safe place in this callback, is very useful +* for cases like kexec. Stopping cpu is the victim cpu in this case. +*/ +int (*smp_notify_cpu_stop)(unsigned int cpu); +#ifdef CONFIG_HOTPLUG_CPU int (*cpu_kill)(unsigned int cpu);void (*cpu_die)(unsigned int cpu);
arch/arm/kernel/smp.c +3 -0
@@ -564,6 +564,9 @@ static void ipi_cpu_stop(unsigned int cpu)local_fiq_disable ();local_irq_disable (); +if (smp_ops.smp_notify_cpu_stop) + smp_ops.smp_notify_cpu_stop (cpu); +while (1 )cpu_relax (); }
Signature · 1 line
1.7.0.4
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
Hello,
Quoted 11 lines · Srinivas KANDAGATLA
From: Srinivas Kandagatla <[email protected]>
Use-case is simple kexec on smp system.
Lets say there are two cores core-0 and core-1.
sequence here is:
As part of kexec call, the secondary cores are stopped by callingmachine_shutdown(), which in turn issues IPIs to off-line the other CPUs and then call smp_ops.cpu_kill.
Quoted 6 lines
machine_shutdowninvokessmp_send_stopfrom core-0 which then sendsIPI_CPU_STOPipi message for core-1 and thensmp_kill_cpuscall smp_ops.cpu_kill from core-0 with cpu=1.The expectation here is that
cpu_killwill relocate secondary-core to safe location from primary-core(core-0).
Not quite. The expectation is actually that you wiggle a line to a power controller and the secondary core either disappears or resets.
Quoted 4 lines
This is a catch 22 situation for SOCs, which can't change PC's of other cores. At the point when
cpu_killis invoked, it cant execute any function calls for core-1, as it is off-line and is spinning incpu_relax.
I thought these SoCs were fairly rare in production -- usually there is some way to power down secondary cores.
Quoted 6 lines
So the only way is to put the core into safe place by executing some SOC specific relocation code.
Having a callback from ipi handler when cpu is stopped would be very usefull for SOC's which want to relocate the secondary-core to safe location.
I thought about this for Versatile Express but even if you do that, your problems are just beginning. The bigger issue is co-ordinating with the secondaries so you know when they're safely out of the way and you can proceed with the kexec. Remember that you won't have working locks and you can't cross-call a function because it won't actually return.
If you can solve the synchronisation problem then we can think about adding these hooks.
Will
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
Quoted 14 lines · Will Deacon
Hello,
Quoted 11 lines · Srinivas KANDAGATLA
From: Srinivas Kandagatla <[email protected]>
Use-case is simple kexec on smp system.
Lets say there are two cores core-0 and core-1.
sequence here is:
As part of kexec call, the secondary cores are stopped by callingmachine_shutdown(), which in turn issues IPIs to off-line the other CPUs and then call smp_ops.cpu_kill.
Quoted 8 lines
Quoted 6 lines
machine_shutdowninvokessmp_send_stopfrom core-0 which then sendsIPI_CPU_STOPipi message for core-1 and thensmp_kill_cpuscall smp_ops.cpu_kill from core-0 with cpu=1.The expectation here is that
cpu_killwill relocate secondary-core to safe location from primary-core(core-0).Not quite. The expectation is actually that you wiggle a line to a power controller and the secondary core either disappears or resets.
Ok, I got this. But In our chips we can't control power/reset to individual cores, reset is tied up to both the cores.
Quoted 7 lines
Quoted 5 lines
This is a catch 22 situation for SOCs, which can't change PC's of other cores. At the point when
cpu_killis invoked, it cant execute any function calls for core-1, as it is off-line and is spinning incpu_relax.I thought these SoCs were fairly rare in production -- usually there is some way to power down secondary cores.
We have at-least 3 chips of this category, and we are looking forward to mainline the linux-support soon.
Quoted 9 lines
Quoted 7 lines
So the only way is to put the core into safe place by executing some SOC specific relocation code.
Having a callback from ipi handler when cpu is stopped would be very usefull for SOC's which want to relocate the secondary-core to safe location.
I thought about this for Versatile Express but even if you do that, your problems are just beginning.
Ok, atleast some SOC has similar requirement.
Quoted 4 lines
The bigger issue is co-ordinating with the
secondaries so you know when they're safely out of the way and you can proceed with the kexec. Remember that you won't have working locks and you can't cross-call a function because it won't actually return.
Yes I agree and understand the limitation.
The callback should simple and just relocate the secondary core to a safe place which in our case is a holding pen, which is safe and not in the system ram.
After the callback, the secondary core will be spinning in the holding pen which will be released once they get a matching cpu-id. And this approach works perfectly ok on our chips and I guess should work for other chips aswell.
Quoted 2 lines
If you can solve the synchronisation problem then we can think about adding these hooks.
I think the call back code should not do anything complex other than few lines of assembly or jump to a holding pen, this way it does not need any synchronization calls or system infrastructure.
Most of the SOCs in the current tree return 1 for cpu_kill callback, Am not sure if they fit in this category.
I think adding the callback hook would also benefit other chips which fit in this category.
??
Thanks,
srini
Quoted 3 lines
Will
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
Quoted 7 lines · Srinivas KANDAGATLA
Quoted 5 lines · Will Deacon
The bigger issue is co-ordinating with the
secondaries so you know when they're safely out of the way and you can proceed with the kexec. Remember that you won't have working locks and you can't cross-call a function because it won't actually return.
Quoted 8 lines
Yes I agree and understand the limitation.
The callback should simple and just relocate the secondary core to a safe place which in our case is a holding pen, which is safe and not in the system ram.
After the callback, the secondary core will be spinning in the holding pen which will be released once they get a matching cpu-id. And this approach works perfectly ok on our chips and I guess should work for other chips aswell.
But how do you know when the callback is complete? That's the part that is tricky as you need to avoid clobbering the kernel image before you know for sure that all the secondaries are out of the way. I think you either need some horrible homebrew locking primitives or something in hardware to signal back to the primary CPU.
Quoted 6 lines
Quoted 2 lines
If you can solve the synchronisation problem then we can think about adding these hooks.
I think the call back code should not do anything complex other than few lines of assembly or jump to a holding pen, this way it does not need any synchronization calls or system infrastructure.
See above.
Will
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
Quoted 14 lines · Will Deacon
Quoted 5 lines · Srinivas KANDAGATLA
Quoted 4 lines · Will Deacon
The bigger issue is co-ordinating with the
secondaries so you know when they're safely out of the way and you can proceed with the kexec. Remember that you won't have working locks and you can't cross-call a function because it won't actually return.Quoted 8 lines
Yes I agree and understand the limitation.
The callback should simple and just relocate the secondary core to a safe place which in our case is a holding pen, which is safe and not in the system ram.
After the callback, the secondary core will be spinning in the holding pen which will be released once they get a matching cpu-id. And this approach works perfectly ok on our chips and I guess should work for other chips aswell.
Quoted 3 lines
But how do you know when the callback is complete? That's the part that is tricky as you need to avoid clobbering the kernel image before you know for sure that all the secondaries are out of the way.
Yes, there is a very small window of opportunity for the primary core to continue with new kernel image.
We have two options here:
1> I think the existing infrastructure of cpu_kill can be re-used to know if the callback is complete and is very much specific to platform implementation.
2> Add a is_cpu_stopped() in smp_operations and call it from primary core with cpu set to secondary cores, similar to smp_kill_cpus call. This is also specific to platform implementation.
Both of them will address the concern.
???
Quoted 12 lines
I think you either need
some horrible homebrew locking primitives or something in hardware to signal back to the primary CPU.Quoted 5 lines
Quoted 2 lines
If you can solve the synchronisation problem then we can think about adding these hooks.
I think the call back code should not do anything complex other than few lines of assembly or jump to a holding pen, this way it does not need any synchronization calls or system infrastructure.
See above.
Will
RFC PATCH 1/1 ARM:smp: introduce smp_notify_cpu_stop to fix kexec smp case
Quoted 15 lines · Srinivas KANDAGATLA
Quoted 14 lines · Will Deacon
Quoted 13 lines · Srinivas KANDAGATLA
Quoted 4 lines · Will Deacon
The bigger issue is co-ordinating with the
secondaries so you know when they're safely out of the way and you can proceed with the kexec. Remember that you won't have working locks and you can't cross-call a function because it won't actually return.Yes I agree and understand the limitation.
The callback should simple and just relocate the secondary core to a safe place which in our case is a holding pen, which is safe and not in the system ram.
After the callback, the secondary core will be spinning in the holding pen which will be released once they get a matching cpu-id. And this approach works perfectly ok on our chips and I guess should work for other chips aswell.
Quoted 5 lines
Quoted 3 lines
But how do you know when the callback is complete? That's the part that is tricky as you need to avoid clobbering the kernel image before you know for sure that all the secondaries are out of the way.
Yes, there is a very small window of opportunity for the primary core to continue with new kernel image.
It might not be that small, particulary if the secondaries are doing something with interrupts disabled and the signal from the primary is delayed.
Quoted 5 lines
We have two options here:
1> I think the existing infrastructure of
cpu_killcan be re-used to know if the callback is complete and is very much specific to platform implementation.
How? I would just like a description of one concrete implementation so that I know we're adding something which is actually usable.
Quoted 3 lines
2> Add a
is_cpu_stopped()insmp_operationsand call it from primary core with cpu set to secondary cores, similar tosmp_kill_cpuscall. This is also specific to platform implementation.
Again, how does is_cpu_stopped() work? I'm not even sure we can guarantee coherency on this path.
Perhaps it's best if you include these patches with the platform support for a board that uses it, then we can see the bigger picture.
Will