ARM: avoid Cortex-A9 livelock on tight dmb loops
Subsystems (4)
PATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Executing loops such as:
while (1) cpu_relax();
with interrupts disabled results in a livelock of the entire system, as other CPUs are prevented making progress. This is most noticable as a failure of crashdump kexec, which stops just after issuing:
Loading crashdump kernel...
to the system console. Two other locations of these loops within the ARM code have been identified and fixed up.
--- arch/arm/kernel/machine_kexec.c | 3 ++- arch/arm/kernel/smp.c | 2 +- arch/arm/mach-omap2/prm_common.c | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-)
arch/arm/kernel/machine_kexec.c +2 -1
@@ -91,8 +91,9 @@ void machine_crash_nonpanic_core(void *unused)set_cpu_online (smp_processor_id (), false);atomic_dec (&waiting_for_crash_ipi); +while (1 ) -cpu_relax (); +cpu_do_idle (); }static void machine_kexec_mask_interrupts (void )
arch/arm/kernel/smp.c +1 -1
@@ -568,7 +568,7 @@ static void ipi_cpu_stop(unsigned int cpu)local_irq_disable ();while (1 ) -cpu_relax (); +cpu_do_idle (); } static DEFINE_PER_CPU(struct completion *, cpu_completion);
arch/arm/mach-omap2/prm_common.c +1 -1
@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }/**
Signature · 1 line
2.7.4
Tony Lindgren
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 528
Quoted 12 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }
Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Regards,
Tony
PATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 8 lines · Tony Lindgren
Quoted 7 lines · Russell King
arch/arm/mach-omap2/prm_common.c +0
-0@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();
Quoted 8 lines
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 528
Quoted 5 lines
arch/arm/mach-omap2/prm_common.c +1
-1while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
-Tero
--
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
PATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 16 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Quoted 6 lines
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
An alternative to WFI would be a DSB, which should avoid unexpected interactions with power management.
Will
Russell King - ARM Linux
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
In reply to Tero KristoQuoted 16 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Quoted 6 lines
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up According to speedtest.net: 8.21Mbps down 510kbps up
PATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 21 lines · Russell King - ARM Linux
Quoted 20 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Quoted 8 lines
Quoted 2 lines
Has this patch been tested on any platform that supports proper power management?
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
Yea, I would definitely prefer this over adding arbitrary WFIs in the kernel.
-Tero
Quoted 7 lines
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
--
Texas Instruments Finland Oy, Porkkalankatu 22, 00180 Helsinki. Y-tunnus/Business ID: 0615521-4. Kotipaikka/Domicile: Helsinki
PATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
In reply to Russell King - ARM LinuxQuoted 27 lines · Russell King - ARM Linux
Quoted 16 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Quoted 6 lines
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
Quoted 8 lines
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
Yes. If the ondemand governor detects that CPU was busy greater than 80% of the time it bumps to the highest OPP and can lead to higher temperatures though CPU might not be doing anything useful.
Quoted 1 line
Russell King - ARM Linux
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 13 lines · Keerthy
Quoted 11 lines · Russell King - ARM Linux
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
Quoted 3 lines
Yes. If the ondemand governor detects that CPU was busy greater than 80% of the time it bumps to the highest OPP and can lead to higher temperatures though CPU might not be doing anything useful.
That probably wouldn't happen - all these paths are concerned with stopping CPUs doing something as a result of either a panic, a crash or a failed attempt to reset the system.
We'd enter them in whatever operating state the system was in at the time, which is indeterminant. What we can be relatively sure about is that no further operating state transitions will occur.
For example, in the case of a crash with kexec and a crashdump kernel loaded, the non-crashing CPUs end up in machine_crash_nonpanic_core(). Should kexec fail, then the system stops leaving all but one CPU spinning in that function in whatever operating state they were in, which could be the highest OPP.
This means that, for example, in the case of a four CPU system, three CPUs will be spinning hard on whatever instructions we have there, while one CPU is trying to perform cache operations to prepare to boot the crashdump kernel.
For a panic, it's very similar - the CPUs which didn't call panic() are directed to ipi_cpu_stop() where they spin. By default, a panic() halts the panicing CPU and nothing further happens, so the other CPUs will endlessly spin in the same way as above. The panicing CPU may be waiting for the panic timeout to expire before trying to reboot the system.
The OMAP reset case is slightly different, because that's a case of failure-to-reboot - combine that with a panic timeout, and you can end up with _all_ CPUs in the system indefinitely spinning hard in a tight loop.
Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up According to speedtest.net: 8.21Mbps down 510kbps up
Tony Lindgren
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
In reply to Russell King - ARM LinuxQuoted 27 lines · Russell King - ARM Linux
Quoted 16 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Quoted 6 lines
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
Quoted 8 lines
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
We already have code in the kernel (and in the bootrom) to "park" a cpu after starting. But using it without resetting the cpu would require 1-1 memory mapping or modifying the code. That is if we wanted to use the same code also for parking the cpus for kexec without resetting them.
Regards,
Tony
Russell King - ARM Linux
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 37 lines · Tony Lindgren
Quoted 27 lines · Russell King - ARM Linux
Quoted 16 lines · Tero Kristo
Quoted 14 lines · Tony Lindgren
arch/arm/mach-omap2/prm_common.c void omap_prm_reset_system(void) · line 526
Quoted 10 lines · Russell King
arch/arm/mach-omap2/prm_common.c +1
-1@@ -523,7 +523,7 @@ void omap_prm_reset_system(void) prm_ll_data->reset_system ();while (1 ) -cpu_relax (); +cpu_do_idle (); }Hmm we need to check so the added WFI here does not cause an undesired change to a low power state. Adding Tero to Cc also.
Quoted 6 lines
Generally it is a bad idea to call arbitrary WFI within OMAP architecture, as this triggers a PRCM power transition and will most likely cause a hang if not controlled properly.
Has this patch been tested on any platform that supports proper power management?
That will also go for the other locations in this patch too, as they are all callable on _any_ platform.
Quoted 8 lines
It sounds like we need to abstract this so that platforms where "wfi" is complex can handle the "spin on this CPU forever" appropriately.
While we could use dsb, we're asking a CPU to indefinitely spin in a tight loop, which isn't going to be good for power consumption - what if we have three CPUs doing that, could it push a SoC over the thermal limits? I don't think that's a question we can confidently answer except for specific SoCs.
Quoted 5 lines
We already have code in the kernel (and in the bootrom) to "park" a cpu after starting. But using it without resetting the cpu would require 1-1 memory mapping or modifying the code. That is if we wanted to use the same code also for parking the cpus for kexec without resetting them.
In which case, how about using:
while (1) {
cpu_relax();
wfe();
}instead - that appears to also have the desired effect, allowing kdump to work on the SDP4430.
Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up According to speedtest.net: 8.21Mbps down 510kbps up
Russell King - ARM Linux
maintainerPATCH 1/1 ARM: avoid Cortex-A9 livelock on tight dmb loops
Quoted 9 lines · Russell King - ARM Linux
Quoted 5 lines · Tony Lindgren
We already have code in the kernel (and in the bootrom) to "park" a cpu after starting. But using it without resetting the cpu would require 1-1 memory mapping or modifying the code. That is if we wanted to use the same code also for parking the cpus for kexec without resetting them.
In which case, how about using:
Quoted 7 lines
while (1) { cpu_relax(); wfe(); }instead - that appears to also have the desired effect, allowing kdump to work on the SDP4430.
... but results in compile failures on non-ARMv7 targets.
Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 8.8Mbps down 630kbps up According to speedtest.net: 8.21Mbps down 510kbps up