ARM: Temporary fix for broken arch reboot
PATCH 1/1 ARM: Temporary fix for broken arch reboot
With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
---
Tested with OMAP4 device.
arch/arm/kernel/process.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-)
arch/arm/kernel/process.c +0 -1
@@ -247,7 +247,6 @@ void machine_power_off(void)void machine_restart (char *cmd) { -machine_shutdown ();arm_pm_restart (reboot_mode, cmd); }
Signature · 1 line
1.6.0.4
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 6 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
Catalin?
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 12 lines
-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Tuesday, November 09, 2010 6:49 PM
To: Shilimkar, Santosh; Catalin Marinas
Cc: linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 4 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()
Quoted 6 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
This behaviour depends on the reset mechanism implemented. On OMAP, we have a way to do a reset on all the CPUs with single register bit, like a global warm/cold reset.
Quoted 5 lines
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
Catalin?
PATCH 1/1 ARM: Temporary fix for broken arch reboot
In reply to Russell King - ARM LinuxQuoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()
What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?
Quoted 9 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
I can't think of anything. Did the other CPUs print 'stopping'?
Catalin
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?
I didn't try connecting with a debugger - will try this.
Quoted 12 lines
Quoted 9 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
I can't think of anything. Did the other CPUs print 'stopping'?
I'm fairly certain the other CPUs did not print 'stopping'. Will double check.
- Anand
PATCH 1/1 ARM: Temporary fix for broken arch reboot
In reply to Catalin MarinasQuoted 8 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot
Quoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?
It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
I can't think of anything. Did the other CPUs print 'stopping'?
No it doesn't not print anything.
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 26 lines · Santosh
Quoted 8 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?
It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
I can't think of anything. Did the other CPUs print 'stopping'?
No it doesn't not print anything.
The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.
You might be able to get something from the other CPUs by inserting a mdelay(100) after the call to machine_shutdown(), to give the other CPUs time to respond.
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 31 lines · Russell King - ARM Linux
Quoted 30 lines · Santosh
Quoted 8 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?
It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
I can't think of anything. Did the other CPUs print 'stopping'?
No it doesn't not print anything.
The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.
The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.
I also wonder what happens when you stop a secondary CPU without clearing the SMP/nAMP bit first. The SCU may still consider it as part of the inner coherency domain. We were clearing this in some ARM tests for CPU hotplug but mainly to improve the SCU speed.
BTW, does setup_mm_for_reboot works if PHYS_OFFSET > PAGE_OFFSET?
Catalin
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 55 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 6 lines · Russell King - ARM Linux
Quoted 5 lines · Santosh
Quoted 5 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org;Gadiyar,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 1 line · Russell King - ARM Linux
On Tue, Nov 09, 2010 at 06:40:39PM +0530, Shilimkar, Santosh
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
With commit
3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doingflush_pmd_entry()as part ofsetup_mm_for_reboot()What do you mean by 'dies'? Can you still connect with a debugger or
it
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
got to some weird state?
It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly
So you're now rebooting with the secondary CPUs still running. I
guess
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
that the secondary CPUs end up crashing and don't restart.
I think more the question is why the CP15 cache clean/flush is
hanging
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
with the other CPUs taken down. All the other CPUs will be doing
is
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
sitting in a loop doing nothing.
I can't think of anything. Did the other CPUs print 'stopping'?
No it doesn't not print anything.
The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.
Quoted 6 lines
The printing only happens for
SYSTEM_BOOTINGorSYSTEM_RUNNING. I suspect in this case we haveSYSTEM_RESTARTINGand the condition inipi_cpu_stop()is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.
While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.
Just adding a delay in the ipi_cpu_stop() makes the reboot work as well
arch/arm/kernel/smp.c +2 -0
@@ -526,6 +526,8 @@ static void ipi_cpu_stop(unsigned int cpu)spin_unlock (&stop_lock); } +udelay (500 ); +set_cpu_online (cpu, false);local_fiq_disable ();
Regards,
Santosh
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 16 lines · Santosh
Quoted 10 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 6 lines
The printing only happens for
SYSTEM_BOOTINGorSYSTEM_RUNNING. I suspect in this case we haveSYSTEM_RESTARTINGand the condition inipi_cpu_stop()is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.
Quoted 6 lines
While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.
Just adding a delay in the
ipi_cpu_stop()makes the reboot work as well
That could just be that it's taking longer than 500us for the reboot CPU to get through setup_mm_for_reboot().
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 29 lines
-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Wednesday, November 10, 2010 9:16 PM
To: Shilimkar, Santosh
Cc: Catalin Marinas; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 10 lines · Santosh
Quoted 10 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 3 lines
Quoted 3 lines
The printing only happens for
SYSTEM_BOOTINGorSYSTEM_RUNNING. I suspect in this case we haveSYSTEM_RESTARTINGand the condition inipi_cpu_stop()is false, therefore no printing. It may be worthputting
Quoted 7 lines
Quoted 3 lines
some printks outside the 'if' to see whether the secondary CPUs get there.
While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.
Quoted 5 lines
Quoted 2 lines
Just adding a delay in the
ipi_cpu_stop()makes the reboot work as wellThat could just be that it's taking longer than 500us for the reboot CPU to get through
setup_mm_for_reboot().
It not seems to be the case otherwise below patch should have worked. Right ? Did I miss your point ?
arch/arm/kernel/process.c +2 -1
@@ -103,6 +103,8 @@ void arm_machine_restart(char mode, const char *cmd)* soft boot works. */ setup_mm_for_reboot (mode); + +machine_shutdown ();/* Clean and invalidate caches */ flush_cache_all ();
@@ -247,7 +249,6 @@ void machine_power_off(void)void machine_restart (char *cmd) { -machine_shutdown ();arm_pm_restart (reboot_mode, cmd); }
PATCH 1/1 ARM: Temporary fix for broken arch reboot
Quoted 29 lines · Santosh
Quoted 29 lines
-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Wednesday, November 10, 2010 9:16 PM
To: Shilimkar, Santosh
Cc: Catalin Marinas; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 10 lines · Santosh
Quoted 10 lines
-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch rebootQuoted 3 lines
Quoted 3 lines
The printing only happens for
SYSTEM_BOOTINGorSYSTEM_RUNNING. I suspect in this case we haveSYSTEM_RESTARTINGand the condition inipi_cpu_stop()is false, therefore no printing. It may be worthputting
Quoted 7 lines
Quoted 3 lines
some printks outside the 'if' to see whether the secondary CPUs get there.
While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Just adding a delay in the
ipi_cpu_stop()makes the reboot work as wellThat could just be that it's taking longer than 500us for the reboot CPU to get through
setup_mm_for_reboot().It not seems to be the case otherwise below patch should have worked. Right ? Did I miss your point ?
Yes you did. Maybe all this code completes before your 500us in the secondary CPUs complete. So the delay in the secondary CPU path proves nothing.
A delay in the reboot CPU path ensures that the other CPUs have time to react to the stop IPI, so we know what state the system is in.