Skip to contentSkip to conversation

ARM: Temporary fix for broken arch reboot

PATCH New Shilimkar, Santosh 4 participants 12 messages Nov 9, 2010 – Nov 15, 2010 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

---
Tested with OMAP4 device.

 arch/arm/kernel/process.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)
arch/arm/kernel/process.c +0 -1
index e76fcaa..ac370b2 100644
@@ -247,7 +247,6 @@ void machine_power_off(void)
 
 void machine_restart(char *cmd)
 {
-	machine_shutdown();
 	arm_pm_restart(reboot_mode, cmd);
 }
 
Signature · 1 line
1.6.0.4
#1 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 6 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

Catalin?

#2 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 12 lines

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Tuesday, November 09, 2010 6:49 PM
To: Shilimkar, Santosh; Catalin Marinas
Cc: linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 4 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

Quoted 6 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

This behaviour depends on the reset mechanism implemented. On OMAP, we have a way to do a reset on all the CPUs with single register bit, like a global warm/cold reset.

Quoted 5 lines

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

Catalin?

#3 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

In reply to Russell King - ARM Linux
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?

Quoted 9 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

I can't think of anything. Did the other CPUs print 'stopping'?

Catalin

#4 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 9 lines · Catalin Marinas
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?

I didn't try connecting with a debugger - will try this.

Quoted 12 lines
Quoted 9 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

I can't think of anything. Did the other CPUs print 'stopping'?

I'm fairly certain the other CPUs did not print 'stopping'. Will double check.

- Anand

#5 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

In reply to Catalin Marinas
Quoted 8 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?

It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.

Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

I can't think of anything. Did the other CPUs print 'stopping'?

No it doesn't not print anything.

#6 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 26 lines · Santosh
Quoted 8 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?

It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

I can't think of anything. Did the other CPUs print 'stopping'?

No it doesn't not print anything.

The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.

You might be able to get something from the other CPUs by inserting a mdelay(100) after the call to machine_shutdown(), to give the other CPUs time to respond.

#7 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 31 lines · Russell King - ARM Linux
Quoted 30 lines · Santosh
Quoted 8 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 9 lines
Quoted 4 lines · Russell King - ARM Linux
Quoted 3 lines · Santosh

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or it got to some weird state?

It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.

Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I guess that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is hanging with the other CPUs taken down. All the other CPUs will be doing is sitting in a loop doing nothing.

Quoted 6 lines
Quoted 2 lines
Quoted 1 line

I can't think of anything. Did the other CPUs print 'stopping'?

No it doesn't not print anything.

The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.

The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.

I also wonder what happens when you stop a secondary CPU without clearing the SMP/nAMP bit first. The SCU may still consider it as part of the inner coherency domain. We were clearing this in some ARM tests for CPU hotplug but mainly to improve the SCU speed.

BTW, does setup_mm_for_reboot works if PHYS_OFFSET > PAGE_OFFSET?

Catalin

#8 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 55 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 6 lines · Russell King - ARM Linux
Quoted 5 lines · Santosh
Quoted 5 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Tuesday, November 09, 2010 10:08 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org;

Gadiyar,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 1 line · Russell King - ARM Linux

On Tue, Nov 09, 2010 at 06:40:39PM +0530, Shilimkar, Santosh

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

With commit 3d3f78d752bf, reboot seems to broken on ARM machines. CPU dies while doing flush_pmd_entry() as part of setup_mm_for_reboot()

What do you mean by 'dies'? Can you still connect with a debugger or

it

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

got to some weird state?

It goes to some weird state. Basically the emulation connection dies, and debugger gets disconnected.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

I know this is not the fix but intention is to report the issue and also provide temporary fix till it get fixed correctly

So you're now rebooting with the secondary CPUs still running. I

guess

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

that the secondary CPUs end up crashing and don't restart.

I think more the question is why the CP15 cache clean/flush is

hanging

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

with the other CPUs taken down. All the other CPUs will be doing

is

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

sitting in a loop doing nothing.

I can't think of anything. Did the other CPUs print 'stopping'?

No it doesn't not print anything.

The processing of the IPI is asynchronous to the CPU which is rebooting continuing - which means that if there is some kind of bus lockup, you won't get anything from any of the CPUs.

Quoted 6 lines

The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.

While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.

Just adding a delay in the ipi_cpu_stop() makes the reboot work as well

arch/arm/kernel/smp.c +2 -0
index 8c19595..f7dadbf 100644
@@ -526,6 +526,8 @@ static void ipi_cpu_stop(unsigned int cpu)
                spin_unlock(&stop_lock);
        }
 
+       udelay(500);
+
        set_cpu_online(cpu, false);
 
        local_fiq_disable();

Regards,
Santosh

#9 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 16 lines · Santosh
Quoted 10 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 6 lines

The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth putting some printks outside the 'if' to see whether the secondary CPUs get there.

Quoted 6 lines

While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.

Just adding a delay in the ipi_cpu_stop() makes the reboot work as well

That could just be that it's taking longer than 500us for the reboot CPU to get through setup_mm_for_reboot().

#10 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 29 lines

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Wednesday, November 10, 2010 9:16 PM
To: Shilimkar, Santosh
Cc: Catalin Marinas; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 10 lines · Santosh
Quoted 10 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 3 lines
Quoted 3 lines

The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth

putting

Quoted 7 lines
Quoted 3 lines

some printks outside the 'if' to see whether the secondary CPUs get there.

While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.

Quoted 5 lines
Quoted 2 lines

Just adding a delay in the ipi_cpu_stop() makes the reboot work as well

That could just be that it's taking longer than 500us for the reboot CPU to get through setup_mm_for_reboot().

It not seems to be the case otherwise below patch should have worked. Right ? Did I miss your point ?

arch/arm/kernel/process.c +2 -1
index e76fcaa..c3ae1d6 100644
@@ -103,6 +103,8 @@ void arm_machine_restart(char mode, const char *cmd)
         * soft boot works.
         */
        setup_mm_for_reboot(mode);
+
+       machine_shutdown();
 
        /* Clean and invalidate caches */
        flush_cache_all();
@@ -247,7 +249,6 @@ void machine_power_off(void)
 
 void machine_restart(char *cmd)
 {
-       machine_shutdown();
        arm_pm_restart(reboot_mode, cmd);
 }
#11 lore raw

PATCH 1/1 ARM: Temporary fix for broken arch reboot

Quoted 29 lines · Santosh
Quoted 29 lines

-----Original Message-----
From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
Sent: Wednesday, November 10, 2010 9:16 PM
To: Shilimkar, Santosh
Cc: Catalin Marinas; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 10 lines · Santosh
Quoted 10 lines

-----Original Message-----
From: Catalin Marinas [mailto:catalin.marinas at arm.com]
Sent: Wednesday, November 10, 2010 7:35 PM
To: Russell King - ARM Linux
Cc: Shilimkar, Santosh; linux-arm-kernel at lists.infradead.org; Gadiyar, Anand
Subject: Re: [PATCH] ARM: Temporary fix for broken arch reboot

Quoted 3 lines
Quoted 3 lines

The printing only happens for SYSTEM_BOOTING or SYSTEM_RUNNING. I suspect in this case we have SYSTEM_RESTARTING and the condition in ipi_cpu_stop() is false, therefore no printing. It may be worth

putting

Quoted 7 lines
Quoted 3 lines

some printks outside the 'if' to see whether the secondary CPUs get there.

While doing some experiments on this issue, one interesting observation I made. Looks like there is race between two Cores which makes system behave badly in reboot path.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Just adding a delay in the ipi_cpu_stop() makes the reboot work as well

That could just be that it's taking longer than 500us for the reboot CPU to get through setup_mm_for_reboot().

It not seems to be the case otherwise below patch should have worked. Right ? Did I miss your point ?

Yes you did. Maybe all this code completes before your 500us in the secondary CPUs complete. So the delay in the secondary CPU path proves nothing.

A delay in the reboot CPU path ensures that the other CPUs have time to react to the stop IPI, so we know what state the system is in.