imx6: PCIe imx6_pcie_assert_core_reset() hangs after watchdog reset

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

		val = readl(pp->dbi_base + PCIE_PL_PFLR);

The watchdog will issue a new reset once it times out again. But Linux will hang here the next time as well. Only a power-cycle helps. For a test purpose, I've removed the following part from this function:

	if ((gpr1 & IMX6Q_GPR1_PCIE_REF_CLK_EN) &&
	    (gpr12 & IMX6Q_GPR12_PCIE_CTL_2)) {
		...
	}

Without it, Linux boots up fine. And PCIe also seems to be fully functional.

I have to admit that I have no real insight in the imx6 / DW PCIe driver. So I'm checking with you experts, if you have any ideas why this could be the case? And how this could be solved.

BTW: This happens with v3.19 and v4.0-rc5 on a i.MX6Solo based board (TQMa6S SoM). With no PCIe support enabled in the bootloader (U-Boot).

Am Mittwoch, den 25.03.2015, 10:29 +0100 schrieb Stefan Roese:

Quoted 23 lines

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

		val = readl(pp->dbi_base + PCIE_PL_PFLR);

The watchdog will issue a new reset once it times out again. But Linux will hang here the next time as well. Only a power-cycle helps. For a test purpose, I've removed the following part from this function:

	if ((gpr1 & IMX6Q_GPR1_PCIE_REF_CLK_EN) &&
	    (gpr12 & IMX6Q_GPR12_PCIE_CTL_2)) {
		...
	}

Without it, Linux boots up fine. And PCIe also seems to be fully functional.

Quoted 6 lines

I have to admit that I have no real insight in the imx6 / DW PCIe driver. So I'm checking with you experts, if you have any ideas why this could be the case? And how this could be solved.

BTW: This happens with v3.19 and v4.0-rc5 on a i.MX6Solo based board (TQMa6S SoM). With no PCIe support enabled in the bootloader (U-Boot).

The problem here is that the LTSSM state machine stays in "Link" state after the WD reset and no configuration of the core can happen in this state. The path inside the above if clause applies a workaround the get LTSSM back into the initial state.

As this is a bit racy and not the intended way to do things we only apply this workaround if we are relatively certain that the PCIe link had been enabled without a reset to the core before.

So what you are saying is that after the WD reset the GPR registers are cleared?

If so we may need to make the logic to decide when to apply this WAR a bit smarter.

Quoted 31 lines

Am Mittwoch, den 25.03.2015, 10:29 +0100 schrieb Stefan Roese:

Quoted 23 lines

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

		val = readl(pp->dbi_base + PCIE_PL_PFLR);

The watchdog will issue a new reset once it times out again. But Linux will hang here the next time as well. Only a power-cycle helps. For a test purpose, I've removed the following part from this function:

	if ((gpr1 & IMX6Q_GPR1_PCIE_REF_CLK_EN) &&
	    (gpr12 & IMX6Q_GPR12_PCIE_CTL_2)) {
		...
	}

Without it, Linux boots up fine. And PCIe also seems to be fully functional.

Quoted 6 lines

I have to admit that I have no real insight in the imx6 / DW PCIe driver. So I'm checking with you experts, if you have any ideas why this could be the case? And how this could be solved.

BTW: This happens with v3.19 and v4.0-rc5 on a i.MX6Solo based board (TQMa6S SoM). With no PCIe support enabled in the bootloader (U-Boot).

Quoted 8 lines

The problem here is that the LTSSM state machine stays in "Link" state after the WD reset and no configuration of the core can happen in this state. The path inside the above if clause applies a workaround the get LTSSM back into the initial state.

As this is a bit racy and not the intended way to do things we only apply this workaround if we are relatively certain that the PCIe link had been enabled without a reset to the core before.

Thanks for the explanation.

Quoted 2 lines

So what you are saying is that after the WD reset the GPR registers are cleared?

No, they are not cleared.

After a power-on reset, these are the GPR values:
gpr1=48401005 gpr12=0f000000

Here the values after a "normal" reset issued via the reboot command:
gpr1=48441005 gpr12=0f004090

This is because of the shutdown handler calling
imx6_pcie_assert_core_reset().

Now after the watchdog reset, its a bit different:
gpr1=48411005 gpr12=0f004490

So this is only time (in my test cases), that the if clause is entered (and hangs).

Quoted 31 lines

Am Mittwoch, den 25.03.2015, 10:29 +0100 schrieb Stefan Roese:

Quoted 23 lines

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

		val = readl(pp->dbi_base + PCIE_PL_PFLR);

The watchdog will issue a new reset once it times out again. But Linux will hang here the next time as well. Only a power-cycle helps. For a test purpose, I've removed the following part from this function:

	if ((gpr1 & IMX6Q_GPR1_PCIE_REF_CLK_EN) &&
	    (gpr12 & IMX6Q_GPR12_PCIE_CTL_2)) {
		...
	}

Without it, Linux boots up fine. And PCIe also seems to be fully functional.

Quoted 6 lines

I have to admit that I have no real insight in the imx6 / DW PCIe driver. So I'm checking with you experts, if you have any ideas why this could be the case? And how this could be solved.

BTW: This happens with v3.19 and v4.0-rc5 on a i.MX6Solo based board (TQMa6S SoM). With no PCIe support enabled in the bootloader (U-Boot).

Quoted 8 lines

The problem here is that the LTSSM state machine stays in "Link" state after the WD reset and no configuration of the core can happen in this state. The path inside the above if clause applies a workaround the get LTSSM back into the initial state.

As this is a bit racy and not the intended way to do things we only apply this workaround if we are relatively certain that the PCIe link had been enabled without a reset to the core before.

Thanks for the explanation.

Quoted 2 lines

So what you are saying is that after the WD reset the GPR registers are cleared?

No, they are not cleared.

After a power-on reset, these are the GPR values:
gpr1=48401005 gpr12=0f000000

Here the values after a "normal" reset issued via the reboot command:
gpr1=48441005 gpr12=0f004090

This is because of the shutdown handler calling
imx6_pcie_assert_core_reset().

Now after the watchdog reset, its a bit different:
gpr1=48411005 gpr12=0f004490

So this is only time (in my test cases), that the if clause is entered (and hangs).

Quoted 2 lines

If so we may need to make the logic to decide when to apply this WAR a bit smarter.

Am Mittwoch, den 25.03.2015, 11:32 +0100 schrieb Stefan Roese:
Okay, I've looked a bit into this and it seems there is no easy solution available. It is really unfortunate that the WD reset doesn't reset the GPR registers. Also I have no idea if the WD reset properly resets the PCIe core, as the reset signal of this core is only wired to the POR line.

To fix this (almost) properly we would have to change the complete init order of the core, which isn't an easy task, as experience has shown that even small changes in that area can prevent the link from coming up under certain circumstances.

Which brings me back to my earlier assertion that WD reset should really be done through the PMIC. That's yet another case of a WD making the overall system more instable.

I think the easiest workaround for now is to detect the WD reset in your bootloader and bash the expected default values into the GPR bits.

Regards,
Lucas

Quoted 16 lines

Am Mittwoch, den 25.03.2015, 11:32 +0100 schrieb Stefan Roese:
Okay, I've looked a bit into this and it seems there is no easy solution available. It is really unfortunate that the WD reset doesn't reset the GPR registers. Also I have no idea if the WD reset properly resets the PCIe core, as the reset signal of this core is only wired to the POR line.

To fix this (almost) properly we would have to change the complete init order of the core, which isn't an easy task, as experience has shown that even small changes in that area can prevent the link from coming up under certain circumstances.

Which brings me back to my earlier assertion that WD reset should really be done through the PMIC. That's yet another case of a WD making the overall system more instable.

Quoted 5 lines

I think the easiest workaround for now is to detect the WD reset in your bootloader and bash the expected default values into the GPR bits.

Regards,
Lucas

Lucas,

There are many boards out there that unfortunately don't reset PMIC's properly, IMHO due to a confusing reference design from Freescale.

Using the WDOGx_WRSR register we can detect the reason for reset:
Bit 4 - POR - Power on reset
Bit 1 - TOUT - Watchdog timeout
Bit 0 - SFTW - Software reset (used in machine_restart)

Can we reset the GPR registers based on bits 0 or 1 set, or use these as further qualifiers in the WAR?

Hi Tim,

Am Donnerstag, den 26.03.2015, 06:23 -0700 schrieb Tim Harvey:

Quoted 28 lines · Lucas Stach

Quoted 16 lines

Am Mittwoch, den 25.03.2015, 11:32 +0100 schrieb Stefan Roese:
Okay, I've looked a bit into this and it seems there is no easy solution available. It is really unfortunate that the WD reset doesn't reset the GPR registers. Also I have no idea if the WD reset properly resets the PCIe core, as the reset signal of this core is only wired to the POR line.

To fix this (almost) properly we would have to change the complete init order of the core, which isn't an easy task, as experience has shown that even small changes in that area can prevent the link from coming up under certain circumstances.

Which brings me back to my earlier assertion that WD reset should really be done through the PMIC. That's yet another case of a WD making the overall system more instable.

Quoted 5 lines

I think the easiest workaround for now is to detect the WD reset in your bootloader and bash the expected default values into the GPR bits.

Regards,
Lucas

Lucas,

There are many boards out there that unfortunately don't reset PMIC's properly, IMHO due to a confusing reference design from Freescale.

Quoted 8 lines

Using the WDOGx_WRSR register we can detect the reason for reset:
Bit 4 - POR - Power on reset
Bit 1 - TOUT - Watchdog timeout
Bit 0 - SFTW - Software reset (used in machine_restart)

Can we reset the GPR registers based on bits 0 or 1 set, or use these as further qualifiers in the WAR?

Doing it in the kernel is too late. This WAR is specifically to work around bootloaders leaving the PCI link running when jumping into the kernel. We use the GPR bits to detect if the bootloader has touched the PCIe core. Unfortunately we see the same signature if the kernel touched PCIe and the system got reset by the WD.

If we reset the GPR bits depending on the reset reason register in the kernel we have no way to know if the bootloader has touched the PCIe core after a WD reset (in which case we still need to apply the WAR). So the only way to keep the WAR working while cleaning out bad WD reset behavior is to reset the GPR bits in the bootloader, before the bootloader itself may touch the PCIe core.

It may be possible to come up with a solution for this in the kernel by looking at the PCIe clock status when entering the kernel, but this means that we scatter even more WAR code for the bad Freescale PCIe integration throughout the kernel, as such code has to go into the platform as we don't have the required information at hand in the PCI driver. So I'm not really thrilled by thinking about doing it in the kernel.

Regards,
Lucas

Signature · 8 lines

Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

I use either v3.17 or Freescale's 4.1.15_1.2.0_ga

Quoted 2 lines

I use either v3.17 or Freescale's 4.1.15_1.2.0_ga

These kernel versions are not supported by the kernel community.

Do you observe issues with 4.8.6 or 4.9-rc4?

The problem is definitely present in current mainline Linux. I intent to remove this workaround from the kernel, but we need to make sure that the bootloader properly disables PCIe before jumping to the kernel image.

I don't know what the status of this is in U-Boot, but Barebox already does this correctly.

Fabio, would you mind to port this coed to U-Boot, or at least check if it does the right thing?

Hi Lucas,

Quoted 5 lines · Lucas Stach

The problem is definitely present in current mainline Linux. I intent to remove this workaround from the kernel, but we need to make sure that the bootloader properly disables PCIe before jumping to the kernel image.

Quoted 5 lines

I don't know what the status of this is in U-Boot, but Barebox already does this correctly.

Fabio, would you mind to port this coed to U-Boot, or at least check if it does the right thing?

Sure, could you point me to the Barebox fix so I can take a look?

Hi Lucas,

Quoted 5 lines · Lucas Stach

The problem is definitely present in current mainline Linux. I intent to remove this workaround from the kernel, but we need to make sure that the bootloader properly disables PCIe before jumping to the kernel image.

Quoted 5 lines

I don't know what the status of this is in U-Boot, but Barebox already does this correctly.

Fabio, would you mind to port this coed to U-Boot, or at least check if it does the right thing?

Sure, could you point me to the Barebox fix so I can take a look?

Did you have some time to validate/fix this in U-Boot? I would like to get the workaround removed from the kernel.

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

Hi Stefan,

Quoted 7 lines · Stefan Roese

Hi!

I'm currently facing a problem in the imx6 PCIe driver. The problem happens, after the Linux imx6 internal watchdog has timed-out and issues a reset (this is intended, as I'm testing this wdog right now). But the next Linux boot hangs in imx6_pcie_assert_core_reset() while accessing a register from the PCIe port logic memory:

I am not able to reproduce this problem on a imx6q-sabresd.

Even with PCI enabled in U-boot, when I run 'reboot' the kernel re-starts fine here.

The problem appears when a watchdog reset happens, so the kernel has no chance to clean things up before rebooting. This isn't a problem if you do a full system reset via the watchdog out line to the PMIC, but gets ugly if only the i.MX6 is reset (which apparently doesn't clear the GPR bits).