Skip to contentSkip to conversation

msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

DISCUSSION Paolo Pisati 3 participants 7 messages Jun 10, 2019 – Jun 13, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.

The issue is present in mainline up to 5.2-rc4, using defconfig and:

CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y

but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).

These are the last lines printed out:
...

[    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    7.538596] phy phy-34000.phy.0: phy init failed --> -110
[    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
[    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,

using dummy regulator

Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...

Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Any idea what is going on? Am i doing something wrong?

Signature · 2 lines
bye,
p.
#1 lore raw
Quoted 7 lines · Paolo Pisati

From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.

The issue is present in mainline up to 5.2-rc4, using defconfig and:

Quoted 7 lines

CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y

but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).

Hello Paolo,

I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets). The QMP block has 3 PCIe PHYs, that have their own clocks and resets.

Quoted 7 lines

These are the last lines printed out:
...

[    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out

^^ here the phy_init() called from pcie-qcom.c
which ends up to a call to qcom_qmp_phy_enable()

which has this code:

        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
                return ret;

qcom_qmp_phy_com_init() has this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

qcom_qmp_phy_com_init() later fails,
since the common block init time out, so the qmp driver disables clocks, asserts reset, and disables regulators

Quoted 2 lines
[    7.538596] phy phy-34000.phy.0: phy init failed --> -110
[    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110

^^ here the first PCIe controller instance fails to probe

Quoted 2 lines
[    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,

using dummy regulator

^^ here the second PCIe controller is probed.

it will call phy_init()

which will again call qcom_qmp_phy_enable() which will call qcom_qmp_phy_com_init()

where this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

now will return 0,

so clocks will never be enabled, resets never deasserted, regulators never enabled.

since qcom_qmp_phy_com_init() returns success in this case, qcom_qmp_phy_enable() will try to continue with the init, and writes to disabled hardware is usually not a good idea.

I think the proper fix for this is:

drivers/phy/qualcomm/phy-qcom-qmp.c +1 -1
index cd91b4179b10..22352e3b0ec5 100644
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
 
        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
-               return ret;
+               goto err_lane_rst;
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

Kind regards,
Niklas

Quoted 15 lines

Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...

Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Any idea what is going on? Am i doing something wrong?

Signature · 2 lines
bye,
p.
#2 lore raw
Quoted 101 lines · Niklas Cassel
Quoted 7 lines · Paolo Pisati

From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.

The issue is present in mainline up to 5.2-rc4, using defconfig and:

Quoted 7 lines

CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y

but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).

Hello Paolo,

I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets). The QMP block has 3 PCIe PHYs, that have their own clocks and resets.

Quoted 7 lines

These are the last lines printed out:
...

[    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out

^^ here the phy_init() called from pcie-qcom.c
which ends up to a call to qcom_qmp_phy_enable()

which has this code:

        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
                return ret;

qcom_qmp_phy_com_init() has this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

qcom_qmp_phy_com_init() later fails,
since the common block init time out, so the qmp driver disables clocks, asserts reset, and disables regulators

Quoted 2 lines
[    7.538596] phy phy-34000.phy.0: phy init failed --> -110
[    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110

^^ here the first PCIe controller instance fails to probe

Quoted 2 lines
[    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,

using dummy regulator

^^ here the second PCIe controller is probed.

it will call phy_init()

which will again call qcom_qmp_phy_enable() which will call qcom_qmp_phy_com_init()

where this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

now will return 0,

so clocks will never be enabled, resets never deasserted, regulators never enabled.

since qcom_qmp_phy_com_init() returns success in this case, qcom_qmp_phy_enable() will try to continue with the init, and writes to disabled hardware is usually not a good idea.

I think the proper fix for this is:

drivers/phy/qualcomm/phy-qcom-qmp.c +1 -1
index cd91b4179b10..22352e3b0ec5 100644
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
 
        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
-               return ret;
+               goto err_lane_rst;
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);
Quoted 18 lines

Kind regards,
Niklas

Quoted 15 lines

Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...

Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Any idea what is going on? Am i doing something wrong?

Signature · 2 lines
bye,
p.

Adding Vivek.

#3 lore raw
Quoted 6 lines · Niklas Cassel
Quoted 6 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +0 -0
index cd91b4179b10..22352e3b0ec5 100644
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
 
Quoted 7 lines

drivers/phy/qualcomm/phy-qcom-qmp.c static int qcom_qmp_phy_enable(struct phy *phy) · line 1496

Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +1 -1
        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
-               return ret;
+               goto err_lane_rst;
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:

drivers/phy/qualcomm/phy-qcom-qmp.c +4 -2
@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
        }
 
        ret = qcom_qmp_phy_com_init(qphy);
-       if (ret)
-               return ret;
+       if (ret) {
+               dev_err(qmp->dev, "qphy initialization failed\n");
+               goto err_lane_rst;
+       }
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

After several reboots i was able to trigger the phy init failure again:

...

[    2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed

...

these are the last lines printed, before rebooting in fastboot and locking up there (as before[*]).

So, as far as i understand there are two distinct problems:

1) sometimes, qcom-qmp-phy fails to initialize

2) and when that happens, the failure is fatal and it led to a reboot & lockup in fastboot

1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Signature · 2 lines
bye,
p.
#4 lore raw
Quoted 36 lines · Paolo Pisati
Quoted 6 lines · Niklas Cassel
Quoted 6 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +0 -0
index cd91b4179b10..22352e3b0ec5 100644
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
 
Quoted 7 lines

drivers/phy/qualcomm/phy-qcom-qmp.c static int qcom_qmp_phy_enable(struct phy *phy) · line 1496

Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +1 -1
        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
-               return ret;
+               goto err_lane_rst;
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:

drivers/phy/qualcomm/phy-qcom-qmp.c +4 -2
@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
        }
 
        ret = qcom_qmp_phy_com_init(qphy);
-       if (ret)
-               return ret;
+       if (ret) {
+               dev_err(qmp->dev, "qphy initialization failed\n");
+               goto err_lane_rst;
+       }
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

After several reboots i was able to trigger the phy init failure again:

Quoted 7 lines

...

[    2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed

...

I still think that the above patch is correct,
even though it didn't fix your problem.

If you try to disable the two other PCIe controllers:

arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +2 -2
index 943f69912074..95900fe99f89 100644
@@ -455,12 +455,12 @@
                        };
 
                        pcie@608000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
                        };
 
                        pcie@610000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
                        };
                };

Can you still reproduce the reboot?

Kind regards,
Niklas

Quoted 15 lines

these are the last lines printed, before rebooting in fastboot and locking up there (as before[*]).

So, as far as i understand there are two distinct problems:

1) sometimes, qcom-qmp-phy fails to initialize

2) and when that happens, the failure is fatal and it led to a reboot & lockup in fastboot

1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Signature · 2 lines
bye,
p.
#5 lore raw
Quoted 2 lines · Niklas Cassel

Can you still reproduce the reboot?

arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +2 -2
@@ -447,12 +447,12 @@
                        };
 
                        pcie@608000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
                        };
 
                        pcie@610000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
                        };
                };

disabling the two other pci controllers (and leaving your previous patch applied), made the issue more frequent: i was never able to boot to user space three times in a row without experiencing a crash, previously it was more sporadic.

On the other hand, now i can reproduce the issue more easily this way.

Signature · 2 lines
bye,
p.
#6 lore raw
In reply to Paolo Pisati
Quoted 34 lines · Paolo Pisati

From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.

The issue is present in mainline up to 5.2-rc4, using defconfig and:

CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y

but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).

These are the last lines printed out:
...

[    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    7.538596] phy phy-34000.phy.0: phy init failed --> -110
[    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
[    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,

using dummy regulator

Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...

Quoted 3 lines

Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Any idea what is going on? Am i doing something wrong?

Hmmm... this might be related to:

https://patchwork.kernel.org/patch/10976217/

Regards.