msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
Subsystems (4)
From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.
The issue is present in mainline up to 5.2-rc4, using defconfig and:
CONFIG_SCSI_UFS_QCOM=yCONFIG_PHY_QCOM_QMP=yCONFIG_PHY_QCOM_UFS=y
but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).
These are the last lines printed out:
...
[ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out [ 7.538596] phy phy-34000.phy.0: phy init failed --> -110 [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110 [ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
using dummy regulator
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...
Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Any idea what is going on? Am i doing something wrong?
Signature · 2 lines
bye, p.
Quoted 7 lines · Paolo Pisati
From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.
The issue is present in mainline up to 5.2-rc4, using defconfig and:
Quoted 7 lines
CONFIG_SCSI_UFS_QCOM=yCONFIG_PHY_QCOM_QMP=yCONFIG_PHY_QCOM_UFS=ybut was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).
Hello Paolo,
I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets). The QMP block has 3 PCIe PHYs, that have their own clocks and resets.
Quoted 7 lines
These are the last lines printed out:
...[ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
^^ here the phy_init() called from pcie-qcom.c
which ends up to a call to qcom_qmp_phy_enable()
which has this code:
ret = qcom_qmp_phy_com_init(qphy);
if (ret)
return ret;qcom_qmp_phy_com_init() has this code:
if (qmp->init_count++) {
mutex_unlock(&qmp->phy_mutex);
return 0;
}qcom_qmp_phy_com_init() later fails,
since the common block init time out, so the qmp driver disables clocks, asserts reset, and disables regulators
Quoted 2 lines
[ 7.538596] phy phy-34000.phy.0: phy init failed --> -110 [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
^^ here the first PCIe controller instance fails to probe
Quoted 2 lines
[ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,using dummy regulator
^^ here the second PCIe controller is probed.
it will call phy_init()
which will again call qcom_qmp_phy_enable() which will call qcom_qmp_phy_com_init()
where this code:
if (qmp->init_count++) {
mutex_unlock(&qmp->phy_mutex);
return 0;
}now will return 0,
so clocks will never be enabled, resets never deasserted, regulators never enabled.
since qcom_qmp_phy_com_init() returns success in this case, qcom_qmp_phy_enable() will try to continue with the init, and writes to disabled hardware is usually not a good idea.
I think the proper fix for this is:
drivers/phy/qualcomm/phy-qcom-qmp.c +1 -1
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) ret =qcom_qmp_phy_com_init (qphy);if (ret) -return ret; + goto err_lane_rst;if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);
Kind regards,
Niklas
Quoted 15 lines
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Any idea what is going on? Am i doing something wrong?
Signature · 2 lines
bye, p.
Quoted 101 lines · Niklas Cassel
Quoted 7 lines · Paolo Pisati
From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.
The issue is present in mainline up to 5.2-rc4, using defconfig and:
Quoted 7 lines
CONFIG_SCSI_UFS_QCOM=yCONFIG_PHY_QCOM_QMP=yCONFIG_PHY_QCOM_UFS=ybut was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).
Hello Paolo,
I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets). The QMP block has 3 PCIe PHYs, that have their own clocks and resets.Quoted 7 lines
These are the last lines printed out:
...[ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out^^ here the
phy_init()called frompcie-qcom.c
which ends up to a call toqcom_qmp_phy_enable()which has this code:
ret = qcom_qmp_phy_com_init(qphy); if (ret) return ret;
qcom_qmp_phy_com_init()has this code:if (qmp->init_count++) { mutex_unlock(&qmp->phy_mutex); return 0; }
qcom_qmp_phy_com_init()later fails,
since the common block init time out, so the qmp driver disables clocks, asserts reset, and disables regulatorsQuoted 2 lines
[ 7.538596] phy phy-34000.phy.0: phy init failed --> -110 [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110^^ here the first PCIe controller instance fails to probe
Quoted 2 lines
[ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,using dummy regulator
^^ here the second PCIe controller is probed.
it will call
phy_init()which will again call
qcom_qmp_phy_enable()which will callqcom_qmp_phy_com_init()where this code:
if (qmp->init_count++) { mutex_unlock(&qmp->phy_mutex); return 0; }now will return 0,
so clocks will never be enabled, resets never deasserted, regulators never enabled.
since
qcom_qmp_phy_com_init()returns success in this case,qcom_qmp_phy_enable()will try to continue with the init, and writes to disabled hardware is usually not a good idea.I think the proper fix for this is:
drivers/phy/qualcomm/phy-qcom-qmp.c +1
-1@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) ret =qcom_qmp_phy_com_init (qphy);if (ret) -return ret; + goto err_lane_rst;if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);
Quoted 18 lines
Kind regards,
NiklasQuoted 15 lines
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Any idea what is going on? Am i doing something wrong?
Signature · 2 lines
bye, p.
Adding Vivek.
Quoted 6 lines · Niklas Cassel
Quoted 6 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +0
-0@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c static int qcom_qmp_phy_enable(struct phy *phy) · line 1496
Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +1
-1ret =qcom_qmp_phy_com_init (qphy);if (ret) -return ret; + goto err_lane_rst;if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);
Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:
drivers/phy/qualcomm/phy-qcom-qmp.c +4 -2
@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy) } ret =qcom_qmp_phy_com_init (qphy); -if (ret) -return ret; +if (ret) { +dev_err (qmp->dev,"qphy initialization failed\n" ); + goto err_lane_rst; + }if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);
After several reboots i was able to trigger the phy init failure again:
...
[ 2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out [ 2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed
...
these are the last lines printed, before rebooting in fastboot and locking up there (as before[*]).
So, as far as i understand there are two distinct problems:
1) sometimes, qcom-qmp-phy fails to initialize
2) and when that happens, the failure is fatal and it led to a reboot & lockup in fastboot
1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Signature · 2 lines
bye, p.
Quoted 36 lines · Paolo Pisati
Quoted 6 lines · Niklas Cassel
Quoted 6 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +0
-0@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c static int qcom_qmp_phy_enable(struct phy *phy) · line 1496
Quoted 7 lines
drivers/phy/qualcomm/phy-qcom-qmp.c +1
-1ret =qcom_qmp_phy_com_init (qphy);if (ret) -return ret; + goto err_lane_rst;if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:drivers/phy/qualcomm/phy-qcom-qmp.c +4
-2@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy) } ret =qcom_qmp_phy_com_init (qphy); -if (ret) -return ret; +if (ret) { +dev_err (qmp->dev,"qphy initialization failed\n" ); + goto err_lane_rst; + }if (cfg->has_lane_rst) { ret =reset_control_deassert (qphy->lane_rst);After several reboots i was able to trigger the phy init failure again:
Quoted 7 lines
...
[ 2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out [ 2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed...
I still think that the above patch is correct,
even though it didn't fix your problem.
If you try to disable the two other PCIe controllers:
arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +2 -2
@@ -455,12 +455,12 @@ }; pcie@608000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; }; pcie@610000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; }; };
Can you still reproduce the reboot?
Kind regards,
Niklas
Quoted 15 lines
these are the last lines printed, before rebooting in fastboot and locking up there (as before[*]).
So, as far as i understand there are two distinct problems:
1) sometimes, qcom-qmp-phy fails to initialize
2) and when that happens, the failure is fatal and it led to a reboot & lockup in fastboot
1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Signature · 2 lines
bye, p.
Quoted 2 lines · Niklas Cassel
Can you still reproduce the reboot?
arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +2 -2
@@ -447,12 +447,12 @@ }; pcie@608000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; }; pcie@610000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; }; };
disabling the two other pci controllers (and leaving your previous patch applied), made the issue more frequent: i was never able to boot to user space three times in a row without experiencing a crash, previously it was more sporadic.
On the other hand, now i can reproduce the issue more easily this way.
Signature · 2 lines
bye, p.
Quoted 34 lines · Paolo Pisati
From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc.
The issue is present in mainline up to 5.2-rc4, using defconfig and:
CONFIG_SCSI_UFS_QCOM=yCONFIG_PHY_QCOM_QMP=yCONFIG_PHY_QCOM_UFS=ybut was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it).
These are the last lines printed out:
...[ 7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [ 7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy [ 7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [ 7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out [ 7.538596] phy phy-34000.phy.0: phy init failed --> -110 [ 7.550891] qcom-pcie: probe of 600000.pcie failed with error -110 [ 7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,using dummy regulator
Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...
Quoted 3 lines
Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
Any idea what is going on? Am i doing something wrong?
Hmmm... this might be related to:
https://patchwork.kernel.org/patch/10976217/
Regards.