Skip to contentSkip to conversation

sc7180 kernel hang with linux-next

DISCUSSION Laura Nao 2 participants 5 messages Dec 18, 2023 – Jan 22, 2024 Arm Qualcomm mailing list

lore
Share & tools
lore

Laura Nao

author
#0 lore raw

Hello,

KernelCI has reported a regression on some sc7180 based platforms (lazor and kingoftown Chromebooks) for linux-next: the kernel seems to hang after initializing the SDHCI controller (~2 seconds in the boot), nothing is reported on the console after unused clocks are disabled:

[    2.241767] mmc1: Command Queue Engine enabled
[    2.257574] dwc3 a600000.usb: Adding to iommu group 9
[    2.261398] mmc1: new HS400 Enhanced strobe MMC card at address 0001
[    2.270452] msm_dsi ae94000.dsi: supply refgen not found, using dummy
regulator
[    2.274496] mmcblk1: mmc1:0001 DA4064 58.2 GiB
[    2.294482]  mmcblk1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
[    2.301798] mmcblk1boot0: mmc1:0001 DA4064 4.00 MiB
[    2.307847] mmcblk1boot1: mmc1:0001 DA4064 4.00 MiB
[    2.313799] mmcblk1rpmb: mmc1:0001 DA4064 16.0 MiB, chardev (507:0)
[   14.899579] clk: Disabling unused clocks

This was observed on next-20231123 first and is still present on next-20231218.

Full kernel log from a couple examples:
- next-20231205:
https://storage.kernelci.org/next/master/next-20231205/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/baseline-nfs-sc7180-trogdor-kingoftown.html
- next-20231215:
https://storage.kernelci.org/next/master/next-20231215/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-lazor-limozeen.html

Sometimes the kernel is able to get past that point, but crashes a bit later - here's an example from a decoder conformance tests, the kernel boots fine but crashes shortly afterwards:
- next-20231214:
https://storage.kernelci.org/next/master/next-20231214/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-kingoftown.html

Any idea on what might be causing this issue?

Thanks,

Laura

#1 lore raw

Hi,

Quoted 20 lines · Laura Nao

Hello,

KernelCI has reported a regression on some sc7180 based platforms (lazor and kingoftown Chromebooks) for linux-next: the kernel seems to hang after initializing the SDHCI controller (~2 seconds in the boot), nothing is reported on the console after unused clocks are disabled:

[    2.241767] mmc1: Command Queue Engine enabled
[    2.257574] dwc3 a600000.usb: Adding to iommu group 9
[    2.261398] mmc1: new HS400 Enhanced strobe MMC card at address 0001
[    2.270452] msm_dsi ae94000.dsi: supply refgen not found, using dummy
regulator
[    2.274496] mmcblk1: mmc1:0001 DA4064 58.2 GiB
[    2.294482]  mmcblk1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
[    2.301798] mmcblk1boot0: mmc1:0001 DA4064 4.00 MiB
[    2.307847] mmcblk1boot1: mmc1:0001 DA4064 4.00 MiB
[    2.313799] mmcblk1rpmb: mmc1:0001 DA4064 16.0 MiB, chardev (507:0)
[   14.899579] clk: Disabling unused clocks
Quoted 8 lines

This was observed on next-20231123 first and is still present on next-20231218.

Full kernel log from a couple examples:
- next-20231205:
https://storage.kernelci.org/next/master/next-20231205/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/baseline-nfs-sc7180-trogdor-kingoftown.html
- next-20231215:
https://storage.kernelci.org/next/master/next-20231215/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-lazor-limozeen.html

Is it really hanging? I haven't fully dug into all the logs, but it sure seems like the kernel is not hung, it just isn't doing anything. This looks like the state where the kernel is sitting waiting for the root filesystem to become available so that it can run the init process.

From your command line I see "root=/dev/nfs". Yet nowhere in your boot log do I see a USB network adapter register. I'm going to assume that's the problem.

Quoted 5 lines

Sometimes the kernel is able to get past that point, but crashes a bit later - here's an example from a decoder conformance tests, the kernel boots fine but crashes shortly afterwards:
- next-20231214:
https://storage.kernelci.org/next/master/next-20231214/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-kingoftown.html

This log has all kinds of badness. I see the "stuck clock" on the display that Stephen has been talking about for a while. I couldn't reproduce it for a while but I saw it the other day. This needs to be figured out. I then see an "oops" in qcom_stats_probe() that should be fixed by the revert that landed in Bjorn's tree over the weekend:

a7dc63435197 Revert "soc: qcom: stats: Add DDR sleep stats"

...then I'm at least slightly shocked that the kernel continues on past an oops. You really don't panic on oops?

You then seem to load the r8152 USB Ethernet driver which lets you get the rootfs. Then you're hitting a totally different crash in venus (video decoder/encoder) that needs to be debugged.

Quoted 1 line

Any idea on what might be causing this issue?

This seems like the perfect thing to bisect. Is it possible you could do that?

-Doug

author
#2 lore raw

Hi - thanks for the feedback!

Quoted 43 lines · Doug Anderson

Hi,

Quoted 24 lines · Laura Nao

Hello,

KernelCI has reported a regression on some sc7180 based platforms (lazor
and kingoftown Chromebooks) for linux-next: the kernel seems to hang after initializing the SDHCI controller (~2 seconds in the boot), nothing is reported on the console after unused clocks are disabled:

[    2.241767] mmc1: Command Queue Engine enabled
[    2.257574] dwc3 a600000.usb: Adding to iommu group 9
[    2.261398] mmc1: new HS400 Enhanced strobe MMC card at address
0001
[    2.270452] msm_dsi ae94000.dsi: supply refgen not found, using
dummy
regulator
[    2.274496] mmcblk1: mmc1:0001 DA4064 58.2 GiB
[    2.294482]  mmcblk1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
[    2.301798] mmcblk1boot0: mmc1:0001 DA4064 4.00 MiB
[    2.307847] mmcblk1boot1: mmc1:0001 DA4064 4.00 MiB
[    2.313799] mmcblk1rpmb: mmc1:0001 DA4064 16.0 MiB, chardev
(507:0)
[   14.899579] clk: Disabling unused clocks
Quoted 8 lines

This was observed on next-20231123 first and is still present on next-20231218.

Full kernel log from a couple examples:
- next-20231205:

   https://storage.kernelci.org/next/master/next-20231205/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/baseline-nfs-sc7180-trogdor-kingoftown.html
- next-20231215:
   https://storage.kernelci.org/next/master/next-20231215/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-lazor-limozeen.html

Is it really hanging? I haven't fully dug into all the logs, but it sure seems like the kernel is not hung, it just isn't doing anything. This looks like the state where the kernel is sitting waiting for the root filesystem to become available so that it can run the init process.

Quoted 6 lines

From your command line I see "root=/dev/nfs". Yet nowhere in your boot
log do I see a USB network adapter register. I'm going to assume that's the problem.

These tests all use an initrd, so I'd expect the kernel to spit out a few more lines before waiting for the network to come up (log from a good run as reference: https://pastebin.com/raw/rhPsLy1q ).

The USB network adapter is usually initialized a bit later in the boot process indeed.

Quoted 24 lines
Quoted 7 lines

Sometimes the kernel is able to get past that point, but crashes a bit
later - here's an example from a decoder conformance tests, the kernel
boots fine but crashes shortly afterwards:
- next-20231214:

   https://storage.kernelci.org/next/master/next-20231214/arm64/defconfig+arm64-chromebook+videodec/gcc-10/lab-collabora/v4l2-decoder-conformance-h265-sc7180-trogdor-kingoftown.html

This log has all kinds of badness. I see the "stuck clock" on the display that Stephen has been talking about for a while. I couldn't reproduce it for a while but I saw it the other day. This needs to be figured out. I then see an "oops" in qcom_stats_probe() that should be fixed by the revert that landed in Bjorn's tree over the weekend:

a7dc63435197 Revert "soc: qcom: stats: Add DDR sleep stats"

...then I'm at least slightly shocked that the kernel continues on past an oops. You really don't panic on oops?

You then seem to load the r8152 USB Ethernet driver which lets you get the rootfs. Then you're hitting a totally different crash in venus (video decoder/encoder) that needs to be debugged.

Quoted 4 lines
Quoted 1 line

Any idea on what might be causing this issue?

This seems like the perfect thing to bisect. Is it possible you could do that?

Thanks for the useful pointers!

Yes, I'll try to bisect this through KernelCI and report back.

Pasting new links to the logs mentioned above for future reference, as they were removed from the KernelCI storage:

- next-20231215: https://pastebin.com/raw/C7RzgHn0
- next-20231214: https://pastebin.com/raw/VtpUS5D9

Best,

Laura

author
#3 lore raw
Quoted 3 lines · Laura Nao

Yes, I'll try to bisect this through KernelCI and report back.

KernelCI has not bisected this regression yet. However, upon further investigation I noticed the kernel was getting consistently stuck while disabling the display clocks (namely disp_cc_mdss_pclk0_clk ) and booted consistently without issues after adding clk_ignore_unused to the cmdline.

The kernel configuration used by KernelCI had CONFIG_SC_DISPCC_7180=y ; setting CONFIG_SC_DISPCC_7180=m fixed the issue as the display clock controller is being initialized a bit later in the boot.

We're going to set CONFIG_SC_DISPCC_7180=m in the configuration used by KernelCI and monitor the results for a while, I'll report back to confirm whether the issue is still present or not.

Best,

Laura

#4 lore raw

Hi,

Quoted 12 lines · Laura Nao
Quoted 3 lines · Laura Nao

Yes, I'll try to bisect this through KernelCI and report back.

KernelCI has not bisected this regression yet. However, upon further investigation I noticed the kernel was getting consistently stuck while disabling the display clocks (namely disp_cc_mdss_pclk0_clk ) and booted consistently without issues after adding clk_ignore_unused to the cmdline.

Quoted 7 lines

The kernel configuration used by KernelCI had CONFIG_SC_DISPCC_7180=y ; setting CONFIG_SC_DISPCC_7180=m fixed the issue as the display clock controller is being initialized a bit later in the boot.

We're going to set CONFIG_SC_DISPCC_7180=m in the configuration used by KernelCI and monitor the results for a while, I'll report back to confirm whether the issue is still present or not.

Ick. I wonder if this is related to the issue that Stephen filed:
https://issuetracker.google.com/issues/319956935