New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]/
Quoted 65 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
Hi, this is your Linux kernel regression tracker. Partly top-posting to mnake this easily accessible.
Jim, what's up here? The regression was reported more than a week ago and it seems nothing happened since then. Or was there progress and I just missed it?
Anyway:
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.
Quoted 5 lines
Quoted 5 lines
I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
CCing the regression mailing list, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced 830aa6f29f07a4e2f1a
#regzbot title pci: brcmstb: CM4 no longer boots from SD card #regzbot ignore-activity
#regzbot from: Cyril Brulebois <[email protected]> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925
This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/
Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replied to), as the kernel's documentation call for; above page explains why this is important for tracked regressions.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
Quoted 7 lines · Thorsten Leemhuis
Hi, this is your Linux kernel regression tracker. Partly top-posting to mnake this easily accessible.
Jim, what's up here? The regression was reported more than a week ago and it seems nothing happened since then. Or was there progress and I just missed it?
Quoted 73 lines
Anyway:
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 66 lines · Bjorn Helgaas
Quoted 65 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
Cyril, 830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two funcs") reverts cleanly as of 57ae8a492116. Does reverting it avoid the regression?
Hi Bjorn,
Bjorn Helgaas <[email protected]> (2022-05-09):
Quoted 3 lines
Cyril,
830aa6f29f07("PCI: brcmstb: Splitbrcm_pcie_setup()into two funcs") reverts cleanly as of57ae8a492116. Does reverting it avoid the regression?
I didn't even try and revert this commit before you suggested doing so since it was supposed to be some preliminary work. Quoting a part of its commit message:
In future commits the brcm_pcie_linkup() function will be called indirectly by pci_host_probe() as opposed to the host driver invoking it directly.
Anyway, the patch can indeed be reverted on top of v5.18-rc4 or v5.18-rc6 but the build fails due to the former function being removed, while being still called from other places:
CC drivers/pci/controller/pcie-brcmstb.o
drivers/pci/controller/pcie-brcmstb.c:199:12: warning: ‘brcm_pcie_linkup’ used but never defined
199 | static int brcm_pcie_linkup(struct brcm_pcie *pcie);
| ^~~~~~~~~~~~~~~~
…
aarch64-linux-gnu-ld: drivers/pci/controller/pcie-brcmstb.o: in function `brcm_pcie_add_bus':
/home/kibi/hack/linux.git/drivers/pci/controller/pcie-brcmstb.c:527: undefined reference to `brcm_pcie_linkup'
See for example:
commit 93e41f3fca3d4a0f927b784012338c37f80a8a80
Author: Jim Quinlan <[email protected]>
Date: Thu Jan 6 11:03:29 2022 -0500
PCI: brcmstb: Add control of subdevice voltage regulators(And that one cannot be trivially reverted.)
Cheers,
Signature · 2 lines
Cyril Brulebois ([email protected]) <https://debamax.com/> D-I release manager -- Release team member -- Freelance Consultant
Quoted 31 lines · Cyril Brulebois
Hi Bjorn,
Bjorn Helgaas <[email protected]> (2022-05-09):
Quoted 3 lines
Cyril,
830aa6f29f07("PCI: brcmstb: Splitbrcm_pcie_setup()into two funcs") reverts cleanly as of57ae8a492116. Does reverting it avoid the regression?I didn't even try and revert this commit before you suggested doing so since it was supposed to be some preliminary work. Quoting a part of its commit message:
In future commits the
brcm_pcie_linkup()function will be called indirectly bypci_host_probe()as opposed to the host driver invoking it directly.Anyway, the patch can indeed be reverted on top of v5.18-rc4 or v5.18-rc6 but the build fails due to the former function being removed, while being still called from other places:
CC
drivers/pci/controller/pcie-brcmstb.o
drivers/pci/controller/pcie-brcmstb.c:199:12: warning: ‘brcm_pcie_linkup’ used but never defined
199 | static intbrcm_pcie_linkup(struct brcm_pcie *pcie);
| ^~~~~~~~~~~~~~~~
…
aarch64-linux-gnu-ld:drivers/pci/controller/pcie-brcmstb.o: in function `brcm_pcie_add_bus':
/home/kibi/hack/linux.git/drivers/pci/controller/pcie-brcmstb.c:527: undefined reference to `brcm_pcie_linkup'See for example:
Quoted 7 lines
commit 93e41f3fca3d4a0f927b784012338c37f80a8a80 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:29 2022 -0500 PCI: brcmstb: Add control of subdevice voltage regulators(And that one cannot be trivially reverted.)
What if you revert 830aa6f29f07 and the subsequent brcmstb patches?
11ed8b8624b8 ("PCI: brcmstb: Do not turn off WOL regulators on suspend") 93e41f3fca3d ("PCI: brcmstb: Add control of subdevice voltage regulators") 67211aadcb4b ("PCI: brcmstb: Add mechanism to turn on subdev regulators") 830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two funcs")
$ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07
I did that on current upstream: 9be9ed2612b5 ("Merge tag 'platform-drivers-x86-v5.18-4' ofgit://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
and it built fine on x86.
Bjorn Helgaas <[email protected]> (2022-05-10):
Quoted 7 lines
What if you revert
830aa6f29f07and the subsequent brcmstb patches?
11ed8b8624b8("PCI: brcmstb: Do not turn off WOL regulators on suspend")93e41f3fca3d("PCI: brcmstb: Add control of subdevice voltage regulators")67211aadcb4b("PCI: brcmstb: Add mechanism to turn on subdev regulators")830aa6f29f07("PCI: brcmstb: Splitbrcm_pcie_setup()into two funcs")
Quoted 6 lines
$ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07I did that on current upstream:
9be9ed2612b5("Merge tag 'platform-drivers-x86-v5.18-4' ofgit://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
and it built fine on x86.
I've done exactly that, and it boots again. Comparing kernel messages against an older version (5.10.106), I'm getting the same output on that bare CM4 on CM4 IO Board setup:
# dmesg|grep -i pcie
[ 3.368620] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 3.368654] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 3.368703] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[ 3.368748] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
[ 3.421094] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[ 3.421341] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00And with a PCIe → quad-USB board plugged in, I'm getting those additional lines:
[ 3.426842] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[ 3.427072] pcieport 0000:00:00.0: PME: Signaling with IRQ 51
[ 3.427472] pcieport 0000:00:00.0: AER: enabled with IRQ 51(It seems to be consistently IRQ 51 with 5.18.0-rc6+ while it seems to be consistently IRQ 52 on 5.10.106, but the output is very similar in both cases.)
And plugging a keyboard on one of those USB ports works fine:
[ 13.406351] input: Logitech USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:046D:C31C.0001/input/input0
[ 13.510144] input: Logitech USB Keyboard Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input2
[ 13.591345] input: Logitech USB Keyboard System Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input3Wrapping up: it boots again (with or without PCIe equipment plugged in), and PCIe seems functional.
I'm happy to test more patches (e.g. trying to fix the actual issue without going for a set of reverts).
Cheers,
Signature · 2 lines
Cyril Brulebois ([email protected]) <https://debamax.com/> D-I release manager -- Release team member -- Freelance Consultant
[+to Jim, heads-up about reverting commits to avoid the regression]
Quoted 45 lines · Cyril Brulebois
Bjorn Helgaas <[email protected]> (2022-05-10):
Quoted 7 lines
What if you revert
830aa6f29f07and the subsequent brcmstb patches?
11ed8b8624b8("PCI: brcmstb: Do not turn off WOL regulators on suspend")93e41f3fca3d("PCI: brcmstb: Add control of subdevice voltage regulators")67211aadcb4b("PCI: brcmstb: Add mechanism to turn on subdev regulators")830aa6f29f07("PCI: brcmstb: Splitbrcm_pcie_setup()into two funcs")Quoted 6 lines
$ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07I did that on current upstream:
9be9ed2612b5("Merge tag 'platform-drivers-x86-v5.18-4' ofgit://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
and it built fine on x86.I've done exactly that, and it boots again. Comparing kernel messages against an older version (5.10.106), I'm getting the same output on that bare CM4 on CM4 IO Board setup:
# dmesg|grep -i pcie [ 3.368620] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges: [ 3.368654] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff] [ 3.368703] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x0603ffffff -> 0x00f8000000 [ 3.368748] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x003fffffff -> 0x0400000000 [ 3.421094] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC) [ 3.421341] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00And with a PCIe → quad-USB board plugged in, I'm getting those additional lines:
[ 3.426842] pcieport 0000:00:00.0: enabling device (0000 -> 0002) [ 3.427072] pcieport 0000:00:00.0: PME: Signaling with IRQ 51 [ 3.427472] pcieport 0000:00:00.0: AER: enabled with IRQ 51(It seems to be consistently IRQ 51 with 5.18.0-rc6+ while it seems to be consistently IRQ 52 on 5.10.106, but the output is very similar in both cases.)
And plugging a keyboard on one of those USB ports works fine:
[ 13.406351] input: Logitech USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:046D:C31C.0001/input/input0 [ 13.510144] input: Logitech USB Keyboard Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input2 [ 13.591345] input: Logitech USB Keyboard System Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input3
Quoted 5 lines
Wrapping up: it boots again (with or without PCIe equipment plugged in), and PCIe seems functional.
I'm happy to test more patches (e.g. trying to fix the actual issue without going for a set of reverts).
We're at -rc6 already, but no word from Jim yet. I think all we can do at this point is queue up reverts of these patches for v5.18. The reverts on now on my for-linus branch.
If we get a fix, I can easily drop the reverts, but I don't want v5.18 to release with PCI completely broken on the Compute Module 4.
Bjorn
Hi Bjorn, Thorsten,
I apologize -- I did not see this email until now; I think I have to work on my gmail filters and labels.
I've just made a post on the Bugzilla website regarding this regression and have ideas on what may be causing the problem. Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom STB version of the 2711.
Hopefully Cyril can help me identify the issue.
I will try to get a Fixup ASAP.
Regards,
Jim Quinlan
Broadcom STB
Quoted 113 lines · Thorsten Leemhuis
Hi, this is your Linux kernel regression tracker. Partly top-posting to mnake this easily accessible.
Jim, what's up here? The regression was reported more than a week ago and it seems nothing happened since then. Or was there progress and I just missed it?
Anyway:
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.Quoted 5 lines
Quoted 5 lines
I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
CCing the regression mailing list, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.htmlTo be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced
830aa6f29f07a4e2f1a
#regzbot title pci: brcmstb: CM4 no longer boots from SD card #regzbot ignore-activity
#regzbot from: Cyril Brulebois <[email protected]> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replied to), as the kernel's documentation call for; above page explains why this is important for tracked regressions.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
Hello,
I have just sent a pullrequest to linu-pci@[email protected] to address this regression.
Please let me know if I have to do anything else besides addressing reviewers concerns
for this pullreq.
Thanks,
Jim Quinlan
Broadcom STB
Quoted 133 lines · Jim Quinlan
Hi Bjorn, Thorsten,
I apologize -- I did not see this email until now; I think I have to work on my gmail filters and labels.
I've just made a post on the Bugzilla website regarding this regression and have ideas on what may be causing the problem. Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom STB version of the 2711.
Hopefully Cyril can help me identify the issue.I will try to get a Fixup ASAP.
Regards,
Jim Quinlan
Broadcom STBQuoted 113 lines · Thorsten Leemhuis
Hi, this is your Linux kernel regression tracker. Partly top-posting to mnake this easily accessible.
Jim, what's up here? The regression was reported more than a week ago and it seems nothing happened since then. Or was there progress and I just missed it?
Anyway:
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.Quoted 5 lines
Quoted 5 lines
I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
CCing the regression mailing list, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.htmlTo be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced
830aa6f29f07a4e2f1a
#regzbot title pci: brcmstb: CM4 no longer boots from SD card #regzbot ignore-activity
#regzbot from: Cyril Brulebois <[email protected]> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replied to), as the kernel's documentation call for; above page explains why this is important for tracked regressions.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
Quoted 2 lines · Jim Quinlan
I have just sent a pullrequest to linu-pci@[email protected] to address this regression.
Great, thx
Quoted 3 lines
Please let me know if I have to do anything else besides addressing reviewers concerns
for this pullreq.
No need to, everything fine. I just have to tell regzbot manually about the patch (and the commit-id once it landed), as it doesn't handle links to bugzilla ticket to well yet, but that hopefully should be history in a week or two (or three).
#regzbot monitor:
https://lore.kernel.org/linux-pci/CANCKTBuz9FG-aGuwM-thFgkAyTi480=1j78DpjWUkgU4UDuurg@mail.gmail.com/
Ciao, Thorsten
Quoted 135 lines · Jim Quinlan
Quoted 133 lines
Hi Bjorn, Thorsten,
I apologize -- I did not see this email until now; I think I have to work on my gmail filters and labels.
I've just made a post on the Bugzilla website regarding this regression and have ideas on what may be causing the problem. Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom STB version of the 2711.
Hopefully Cyril can help me identify the issue.I will try to get a Fixup ASAP.
Regards,
Jim Quinlan
Broadcom STBQuoted 113 lines · Thorsten Leemhuis
Hi, this is your Linux kernel regression tracker. Partly top-posting to mnake this easily accessible.
Jim, what's up here? The regression was reported more than a week ago and it seems nothing happened since then. Or was there progress and I just missed it?
Anyway:
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 61 lines · Bjorn Helgaas
Quoted 60 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215925
Bug ID: 215925 Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting Product: Drivers Version: 2.5 Kernel Version: v5.17-rc1 Hardware: ARM OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: PCI Assignee: [email protected] Reporter: [email protected] Regression: NoCatching up with latest kernel releases in Debian, it turned out that my Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO Board,
and booting from an SD card, no longer boots: this means a black screen on the HDMI output, and no output on the serial console.Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the first (pre)release that wasn't.
After some git bisect, it turns out the cause seems to be the following commit
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):``` commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21 Author: Jim Quinlan <[email protected]> Date: Thu Jan 6 11:03:27 2022 -0500 PCI: brcmstb: Split brcm_pcie_setup() into two funcs ```Starting with this commit, the kernel panics early (before 0.30 seconds), with an
Asynchronous SError Interrupt. The backtrace references variousbrcm_pcie_*functions; I can share a picture or try and transcribe it manually if that helps (nothing on the serial console…).This commit is part of a branch that was ultimately merged as
d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's not even a backtrace anymore, the screen stays black after the usual “boot-up rainbow”, and there's still nothing on the serial console.I confirmed that
88db8458086b1dcf20b56682504bdb34d2bca0e2(on the master side) was still booting properly, and that87c71931633bd15e9cfd51d4a4d9cd685e8cdb55(from the branch being merged into master) is the last commit showing the panic.Since
d0a231f01e5b25bacd23e6edc7c979a18a517b2bis a merge commit that includes conflict resolutions indrivers/pci/controller/pcie-brcmstb.c, I suppose this could be consistent with the initial panic being “upgraded” into an even more serious issue.Quoted 5 lines
Quoted 5 lines
I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still affected by this issue.
The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the exact same image on the same SD card (with latest master) boots fine on it.
CCing the regression mailing list, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.htmlTo be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:
#regzbot ^introduced
830aa6f29f07a4e2f1a
#regzbot title pci: brcmstb: CM4 no longer boots from SD card #regzbot ignore-activity
#regzbot from: Cyril Brulebois <[email protected]> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replied to), as the kernel's documentation call for; above page explains why this is important for tracked regressions.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting #forregzbot
In reply to Thorsten LeemhuisTWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.
Quoted 8 lines · Thorsten Leemhuis
[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]
Quoted 3 lines · Bjorn Helgaas
Quoted 2 lines · On Sat, Apr 30, 2022 at 2:53 PM <[email protected]> wrote:
#regzbot fixed-by: f4fd559de3434c44