New: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe
A message that is not in the archive (sent off-list or not yet received): [email protected]/
Quoted 14 lines · On Tue, Jun 07, 2022 at 07:29:03AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216094
Summary: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe Kernel Version: 3.16 ... 5.10 Reporter: [email protected] CC: [email protected] Regression: NoI would like to continue the SATA-related topic started with Pali Rohár at the U-Boot mailing list [1]. I have analysed the issue further and come the following conclusion that it is related to the PCIe subsystem:
Quoted 4 lines
SATA-2 and SATA-3 hard disks connected to a 88SE6121 (AHCI) controller, wired via PCIe to the 88F6281 SoC fail to operate ("failed to IDENTIFY" ... "qc timeout") when the pci-mvebu driver (Kernel 3.16 .. 5.10 Debian) is in use.
Please attach the complete dmesg logs showing this issue.
From your lspci output with v3.2, the SATA controller is at 00:01.0:
00:01.0 IDE [0101]: Marvell 88SE6121 SATA II
The v3.16 (with DTB) lspci output is essentially the same except the controller is at 01:00.0:
01:00.0 IDE [0101]: Marvell 88SE6121 SATA II
The ahci driver is bound in both cases. The PCI address and I/O port assignment differences are of no consequence unless some mvebu driver defect keeps them from working.
I think what we need is a complete dmesg log and DTB from the newest possible kernel that fails, plus the same logs from the newest kernel that works.
Quoted 46 lines
More details:
- The problem does not exist in 2.6 and 3.16 kernels. With the old
mach-kirkwood/pcie.cdriver all SATA-2/3 hard disks work correctly. Especially with a 3.16 kernel it is possible to have identical ATA/AHCI drivers but try both PCIe drivers: without DTB -> mach-kirkwood -> SATA-2/3 HDDs work; with DTB -> mach-mvebu -> HDDs fail.- The problem is specific to SATA-2/3 HDDs. Very old SATA-1-only HDDs work without problems. This might be related to the available data lanes, DMA or other bandwidth-related things -- I can only guess. Interestingly it does not help to limit SATA speed (libata.force=1.5G ...) with SATA-2/3 HDDs, only 'pure' SATA-1 HDDs work with pci-mvebu.
- The problem was identified with the Seagate Blackarmor NAS440 hardware. Forum posts show that other users experience similar problems with the (very similar) Iomega ix4-200d NAS [2].
- Within patched U-Boot [3] all (Sata-1/2/3) HDDs always work. Same for the 88F6281 SoC onboard SATA ports (
sata_mv- not connected via PCIe).- The mach-kirkwood driver operates the 6281 as class "Host bridge [0600]" with Cap "Express (v1) Root Port", the mach-mvebu driver as class "PCI bridge [0604]" with "Express (v2) Root Port" [4][5][6][7]. Notably the v1/v2, cache line size 32/64 or the missing interrupt route might be a key difference.
From the sources I see that all PCI drivers (mach-kirkwood, mach-mvebu and U-Boot) do various unconventional 'magic' things (rewriting PCI class of the root complex, changing capabilitys, host emulation and so on). This is the point where I currently get lost and ask for your help.
[1] https://lists.denx.de/pipermail/u-boot/2022-March/479197.html
[2] https://forum.doozan.com/read.php?2,94079,95519#msg-95519
[3] https://lists.denx.de/pipermail/u-boot/2022-March/479227.html
[4] lspci Linux version 3.2.0-4-kirkwoodmach-kirkwood/pci.c: HDDs ok
[5] lspci Linux version 3.16.0-0.bpo.4-kirkwood - with DTB -> mvebu-pci: HDDs fail
[6] lspci Linux version 3.16.0-0.bpo.4-kirkwood - without DTB ->mach-kirkwood/pci.c: HDDs ok
[7] lspci Linux version 5.10.0-11-marvell (Debian bullseye) mvebu-pci: HDDs fail
Quoted 20 lines · On Tuesday 07 June 2022 18:49:37 Bjorn Helgaas wrote:
Quoted 14 lines · On Tue, Jun 07, 2022 at 07:29:03AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216094
Summary: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe Kernel Version: 3.16 ... 5.10 Reporter: [email protected] CC: [email protected] Regression: NoI would like to continue the SATA-related topic started with Pali Rohár at the U-Boot mailing list [1]. I have analysed the issue further and come the following conclusion that it is related to the PCIe subsystem:
Quoted 4 lines
SATA-2 and SATA-3 hard disks connected to a 88SE6121 (AHCI) controller, wired via PCIe to the 88F6281 SoC fail to operate ("failed to IDENTIFY" ... "qc timeout") when the pci-mvebu driver (Kernel 3.16 .. 5.10 Debian) is in use.
Quoted 8 lines
Please attach the complete dmesg logs showing this issue.
From your lspci output with v3.2, the SATA controller is at 00:01.0:
00:01.0 IDE [0101]: Marvell 88SE6121 SATA II
The v3.16 (with DTB) lspci output is essentially the same except the controller is at 01:00.0:
Old kernel (incorrectly) reports Root Port and Endpoint device from the other end of the link to the same bus zero. Hence the difference in bus number between old kernel and new kernel.
Quoted 9 lines
01:00.0 IDE [0101]: Marvell 88SE6121 SATA II
The ahci driver is bound in both cases. The PCI address and I/O port assignment differences are of no consequence unless some mvebu driver defect keeps them from working.
I think what we need is a complete dmesg log and DTB from the newest possible kernel that fails, plus the same logs from the newest kernel that works.
+1
Quoted 8 lines
Quoted 8 lines
More details:
- The problem does not exist in 2.6 and 3.16 kernels. With the old
mach-kirkwood/pcie.cdriver all SATA-2/3 hard disks work correctly. Especially with a 3.16 kernel it is possible to have identical ATA/AHCI drivers but try both PCIe drivers: without DTB -> mach-kirkwood -> SATA-2/3 HDDs work; with DTB -> mach-mvebu -> HDDs fail.
So it looks like that issue is with DT based setup.
Just by a chance, could you try to boot kernel with pcie_aspm=off or pci=nomsi cmdline options? There are some issues in driver/controller related to link retraining and MSI interrupts which ASPM kernel code can trigger.
Quoted 21 lines
Quoted 21 lines
- The problem is specific to SATA-2/3 HDDs. Very old SATA-1-only HDDs work without problems. This might be related to the available data lanes, DMA or other bandwidth-related things -- I can only guess. Interestingly it does not help to limit SATA speed (libata.force=1.5G ...) with SATA-2/3 HDDs, only 'pure' SATA-1 HDDs work with pci-mvebu.
- The problem was identified with the Seagate Blackarmor NAS440 hardware. Forum posts show that other users experience similar problems with the (very similar) Iomega ix4-200d NAS [2].
- Within patched U-Boot [3] all (Sata-1/2/3) HDDs always work. Same for the 88F6281 SoC onboard SATA ports (
sata_mv- not connected via PCIe).- The mach-kirkwood driver operates the 6281 as class "Host bridge [0600]" with Cap "Express (v1) Root Port", the mach-mvebu driver as class "PCI bridge [0604]" with "Express (v2) Root Port" [4][5][6][7]. Notably the v1/v2, cache line size 32/64 or the missing interrupt route might be a key difference.
Quoted 5 lines
Quoted 5 lines
From the sources I see that all PCI drivers (mach-kirkwood, mach-mvebu and U-Boot) do various unconventional 'magic' things (rewriting PCI class of the root complex, changing capabilitys, host emulation and so on). This is the point where I currently get lost and ask for your help.
This 'magic' is there because of broken PCIe controller and its PCIe Root Port in all 32-bit Marvell SoCs.
Quoted 10 lines
Quoted 10 lines
[1] https://lists.denx.de/pipermail/u-boot/2022-March/479197.html
[2] https://forum.doozan.com/read.php?2,94079,95519#msg-95519
[3] https://lists.denx.de/pipermail/u-boot/2022-March/479227.html
[4] lspci Linux version 3.2.0-4-kirkwoodmach-kirkwood/pci.c: HDDs ok
[5] lspci Linux version 3.16.0-0.bpo.4-kirkwood - with DTB -> mvebu-pci: HDDs fail
[6] lspci Linux version 3.16.0-0.bpo.4-kirkwood - without DTB ->mach-kirkwood/pci.c: HDDs ok
[7] lspci Linux version 5.10.0-11-marvell (Debian bullseye) mvebu-pci: HDDs fail