PCI trouble on mvebu (Turris Omnia)

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

[0] https://lore.kernel.org/linux-pci/[email protected]/

Full dmesg:

[    1.546457] pci 0000:00:02.0: [11ab:6820] type 01 class 0x060400
[    1.546469] pci 0000:00:02.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    1.546615] pci 0000:00:03.0: [11ab:6820] type 01 class 0x060400
[    1.546627] pci 0000:00:03.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    1.547341] PCI: bus0: Fast back to back transfers disabled
[    1.547349] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547356] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547363] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547444] pci 0000:01:00.0: [168c:002e] type 00 class 0x028000
[    1.547466] pci 0000:01:00.0: reg 0x10: [mem 0xe8000000-0xe800ffff 64bit]
[    1.547576] pci 0000:01:00.0: supports D1
[    1.547581] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[    1.547692] pci 0000:00:01.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[    1.601932] PCI: bus1: Fast back to back transfers enabled
[    1.601941] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.602039] pci 0000:02:00.0: [168c:003c] type 00 class 0x028000
[    1.602063] pci 0000:02:00.0: reg 0x10: [mem 0xea000000-0xea1fffff 64bit]
[    1.602096] pci 0000:02:00.0: reg 0x30: [mem 0xea200000-0xea20ffff pref]
[    1.602174] pci 0000:02:00.0: supports D1 D2
[    1.602273] pci 0000:00:02.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[    1.631918] PCI: bus2: Fast back to back transfers enabled
[    1.631926] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.632623] PCI: bus3: Fast back to back transfers enabled
[    1.632630] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    1.632663] pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[    1.632671] pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
[    1.632679] pci 0000:00:01.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff pref]
[    1.632687] pci 0000:00:02.0: BAR 6: assigned [mem 0xe0500000-0xe05007ff pref]
[    1.632694] pci 0000:00:03.0: BAR 6: assigned [mem 0xe0600000-0xe06007ff pref]
[    1.632701] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe000ffff 64bit]
[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632720] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.632728] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe00fffff]
[    1.632737] pci 0000:02:00.0: BAR 0: assigned [mem 0xe0200000-0xe03fffff 64bit]
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632757] pci 0000:02:00.0: BAR 6: assigned [mem 0xe0400000-0xe040ffff pref]
[    1.632762] pci 0000:00:02.0: PCI bridge to [bus 02]
[    1.632768] pci 0000:00:02.0:   bridge window [mem 0xe0200000-0xe04fffff]
[    1.632774] pci 0000:00:03.0: PCI bridge to [bus 03]
[    1.633030] mv_xor f1060800.xor: Marvell shared XOR driver
[    1.691640] mv_xor f1060800.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    1.691756] mv_xor f1060900.xor: Marvell shared XOR driver
[    1.751635] mv_xor f1060900.xor: Marvell XOR (Descriptor Mode): ( xor cpy intr )
[    1.769386] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.770240] printk: console [ttyS0] disabled
[    1.790351] f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 30, base_baud = 15625000) is a 16550A
[    3.040783] printk: console [ttyS0] enabled
[    3.065621] f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 31, base_baud = 15625000) is a 16550A
[    3.075329] ahci-mvebu f10a8000.sata: supply ahci not found, using dummy regulator
[    3.082990] ahci-mvebu f10a8000.sata: supply phy not found, using dummy regulator
[    3.090499] ahci-mvebu f10a8000.sata: supply target not found, using dummy regulator
[    3.098335] ahci-mvebu f10a8000.sata: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl platform mode
[    3.107411] ahci-mvebu f10a8000.sata: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    3.116657] scsi host0: ahci-mvebu
[    3.120302] scsi host1: ahci-mvebu
[    3.123825] ata1: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x100 irq 53
[    3.131768] ata2: SATA max UDMA/133 mmio [mem 0xf10a8000-0xf10a9fff] port 0x180 irq 53
[    3.140560] spi-nor spi0.0: s25fl164k (8192 Kbytes)
[    3.145494] 2 fixed-partitions partitions found on MTD device spi0.0
[    3.151868] Creating 2 MTD partitions on "spi0.0":
[    3.156671] 0x000000000000-0x000000100000 : "U-Boot"
[    3.171461] 0x000000100000-0x000000800000 : "Rescue system"
[    3.191747] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[    3.199597] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
[    3.209859] libphy: Fixed MDIO Bus: probed
[    3.214141] tun: Universal TUN/TAP device driver, 1.6
[    3.219584] libphy: orion_mdio_bus: probed
[    3.224542] mv88e6085 f1072004.mdio-mii:10: switch 0x1760 detected: Marvell 88E6176, revision 1
[    3.450274] libphy: mv88e6xxx SMI: probed
[    3.461815] mvneta f1070000.ethernet eth0: Using hardware mac address d8:58:d7:00:4e:98
[    3.470606] mvneta f1030000.ethernet eth1: Using hardware mac address d8:58:d7:00:4e:96
[    3.479356] mvneta f1034000.ethernet eth2: Using hardware mac address d8:58:d7:00:4e:97
[    3.482630] ata1: SATA link down (SStatus 0 SControl 300)
[    3.487588] pci 0000:00:01.0: enabling device (0140 -> 0142)
[    3.492831] ata2: SATA link down (SStatus 0 SControl 300)
[    3.498496] ath9k 0000:01:00.0: enabling device (0000 -> 0002)
[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110
[    3.601529] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.608072] ehci-pci: EHCI PCI platform driver
[    3.612553] ehci-orion: EHCI orion driver
[    3.616675] orion-ehci f1058000.usb: EHCI Host Controller
[    3.622105] orion-ehci f1058000.usb: new USB bus registered, assigned bus number 1
[    3.629733] orion-ehci f1058000.usb: irq 49, io mem 0xf1058000
[    3.661261] orion-ehci f1058000.usb: USB 2.0 started, EHCI 1.00
[    3.667530] hub 1-0:1.0: USB hub found
[    3.671321] hub 1-0:1.0: 1 port detected
[    3.675700] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    3.681034] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 2
[    3.688599] xhci-hcd f10f0000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
[    3.697867] xhci-hcd f10f0000.usb3: irq 55, io mem 0xf10f0000
[    3.703905] hub 2-0:1.0: USB hub found
[    3.707678] hub 2-0:1.0: 1 port detected
[    3.711767] xhci-hcd f10f0000.usb3: xHCI Host Controller
[    3.717096] xhci-hcd f10f0000.usb3: new USB bus registered, assigned bus number 3
[    3.724621] xhci-hcd f10f0000.usb3: Host supports USB 3.0 SuperSpeed
[    3.731026] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.739388] hub 3-0:1.0: USB hub found
[    3.743167] hub 3-0:1.0: 1 port detected
[    3.747339] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    3.752684] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 4
[    3.760230] xhci-hcd f10f8000.usb3: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
[    3.769502] xhci-hcd f10f8000.usb3: irq 56, io mem 0xf10f8000
[    3.775527] hub 4-0:1.0: USB hub found
[    3.779298] hub 4-0:1.0: 1 port detected
[    3.783756] xhci-hcd f10f8000.usb3: xHCI Host Controller
[    3.789086] xhci-hcd f10f8000.usb3: new USB bus registered, assigned bus number 5
[    3.796610] xhci-hcd f10f8000.usb3: Host supports USB 3.0 SuperSpeed
[    3.803012] usb usb5: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.811375] hub 5-0:1.0: USB hub found
[    3.815147] hub 5-0:1.0: 1 port detected
[    3.819312] usbcore: registered new interface driver usb-storage
[    3.826044] armada38x-rtc f10a3800.rtc: registered as rtc0
[    3.831632] armada38x-rtc f10a3800.rtc: setting system clock to 2020-10-27T15:31:52 UTC (1603812712)
[    3.840905] i2c /dev entries driver
[    3.846565] orion_wdt: Initial timeout 171 sec
[    3.851350] sdhci: Secure Digital Host Controller Interface driver
[    3.857544] sdhci: Copyright(c) Pierre Ossman
[    3.862041] sdhci-pltfm: SDHCI platform and OF driver helper
[    3.868792] marvell-cesa f1090000.crypto: CESA device successfully registered
[    3.876106] usbcore: registered new interface driver usbhid
[    3.881715] usbhid: USB HID core driver
[    3.885678] GACT probability on
[    3.888837] Mirror/redirect action on
[    3.892589] Simple TC action Loaded
[    3.893793] mmc0: SDHCI controller on f10d8000.sdhci [f10d8000.sdhci] using ADMA
[    3.896117] u32 classifier
[    3.906258]     Performance counters on
[    3.910113]     input device check on
[    3.913812]     Actions configured
[    3.917606] NET: Registered protocol family 10
[    3.922867] Segment Routing with IPv6
[    3.926605] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    3.932956] NET: Registered protocol family 17
[    3.937500] 8021q: 802.1Q VLAN Support v1.8
[    3.941850] ThumbEE CPU extension supported.
[    3.946133] Registering SWP/SWPB emulation handler
[    3.951024] Loading compiled-in X.509 certificates
[    3.956916] Btrfs loaded, crc32c=crc32c-generic
[    3.961872] mv88e6085 f1072004.mdio-mii:10: switch 0x1760 detected: Marvell 88E6176, revision 1
[    4.027817] mmc0: new high speed MMC card at address 0001
[    4.033650] mmcblk0: mmc0:0001 H8G4a\x92 7.28 GiB 
[    4.038323] mmcblk0boot0: mmc0:0001 H8G4a\x92 partition 1 4.00 MiB
[    4.044421] mmcblk0boot1: mmc0:0001 H8G4a\x92 partition 2 4.00 MiB
[    4.050457] mmcblk0rpmb: mmc0:0001 H8G4a\x92 partition 3 4.00 MiB, chardev (250:0)
[    4.059708]  mmcblk0: p1
[    4.081276] usb 2-1: new high-speed USB device number 2 using xhci-hcd
[    4.169488] libphy: mv88e6xxx SMI: probed
[    4.261911] usb-storage 2-1:1.0: USB Mass Storage device detected
[    4.268229] scsi host2: usb-storage 2-1:1.0
[    4.816096] mv88e6085 f1072004.mdio-mii:10 lan0 (uninitialized): PHY [mv88e6xxx-1:00] driver [Marvell 88E1540] (irq=70)
[    4.842702] mv88e6085 f1072004.mdio-mii:10 lan1 (uninitialized): PHY [mv88e6xxx-1:01] driver [Marvell 88E1540] (irq=71)
[    4.869246] mv88e6085 f1072004.mdio-mii:10 lan2 (uninitialized): PHY [mv88e6xxx-1:02] driver [Marvell 88E1540] (irq=72)
[    4.895772] mv88e6085 f1072004.mdio-mii:10 lan3 (uninitialized): PHY [mv88e6xxx-1:03] driver [Marvell 88E1540] (irq=73)
[    4.920733] mv88e6085 f1072004.mdio-mii:10 lan4 (uninitialized): PHY [mv88e6xxx-1:04] driver [Marvell 88E1540] (irq=74)
[    4.939701] mv88e6085 f1072004.mdio-mii:10: configuring for fixed/rgmii-id link mode
[    4.950089] mv88e6085 f1072004.mdio-mii:10: Link is Up - 1Gbps/Full - flow control off
[    4.958047] DSA: tree 0 setup
[    4.961339] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    4.970623] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    4.977231] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    4.985879] cfg80211: failed to load regulatory.db
[    4.990987] Waiting 2 sec before mounting root device...
[    5.351539] scsi 2:0:0:0: Direct-Access     General  UDisk            5.00 PQ: 0 ANSI: 2
[    5.360060] sd 2:0:0:0: [sda] 7987200 512-byte logical blocks: (4.09 GB/3.81 GiB)
[    5.367691] sd 2:0:0:0: [sda] Write Protect is off
[    5.372503] sd 2:0:0:0: [sda] Mode Sense: 0b 00 00 08
[    5.372605] sd 2:0:0:0: [sda] No Caching mode page found
[    5.377931] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    5.435076]  sda: sda1
[    5.438130] sd 2:0:0:0: [sda] Attached SCSI removable disk
[    7.047873] BTRFS: device fsid 448334b8-1b27-4738-8118-9e70b56b1e58 devid 1 transid 680 /dev/root scanned by swapper/0 (1)
[    7.059562] BTRFS info (device mmcblk0p1): disk space caching is enabled
[    7.066294] BTRFS info (device mmcblk0p1): has skinny extents
[    7.078585] BTRFS info (device mmcblk0p1): enabling ssd optimizations
[    7.087624] VFS: Mounted root (btrfs filesystem) on device 0:12.
[    7.094044] devtmpfs: mounted
[    7.097581] Freeing unused kernel memory: 1024K
[    7.131431] Run /sbin/init as init process
[    7.135536]   with arguments:
[    7.135539]     /sbin/init
[    7.135541]     earlyprintk
[    7.135543]   with environment:
[    7.135545]     HOME=/
[    7.135548]     TERM=linux
[    7.220335] random: fast init done
[    7.650974] systemd[1]: systemd 246.6-1.1-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    7.674141] systemd[1]: Detected architecture arm.
[    7.752534] systemd[1]: Set hostname to <omnia-arch>.
[    7.938493] systemd[164]: /usr/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1.
[    8.148416] systemd[1]: Queued start job for default target Graphical Interface.
[    8.156570] random: systemd: uninitialized urandom read (16 bytes read)
[    8.164923] systemd[1]: Created slice system-getty.slice.
[    8.201373] random: systemd: uninitialized urandom read (16 bytes read)
[    8.208682] systemd[1]: Created slice system-modprobe.slice.
[    8.241347] random: systemd: uninitialized urandom read (16 bytes read)
[    8.248610] systemd[1]: Created slice system-serial\x2dgetty.slice.
[    8.281970] systemd[1]: Created slice User and Session Slice.
[    8.321507] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    8.371436] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    8.421373] systemd[1]: Condition check resulted in Arbitrary Executable File Formats File System Automount Point being skipped.
[    8.433099] systemd[1]: Reached target Local Encrypted Volumes.
[    8.481453] systemd[1]: Reached target Paths.
[    8.521358] systemd[1]: Reached target Remote File Systems.
[    8.571330] systemd[1]: Reached target Slices.
[    8.611374] systemd[1]: Reached target Swap.
[    8.641568] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    8.693521] systemd[1]: Listening on Process Core Dump Socket.
[    8.745061] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[    8.759882] systemd[1]: Listening on Journal Socket (/dev/log).
[    8.801664] systemd[1]: Listening on Journal Socket.
[    8.848051] systemd[1]: Listening on Network Service Netlink Socket.
[    8.892567] systemd[1]: Listening on udev Control Socket.
[    8.941553] systemd[1]: Listening on udev Kernel Socket.
[    8.981628] systemd[1]: Condition check resulted in Huge Pages File System being skipped.
[    8.990034] systemd[1]: Condition check resulted in POSIX Message Queue File System being skipped.
[    8.999279] systemd[1]: Condition check resulted in Kernel Debug File System being skipped.
[    9.010575] systemd[1]: Mounting Kernel Trace File System...
[    9.043918] systemd[1]: Mounting Temporary Directory (/tmp)...
[    9.081515] systemd[1]: Condition check resulted in Create list of static device nodes for the current kernel being skipped.
[    9.095686] systemd[1]: Starting Load Kernel Module drm...
[    9.138063] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[    9.148885] systemd[1]: Condition check resulted in Load Kernel Modules being skipped.
[    9.157149] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
[    9.165757] systemd[1]: Condition check resulted in Kernel Configuration File System being skipped.
[    9.177585] systemd[1]: Starting Remount Root and Kernel File Systems...
[    9.211505] systemd[1]: Condition check resulted in Repartition Root Disk being skipped.
[    9.222607] systemd[1]: Starting Apply Kernel Variables...
[    9.264359] systemd[1]: Starting Coldplug All udev Devices...
[    9.305343] systemd[1]: Mounted Kernel Trace File System.
[    9.352014] systemd[1]: Mounted Temporary Directory (/tmp).
[    9.391997] systemd[1]: [email protected]: Succeeded.
[    9.398342] systemd[1]: Finished Load Kernel Module drm.
[    9.436436] systemd[1]: Finished Remount Root and Kernel File Systems.
[    9.472894] systemd[1]: Finished Apply Kernel Variables.
[    9.514310] systemd[1]: Condition check resulted in First Boot Wizard being skipped.
[    9.529349] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[    9.540514] systemd[1]: Starting Load/Save Random Seed...
[    9.561757] systemd[1]: Condition check resulted in Create System Users being skipped.
[    9.578340] systemd[1]: Starting Create Static Device Nodes in /dev...
[    9.639784] systemd[1]: Finished Create Static Device Nodes in /dev.
[    9.692025] systemd[1]: Reached target Local File Systems (Pre).
[    9.741485] systemd[1]: Condition check resulted in Virtual Machine and Container Storage (Compatibility) being skipped.
[    9.752637] systemd[1]: Reached target Local File Systems.
[    9.794672] systemd[1]: Started Entropy Daemon based on the HAVEGE algorithm.
[    9.831672] systemd[1]: Condition check resulted in Rebuild Dynamic Linker Cache being skipped.
[    9.844130] systemd[1]: Starting Journal Service...
[    9.861510] systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
[    9.885260] systemd[1]: Starting Rule-based Manager for Device Events and Files...
[    9.932999] systemd[1]: Finished Coldplug All udev Devices.
[   10.175983] systemd[1]: Started Journal Service.
[   11.579842] mvneta f1070000.ethernet eth0: configuring for fixed/rgmii link mode
[   11.607754] mvneta f1070000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   11.787479] mvneta f1034000.ethernet eth2: PHY [f1072004.mdio-mii:01] driver [Marvell 88E1510] (irq=POLL)
[   11.817734] mvneta f1034000.ethernet eth2: configuring for phy/sgmii link mode
[   12.102369] BTRFS info (device mmcblk0p1): devid 1 device path /dev/root changed to /dev/mmcblk0p1 scanned by systemd-udevd (194)
[   13.131291] random: crng init done
[   13.134710] random: 7 urandom warning(s) missed due to ratelimiting
[   14.961639] mvneta f1034000.ethernet eth2: Link is Up - 1Gbps/Full - flow control rx/tx
[   14.969684] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Quoted 44 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 41 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Quoted 4 lines

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Got the same device working with > 5.10.0-rc1-next-20201027-to-dirty < but ASPM turned off, as mentioned in the cited bug report.

Note: related issues - workaround compile ath and cfg80211 as modules

(1) https://bugzilla.kernel.org/show_bug.cgi?id=209863
(2) https://bugzilla.kernel.org/show_bug.cgi?id=209855
(3) https://bugzilla.kernel.org/show_bug.cgi?id=209853

Quoted 5 lines

Note: related issues - workaround compile ath and cfg80211 as modules

(1) https://bugzilla.kernel.org/show_bug.cgi?id=209863
(2) https://bugzilla.kernel.org/show_bug.cgi?id=209855
(3) https://bugzilla.kernel.org/show_bug.cgi?id=209853

Yeah, I had noticed the regdb failure but put off debugging that until the PCI issue was resolved. So guess that's next on my list - thanks for the pointer (although I'd rather avoid the module approach as booting the kernel directly from my build box over tftp is quite convenient... Let's see if there isn't another way to fix this)

Quoted 10 lines · Toke Høiland-Jørgensen

Quoted 5 lines

Note: related issues - workaround compile ath and cfg80211 as modules

(1) https://bugzilla.kernel.org/show_bug.cgi?id=209863
(2) https://bugzilla.kernel.org/show_bug.cgi?id=209855
(3) https://bugzilla.kernel.org/show_bug.cgi?id=209853

Yeah, I had noticed the regdb failure but put off debugging that until the PCI issue was resolved. So guess that's next on my list - thanks for the pointer (although I'd rather avoid the module approach as booting the kernel directly from my build box over tftp is quite convenient... Let's see if there isn't another way to fix this)

To follow up on this, everything seems to work just fine (ath10k init at boot + regulatory db load) if I simply set:

CONFIG_EXTRA_FIRMWARE="ath10k/QCA988X/hw2.0/board.bin ath10k/QCA988X/hw2.0/firmware-5.bin regulatory.db regulatory.db.p7s"

-Toke

Quoted 12 lines · Toke Høiland-Jørgensen

Quoted 10 lines · Toke Høiland-Jørgensen

Quoted 5 lines

Note: related issues - workaround compile ath and cfg80211 as modules

(1) https://bugzilla.kernel.org/show_bug.cgi?id=209863
(2) https://bugzilla.kernel.org/show_bug.cgi?id=209855
(3) https://bugzilla.kernel.org/show_bug.cgi?id=209853

Yeah, I had noticed the regdb failure but put off debugging that until the PCI issue was resolved. So guess that's next on my list - thanks for the pointer (although I'd rather avoid the module approach as booting the kernel directly from my build box over tftp is quite convenient... Let's see if there isn't another way to fix this)

Quoted 7 lines

To follow up on this, everything seems to work just fine (ath10k init at boot + regulatory db load) if I simply set:

CONFIG_EXTRA_FIRMWARE="ath10k/QCA988X/hw2.0/board.bin ath10k/QCA988X/hw2.0/firmware-5.bin regulatory.db regulatory.db.p7s"

-Toke

That works on my node only for the regulatory files but not the ath10 firmware with kconfig:

 Symbol: EXTRA_FIRMWARE_DIR [=/srv/fw]
 Type  : string
 Defined at drivers/base/firmware_loader/Kconfig:63
   Prompt: Firmware blobs root directory
   Depends on: FW_LOADER [=y] && EXTRA_FIRMWARE [=regulatory.db
regulatory.db.p7s board.bin firmware-5.bin]!=
   Location:
    -> Device Drivers
      -> Generic Driver Options
        -> Firmware loader
          -> Firmware loading facility (FW_LOADER [=y]) -> Build named firmware blobs into the kernel binary
(EXTRA_FIRMWARE [=regulatory.db regulatory.db.p7s board.bin firmware-5.bin])

Quoted 10 lines

Quoted 7 lines

To follow up on this, everything seems to work just fine (ath10k init at boot + regulatory db load) if I simply set:

CONFIG_EXTRA_FIRMWARE="ath10k/QCA988X/hw2.0/board.bin ath10k/QCA988X/hw2.0/firmware-5.bin regulatory.db regulatory.db.p7s"

-Toke

That works on my node only for the regulatory files but not the ath10 firmware with kconfig:

Quoted 14 lines

 Symbol: EXTRA_FIRMWARE_DIR [=/srv/fw]
 Type  : string
 Defined at drivers/base/firmware_loader/Kconfig:63
   Prompt: Firmware blobs root directory
   Depends on: FW_LOADER [=y] && EXTRA_FIRMWARE [=regulatory.db
regulatory.db.p7s board.bin firmware-5.bin]!=
   Location:
    -> Device Drivers
      -> Generic Driver Options
        -> Firmware loader
          -> Firmware loading facility (FW_LOADER [=y]) -> Build named firmware blobs into the kernel binary
(EXTRA_FIRMWARE [=regulatory.db regulatory.db.p7s board.bin firmware-5.bin])

I think that's because you're missing the path prefix (ath10k/QCA988X/hw2.0/) from board.bin and firmware-5.bin? request_firmware() uses the full path...

-Toke

Quoted 24 lines · Toke Høiland-Jørgensen

Quoted 10 lines

Quoted 7 lines

To follow up on this, everything seems to work just fine (ath10k init at boot + regulatory db load) if I simply set:

CONFIG_EXTRA_FIRMWARE="ath10k/QCA988X/hw2.0/board.bin ath10k/QCA988X/hw2.0/firmware-5.bin regulatory.db regulatory.db.p7s"

-Toke

That works on my node only for the regulatory files but not the ath10 firmware with kconfig:

Quoted 14 lines

 Symbol: EXTRA_FIRMWARE_DIR [=/srv/fw]
 Type  : string
 Defined at drivers/base/firmware_loader/Kconfig:63
   Prompt: Firmware blobs root directory
   Depends on: FW_LOADER [=y] && EXTRA_FIRMWARE [=regulatory.db
regulatory.db.p7s board.bin firmware-5.bin]!=
   Location:
    -> Device Drivers
      -> Generic Driver Options
        -> Firmware loader
          -> Firmware loading facility (FW_LOADER [=y]) -> Build named firmware blobs into the kernel binary
(EXTRA_FIRMWARE [=regulatory.db regulatory.db.p7s board.bin firmware-5.bin])

Quoted 5 lines

I think that's because you're missing the path prefix (ath10k/QCA988X/hw2.0/) from board.bin and firmware-5.bin? request_firmware() uses the full path...

-Toke

Well, that would be weird/strange having to specify the path prefix for build-in firmware,considering:

 CONFIG_FW_LOADER:

 This enables the firmware loading facility in the kernel. The kernel will first look for built-in firmware, if it has any. Next, it will look for the requested firmware in a series of filesystem paths:

       o firmware_class path module parameter or kernel boot param
       o /lib/firmware/updates/UTS_RELEASE
       o /lib/firmware/updates
       o /lib/firmware/UTS_RELEASE
       o /lib/firmware

----

Nevertheless, I tried with same path prefix as per your kconfig but the compilation fails, which I am not surprised since the ath10 blobs are not located at that path

  UPD     drivers/base/firmware_loader/builtin/regulatory.db.gen.S UPD drivers/base/firmware_loader/builtin/regulatory.db.p7s.gen.S
make[4]: *** No rule to make target '/srv/fw/ath10k/QCA988X/hw2.0/board.bin', needed by

'drivers/base/firmware_loader/builtin/ath10k/QCA988X/hw2.0/board.bin.gen.o'. Stop.
make[4]: *** Waiting for unfinished jobs....
  UPD
drivers/base/firmware_loader/builtin/ath10k/QCA988X/hw2.0/board.bin.gen.S make[3]: *** [scripts/Makefile.build:500:
drivers/base/firmware_loader/builtin] Error 2
make[2]: *** [scripts/Makefile.build:500: drivers/base/firmware_loader] Error 2
make[1]: *** [scripts/Makefile.build:500: drivers/base] Error 2 make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1799: drivers] Error 2
make: *** Waiting for unfinished jobs....

I suspect that since you are booting the kernel directly from my build box over tftp it accesses the ath10 firmware blobs on the build box.

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Quoted 45 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 41 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Quoted 3 lines

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

Quoted 53 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 8 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 55 lines · Toke Høiland-Jørgensen

Quoted 53 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 8 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 8 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Quoted 61 lines · Bjorn Helgaas

Quoted 53 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

-Toke

Quoted 76 lines · Toke Høiland-Jørgensen

Quoted 60 lines · Bjorn Helgaas

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

[+cc Pali, Marek, Thomas, Jason]

Quoted 54 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 53 lines · Toke Høiland-Jørgensen

Quoted 52 lines · Bjorn Helgaas

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 50 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 23 lines

Quoted 23 lines

Quoted 8 lines

Quoted 8 lines

Quoted 3 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 8 lines

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Interesting. Indeed, the Compex WLE900VX card seems to have the Qualcomm Atheros QCA9880 on it, and it looks like Toke's system has the same device in it.

The patch you mention (https://git.kernel.org/linus/43fc679ced18) is for aardvark, so of course doesn't help mvebu.

PCIe hardware is supposed to automatically negotiate the highest link speed supported by both ends. But software *is* allowed to set an upper limit (the Target Link Speed in Link Control 2). If we initiate a retrain and the link doesn't come back up, I wonder if we should try to help the hardware out by using Target Link Speed to limit to a lower speed and attempting another retrain, something like this hacky
patch: (please collect the dmesg log if you try this)

drivers/pci/pcie/aspm.c +36 -2

index ac0557a305af..fb6e13532a2c 100644
@@ -192,12 +192,42 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
 	link->clkpm_disable = blacklist ? 1 : 0;
 }
 
+#define PCI_EXP_LNKCAP2_SLS	0x000000fe
+
+static int decrease_tls(struct pci_dev *pdev)
+{
+	u32 lnkcap2;
+	u16 lnkctl2, tls;
+
+	pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP2, &lnkcap2);
+
+	pcie_capability_read_word(pdev, PCI_EXP_LNKCTL2, &lnkctl2);
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(pdev, "lnkcap2 %#010x sls %#04x lnkctl2 %#06x tls %#03x\n",
+		 lnkcap2, (lnkcap2 & PCI_EXP_LNKCAP2_SLS) >> 1,
+		 lnkctl2, tls);
+
+	if (tls < 2)
+		return -EINVAL;
+
+	tls--;
+	pcie_capability_clear_and_set_word(pdev, PCI_EXP_LNKCTL2,
+					   PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(pdev, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(pdev, "lnkctl2 %#010x new tls %#03x\n",
+		 lnkctl2, tls);
+
+	return 0;
+}
+
 static bool pcie_retrain_link(struct pcie_link_state *link)
 {
 	struct pci_dev *parent = link->pdev;
 	unsigned long end_jiffies;
 	u16 reg16;
 
+top:
 	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
 	reg16 |= PCI_EXP_LNKCTL_RL;
 	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

@@ -216,10 +246,14 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
 	do {
 		pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &reg16);
 		if (!(reg16 & PCI_EXP_LNKSTA_LT))
-			break;
+			return true;	/* success */
 		msleep(1);
 	} while (time_before(jiffies, end_jiffies));
-	return !(reg16 & PCI_EXP_LNKSTA_LT);
+
+	if (decrease_tls(parent))
+		return false;	/* can't decrease any more */
+
+	goto top;
 }
 
 /*

Hello!

Quoted 56 lines · On Wednesday 28 October 2020 18:16:26 Bjorn Helgaas wrote:

[+cc Pali, Marek, Thomas, Jason]

Quoted 53 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Bjorn Helgaas

Quoted 50 lines · Toke Høiland-Jørgensen

Quoted 49 lines · Toke Høiland-Jørgensen

Quoted 47 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yes! Turning that off does indeed help! Thanks a bunch :)

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Hello!

Quoted 56 lines · On Wednesday 28 October 2020 18:16:26 Bjorn Helgaas wrote:

[+cc Pali, Marek, Thomas, Jason]

Quoted 53 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Bjorn Helgaas

Quoted 50 lines · Toke Høiland-Jørgensen

Quoted 49 lines · Toke Høiland-Jørgensen

Quoted 47 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yes! Turning that off does indeed help! Thanks a bunch :)

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Quoted 4 lines · Pali Rohár

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

Quoted 2 lines

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Yup, exactly. And I'm also currently testing with the default WLE200/900 cards... I just tried sticking an MT76-based WiFi card into the third PCI slot, and that doesn't come up either when I enable PCIEASPM.

Quoted 10 lines · Toke Høiland-Jørgensen

Quoted 6 lines · Pali Rohár

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Yup, exactly. And I'm also currently testing with the default WLE200/900 cards... I just tried sticking an MT76-based WiFi card into the third PCI slot, and that doesn't come up either when I enable PCIEASPM.

Huh. So IIUC, the following cases all try to retrain the link and it fails to come up again:

- aardvark + WLE900VX (see commit 43fc679ced18)
- mvebu + WLE200
- mvebu + WLE900
- mvebu + MT76

In all these cases, Linux was able to enumerate the NIC, which means the link was up when firmware handed it off.

I think Linux decided the Common Clock Configuration was wrong, so it tried to fix it and retrain the link, and the link didn't come back up.

I don't have "lspci -vv" output from all of them, but in vtolkm's case, the firmware handed off with:

   00:02.0 Root Port to [bus 02]  SlotClk+ CommClk+
   02:00.0 QCA986x/988x NIC       SlotClk+ CommClk-

Per spec (PCIe r5, sec 7.5.3.7), SlotClk is HwInit and CommClk is RW and should power up as 0. If I'm reading the implementation note correctly, if SlotClk is set on both ends of the link, software should set CommClk, so the config above *does* look wrong, and CommClk+ on the Root Port suggests that firmware set it.

I think both the aardvark and mvebu systems probably use U-Boot. I don't know U-Boot at all, but I don't see anything in it that touches Link Control. I'm curious what happens if you put one of these cards in a PC. If anybody tries it, please collect the "sudo lspci -vv" and dmesg output.

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection and WLE200/WLE900 and MT76 have no obvious connection, I doubt there's a simple hardware defect that explains all these.

Maybe we're doing something wrong in the retrain, but obviously the link came up in the first place. AFAIK the only thing we're changing is the CommClk setting, and that looks legitimate per spec.

Another experiment: build kernel without CONFIG_PCIEASPM, set $ROOT and $NIC appropriately, and try the following:

   # Set $ROOT and $NIC (update to match your system):

     # ROOT=00:02.0
     # NIC=02:00.0

   # Dump the Root Port and NIC Link registers:

     # setpci -s$ROOT CAP_EXP+0xc.l              # Link Capabilities
     # setpci -s$ROOT CAP_EXP+0x10.w             # Link Control
     # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status

     # setpci -s$NIC  CAP_EXP+0xc.l              # Link Capabilities
     # setpci -s$NIC  CAP_EXP+0x10.w             # Link Control
     # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

   # Retrain the link:

     # setpci -s$ROOT CAP_EXP+0x10.w=0x0020      # Link Control Retrain Link
     # sleep 1
     # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
     # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

   # Set CommClk+ and retrain the link:

     # setpci -s$NIC  CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
     # setpci -s$ROOT CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
     # setpci -s$ROOT CAP_EXP+0x10.w=0x0060      # Link Control RL + CC
     # sleep 1
     # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
     # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection

Quoted 2 lines

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection

Both are Mavell. There could be some shared IP.

Another experiment: build kernel without CONFIG_PCIEASPM, set $ROOT and $NIC appropriately, and try the following:

# Set $ROOT and $NIC (update to match your system):

    # ROOT=00:02.0
    # NIC=02:00.0

# Dump the Root Port and NIC Link registers:

    # setpci -s$ROOT CAP_EXP+0xc.l              # Link Capabilities
    # setpci -s$ROOT CAP_EXP+0x10.w             # Link Control
    # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status

    # setpci -s$NIC  CAP_EXP+0xc.l              # Link Capabilities
    # setpci -s$NIC  CAP_EXP+0x10.w             # Link Control
    # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

# Retrain the link:

    # setpci -s$ROOT CAP_EXP+0x10.w=0x0020      # Link Control Retrain Link
    # sleep 1
    # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
    # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

# Set CommClk+ and retrain the link:

    # setpci -s$NIC  CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
    # setpci -s$ROOT CAP_EXP+0x10.w=0x0040      # Link Control Common Clock
    # setpci -s$ROOT CAP_EXP+0x10.w=0x0060      # Link Control RL + CC
    # sleep 1
    # setpci -s$ROOT CAP_EXP+0x12.w             # Link Status
    # setpci -s$NIC  CAP_EXP+0x12.w             # Link Status

Quoted 7 lines · Bjorn Helgaas

Another experiment: build kernel without CONFIG_PCIEASPM, set $ROOT and $NIC appropriately, and try the following:

# Set $ROOT and $NIC (update to match your system):

    # ROOT=00:02.0
    # NIC=02:00.0

(these matched the ath10k card, so just went with that)

Quoted 6 lines · Toke Høiland-Jørgensen

Quoted 4 lines · Pali Rohár

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

Quoted 6 lines

Quoted 2 lines

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Yup, exactly. And I'm also currently testing with the default WLE200/900 cards... I just tried sticking an MT76-based WiFi card into the third PCI slot, and that doesn't come up either when I enable PCIEASPM.

Huh. So IIUC, the following cases all try to retrain the link and it fails to come up again:

- aardvark + WLE900VX (see commit 43fc679ced18)
- mvebu + WLE200
- mvebu + WLE900
- mvebu + MT76

Quoted 10 lines · On Thu, 29 Oct 2020 14:30:22 -0500 Bjorn Helgaas <[email protected]> wrote:

Quoted 9 lines · Toke Høiland-Jørgensen

Quoted 6 lines · Pali Rohár

I have been testing mainline kernel on Turris Omnia with two PCIe default cards (WLE200 and WLE900) and it worked fine. But I do not know if I had ASPM enabled or not.

So it is working fine for you when CONFIG_PCIEASPM is disabled and whole issue is only when CONFIG_PCIEASPM is enabled?

Quoted 8 lines

Quoted 3 lines

Yup, exactly. And I'm also currently testing with the default WLE200/900 cards... I just tried sticking an MT76-based WiFi card into the third PCI slot, and that doesn't come up either when I enable PCIEASPM.

Huh. So IIUC, the following cases all try to retrain the link and it fails to come up again:

- aardvark + WLE900VX (see commit 43fc679ced18)

Quoted 3 lines

- mvebu + WLE200
- mvebu + WLE900
- mvebu + MT76

Bjorn, IIRC Pali's patches fix the WLE900VX card for Aardvark (both in kernel and in U-Boot).
IMO mvebu has similar issues. Both these drivers handle the PCIe reset signal incorrectly (or at least Aardvark did before Pali's work).

mvebu is used on Turris Omnia, and our HW guys first solved the WLE900VX not working issue by using different capacitors for the SerDeses (this was 5 years ago). But after Pali's work on Aardvark I think this could also be solved for mvebu driver in software.

BTW the WLE900VX card has problems on many systems, it won't work for example on Thinkpad X230. There is a bug on kernel bugzilla reported for this.

My opinion is that many drivers do not respect the PCIe specification for reset and link training totally correctly (Pali was talking about this when he was looking at Aardvark) and that WLE900VX has a bug that in combination with those drivers causes the fail. If you look at the drivers, they are incompatible in how they handle the reset signal and link training.

I am curious what Pali will tell us, he said that he will look into the mvebu driver.

On Turris Omnia (with pci-mvebu) PERST# pin from wifi card is connected to MCU and it asserts/deasserts this pin only after board reset. Also it is shared line across all mPCIe slots and also with other peripherals.

So we cannot issue reset via PERST# signal on Turris Omnia. But there are other ways how to issue fundamental reset, via in band signaling.

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection and WLE200/WLE900 and MT76 have no obvious connection, I doubt there's a simple hardware defect that explains all these.

Hello,

Quoted 4 lines · On Thu, 29 Oct 2020 14:30:22 -0500 Bjorn Helgaas <[email protected]> wrote:

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection and WLE200/WLE900 and MT76 have no obvious connection, I doubt there's a simple hardware defect that explains all these.

aardvark and mvebu have one very strong connection: they are the only two drivers making use of the PCI Bridge emulation logic in drivers/pci/pci-bridge-emul.c:

drivers/pci$ git grep pci-bridge-emul akefile:obj-$(CONFIG_PCI_BRIDGE_EMUL) += pci-bridge-emul.o controller/pci-aardvark.c:#include "../pci-bridge-emul.h" controller/pci-mvebu.c:#include "../pci-bridge-emul.h" pci-bridge-emul.c:#include "pci-bridge-emul.h"

I haven't read the whole thread, but it is important to keep in mind that on those two platforms, the PCI Bridge seen by Linux is *not* a real HW bridge. It is faked by the the pci-bridge-emul code. So if this code has defects/bugs in how it emulates a PCI Bridge behavior, you might see weird things.

Quoted 5 lines

I haven't read the whole thread, but it is important to keep in mind that on those two platforms, the PCI Bridge seen by Linux is *not* a real HW bridge. It is faked by the the pci-bridge-emul code. So if this code has defects/bugs in how it emulates a PCI Bridge behavior, you might see weird things.

Ohh, that's interesting. Why does it need to emulate it?

Quoted 20 lines · Thomas Petazzoni

Hello,

Quoted 4 lines · On Thu, 29 Oct 2020 14:30:22 -0500 Bjorn Helgaas <[email protected]> wrote:

We could quirk these NICs to avoid the retrain, but since aardvark and mvebu have no obvious connection and WLE200/WLE900 and MT76 have no obvious connection, I doubt there's a simple hardware defect that explains all these.

aardvark and mvebu have one very strong connection: they are the only two drivers making use of the PCI Bridge emulation logic in drivers/pci/pci-bridge-emul.c:

drivers/pci$ git grep pci-bridge-emul akefile:obj-$(CONFIG_PCI_BRIDGE_EMUL) += pci-bridge-emul.o controller/pci-aardvark.c:#include "../pci-bridge-emul.h" controller/pci-mvebu.c:#include "../pci-bridge-emul.h" pci-bridge-emul.c:#include "pci-bridge-emul.h"

Quoted 5 lines

I haven't read the whole thread, but it is important to keep in mind that on those two platforms, the PCI Bridge seen by Linux is *not* a real HW bridge. It is faked by the the pci-bridge-emul code. So if this code has defects/bugs in how it emulates a PCI Bridge behavior, you might see weird things.

Ohh, that's interesting. Why does it need to emulate it?

And could this cause things weird interactions like what I'm seeing, where a somewhat buggy device in slot 2 affects the ability to retrain the link also in slot 1, but only if there's no device in slot 3?

[+cc Pali, Marek, Thomas, Jason]

Quoted 77 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 76 lines · Toke Høiland-Jørgensen

Quoted 60 lines · Bjorn Helgaas

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 8 lines

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Interesting. Indeed, the Compex WLE900VX card seems to have the Qualcomm Atheros QCA9880 on it, and it looks like Toke's system has the same device in it.

The patch you mention (https://git.kernel.org/linus/43fc679ced18) is for aardvark, so of course doesn't help mvebu.

PCIe hardware is supposed to automatically negotiate the highest link speed supported by both ends. But software *is* allowed to set an upper limit (the Target Link Speed in Link Control 2). If we initiate a retrain and the link doesn't come back up, I wonder if we should try to help the hardware out by using Target Link Speed to limit to a lower speed and attempting another retrain, something like this hacky
patch: (please collect the dmesg log if you try this)

Quoted 94 lines · Bjorn Helgaas

[+cc Pali, Marek, Thomas, Jason]

Quoted 77 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 76 lines · Toke Høiland-Jørgensen

Quoted 60 lines · Bjorn Helgaas

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 7 lines

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Interesting. Indeed, the Compex WLE900VX card seems to have the Qualcomm Atheros QCA9880 on it, and it looks like Toke's system has the same device in it.

The patch you mention (https://git.kernel.org/linus/43fc679ced18) is for aardvark, so of course doesn't help mvebu.

Quoted 7 lines

PCIe hardware is supposed to automatically negotiate the highest link speed supported by both ends. But software *is* allowed to set an upper limit (the Target Link Speed in Link Control 2). If we initiate a retrain and the link doesn't come back up, I wonder if we should try to help the hardware out by using Target Link Speed to limit to a lower speed and attempting another retrain, something like this hacky
patch: (please collect the dmesg log if you try this)

Well, I tried it, but don't see any of the 'lnkcap2' output from that new function:

[    1.545853] mvebu-pcie soc:pcie: host bridge /soc/pcie ranges:
[    1.545878] mvebu-pcie soc:pcie:      MEM 0x00f1080000..0x00f1081fff -> 0x0000080000
[    1.545894] mvebu-pcie soc:pcie:      MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
[    1.545907] mvebu-pcie soc:pcie:      MEM 0x00f1044000..0x00f1045fff -> 0x0000044000
[    1.545920] mvebu-pcie soc:pcie:      MEM 0x00f1048000..0x00f1049fff -> 0x0000048000
[    1.545933] mvebu-pcie soc:pcie:      MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[    1.545945] mvebu-pcie soc:pcie:       IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[    1.545958] mvebu-pcie soc:pcie:      MEM 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
[    1.545970] mvebu-pcie soc:pcie:       IO 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
[    1.545982] mvebu-pcie soc:pcie:      MEM 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
[    1.545994] mvebu-pcie soc:pcie:       IO 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
[    1.546006] mvebu-pcie soc:pcie:      MEM 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
[    1.546014] mvebu-pcie soc:pcie:       IO 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
[    1.546181] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
[    1.546190] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.546197] pci_bus 0000:00: root bus resource [mem 0xf1080000-0xf1081fff] (bus address [0x00080000-0x00081fff])
[    1.546204] pci_bus 0000:00: root bus resource [mem 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
[    1.546210] pci_bus 0000:00: root bus resource [mem 0xf1044000-0xf1045fff] (bus address [0x00044000-0x00045fff])
[    1.546216] pci_bus 0000:00: root bus resource [mem 0xf1048000-0xf1049fff] (bus address [0x00048000-0x00049fff])
[    1.546220] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
[    1.546225] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
[    1.546294] pci 0000:00:01.0: [11ab:6820] type 01 class 0x060400
[    1.546308] pci 0000:00:01.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    1.546482] pci 0000:00:02.0: [11ab:6820] type 01 class 0x060400
[    1.546495] pci 0000:00:02.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    1.546643] pci 0000:00:03.0: [11ab:6820] type 01 class 0x060400
[    1.546656] pci 0000:00:03.0: reg 0x38: [mem 0x00000000-0x000007ff pref]
[    1.547379] PCI: bus0: Fast back to back transfers disabled
[    1.547387] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547394] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547402] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.547484] pci 0000:01:00.0: [168c:002e] type 00 class 0x028000
[    1.547507] pci 0000:01:00.0: reg 0x10: [mem 0xe8000000-0xe800ffff 64bit]
[    1.547615] pci 0000:01:00.0: supports D1
[    1.547620] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[    1.547730] pci 0000:00:01.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[    1.631937] PCI: bus2: Fast back to back transfers enabled
[    1.631945] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.632655] PCI: bus3: Fast back to back transfers enabled
[    1.632662] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[    1.632694] pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[    1.632702] pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
[    1.632710] pci 0000:00:01.0: BAR 6: assigned [mem 0xe0100000-0xe01007ff pref]
[    1.632718] pci 0000:00:02.0: BAR 6: assigned [mem 0xe0500000-0xe05007ff pref]
[    1.632726] pci 0000:00:03.0: BAR 6: assigned [mem 0xe0600000-0xe06007ff pref]
[    1.632734] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe000ffff 64bit]
[    1.632741] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632746] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632752] pci 0000:00:01.0: PCI bridge to [bus 01]
[    1.632760] pci 0000:00:01.0:   bridge window [mem 0xe0000000-0xe00fffff]
[    1.632769] pci 0000:02:00.0: BAR 0: assigned [mem 0xe0200000-0xe03fffff 64bit]
[    1.632776] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632782] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632788] pci 0000:02:00.0: BAR 6: assigned [mem 0xe0400000-0xe040ffff pref]
[    1.632793] pci 0000:00:02.0: PCI bridge to [bus 02]
[    1.632800] pci 0000:00:02.0:   bridge window [mem 0xe0200000-0xe04fffff]
[    1.632807] pci 0000:00:03.0: PCI bridge to [bus 03]

(and then later, still):

[    3.476364] pci 0000:00:01.0: enabling device (0140 -> 0142)
[    3.477542] ata1: SATA link down (SStatus 0 SControl 300)
[    3.482126] ath9k 0000:01:00.0: enabling device (0000 -> 0002)
[    3.487487] ata2: SATA link down (SStatus 0 SControl 300)
[    3.493379] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.505891] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.513325] ath9k 0000:01:00.0: Failed to initialize device
[    3.518933] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.524862] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.531904] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.537590] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.577436] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.583948] ath10k_pci: probe of 0000:02:00.0 failed with error -110

-Toke

[+cc Pali, Marek, Thomas, Jason]

Quoted 77 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 76 lines · Toke Høiland-Jørgensen

Quoted 60 lines · Bjorn Helgaas

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 8 lines

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Interesting. Indeed, the Compex WLE900VX card seems to have the Qualcomm Atheros QCA9880 on it, and it looks like Toke's system has the same device in it.

The patch you mention (https://git.kernel.org/linus/43fc679ced18) is for aardvark, so of course doesn't help mvebu.

PCIe hardware is supposed to automatically negotiate the highest link speed supported by both ends. But software *is* allowed to set an upper limit (the Target Link Speed in Link Control 2). If we initiate a retrain and the link doesn't come back up, I wonder if we should try to help the hardware out by using Target Link Speed to limit to a lower speed and attempting another retrain, something like this hacky
patch: (please collect the dmesg log if you try this)

drivers/pci/pcie/aspm.c +36 -2

index ac0557a305af..fb6e13532a2c 100644
@@ -192,12 +192,42 @@ static void pcie_clkpm_cap_init(struct pcie_link_state *link, int blacklist)
 	link->clkpm_disable = blacklist ? 1 : 0;
 }
 
+#define PCI_EXP_LNKCAP2_SLS	0x000000fe
+
+static int decrease_tls(struct pci_dev *pdev)
+{
+	u32 lnkcap2;
+	u16 lnkctl2, tls;
+
+	pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP2, &lnkcap2);
+
+	pcie_capability_read_word(pdev, PCI_EXP_LNKCTL2, &lnkctl2);
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(pdev, "lnkcap2 %#010x sls %#04x lnkctl2 %#06x tls %#03x\n",
+		 lnkcap2, (lnkcap2 & PCI_EXP_LNKCAP2_SLS) >> 1,
+		 lnkctl2, tls);
+
+	if (tls < 2)
+		return -EINVAL;
+
+	tls--;
+	pcie_capability_clear_and_set_word(pdev, PCI_EXP_LNKCTL2,
+					   PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(pdev, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(pdev, "lnkctl2 %#010x new tls %#03x\n",
+		 lnkctl2, tls);
+
+	return 0;
+}
+
 static bool pcie_retrain_link(struct pcie_link_state *link)
 {
 	struct pci_dev *parent = link->pdev;
 	unsigned long end_jiffies;
 	u16 reg16;
 
+top:
 	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
 	reg16 |= PCI_EXP_LNKCTL_RL;
 	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

@@ -216,10 +246,14 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
 	do {
 		pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &reg16);
 		if (!(reg16 & PCI_EXP_LNKSTA_LT))
-			break;
+			return true;	/* success */
 		msleep(1);
 	} while (time_before(jiffies, end_jiffies));
-	return !(reg16 & PCI_EXP_LNKSTA_LT);
+
+	if (decrease_tls(parent))
+		return false;	/* can't decrease any more */
+
+	goto top;
 }
 
 /*

Quoted 96 lines · On Wednesday 28 October 2020 18:16:26 Bjorn Helgaas wrote:

[+cc Pali, Marek, Thomas, Jason]

Quoted 77 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 76 lines · Toke Høiland-Jørgensen

Quoted 60 lines · Bjorn Helgaas

Quoted 52 lines · Toke Høiland-Jørgensen

Quoted 51 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 7 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 7 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

Quoted 6 lines

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Quoted 3 lines

Maybe the debug patch below would be worth trying to see if it makes any difference? If it *does* help, try omitting the first hunk to see if we just need to apply the quirk_enable_clear_retrain_link() quirk.

Tried, doesn't help...

Quoted 8 lines

Quoted 1 line

-Toke

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Interesting. Indeed, the Compex WLE900VX card seems to have the Qualcomm Atheros QCA9880 on it, and it looks like Toke's system has the same device in it.

The patch you mention (https://git.kernel.org/linus/43fc679ced18) is for aardvark, so of course doesn't help mvebu.

Quoted 7 lines

PCIe hardware is supposed to automatically negotiate the highest link speed supported by both ends. But software *is* allowed to set an upper limit (the Target Link Speed in Link Control 2). If we initiate a retrain and the link doesn't come back up, I wonder if we should try to help the hardware out by using Target Link Speed to limit to a lower speed and attempting another retrain, something like this hacky
patch: (please collect the dmesg log if you try this)

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

Quoted 74 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 6 lines

drivers/pci/pcie/aspm.c +2 -0

  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 75 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 74 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);

Quoted 8 lines

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 6 lines

drivers/pci/pcie/aspm.c +2 -0

  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Yup, same for me :(

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 1 line

Yup, same for me :(

So then it is different issue and not similar to aardvark one.

Anyway, was ASPM working on some previous kernel version? Or was it always broken on Turris Omnia?

And has somebody other Armada 385 device with mPCIe slots to test if ASPM is working? Or any other 32bit Marvell Armada SOC?

I would like to know if this is issue only on Turris Omnia or also on other Armada 385 SOC device or even on any other device which uses pci-mvebu.c driver.

Quoted 85 lines · Pali Rohár

Quoted 84 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 82 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
   	unsigned long end_jiffies;
   	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
   	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
   	reg16 |= PCI_EXP_LNKCTL_RL;
   	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
   			break;
   		msleep(1);
   	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
   	return !(reg16 & PCI_EXP_LNKSTA_LT);
   }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 5 lines

Quoted 1 line

Yup, same for me :(

So then it is different issue and not similar to aardvark one.

Anyway, was ASPM working on some previous kernel version? Or was it always broken on Turris Omnia?

I tried bisecting and couldn't find a commit that worked. And OpenWrt by default builds with ASPM off, so my best guess is that it was always broken.

However, the two other PCI slots *do* work with ASPM on, as long as they're both occupied when booting. If I only have one card installed apart from the dodge WLE900, both of them fail...

Quoted 96 lines · Toke Høiland-Jørgensen

Quoted 85 lines · Pali Rohár

Quoted 84 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 82 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
   	unsigned long end_jiffies;
   	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
   	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
   	reg16 |= PCI_EXP_LNKCTL_RL;
   	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
   			break;
   		msleep(1);
   	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
   	return !(reg16 & PCI_EXP_LNKSTA_LT);
   }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 5 lines

Quoted 1 line

Yup, same for me :(

So then it is different issue and not similar to aardvark one.

Anyway, was ASPM working on some previous kernel version? Or was it always broken on Turris Omnia?

I tried bisecting and couldn't find a commit that worked. And OpenWrt by default builds with ASPM off, so my best guess is that it was always broken.

Quoted 3 lines

However, the two other PCI slots *do* work with ASPM on, as long as they're both occupied when booting. If I only have one card installed apart from the dodge WLE900, both of them fail...

Just to be sure it is not a (particular) mPCIe slot issue on the TO - did you change the device order in the mPCIe slots?

On my node:

- right slot (next to the CPU) hosts a SSD
- centre slot hosts WLE900VX
- left slot (over the SIM card slot) hosts the WLE200N2

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 1 line

Quoted 1 line

Yup, same for me :(

Quoted 1 line

So then it is different issue and not similar to aardvark one.

Quoted 2 lines

Anyway, was ASPM working on some previous kernel version? Or was it always broken on Turris Omnia?

I tried bisecting and couldn't find a commit that worked. And OpenWrt by default builds with ASPM off, so my best guess is that it was always broken.

However, the two other PCI slots *do* work with ASPM on, as long as they're both occupied when booting. If I only have one card installed apart from the dodge WLE900, both of them fail...

Quoted 6 lines

And has somebody other Armada 385 device with mPCIe slots to test if ASPM is working? Or any other 32bit Marvell Armada SOC?

I would like to know if this is issue only on Turris Omnia or also on other Armada 385 SOC device or even on any other device which uses pci-mvebu.c driver.

See above: It does partly work on my Omnia. Is it possible to define a quirk to just disable it on a per-slot basis for the WLE900 card? Maybe just doing that and calling it a day would be enough...

-Toke

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Quoted 1 line

Quoted 1 line

So then it is different issue and not similar to aardvark one.

... and therefore it can be still same issue which I have debugged on aardvark.

Quoted 6 lines

Quoted 2 lines

Anyway, was ASPM working on some previous kernel version? Or was it always broken on Turris Omnia?

I tried bisecting and couldn't find a commit that worked. And OpenWrt by default builds with ASPM off, so my best guess is that it was always broken.

I see and it makes sense that it does not work in any version.

Quoted 11 lines

However, the two other PCI slots *do* work with ASPM on, as long as they're both occupied when booting. If I only have one card installed apart from the dodge WLE900, both of them fail...

Quoted 6 lines

And has somebody other Armada 385 device with mPCIe slots to test if ASPM is working? Or any other 32bit Marvell Armada SOC?

I would like to know if this is issue only on Turris Omnia or also on other Armada 385 SOC device or even on any other device which uses pci-mvebu.c driver.

Quoted 6 lines

See above: It does partly work on my Omnia. Is it possible to define a quirk to just disable it on a per-slot basis for the WLE900 card? Maybe just doing that and calling it a day would be enough...

-Toke

Toke, can you try to put this WLE900 card into some x86 computer and check if this card works? With ASPM enabled and also with ASPM disabled? Or into any other device which does not have Marvell PCIe controller?

I need to know if problem is with this WLE900 card or with Marvell's PCIe controllers. And based on it I can prepare quirk / hook for either WLE900 card or for Marvell PCIe drivers (or both, based on how it is broken).

PS for all: Please do not put fancy unicode characters into email From:
header as such email would be marked as spam and automatically filtered.

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

Quoted 114 lines · Pali Rohár

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

Quoted 4 lines

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

I pasted this into the top of mvebu_pcie_setup_hw(), and that indeed seems to fix things so that all three PCIE devices work even with ASPM turned on! :)

Do you still need me to test the card on a different machine? Not sure I have an x86 machine with a mini-PCIe slot handy, but I can go hunting if needed...

Quoted 116 lines · On Thursday 18 March 2021 23:43:58 Toke Høiland-Jørgensen wrote:

Quoted 114 lines · Pali Rohár

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

Quoted 8 lines

Quoted 4 lines

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

I pasted this into the top of mvebu_pcie_setup_hw(), and that indeed seems to fix things so that all three PCIE devices work even with ASPM turned on! :)

Perfect! Now I'm sure that it is same issue as in aardvark driver.

I will prepare patches for both pci-aardvark.c and pci-mvebu.c to export PCI_EXP_LNKCTL2 register via emulated bridge. And so aspm.c code would be able to use Bjorn or my patch which I have sent last year.

Question reminds, if this is issue with QCA wifi chip on that Compex card or it is issue with PCIe controllers, now on A38x and A3720 SoC. Note that both A38x and A3720 platforms are from Marvell, but they have different PCIe controllers (so it does not mean that both must have same hw bugs).

Quoted 3 lines

Do you still need me to test the card on a different machine? Not sure I have an x86 machine with a mini-PCIe slot handy, but I can go hunting if needed...

Yes, now it is needed to know if we can find any other PCIe controller in which this card does not work when ASPM is enabled and above "retrain link" kernel code is executed.

It does not have to be x86. But due to how UEFI, ACPI and other firmwares touches PCIe, there is a high chance that on some x86 machine this bug can appear too. More firmwares = more problems.

On arm platforms with native controller drivers there does not have to be any firmware (like in Marvell case) so only kernel touches PCIe HW at the same time.

Note that on x86, ASPM may be disabled (if firmware indicates it), so command line argument like "pcie_aspm=force" is probably required for tests.

Quoted 132 lines · On Friday 19 March 2021 00:16:29 Pali Rohár wrote:

Quoted 116 lines · On Thursday 18 March 2021 23:43:58 Toke Høiland-Jørgensen wrote:

Quoted 114 lines · Pali Rohár

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset. Repeated link retraining fails and it fails even when aardvark is reconfigured to GEN1 mode. Reset via PERST# signal is required to have working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If success read "negotiated link speed" from "Link Control Status Register" (for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then retrain link again (for WLE900VX now it would be at GEN1). After that card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link training fails. And if I change mode to GEN1 after this failed link training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does one link training at GEN2. Then aspm.c is called which is doing second link retraining at GEN2. And if it fails then below patch issue third link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark then second link retraining must be at GEN1 (not GEN2) to workaround this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link retraining at GEN2 speed? And always force GEN1 speed prior link training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

Quoted 8 lines

Quoted 4 lines

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

I pasted this into the top of mvebu_pcie_setup_hw(), and that indeed seems to fix things so that all three PCIE devices work even with ASPM turned on! :)

Perfect! Now I'm sure that it is same issue as in aardvark driver.

I will prepare patches for both pci-aardvark.c and pci-mvebu.c to export PCI_EXP_LNKCTL2 register via emulated bridge. And so aspm.c code would be able to use Bjorn or my patch which I have sent last year.

Quoted 5 lines

Question reminds, if this is issue with QCA wifi chip on that Compex card or it is issue with PCIe controllers, now on A38x and A3720 SoC. Note that both A38x and A3720 platforms are from Marvell, but they have different PCIe controllers (so it does not mean that both must have same hw bugs).

Seems that this is really issue in QCA98xx chips. I have send patch which adds quirk for these wifi chips:

https://lore.kernel.org/linux-pci/[email protected]/

Quoted 139 lines · Pali Rohár

Quoted 137 lines · On Friday 19 March 2021 00:16:29 Pali Rohár wrote:

Quoted 116 lines · On Thursday 18 March 2021 23:43:58 Toke Høiland-Jørgensen wrote:

Quoted 114 lines · Pali Rohár

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 25 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Quoted 24 lines · Pali Rohár

My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset.
Repeated link retraining fails and it fails even when aardvark is
reconfigured to GEN1 mode. Reset via PERST# signal is required to have
working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If
success read "negotiated link speed" from "Link Control Status Register"
(for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then
retrain link again (for WLE900VX now it would be at GEN1). After that
card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link
training fails. And if I change mode to GEN1 after this failed link
training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does
one link training at GEN2. Then aspm.c is called which is doing second
link retraining at GEN2. And if it fails then below patch issue third
link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark
then second link retraining must be at GEN1 (not GEN2) to workaround
this issue.

Quoted 7 lines

Quoted 3 lines

Bjorn, Toke: what about trying to hack aspm.c code to never do link
retraining at GEN2 speed? And always force GEN1 speed prior link
training?

Sounds like a plan. I poked around in aspm.c and must confess to being a bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

Quoted 8 lines

Quoted 4 lines

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

I pasted this into the top of mvebu_pcie_setup_hw(), and that indeed seems to fix things so that all three PCIE devices work even with ASPM turned on! :)

Perfect! Now I'm sure that it is same issue as in aardvark driver.

I will prepare patches for both pci-aardvark.c and pci-mvebu.c to export PCI_EXP_LNKCTL2 register via emulated bridge. And so aspm.c code would be able to use Bjorn or my patch which I have sent last year.

Question reminds, if this is issue with QCA wifi chip on that Compex card or it is issue with PCIe controllers, now on A38x and A3720 SoC. Note that both A38x and A3720 platforms are from Marvell, but they have different PCIe controllers (so it does not mean that both must have same hw bugs).

Quoted 4 lines

Seems that this is really issue in QCA98xx chips. I have send patch which adds quirk for these wifi chips:

https://lore.kernel.org/linux-pci/[email protected]/

I tried applying that, and while it does fix the ath10k card, it seems to break the ath9k card in the slot next to it.

When booting with the
patch applied, I get this in dmesg:

[    3.556599] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver

Could there be some kind of data corruption in play here making the driver think the chip revision is wrong, or something like that? If I boot the same kernel without the patch applied, the ath9k initialisation works fine, but obviously the ath10k is then still broken...

Quoted 144 lines · On Friday 26 March 2021 16:25:27 Toke Høiland-Jørgensen wrote:

Quoted 142 lines · Pali Rohár

Quoted 132 lines · On Friday 19 March 2021 00:16:29 Pali Rohár wrote:

Quoted 116 lines · On Thursday 18 March 2021 23:43:58 Toke Høiland-Jørgensen wrote:

Quoted 114 lines · Pali Rohár

Quoted 90 lines · On Monday 15 March 2021 20:58:06 Pali Rohár wrote:

Quoted 89 lines · On Monday 02 November 2020 16:54:35 Toke Høiland-Jørgensen wrote:

Quoted 87 lines · Pali Rohár

Quoted 86 lines · On Saturday 31 October 2020 13:49:49 Toke Høiland-Jørgensen wrote:

Quoted 83 lines · ™֟☻̭҇ Ѽ ҉ ®

Quoted 72 lines · Pali Rohár

Quoted 32 lines · On Friday 30 October 2020 14:02:22 Toke Høiland-Jørgensen wrote:

Pali Rohár <[email protected]> writes:
My experience with that WLE900VX card, aardvark driver and aspm code:

Link training in GEN2 mode for this card succeed only once after reset.
Repeated link retraining fails and it fails even when aardvark is
reconfigured to GEN1 mode. Reset via PERST# signal is required to have
working link training.

What I did in aardvark driver: Set mode to GEN2, do link training. If
success read "negotiated link speed" from "Link Control Status Register"
(for WLE900VX it is 0x1 - GEN1) and set it into aardvark. And then
retrain link again (for WLE900VX now it would be at GEN1). After that
card is stable and all future retraining (e.g. from aspm.c) also passes.

If I do not change aardvark mode from GEN2 to GEN1 the second link
training fails. And if I change mode to GEN1 after this failed link
training then nothing happen, link training do not success.

So just speculation now... In current setup initialization of card does
one link training at GEN2. Then aspm.c is called which is doing second
link retraining at GEN2. And if it fails then below patch issue third
link retraining at GEN1. If A38x/pci-mvebu has same problem as aardvark
then second link retraining must be at GEN1 (not GEN2) to workaround
this issue.

Bjorn, Toke: what about trying to hack aspm.c code to never do link
retraining at GEN2 speed? And always force GEN1 speed prior link
training?
Sounds like a plan. I poked around in aspm.c and must confess to being a
bit lost in the soup of registers ;)

So if one of you can cook up a patch, that would be most helpful!

I modified Bjorn's patch, explicitly set tls to 1 and added debug info about cls (current link speed, that what is used by aardvark). It is untested, I just tried to compile it.

Can try it?

drivers/pci/pcie/aspm.c +21 -0

index 253c30cc1967..f934c0b52f41 100644
@@ -206,6 +206,27 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  	unsigned long end_jiffies;
  	u16 reg16;
 
+	u32 lnkcap2;
+	u16 lnksta, lnkctl2, cls, tls;
+
+	pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &lnkcap2);
+	pcie_capability_read_word(parent, PCI_EXP_LNKSTA, &lnksta);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	cls = lnksta & PCI_EXP_LNKSTA_CLS;
+	tls = lnkctl2 & PCI_EXP_LNKCTL2_TLS;
+
+	pci_info(parent, "lnkcap2 %#010x sls %#04x lnksta %#06x cls %#03x lnkctl2 %#06x tls %#03x\n",
+		lnkcap2, (lnkcap2 & 0x3F) >> 1,
+		lnksta, cls,
+		lnkctl2, tls);
+
+	tls = 1;
+	pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL2,
+					PCI_EXP_LNKCTL2_TLS, tls);
+	pcie_capability_read_word(parent, PCI_EXP_LNKCTL2, &lnkctl2);
+	pci_info(parent, "lnkctl2 %#010x new tls %#03x\n",
+		lnkctl2, tls);
+
  	pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &reg16);
  	reg16 |= PCI_EXP_LNKCTL_RL;
  	pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16);

drivers/pci/pcie/aspm.c static bool pcie_retrain_link(struct pcie_link_state *link) · line 255

Quoted 8 lines

drivers/pci/pcie/aspm.c +2 -0

@@ -227,6 +248,8 @@ static bool pcie_retrain_link(struct pcie_link_state *link)
  			break;
  		msleep(1);
  	} while (time_before(jiffies, end_jiffies));
+	pci_info(parent, "lnksta %#06x new cls %#03x\n",
+		lnksta, (cls & PCI_EXP_LNKSTA_CLS));
  	return !(reg16 & PCI_EXP_LNKSTA_LT);
  }
 

Still exhibiting the BAR update error, run tested with next--20201030

Quoted 6 lines

Quoted 1 line

Quoted 1 line

Quoted 1 line

Yup, same for me :(

I'm answering my own question. This code does not work on Omnia because A38x pci-mvebu.c driver is using emulator for PCIe root bridge and it does not implement PCI_EXP_LNKCTL2 and PCI_EXP_LNKCTL2 registers. So code for forcing link speed has no effect on Omnia...

Toke, on A38x PCIe controller it is possible to access PCI_EXP_LNKCTL2 register. Just access is not exported via emulated root bridge.

Documentation for this PCIe controller is public, so anybody can look at register description. See page 571, A.7 PCI Express 2.0 Port 0 Registers

http://web.archive.org/web/20200420191927/https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-embedded-processors-armada-38x-functional-specifications-2015-11.pdf

In drivers/pci/controller/pci-mvebu.c you can set a new value for this register via function call:

    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

So, could you try to set PCI_EXP_LNKCTL2_TLS bits to gen1 in some hw init function, e.g. mvebu_pcie_setup_hw()?

Quoted 8 lines

Quoted 4 lines

    u32 val = mvebu_readl(port, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);
    val &= ~PCI_EXP_LNKCTL2_TLS;
    val |= PCI_EXP_LNKCTL2_TLS_2_5GT;
    mvebu_writel(port, val, PCIE_CAP_PCIEXP + PCI_EXP_LNKCTL2);

I pasted this into the top of mvebu_pcie_setup_hw(), and that indeed seems to fix things so that all three PCIE devices work even with ASPM turned on! :)

Perfect! Now I'm sure that it is same issue as in aardvark driver.

I will prepare patches for both pci-aardvark.c and pci-mvebu.c to export PCI_EXP_LNKCTL2 register via emulated bridge. And so aspm.c code would be able to use Bjorn or my patch which I have sent last year.

Quoted 5 lines

Question reminds, if this is issue with QCA wifi chip on that Compex card or it is issue with PCIe controllers, now on A38x and A3720 SoC. Note that both A38x and A3720 platforms are from Marvell, but they have different PCIe controllers (so it does not mean that both must have same hw bugs).

Seems that this is really issue in QCA98xx chips. I have send patch which adds quirk for these wifi chips:

Quoted 4 lines

Quoted 1 line

https://lore.kernel.org/linux-pci/[email protected]/

I tried applying that, and while it does fix the ath10k card, it seems to break the ath9k card in the slot next to it.

Ehm, what?

Patch which I have sent today to mailing list calls quirk code only for PCI device id used by QCA98xx cards. For all other cards it is noop.

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

Quoted 4 lines

When booting with the
patch applied, I get this in dmesg:

[    3.556599] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver

Can you send whole dmesg log? So I can see which new err/info lines are printed.

Quoted 4 lines

Could there be some kind of data corruption in play here making the driver think the chip revision is wrong, or something like that? If I boot the same kernel without the patch applied, the ath9k initialisation works fine, but obviously the ath10k is then still broken...

There is something really strange.

Can you add debug log into pcie_change_tls_to_gen1() function to check for which card is this function called?

Are you testing this new patch with or without changes to mvebu_pcie_setup_hw() function?

Quoted 11 lines · Pali Rohár

Quoted 8 lines · On Friday 26 March 2021 16:25:27 Toke Høiland-Jørgensen wrote:

Quoted 4 lines · Pali Rohár

Seems that this is really issue in QCA98xx chips. I have send patch which adds quirk for these wifi chips:

https://lore.kernel.org/linux-pci/[email protected]/

I tried applying that, and while it does fix the ath10k card, it seems to break the ath9k card in the slot next to it.

Ehm, what?

I know, right?! :/

Quoted 3 lines

Patch which I have sent today to mailing list calls quirk code only for PCI device id used by QCA98xx cards. For all other cards it is noop.

So upon further investigation this seems to be unrelated to the patch. Meaning that I can't reliably get the ath9k device to work again by reverting it. And the patch does seem to fix the ath10k device, so I think that's probably good.

However, the issue with ath9k does seem to be related to ASPM; if I turn that off in .config, I get the ath9k device back.

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

Quoted 7 lines

Quoted 4 lines

When booting with the
patch applied, I get this in dmesg:

[    3.556599] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver

Can you send whole dmesg log? So I can see which new err/info lines are printed.

Pasting all three cases below:

Quoted 9 lines

Quoted 4 lines

Could there be some kind of data corruption in play here making the driver think the chip revision is wrong, or something like that? If I boot the same kernel without the patch applied, the ath9k initialisation works fine, but obviously the ath10k is then still broken...

There is something really strange.

Can you add debug log into pcie_change_tls_to_gen1() function to check for which card is this function called?

Erm, it looks like it's never called? I added this:

drivers/pci/pcie/aspm.c +0 -0

        /* Check if link speed can be forced to 2.5 GT/s */
        pcie_capability_read_dword(parent, PCI_EXP_LNKCAP2, &reg32);
        if (!(reg32 & PCI_EXP_LNKCAP2_SLS_2_5GB)) {

But 'dmesg | grep called' returns nothing...

Quoted 2 lines

Are you testing this new patch with or without changes to mvebu_pcie_setup_hw() function?

I applied your patch on top of latest mac80211-next, which right now is this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211-next.git/commit/?id=4b837ad53be2ab100dfaa99dc73a9443a8a2392d

Quoted 24 lines · On Friday 26 March 2021 17:54:38 Toke Høiland-Jørgensen wrote:

Quoted 11 lines · Pali Rohár

Quoted 8 lines · On Friday 26 March 2021 16:25:27 Toke Høiland-Jørgensen wrote:

Quoted 4 lines · Pali Rohár

Seems that this is really issue in QCA98xx chips. I have send patch which adds quirk for these wifi chips:

https://lore.kernel.org/linux-pci/[email protected]/

I tried applying that, and while it does fix the ath10k card, it seems to break the ath9k card in the slot next to it.

Ehm, what?

I know, right?! :/

Quoted 3 lines

Patch which I have sent today to mailing list calls quirk code only for PCI device id used by QCA98xx cards. For all other cards it is noop.

So upon further investigation this seems to be unrelated to the patch. Meaning that I can't reliably get the ath9k device to work again by reverting it. And the patch does seem to fix the ath10k device, so I think that's probably good.

Quoted 2 lines

However, the issue with ath9k does seem to be related to ASPM; if I turn that off in .config, I get the ath9k device back.

Ok, perfect. So this my patch is does not break ath9k.

Quoted 7 lines

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

Quoted 2 lines

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

I'm not sure. Maybe ASPM is somehow buggy on ath9k or needs some special handling. But issue is not at PCI config space as ath9k driver start initialization of this card. Needs also some debugging in ath9k driver if it prints that strange "mac chip rev" error.

I think this issue should be handled separately. Could you report it also to ath9k mailing list (and CC me)? Maybe other ath developers would know some more details.

Quoted 9 lines

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

That is fine. Also all ath9k testing cards have id 0x002e.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

When booting with the
patch applied, I get this in dmesg:

[    3.556599] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver

Quoted 4 lines

Quoted 2 lines

Can you send whole dmesg log? So I can see which new err/info lines are printed.

Pasting all three cases below:

...

Seem that there is no ASPM related line... But your logs are not complete, beginning is missing. So important lines are maybe trimmed.

Quoted 7 lines

Quoted 7 lines

Quoted 4 lines

Could there be some kind of data corruption in play here making the driver think the chip revision is wrong, or something like that? If I boot the same kernel without the patch applied, the ath9k initialisation works fine, but obviously the ath10k is then still broken...

There is something really strange.

Quoted 4 lines

Quoted 2 lines

Can you add debug log into pcie_change_tls_to_gen1() function to check for which card is this function called?

Erm, it looks like it's never called? I added this:

Ehm? With patch it must be called otherwise ath10k card would not be detected on PCIe bus. And you tested that patch fixes it...

Quoted 8 lines · Pali Rohár

Quoted 7 lines · On Friday 26 March 2021 17:54:38 Toke Høiland-Jørgensen wrote:

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

Quoted 7 lines

Quoted 2 lines

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

I'm not sure. Maybe ASPM is somehow buggy on ath9k or needs some special handling. But issue is not at PCI config space as ath9k driver start initialization of this card. Needs also some debugging in ath9k driver if it prints that strange "mac chip rev" error.

Well that's just being output because it gets a revision that it doesn't recognise - which it seems to be just reading from a register:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L255

The value returned is consistent with the value returned just being 0xffffffff. Which from looking at ioread32() is the value being returned on a failed read. So there's a driver bug there - the check against -EIO here is obviously nonsensical:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L290

But the underlying cause appears to be that the read from the register fails, which I suppose is related to something the PCI bus does?

Quoted 3 lines

I think this issue should be handled separately. Could you report it also to ath9k mailing list (and CC me)? Maybe other ath developers would know some more details.

I'll send a patch for the nonsensical check above, but other than that I think we're still in PCI land here, or?

Quoted 11 lines

Quoted 9 lines

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

That is fine. Also all ath9k testing cards have id 0x002e.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

When booting with the
patch applied, I get this in dmesg:

[    3.556599] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver

Quoted 36 lines · On Friday 26 March 2021 18:51:42 Toke Høiland-Jørgensen wrote:

Quoted 8 lines · Pali Rohár

Quoted 7 lines · On Friday 26 March 2021 17:54:38 Toke Høiland-Jørgensen wrote:

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

Quoted 7 lines

Quoted 2 lines

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

I'm not sure. Maybe ASPM is somehow buggy on ath9k or needs some special handling. But issue is not at PCI config space as ath9k driver start initialization of this card. Needs also some debugging in ath9k driver if it prints that strange "mac chip rev" error.

Well that's just being output because it gets a revision that it doesn't recognise - which it seems to be just reading from a register:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L255

The value returned is consistent with the value returned just being 0xffffffff. Which from looking at ioread32() is the value being returned on a failed read. So there's a driver bug there - the check against -EIO here is obviously nonsensical:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L290

But the underlying cause appears to be that the read from the register fails, which I suppose is related to something the PCI bus does?

Quoted 3 lines

I think this issue should be handled separately. Could you report it also to ath9k mailing list (and CC me)? Maybe other ath developers would know some more details.

Quoted 2 lines

I'll send a patch for the nonsensical check above, but other than that I think we're still in PCI land here, or?

First, can you try to enable my quirk also for this ath9k card with ASPM enabled?

I have there another ath9k card which after toggling link retraining changes PCI device ID (really!) to 0xABCD. But lspci ...

There is long story about broken ath9k cards that are reporting 0xABCD id on x86 machines with specific BIOS versions. It can be find in ath9k-devel mailing list archive:

https://www.mail-archive.com/[email protected]/msg07529.html

Maybe we now found root cause of this ABCD? If yes, then it also answers why above ath9k driver check fails (device id was changed) and also because kernel see correct id (kernel reads id before configuring ASPM and therefore before triggering link retraining).

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

Quoted 1 line

Quoted 1 line

That is fine. Also all ath9k testing cards have id 0x002e.

Today I found out that lspci -nn may lie! Please send output from
command: lspci -nn -x because real PCI device id can read only from -x hexdump output.

Quoted 37 lines · Pali Rohár

Quoted 36 lines · On Friday 26 March 2021 18:51:42 Toke Høiland-Jørgensen wrote:

Quoted 8 lines · Pali Rohár

Quoted 7 lines · On Friday 26 March 2021 17:54:38 Toke Høiland-Jørgensen wrote:

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

Quoted 7 lines

Quoted 2 lines

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

I'm not sure. Maybe ASPM is somehow buggy on ath9k or needs some special handling. But issue is not at PCI config space as ath9k driver start initialization of this card. Needs also some debugging in ath9k driver if it prints that strange "mac chip rev" error.

Well that's just being output because it gets a revision that it doesn't recognise - which it seems to be just reading from a register:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L255

The value returned is consistent with the value returned just being 0xffffffff. Which from looking at ioread32() is the value being returned on a failed read. So there's a driver bug there - the check against -EIO here is obviously nonsensical:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L290

But the underlying cause appears to be that the read from the register fails, which I suppose is related to something the PCI bus does?

Quoted 3 lines

I think this issue should be handled separately. Could you report it also to ath9k mailing list (and CC me)? Maybe other ath developers would know some more details.

Quoted 5 lines

Quoted 2 lines

I'll send a patch for the nonsensical check above, but other than that I think we're still in PCI land here, or?

First, can you try to enable my quirk also for this ath9k card with ASPM enabled?

Yup, with this I get both devices working:

drivers/pci/quirks.c +0 -0

index 8ff690c7679d..7e2f9c69f6b2 100644

Quoted 25 lines

I have there another ath9k card which after toggling link retraining changes PCI device ID (really!) to 0xABCD. But lspci ...

There is long story about broken ath9k cards that are reporting 0xABCD id on x86 machines with specific BIOS versions. It can be find in ath9k-devel mailing list archive:

https://www.mail-archive.com/[email protected]/msg07529.html

Maybe we now found root cause of this ABCD? If yes, then it also answers why above ath9k driver check fails (device id was changed) and also because kernel see correct id (kernel reads id before configuring ASPM and therefore before triggering link retraining).

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

Quoted 5 lines

Quoted 1 line

Quoted 1 line

That is fine. Also all ath9k testing cards have id 0x002e.

Today I found out that lspci -nn may lie! Please send output from
command: lspci -nn -x because real PCI device id can read only from -x hexdump output.

Without the quirk added to the ath9k:

01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 00: 8c 16 2e 00 02 00 10 00 01 00 80 02 10 00 00 00
10: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 a4 30
30: 00 00 00 00 40 00 00 00 00 00 00 00 3d 01 00 00

02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c] 00: 8c 16 3c 00 46 05 10 00 00 00 80 02 10 00 00 00
10: 04 00 20 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 20 ea 40 00 00 00 00 00 00 00 3e 01 00 00

And with:

01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 00: 8c 16 2e 00 46 01 10 00 01 00 80 02 10 00 00 00
10: 04 00 00 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 a4 30
30: 00 00 00 00 40 00 00 00 00 00 00 00 3d 01 00 00

02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c] 00: 8c 16 3c 00 46 05 10 00 00 00 80 02 10 00 00 00
10: 04 00 20 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 20 ea 40 00 00 00 00 00 00 00 3e 01 00 00

Is that change in bytes 5 and 6 significant?

-Toke

Quoted 57 lines · On Wednesday 31 March 2021 16:02:42 Toke Høiland-Jørgensen wrote:

Quoted 37 lines · Pali Rohár

Quoted 36 lines · On Friday 26 March 2021 18:51:42 Toke Høiland-Jørgensen wrote:

Quoted 8 lines · Pali Rohár

Quoted 7 lines · On Friday 26 March 2021 17:54:38 Toke Høiland-Jørgensen wrote:

So we have these
cases:

ASPM disabled: ath9k, ath10k and mt76 cards all work
ASPM enabled, no patch: only mt76 card works
ASPM enabled + patch: ath10k and mt76 cards work

Quoted 7 lines

Quoted 2 lines

So IDK, maybe the ath9k card needs a quirk as well? Or the mvebu board is just generally flaky?

I'm not sure. Maybe ASPM is somehow buggy on ath9k or needs some special handling. But issue is not at PCI config space as ath9k driver start initialization of this card. Needs also some debugging in ath9k driver if it prints that strange "mac chip rev" error.

Well that's just being output because it gets a revision that it doesn't recognise - which it seems to be just reading from a register:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L255

The value returned is consistent with the value returned just being 0xffffffff. Which from looking at ioread32() is the value being returned on a failed read. So there's a driver bug there - the check against -EIO here is obviously nonsensical:

https://elixir.bootlin.com/linux/latest/source/drivers/net/wireless/ath/ath9k/hw.c#L290

But the underlying cause appears to be that the read from the register fails, which I suppose is related to something the PCI bus does?

Quoted 3 lines

I think this issue should be handled separately. Could you report it also to ath9k mailing list (and CC me)? Maybe other ath developers would know some more details.

Quoted 5 lines

Quoted 2 lines

I'll send a patch for the nonsensical check above, but other than that I think we're still in PCI land here, or?

First, can you try to enable my quirk also for this ath9k card with ASPM enabled?

Yup, with this I get both devices working:

drivers/pci/quirks.c +1 -0

index 8ff690c7679d..7e2f9c69f6b2 100644
@@ -3583,6 +3583,7 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset);
  * PCIe bridge has forced link speed to 2.5 GT/s via PCI_EXP_LNKCTL2 register.
  */
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c, quirk_no_bus_reset_and_no_retrain_link);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x002e, quirk_no_bus_reset_and_no_retrain_link);
 

drivers/pci/quirks.c DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset); · line 3589

Quoted 2 lines

drivers/pci/quirks.c +0 -0

 /*
  * Root port on some Cavium CN8xxx chips do not successfully complete a bus

Ok, thank you for testing!

I'm seeing that testing unit 0x0030 (AR93xx) also needs this quirk, so I will mark all Atheros chips in above no bus reset list with no retrain link quirk.

Quoted 53 lines

Quoted 25 lines

I have there another ath9k card which after toggling link retraining changes PCI device ID (really!) to 0xABCD. But lspci ...

There is long story about broken ath9k cards that are reporting 0xABCD id on x86 machines with specific BIOS versions. It can be find in ath9k-devel mailing list archive:

https://www.mail-archive.com/[email protected]/msg07529.html

Maybe we now found root cause of this ABCD? If yes, then it also answers why above ath9k driver check fails (device id was changed) and also because kernel see correct id (kernel reads id before configuring ASPM and therefore before triggering link retraining).

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

Can you send PCI device id of your ath9k card (lspci -nn)? Because all my tested ath9k cards have different PCI device id.

[root@omnia-arch ~]# lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]

Quoted 5 lines

Quoted 1 line

Quoted 1 line

That is fine. Also all ath9k testing cards have id 0x002e.

Today I found out that lspci -nn may lie! Please send output from
command: lspci -nn -x because real PCI device id can read only from -x hexdump output.

Without the quirk added to the ath9k:

01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 00: 8c 16 2e 00 02 00 10 00 01 00 80 02 10 00 00 00
10: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 a4 30
30: 00 00 00 00 40 00 00 00 00 00 00 00 3d 01 00 00

02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c] 00: 8c 16 3c 00 46 05 10 00 00 00 80 02 10 00 00 00
10: 04 00 20 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 20 ea 40 00 00 00 00 00 00 00 3e 01 00 00

And with:

01:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01) 00: 8c 16 2e 00 46 01 10 00 01 00 80 02 10 00 00 00
10: 04 00 00 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 8c 16 a4 30
30: 00 00 00 00 40 00 00 00 00 00 00 00 3d 01 00 00

Quoted 6 lines

02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c] 00: 8c 16 3c 00 46 05 10 00 00 00 80 02 10 00 00 00
10: 04 00 20 e0 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 20 ea 40 00 00 00 00 00 00 00 3e 01 00 00

Yesterday both MJ and Bjorn told me to use lspci '-b' switch which instruct lspci to parse capabilities from config space (instead of kernel cache).

Could you try to run 'lspci -nn -vv' and 'lspci -nn -vv -b' and compare results? If something changes?

Anyway I have discussion with Adrian Chadd about 0xABCD issue and these Qualcomm/Atheros cards. When post-AR9300 card is not initialized it reports PCI device id 0xABCD. Pre-AR9300 cards should report correct PCI device id even when it is not initialized. WLE200 is AR9287-based, so it reports always correct id, should not change it during usage.

But seems that also this AR9287 has issue with EEPROM/OTP as you figured out that ath9k driver is not able to read some device id from internal register. So please prepare patch for fixing -EIO in ath9k.

PCI vendor & device id is in first 4 bytes and as you can see it is correct and was not changed.

So I guess lspci output would not change for this card.

Quoted 1 line

Is that change in bytes 5 and 6 significant?

At offset 0x04 is 16bit PCI Command Register.

In second (with) output is set bit 2 which means that Bus Mastering is enabled. This is normal and required when card communicate with system. Then is enabled bit 6 (Parity Error Response) and bit 8 (SERR# Enable), both for error reporting. This is normal when device is active.

So nothing suspicious here.

Found this patch

https://github.com/openwrt/openwrt/blob/7c0496f29bed87326f1bf591ca25ace82373cfc7/target/linux/mvebu/patches-5.4/405-PCI-aardvark-Improve-link-training.patch

that mentions the Compex WLE900VX card, which reading the lspci verbose output from the bugtracker seems to the device being troubled.

Quoted 68 lines · Bjorn Helgaas

Quoted 55 lines · Toke Høiland-Jørgensen

Quoted 53 lines · Toke Høiland-Jørgensen

Quoted 40 lines · Bjorn Helgaas

[+cc vtolkm]

Quoted 37 lines · Toke Høiland-Jørgensen

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Quoted 8 lines

Quoted 4 lines

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Can you try turning off CONFIG_PCIEASPM? We had a similar recent report at https://bugzilla.kernel.org/show_bug.cgi?id=209833 but I don't think we have a fix yet.

Yes! Turning that off does indeed help! Thanks a bunch :)

Quoted 8 lines

Quoted 2 lines

You mention that bisecting this would be helpful - I can try that tomorrow; any idea when this was last working?

OK, so I tried to bisect this, but, erm, I couldn't find a working revision to start from? I went all the way back to 4.10 (which is the first version to include the device tree file for the Omnia), and even on that, the wireless cards were failing to initialise with ASPM enabled...

I have no personal experience with this device; all I know is that the bugzilla suggests that it worked in v5.4, which isn't much help.

Quoted 3 lines

Possibly the apparent regression was really a .config change, i.e., CONFIG_PCIEASPM was disabled in the v5.4 kernel vtolkm@ tested and it "worked" but got enabled later and it started failing?

Yeah, I suspect so. The OpenWrt config disables CONFIG_PCIEASPM by default and only turns it on for specific targets. So I guess that it's most likely that this has never worked...

Hi everyone

I'm trying to get a mainline kernel to run on my Turris Omnia, and am having some trouble getting the PCI bus to work correctly. Specifically, I'm running a 5.10-rc1 kernel (torvalds/master as of this moment), with the resource request fix[0] applied on top.

The kernel boots fine, and the patch in [0] makes the PCI devices show up. But I'm still getting initialisation errors like these:

[    1.632709] pci 0000:01:00.0: BAR 0: error updating (0xe0000004 != 0xffffffff)
[    1.632714] pci 0000:01:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)
[    1.632745] pci 0000:02:00.0: BAR 0: error updating (0xe0200004 != 0xffffffff)
[    1.632750] pci 0000:02:00.0: BAR 0: error updating (high 0x000000 != 0xffffffff)

and the WiFi drivers fail to initialise with what appears to me to be errors related to the bus rather than to the drivers themselves:

[    3.509878] ath: phy0: Mac Chip Rev 0xfffc0.f is not supported by this driver
[    3.517049] ath: phy0: Unable to initialize hardware; initialization status: -95
[    3.524473] ath9k 0000:01:00.0: Failed to initialize device
[    3.530081] ath9k: probe of 0000:01:00.0 failed with error -95
[    3.536012] ath10k_pci 0000:02:00.0: of_irq_parse_pci: failed with rc=134
[    3.543049] pci 0000:00:02.0: enabling device (0140 -> 0142)
[    3.548735] ath10k_pci 0000:02:00.0: can't change power state from D3hot to D0 (config space inaccessible)
[    3.588592] ath10k_pci 0000:02:00.0: failed to wake up device : -110
[    3.595098] ath10k_pci: probe of 0000:02:00.0 failed with error -110

lspci looks OK, though:

# lspci
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04)
01:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter (rev ff)

Does anyone have any clue what could be going on here? Is this a bug, or did I miss something in my config or other initialisation? I've tried with both the stock u-boot distributed with the board, and with an upstream u-boot from latest master; doesn't seem to make any different.

Any pointers will be greatly appreciated!

Thanks,

-Toke

[0] https://lore.kernel.org/linux-pci/[email protected]/

Are you using stock U-Boot in the Omnia?

Quoted 1 line · Marek Behun

Are you using stock U-Boot in the Omnia?

I've tried both that and the latest upstream - didn't make a difference wrt the PCI issue. Only difference I've noticed other than that (apart from being able to turn more things on when using upstream) is that the upstream u-boot can't seem to find the eMMC chip on the Omnia. Any idea why? It doesn't matter right now since I'm just tftp-booting, but it would be kinda nice to get that fixed as well :)

-Toke

Quoted 11 lines · On Tue, 27 Oct 2020 20:00:58 +0100 Toke Høiland-Jørgensen <[email protected]>…

Quoted 1 line · Marek Behun

Are you using stock U-Boot in the Omnia?

I've tried both that and the latest upstream - didn't make a difference wrt the PCI issue. Only difference I've noticed other than that (apart from being able to turn more things on when using upstream) is that the upstream u-boot can't seem to find the eMMC chip on the Omnia. Any idea why? It doesn't matter right now since I'm just tftp-booting, but it would be kinda nice to get that fixed as well :)

Quoted 2 lines

-Toke

No idea, I will have to look into that.