PCIe issue with NIC card that has 64bit BARs

DISCUSSION Murali Karicheri 3 participants 14 messages May 9, 2016 – May 16, 2016 PCI

Share & tools

Murali Karicheri

author

2016-05-09 19:32 UTC #0 lore raw

Bjorn,

I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.

The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.

Here are the logs.

K2E log with Marvel controller (Good working case).
===================================================

[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]

K2G log with Tealtek NIC card
=============================

[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
[    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Bjorn Helgaas

2016-05-09 20:34 UTC #1 lore raw

Hi Murali,

Quoted 31 lines · Murali Karicheri

Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]

Quoted 6 lines

K2G log with Tealtek NIC card
=============================

[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]

There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.

Quoted 8 lines

[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]

I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.

The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.

Quoted 10 lines

[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286

Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here. Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?

Quoted 1 line

[    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]

Murali Karicheri

author

2016-05-09 21:02 UTC #2 lore raw

Hi Bjorn,

Thanks for your quick response!
See below for some follow up question.

Quoted 73 lines · Bjorn Helgaas

Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286

Quoted 2 lines

Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.

So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?

Quoted 3 lines

Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?

Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?

Thanks

Murali

Quoted 1 line

[    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Alex Williamson

2016-05-09 21:23 UTC #3 lore raw

Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:

Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]  
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]  
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286  
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?

Quoted 7 lines

Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?

I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:

02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)

	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]

Thanks,
Alex

Alex Williamson

2016-05-09 21:31 UTC #4 lore raw

Quoted 100 lines · On Mon, 9 May 2016 15:23:51 -0600 Alex Williamson <[email protected]>…

Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:

Quoted 8 lines

02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)

	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]

Intel has lots apparently, 82574L, 92579LM, I218-LM and even an 82576. IOW grab a random Intel 1G NIC and it probably has 32bit BARs.

Murali Karicheri

author

2016-05-09 21:38 UTC #5 lore raw

In reply to Alex Williamson

Quoted 100 lines · Alex Williamson

Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]  
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]  
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286  
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:

Quoted 8 lines

02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)

	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]

Alex,

Thanks for the response. I see following mini pci card which match with your description.

http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt

Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?

Murali

Quoted 4 lines

Thanks,
Alex

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Alex Williamson

2016-05-09 23:12 UTC #6 lore raw

Quoted 111 lines · On Mon, 9 May 2016 17:38:27 -0400 Murali Karicheri <[email protected]> wrote:

Quoted 100 lines · Alex Williamson
Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:
Quoted 8 lines
02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]  
Alex,

Quoted 7 lines

Thanks for the response. I see following mini pci card which match with your description.
http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?

I would venture to say yes, but I offer no guarantees ;)

Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter

Murali Karicheri

author

2016-05-10 13:56 UTC #7 lore raw

Quoted 119 lines · Alex Williamson

Quoted 116 lines · On Mon, 9 May 2016 17:38:27 -0400 Murali Karicheri <[email protected]> wrote:
Quoted 100 lines · Alex Williamson
Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:
Quoted 8 lines
02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]  
Alex,
Thanks for the response. I see following mini pci card which match with your description.
http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt

Quoted 8 lines

Quoted 2 lines
Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?
I would venture to say yes, but I offer no guarantees ;)
Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter

Thanks Alex!

Murali

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Murali Karicheri

author

2016-05-11 21:46 UTC #8 lore raw

In reply to Alex Williamson

Bjorn, Alex,

Quoted 119 lines · Alex Williamson

Quoted 116 lines · On Mon, 9 May 2016 17:38:27 -0400 Murali Karicheri <[email protected]> wrote:
Quoted 100 lines · Alex Williamson
Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:
Quoted 8 lines
02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]  
Alex,
Thanks for the response. I see following mini pci card which match with your description.
http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt

Quoted 8 lines

Quoted 2 lines
Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?
I would venture to say yes, but I offer no guarantees ;)
Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter

I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC and I am getting the following error:- This one has a 32bit bar, but similar issue as with r8169 and tg3. The PCI access over the window seems to return zeros. Per suggestion, tried dumping the memory window and got all zeros. See below for additional debug info.

root@k2g-evm:/# insmod e1000e.ko debug=16
[  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
[  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5

root@k2g-evm:/# lspci -vvv
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 287
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-500fffff
        Prefetchable memory behind bridge: 50100000-501fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 0000000021800054  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 319
Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 1000 [disabled] [size=32]
[virtual] Expansion ROM at 50100000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000021800054 Data: 0001
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2

Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with Broadcom tg3 and

root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
50000000:00000000 00000000 00000000 00000000
50000010:00000000 00000000 00000000 00000000
50000020:00000000 00000000 00000000 00000000
50000030:00000000 00000000 00000000 00000000
50000040:00000000 00000000 00000000 00000000
50000050:00000000 00000000 00000000 00000000
50000060:00000000 00000000 00000000 00000000
50000070:00000000 00000000 00000000 00000000
50000080:00000000 00000000 00000000 00000000
50000090:00000000 00000000 00000000 00000000
500000A0:00000000 00000000 00000000 00000000
500000B0:00000000 00000000 00000000 00000000
500000C0:00000000 00000000 00000000 00000000
500000D0:00000000 00000000 00000000 00000000
500000E0:00000000 00000000 00000000 00000000
500000F0:00000000 00000000 00000000 00000000

root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
50020000:00000000 00000000 00000000 00000000
50020010:00000000 00000000 00000000 00000000
50020020:00000000 00000000 00000000 00000000
50020030:00000000 00000000 00000000 00000000
50020040:00000000 00000000 00000000 00000000
50020050:00000000 00000000 00000000 00000000
50020060:00000000 00000000 00000000 00000000
50020070:00000000 00000000 00000000 00000000
50020080:00000000 00000000 00000000 00000000
50020090:00000000 00000000 00000000 00000000
500200A0:00000000 00000000 00000000 00000000
500200B0:00000000 00000000 00000000 00000000
500200C0:00000000 00000000 00000000 00000000
500200D0:00000000 00000000 00000000 00000000
500200E0:00000000 00000000 00000000 00000000
500200F0:00000000 00000000 00000000 00000000

Corresponding PCI boot log

[    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
[    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
[    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
[    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
[    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.361483] PCI: bus0: Fast back to back transfers disabled
[    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.378560] PCI: bus1: Fast back to back transfers disabled
[    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
[    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
[    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

Any idea where to start to debug this issue? Suggestion?

Thanks

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Bjorn Helgaas

2016-05-11 22:55 UTC #9 lore raw

Quoted 296 lines · Murali Karicheri

Bjorn, Alex,
Quoted 119 lines · Alex Williamson
Quoted 116 lines · On Mon, 9 May 2016 17:38:27 -0400 Murali Karicheri <[email protected]> wrote:
Quoted 100 lines · Alex Williamson
Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:
Quoted 8 lines
02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]  
Alex,
Thanks for the response. I see following mini pci card which match with your description.
http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
Quoted 8 lines
Quoted 2 lines
Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?
I would venture to say yes, but I offer no guarantees ;)
Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC and I am getting the following error:- This one has a 32bit bar, but similar issue as with r8169 and tg3. The PCI access over the window seems to return zeros. Per suggestion, tried dumping the memory window and got all zeros. See below for additional debug info.
root@k2g-evm:/# insmod e1000e.ko debug=16
[  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
[  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5
root@k2g-evm:/# lspci -vvv
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 287
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-500fffff
        Prefetchable memory behind bridge: 50100000-501fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 0000000021800054  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12
01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 319
Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 1000 [disabled] [size=32]
[virtual] Expansion ROM at 50100000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000021800054 Data: 0001
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2
Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with Broadcom tg3 and
root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
50000000:00000000 00000000 00000000 00000000
50000010:00000000 00000000 00000000 00000000
50000020:00000000 00000000 00000000 00000000
50000030:00000000 00000000 00000000 00000000
50000040:00000000 00000000 00000000 00000000
50000050:00000000 00000000 00000000 00000000
50000060:00000000 00000000 00000000 00000000
50000070:00000000 00000000 00000000 00000000
50000080:00000000 00000000 00000000 00000000
50000090:00000000 00000000 00000000 00000000
500000A0:00000000 00000000 00000000 00000000
500000B0:00000000 00000000 00000000 00000000
500000C0:00000000 00000000 00000000 00000000
500000D0:00000000 00000000 00000000 00000000
500000E0:00000000 00000000 00000000 00000000
500000F0:00000000 00000000 00000000 00000000
root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
50020000:00000000 00000000 00000000 00000000
50020010:00000000 00000000 00000000 00000000
50020020:00000000 00000000 00000000 00000000
50020030:00000000 00000000 00000000 00000000
50020040:00000000 00000000 00000000 00000000
50020050:00000000 00000000 00000000 00000000
50020060:00000000 00000000 00000000 00000000
50020070:00000000 00000000 00000000 00000000
50020080:00000000 00000000 00000000 00000000
50020090:00000000 00000000 00000000 00000000
500200A0:00000000 00000000 00000000 00000000
500200B0:00000000 00000000 00000000 00000000
500200C0:00000000 00000000 00000000 00000000
500200D0:00000000 00000000 00000000 00000000
500200E0:00000000 00000000 00000000 00000000
500200F0:00000000 00000000 00000000 00000000
Corresponding PCI boot log
[    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
[    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
[    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
[    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
[    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.361483] PCI: bus0: Fast back to back transfers disabled
[    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.378560] PCI: bus1: Fast back to back transfers disabled
[    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
[    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
[    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

Quoted 1 line

Any idea where to start to debug this issue? Suggestion?

If I understand correctly, K2G is the platform where these NICs don't work. Do *any* PCI devices work correctly on that platform? Maybe there's some host controller configuration problem related to the MMIO aperture?

If you have an analyzer, I guess you could look for your Read Requests and Completions, but I doubt you'd learn anything. I assume the NICs are sending data over the link correctly, and something is happening to the data between the device and the CPU.

I would suspect something in the host controller. Most controllers would give you 0xffffffff back instead of zeros if there's a problem, but maybe yours is special.

Bjorn

Murali Karicheri

author

2016-05-12 14:00 UTC #10 lore raw

Bjorn,

Quoted 35 lines · Bjorn Helgaas

Quoted 34 lines · Murali Karicheri

Bjorn, Alex,

500200D0:00000000 00000000 00000000 00000000
500200E0:00000000 00000000 00000000 00000000
500200F0:00000000 00000000 00000000 00000000

Corresponding PCI boot log

[    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
[    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
[    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
[    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
[    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.361483] PCI: bus0: Fast back to back transfers disabled
[    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.378560] PCI: bus1: Fast back to back transfers disabled
[    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
[    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
[    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

Quoted 4 lines

Quoted 1 line
Any idea where to start to debug this issue? Suggestion?
If I understand correctly, K2G is the platform where these NICs don't work.

Yes.

Quoted 1 line

Do *any* PCI devices work correctly on that platform?

No. I am trying to test a SATA controller and see if that works. Marvel SATA controller works fine using Keystone PCIe controller, but this is on K2E EVM and this shares the same PCIe controller hardware (designware).

Quoted 4 lines

Maybe
there's some host controller configuration problem related to the MMIO aperture?

Quoted 8 lines

If you have an analyzer, I guess you could look for your Read Requests and Completions, but I doubt you'd learn anything. I assume the NICs are sending data over the link correctly, and something is happening to the data between the device and the CPU.
I would suspect something in the host controller. Most controllers would give you 0xffffffff back instead of zeros if there's a problem, but maybe yours is special.

I will work with our hardware folks to see if they can suggest something to debug.

Thanks for you response.

Murali

Quoted 3 lines

Bjorn

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Murali Karicheri

author

2016-05-16 16:30 UTC #11 lore raw

In reply to Bjorn Helgaas

Quoted 304 lines · Bjorn Helgaas

Quoted 296 lines · Murali Karicheri
Bjorn, Alex,
Quoted 119 lines · Alex Williamson
Quoted 116 lines · On Mon, 9 May 2016 17:38:27 -0400 Murali Karicheri <[email protected]> wrote:
Quoted 100 lines · Alex Williamson
Quoted 86 lines · On Mon, 9 May 2016 17:02:23 -0400 Murali Karicheri <[email protected]> wrote:
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right? So is there a chance it is making access to upper 4 bytes and hence reading all zeros?
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server Adapter) that only has 32bit BARs:
Quoted 8 lines
02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]  
Alex,
Thanks for the response. I see following mini pci card which match with your description.
http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
Quoted 8 lines
Quoted 2 lines
Is this same thing? I assume e1000e/82571.c is the driver for this. Could you please confirm so that I can procure the same?
I would venture to say yes, but I offer no guarantees ;)
Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC and I am getting the following error:- This one has a 32bit bar, but similar issue as with r8169 and tg3. The PCI access over the window seems to return zeros. Per suggestion, tried dumping the memory window and got all zeros. See below for additional debug info.
root@k2g-evm:/# insmod e1000e.ko debug=16
[  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
[  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5
root@k2g-evm:/# lspci -vvv
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 287
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-500fffff
        Prefetchable memory behind bridge: 50100000-501fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 0000000021800054  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12
01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Server Adapter
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 319
Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 1000 [disabled] [size=32]
[virtual] Expansion ROM at 50100000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000021800054 Data: 0001
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2
Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with Broadcom tg3 and
root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
50000000:00000000 00000000 00000000 00000000
50000010:00000000 00000000 00000000 00000000
50000020:00000000 00000000 00000000 00000000
50000030:00000000 00000000 00000000 00000000
50000040:00000000 00000000 00000000 00000000
50000050:00000000 00000000 00000000 00000000
50000060:00000000 00000000 00000000 00000000
50000070:00000000 00000000 00000000 00000000
50000080:00000000 00000000 00000000 00000000
50000090:00000000 00000000 00000000 00000000
500000A0:00000000 00000000 00000000 00000000
500000B0:00000000 00000000 00000000 00000000
500000C0:00000000 00000000 00000000 00000000
500000D0:00000000 00000000 00000000 00000000
500000E0:00000000 00000000 00000000 00000000
500000F0:00000000 00000000 00000000 00000000
root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
50020000:00000000 00000000 00000000 00000000
50020010:00000000 00000000 00000000 00000000
50020020:00000000 00000000 00000000 00000000
50020030:00000000 00000000 00000000 00000000
50020040:00000000 00000000 00000000 00000000
50020050:00000000 00000000 00000000 00000000
50020060:00000000 00000000 00000000 00000000
50020070:00000000 00000000 00000000 00000000
50020080:00000000 00000000 00000000 00000000
50020090:00000000 00000000 00000000 00000000
500200A0:00000000 00000000 00000000 00000000
500200B0:00000000 00000000 00000000 00000000
500200C0:00000000 00000000 00000000 00000000
500200D0:00000000 00000000 00000000 00000000
500200E0:00000000 00000000 00000000 00000000
500200F0:00000000 00000000 00000000 00000000
Corresponding PCI boot log
[    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
[    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
[    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
[    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
[    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.361483] PCI: bus0: Fast back to back transfers disabled
[    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.378560] PCI: bus1: Fast back to back transfers disabled
[    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
[    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
[    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
Quoted 1 line
Any idea where to start to debug this issue? Suggestion?
If I understand correctly, K2G is the platform where these NICs don't work. Do *any* PCI devices work correctly on that platform? Maybe there's some host controller configuration problem related to the MMIO aperture?

Quoted 9 lines

If you have an analyzer, I guess you could look for your Read Requests and Completions, but I doubt you'd learn anything. I assume the NICs are sending data over the link correctly, and something is happening to the data between the device and the CPU.
I would suspect something in the host controller. Most controllers would give you 0xffffffff back instead of zeros if there's a problem, but maybe yours is special.

Bjorn,

I am able to bring up PCIe last week. There was a PCIe memory address incorrect in the UG that was pointing to a reserved area. So was reading all zeros. With that fixed, PCIe devices such as e1000e and tg3 worked fine and I could pass network traffic over the PCIe interface without issues. Also a USB device worked fine. But I have some issues with the Marvell 88SE9120 SATA 6Gb/s SATA (link never comes up) and also one of the NIC card RTL8168c/8111c (no data over the network interface)

root@k2g-evm:/# insmod r8169.ko
[   43.980877] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   43.986609] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[   43.993132] r8169 0000:01:00.0: limiting MRRS to 256
[   44.004152] r8169 0000:01:00.0 eth1: RTL8168c/8111c at 0xf0f56000,
00:e0:4c:21:16:2c, XID 1c4000c0 IRQ 288
[   44.024167] r8169 0000:01:00.0 eth1: jumbo features [frames: 6128 bytes, tx
checksumming: ko]
root@k2g-evm:/# ifup eth1
[   51.782650] r8169 0000:01:00.0 eth1: link down
[   51.787121] r8169 0000:01:00.0 eth1: link down
[   51.792348] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
udhcpc (v1.23.1) started
Sending discover...
[   54.158011] r8169 0000:01:00.0 eth1: link up
[   54.162344] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sending discover...
Sending discover...
No lease, forking to background
root@k2g-evm:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 50:72:24:D6:DE:ED
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1 Link encap:Ethernet HWaddr 00:E0:4C:21:16:2C
inet6 addr: fe80::2e0:4cff:fe21:162c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

root@k2g-evm:/# insmod ahci.ko
[   79.283747] ahci 0000:01:00.0: limiting MRRS to 256
[   79.288878] ahci 0000:01:00.0: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3
impl SATA mode
[   79.297868] ahci 0000:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio
slum part sxs
[   79.327629] scsi host0: ahci
[   79.344756] scsi host1: ahci
[   79.350808] ata1: SATA max UDMA/133 abar m2048@0x70000000 port 0x70000100 irq
288
[   79.358973] ata2: SATA max UDMA/133 abar m2048@0x70000000 port 0x70000180 irq
288
[   79.711079] ata2: SATA link down (SStatus 0 SControl 330)
[   79.716555] ata1: SATA link down (SStatus 0 SControl 330)

Will need to debug the above, but for now we have few NIC devices that works and USB. I would like to try the Marvel 918x as well which is known to work on K2E EVM.

Thanks once again for the right pointers that helped me debug this issue.

Murali

Quoted 2 lines

Bjorn

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone

Bjorn Helgaas

2016-05-09 21:33 UTC #12 lore raw

In reply to Murali Karicheri

Quoted 79 lines · Murali Karicheri

Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286

Quoted 5 lines

Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right?

I'm not sure I understand the question, so correct me if I'm going astray.

r8169.c assumes device registers are mapped by BAR 1 or 2 (see rtl_cfg_infos []). Your device has no BAR 1, so the registers must be mapped by BAR 2.

"BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]" means the *BAR* is 64 bits wide. This has nothing to do with the access size for reads & writes to the area mapped by the BAR. The BAR currently contains 0x50000000, which is only 32 bits, but it *could* contain any 64-bit address.

This BAR is 4KB in size, so it maps 4KB of register space. The BAR places it at bus address 0x50000000 (which in this case is also at CPU physical address 0x50000000, since there's no host bridge address translation).

The driver ioremaps that space, and it apparently got 0xf0d6a000, so it can access the registers by reading and writing at virtual addresses 0xf0d6a000-0xf0d6afff.

You should be able to manually read the register space with a program like http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c, which uses /dev/mem. You would use the CPU physical addresses, e.g., 0x50000000.

Quoted 2 lines

So is there a chance
it is making access to upper 4 bytes and hence reading all zeros?

If you read the entire 64-bit BAR itself, you should see zeros in the upper 32 bits, e.g.,

  $ setpci -sBB:DD.F BASE_ADDRESS_2 BASE_ADDRESS_3
  50000000
  00000000

since the address fits in the low 32 bits. But accesses to the registers themselves would use 32-bit addresses starting at 0x50000000.

Quoted 7 lines

Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?

Nope, sorry, I don't pay much attention to specific devices.

Bjorn

Murali Karicheri

author

2016-05-09 21:49 UTC #13 lore raw

Bjorn,

Quoted 86 lines · Bjorn Helgaas

Quoted 79 lines · Murali Karicheri
Hi Bjorn,
Thanks for your quick response!
See below for some follow up question.
Quoted 73 lines · Bjorn Helgaas
Hi Murali,
Quoted 31 lines · Murali Karicheri
Bjorn,
I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe. It works for 32bit BARs such as the Marvel SATA controller on K2E. On another recent SoC (K2G) that re-uses the same driver and hardware, I have issues bringing up PCIe.
The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell controller and that on K2G with rtk8168. Is there way we can get it assigned 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
Here are the logs.
K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
Quoted 6 lines
K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
There's no "(bus address ...)" annotation here, which means these windows map CPU addresses to identical bus addresses. The "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
Quoted 8 lines
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
I assume these BARs (2 and 4) are the ones in question. They are 64-bit BARs, but the addresses currently assigned to them fit in 32 bits, so the space is already all below 4GB.
The BAR type (32-bit or 64-bit) is built into the device; Linux doesn't have any influence on that. The type is encoded in the low-order four bits of the BAR, which are read-only.
Quoted 10 lines
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
Quoted 5 lines
Quoted 2 lines
Looks like this is printed by rtl_init_one(). I see that we read zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped to the window to have 64bit access, they are to be accessed with proper address to get the lower 32 bits of the data, right?

Quoted 7 lines

I'm not sure I understand the question, so correct me if I'm going astray.
r8169.c assumes device registers are mapped by BAR 1 or 2 (see rtl_cfg_infos []). Your device has no BAR 1, so the registers must be mapped by BAR 2.

Ok. I will review and get back. Thanks for your response!

Regards,

Murali

Quoted 45 lines

"BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]" means the *BAR* is 64 bits wide. This has nothing to do with the access size for reads & writes to the area mapped by the BAR. The BAR currently contains 0x50000000, which is only 32 bits, but it *could* contain any 64-bit address.
This BAR is 4KB in size, so it maps 4KB of register space. The BAR places it at bus address 0x50000000 (which in this case is also at CPU physical address 0x50000000, since there's no host bridge address translation).
The driver ioremaps that space, and it apparently got 0xf0d6a000, so it can access the registers by reading and writing at virtual addresses 0xf0d6a000-0xf0d6afff.
You should be able to manually read the register space with a program like http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c, which uses /dev/mem. You would use the CPU physical addresses, e.g., 0x50000000.
Quoted 2 lines
So is there a chance
it is making access to upper 4 bytes and hence reading all zeros?
If you read the entire 64-bit BAR itself, you should see zeros in the upper 32 bits, e.g.,
  $ setpci -sBB:DD.F BASE_ADDRESS_2 BASE_ADDRESS_3
  50000000
  00000000
since the address fits in the low 32 bits. But accesses to the registers themselves would use 32-bit addresses starting at 0x50000000.
Quoted 7 lines
Quoted 3 lines
Could it be that some EEPROM on the card hasn't been programmed yet? Does the card work if you put it in a different machine?
Probably. I will ask this to the owner of r8169 driver. He seems to think the access is not right. I need to find a PC that has PCIe slot to do this test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and driver in Linux?
Nope, sorry, I don't pay much attention to specific devices.
Bjorn

Signature · 2 lines

Murali Karicheri
Linux Kernel, Keystone