high latency on 82573L
Hi.
Since commit 6f461f6c7c (e1000e: enable/disable ASPM L0s and L1 and ERT according to hardware errata) I'm seeing high latencies on my Thinkpad T60p.
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
Subsystem: Lenovo ThinkPad T60
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at ee000000 (32-bit, non-prefetchable) [size=128K] I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e
# uname -r 2.6.36-rc3-0-default # ping -c 20 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms 64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms 64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms 64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms 64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms 64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms 64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms 64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms 64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms 64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms 64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms 64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms 64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms
--- 192.168.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19063ms rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2
This is 2.6.36-rc3 yet commit 19833b5dff (e1000e: disable ASPM L1 on 82573) isn't having any effect for me.
For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just 6f461f6c7c solves the issue. For .36-rc3 reverting 6f461f6c7c plus of course 19833b5dff does the trick.
I'm happy to perform any testing to help narrow this down. LMK.
Tony
Quoted 50 lines · Tony Jones
Hi.
Since commit
6f461f6c7c(e1000e: enable/disable ASPM L0s and L1 and ERT according to hardware errata) I'm seeing high latencies on my Thinkpad T60p.02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
Ethernet Controller Subsystem: Lenovo ThinkPad T60
Flags: bus master, fast devsel, latency 0, IRQ 30 Memory atee000000(32-bit, non-prefetchable) [size=128K] I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e# uname -r 2.6.36-rc3-0-default # ping -c 20 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms 64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms 64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms 64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms 64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms 64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms 64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms 64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms 64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms 64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms 64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms 64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms 64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms 64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms--- 192.168.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19063ms rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2This is 2.6.36-rc3 yet commit
19833b5dff(e1000e: disable ASPM L1 on 82573) isn't having any effect for me.
Quoted 8 lines
For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just
6f461f6c7c
solves the issue. For .36-rc3 reverting6f461f6c7cplus of course19833b5dffdoes the trick.I'm happy to perform any testing to help narrow this down. LMK.
Tony
Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded, your kernel .config and the full version number of the OpenSUSE kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?
I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit 19833b5dff is not present, but I do not understand the latency with 2.6.36-rc3.
Thanks,
Bruce.
Quoted 2 lines · Bruce W
Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
# lspci -s 2:0.0 -vvv -xxx 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller Subsystem: Lenovo ThinkPad T60 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 46 Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0100c Data: 41c9 Capabilities: [e0] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00 00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00 e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00 f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
Quoted 2 lines
kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?
It would appear it is being disabled:
[ 0.194271] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.299123] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0: Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM L0s
but I see the same high ping latencies.
Quoted 3 lines
I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit
19833b5dffis not present, but I do not understand the latency with 2.6.36-rc3.
The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing it was 6f461f6c7c.
Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE kernels.
http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3 generated using localmodconfig, defaults chosen for all prompts.
http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
Tony
Quoted 87 lines · Tony Jones
Quoted 2 lines · Bruce W
Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
# lspci -s 2:0.0 -vvv -xxx 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller Subsystem: Lenovo ThinkPad T60 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 46 Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0100c Data: 41c9 Capabilities: [e0] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00 00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00 e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00 f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00Quoted 2 lines
kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?
It would appear it is being disabled:
[ 0.194271] ACPI FADT declares the system doesn't support PCIeASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
You can enable it with 'pcie_aspm=force' [ 0.299123] pci
0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
L0sbut I see the same high ping latencies.
Quoted 3 lines
I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit
19833b5dffis not present, but I do not understand the latency with 2.6.36-rc3.The first thing I tried was OpenSUSE 2.6.34 plus
19833b5dff. This
led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing
it was6f461f6c7c.Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE
kernels.
Quoted 7 lines
http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3
generated using localmodconfig, defaults chosen for all prompts.http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
Tony
ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is calling the kernel function pci_disable_link_state() to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).
I'm unclear on whether the aspm_disabled variable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?
Alternatively in the meantime, if you disable CONFIG_PCIEASPM the e100e driver will act how it did before 6f461f6c7c, i.e. it will directly write the PCI config space to disable ASPM L1.
Thanks,
Bruce.
Quoted 106 lines · Bruce W
Quoted 87 lines · Tony Jones
Quoted 2 lines · Bruce W
Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,
# lspci -s 2:0.0 -vvv -xxx 02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller Subsystem: Lenovo ThinkPad T60 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 46 Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0100c Data: 41c9 Capabilities: [e0] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00 00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00 e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00 f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00Quoted 2 lines
kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?
It would appear it is being disabled:
[ 0.194271] ACPI FADT declares the system doesn't support PCIeASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
You can enable it with 'pcie_aspm=force' [ 0.299123] pci
0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
L0sbut I see the same high ping latencies.
Quoted 3 lines
I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit
19833b5dffis not present, but I do not understand the latency with 2.6.36-rc3.The first thing I tried was OpenSUSE 2.6.34 plus
19833b5dff. This
led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing
it was6f461f6c7c.Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE
kernels.Quoted 7 lines
http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3
generated using localmodconfig, defaults chosen for all prompts.http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg
Tony
ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since
CONFIG_PCIEASPMis enabled in your kernel config, the driver is calling the kernel functionpci_disable_link_state()to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).
Quoted 8 lines
I'm unclear on whether the
aspm_disabledvariable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before6f461f6c7c, or b) fixpci_disable_link_state()et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?
Yeah, I'd prefer this code to be in the core. Are there any patches available yet?
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 24 lines · Jesse Barnes
Quoted 11 lines · Bruce W
ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since
CONFIG_PCIEASPMis enabled in your kernel config, the driver is calling the kernel functionpci_disable_link_state()to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).Quoted 9 lines
I'm unclear on whether the
aspm_disabledvariable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before6f461f6c7c, or b) fixpci_disable_link_state()et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?
Quoted 2 lines
Yeah, I'd prefer this code to be in the core. Are there any patches available yet?
Nothing from me at this point (been on vacation). I might be able to look into this further but not for another week or so (need to catch up on the backlog that piled up while away).
Bruce.