Skip to contentSkip to conversation

high latency on 82573L

DISCUSSION Tony Jones 3 participants 6 messages Sep 2, 2010 – Sep 20, 2010

lore
Share & tools
lore

Tony Jones

author
#0 lore raw

Hi.

Since commit 6f461f6c7c (e1000e: enable/disable ASPM L0s and L1 and ERT according to hardware errata) I'm seeing high latencies on my Thinkpad T60p.

02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
Subsystem: Lenovo ThinkPad T60
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at ee000000 (32-bit, non-prefetchable) [size=128K] I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e

# uname -r
2.6.36-rc3-0-default
# ping -c 20 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms
64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms
64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms

--- 192.168.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19063ms rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2

This is 2.6.36-rc3 yet commit 19833b5dff (e1000e: disable ASPM L1 on 82573) isn't having any effect for me.

For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just 6f461f6c7c solves the issue. For .36-rc3 reverting 6f461f6c7c plus of course 19833b5dff does the trick.

I'm happy to perform any testing to help narrow this down. LMK.

Tony

#1 lore raw
Quoted 50 lines · Tony Jones

Hi.

Since commit 6f461f6c7c (e1000e: enable/disable ASPM L0s and L1 and ERT according to hardware errata) I'm seeing high latencies on my Thinkpad T60p.

02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
Ethernet Controller Subsystem: Lenovo ThinkPad T60
Flags: bus master, fast devsel, latency 0, IRQ 30 Memory at ee000000 (32-bit, non-prefetchable) [size=128K] I/O ports at 3000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use: e1000e

# uname -r
2.6.36-rc3-0-default
# ping -c 20 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.06 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=698 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=198 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=697 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=690 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=682 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.03 ms
64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1007 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.874 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=1009 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=1.72 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=1006 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=650 ms
64 bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1008 ms
64 bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=642 ms
64 bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1643 ms

--- 192.168.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19063ms rtt min/avg/max/mdev = 0.874/698.661/1643.392/439.376 ms, pipe 2

This is 2.6.36-rc3 yet commit 19833b5dff (e1000e: disable ASPM L1 on 82573) isn't having any effect for me.

Quoted 8 lines

For our OpenSUSE 11.3 kernels (2.6.34 based), reverting just 6f461f6c7c
solves the issue. For .36-rc3 reverting 6f461f6c7c plus of course 19833b5dff does the trick.

I'm happy to perform any testing to help narrow this down. LMK.

Tony

Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded, your kernel .config and the full version number of the OpenSUSE kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?

I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit 19833b5dff is not present, but I do not understand the latency with 2.6.36-rc3.

Thanks,
Bruce.

author
#2 lore raw
Quoted 2 lines · Bruce W

Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,

# lspci -s 2:0.0 -vvv -xxx
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
	Subsystem: Lenovo ThinkPad T60
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 46
	Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3000 [size=32]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee0100c  Data: 41c9
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4
	Kernel driver in use: e1000e
00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00
10: 00 00 00 ee 00 00 00 00 01 30 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 aa 17 01 20
30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
Quoted 2 lines

kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?

It would appear it is being disabled:

[    0.194271] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.297112] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.299123] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[   18.135907] e1000e 0000:02:00.0: Disabling ASPM  L1
[   18.137262] e1000e 0000:02:00.0: Disabling ASPM L0s 

but I see the same high ping latencies.

Quoted 3 lines

I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit 19833b5dff is not present, but I do not understand the latency with 2.6.36-rc3.

The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing it was 6f461f6c7c.

Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE kernels.

http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3 generated using localmodconfig, defaults chosen for all prompts.

http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg

Tony

#3 lore raw
Quoted 87 lines · Tony Jones
Quoted 2 lines · Bruce W

Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,

# lspci -s 2:0.0 -vvv -xxx
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
	Ethernet Controller Subsystem: Lenovo ThinkPad T60
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
	Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B-
	ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ
	46 Region 0: Memory at ee000000 (32-bit, non-prefetchable)
	[size=128K] Region 2: I/O ports at 3000 [size=32]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
		PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable-
	DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1
		Maskable- 64bit+ Address: 00000000fee0100c  Data: 41c9
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
			<64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0
			<128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
	BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
		MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk:	DLP- SDES- TLP- FCP-
		CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq-
		ACSViol- UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
		RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta:	RxErr+ BadTLP+
		BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk:	RxErr- BadTLP-
	BadDLLP- Rollover- Timeout- NonFatalErr- AERCap:	First Error
	Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1]
Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use:
e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00
00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00
00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01
00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
Quoted 2 lines

kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?

It would appear it is being disabled:

[    0.194271] ACPI FADT declares the system doesn't support PCIe

ASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
You can enable it with 'pcie_aspm=force' [ 0.299123] pci
0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
L0s

but I see the same high ping latencies.

Quoted 3 lines

I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit 19833b5dff is not present, but I do not understand the latency with 2.6.36-rc3.

The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This
led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing
it was 6f461f6c7c.

Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE
kernels.

Quoted 7 lines

http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3
generated using localmodconfig, defaults chosen for all prompts.

http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg

Tony

ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is calling the kernel function pci_disable_link_state() to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).

I'm unclear on whether the aspm_disabled variable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?

Alternatively in the meantime, if you disable CONFIG_PCIEASPM the e100e driver will act how it did before 6f461f6c7c, i.e. it will directly write the PCI config space to disable ASPM L1.

Thanks,
Bruce.

#4 lore raw
Quoted 106 lines · Bruce W
Quoted 87 lines · Tony Jones
Quoted 2 lines · Bruce W

Please provide more verbose lspci output and include the PCI config space, i.e. 'lspci -s 2:0.0 -vvv -xxx' after the driver is loaded,

# lspci -s 2:0.0 -vvv -xxx
02:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
	Ethernet Controller Subsystem: Lenovo ThinkPad T60
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
	Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B-
	ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ
	46 Region 0: Memory at ee000000 (32-bit, non-prefetchable)
	[size=128K] Region 2: I/O ports at 3000 [size=32]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
		PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable-
	DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1
		Maskable- 64bit+ Address: 00000000fee0100c  Data: 41c9
	Capabilities: [e0] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1
			<64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0
			<128ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
	BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
		MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk:	DLP- SDES- TLP- FCP-
		CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq-
		ACSViol- UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
		RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta:	RxErr+ BadTLP+
		BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk:	RxErr- BadTLP-
	BadDLLP- Rollover- Timeout- NonFatalErr- AERCap:	First Error
	Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1]
Device Serial Number 00-1a-6b-ff-ff-6c-7e-a4 Kernel driver in use:
e1000e 00: 86 80 9a 10 07 05 10 00 00 00 00 02 10 00 00 00 10: 00 00
00 ee 00 00 00 00 01 30 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00
00 00 00 00 aa 17 01 20 30: 00 00 00 00 c8 00 00 00 00 00 00 00 0b 01
00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 01 d0 22 c8 00 20 00 0f
d0: 05 e0 81 00 0c 10 e0 fe 00 00 00 00 c9 41 00 00
e0: 10 00 01 00 c1 0c 00 00 1f 28 1a 00 11 1c 07 00
f0: 42 01 11 10 00 00 00 00 00 00 00 00 00 00 00 00
Quoted 2 lines

kernel. Are there any messages in the system log regarding disabling ASPM L0s and/or L1 on that device?

It would appear it is being disabled:

[    0.194271] ACPI FADT declares the system doesn't support PCIe

ASPM, so disable it [ 0.297112] pci 0000:01:00.0: disabling ASPM
on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.298003] pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.
You can enable it with 'pcie_aspm=force' [ 0.299123] pci
0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable
it with 'pcie_aspm=force' [ 18.135907] e1000e 0000:02:00.0:
Disabling ASPM L1 [ 18.137262] e1000e 0000:02:00.0: Disabling ASPM
L0s

but I see the same high ping latencies.

Quoted 3 lines

I can understand the latency with the OpenSUSE 2.6.34-based kernels assuming commit 19833b5dff is not present, but I do not understand the latency with 2.6.36-rc3.

The first thing I tried was OpenSUSE 2.6.34 plus 19833b5dff. This
led me to
think it wasn't related to ASPM so I resorted to a bisect which ended up showing
it was 6f461f6c7c.

Anyways, all of the above is from vanilla 2.6.36-rc3 so lets ignore OpenSUSE
kernels.

Quoted 7 lines

http://ftp.suse.com/pub/people/tonyj/82573L/config is the config for .36-rc3
generated using localmodconfig, defaults chosen for all prompts.

http://ftp.suse.com/pub/people/tonyj/82573L/dmesg is the full dmesg

Tony

ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is calling the kernel function pci_disable_link_state() to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).

Quoted 8 lines

I'm unclear on whether the aspm_disabled variable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?

Yeah, I'd prefer this code to be in the core. Are there any patches available yet?

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#5 lore raw
Quoted 24 lines · Jesse Barnes
Quoted 11 lines · Bruce W

ASPM L1 must be disabled on this device otherwise the latency described above will happen. And even though there are log messages indicating ASPM L1 is disabled, it really isn't according to the verbose lspci output and PCI config space for the 2:0.0 device (see LnkCtl above). Since CONFIG_PCIEASPM is enabled in your kernel config, the driver is calling the kernel function pci_disable_link_state() to disable ASPM L1 which it fails to do because the variable aspm_disabled=1 (as indicated by the "ACPI FADT declares the system doesn't support PCIe ASPM, so disable it" message).

Quoted 9 lines

I'm unclear on whether the aspm_disabled variable is meant to indicate ASPM L0s or both ASPM L0s _and_ L1 are disabled (added PCI maintainer and linux-pci mail-list). To resolve this issue, we need to either a) change e1000e to directly write the PCI config space to disable ASPM L1 as was done before 6f461f6c7c, or b) fix pci_disable_link_state() et. al. to allow for ASPM L1 to be disabled properly. I would prefer the latter option so that other drivers do not have to use the same kludge to write to the PCI config space. Any input from the PCI guys?

Quoted 2 lines

Yeah, I'd prefer this code to be in the core. Are there any patches available yet?

Nothing from me at this point (been on vacation). I might be able to look into this further but not for another week or so (need to catch up on the backlog that piled up while away).

Bruce.