e1000e failure triggered by "PCI: Remove MRRS modification from MPS setting code"

Commit ed2888e906b567 (merged 3.1-rc6) causes a reproducible failure with my e1000e card here.  Starting a kvm guest (which mostly consists of shoving display updates over X over ssh, as far as the network is concerned) locks up the card hard.

The failure is easily reproducible, will gladly test patches.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Avi Kivity

Commit ed2888e906b567 (merged 3.1-rc6) causes a reproducible failure with my e1000e card here. Starting a kvm guest (which mostly consists of shoving display updates over X over ssh, as far as the network is concerned) locks up the card hard.

The failure is easily reproducible, will gladly test patches.

Does the problem occur with the adapter assigned to the kvm guest or the host?

I assume that you are running with the default behavior and not with the boot arg "pci=pcie_bus_perf".

Can you get the dmesg
output (specifically the lines that have "Dev MPS")

and lspci -vvv on
the host?

Is this an e1000e 4port adapter?

Quoted 10 lines · Jon Mason

Quoted 8 lines · Avi Kivity

 Commit ed2888e906b567 (merged 3.1-rc6) causes a reproducible failure with my
 e1000e card here.  Starting a kvm guest (which mostly consists of shoving
 display updates over X over ssh, as far as the network is concerned) locks
 up the card hard.

Quoted 4 lines

Quoted 1 line

 The failure is easily reproducible, will gladly test patches.

Does the problem occur with the adapter assigned to the kvm guest or the host?

The host.  It's not even using vhost-net or tap or bridge.  In fact the failure happens before the guest has set up neworking itself (in grub, while clearing the screen); all of the networking load is due to tunnelling X over ssh.

Quoted 2 lines

I assume that you are running with the default behavior and not with the boot arg "pci=pcie_bus_perf".

Correct.

Quoted 2 lines

Can you get the dmesg
output (specifically the lines that have "Dev MPS")

(this is with the patch reverted)

Sep 27 20:42:48 violet-regb kernel: pci 0000:02:02.0: PCI bridge to [bus
04-04]
Sep 27 20:42:48 violet-regb kernel: pci 0000:01:00.3: PCI bridge to [bus
05-05]
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:03.0: PCI bridge to [bus
06-06]
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:1c.0: PCI bridge to [bus
07-07]
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:1e.0: PCI bridge to [bus
08-08] (subtractive decode)
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:02.0: Dev MPS 128 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:02.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:01:00.0: Dev MPS 128 MPSS 256
MRRS 4096
Sep 27 20:42:48 violet-regb kernel: pci 0000:01:00.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:02:00.0: Dev MPS 128 MPSS 256
MRRS 2048
Sep 27 20:42:48 violet-regb kernel: pci 0000:02:00.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:02:02.0: Dev MPS 128 MPSS 256
MRRS 2048
Sep 27 20:42:48 violet-regb kernel: pci 0000:02:02.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:04:00.0: Dev MPS 128 MPSS 256
MRRS 512
Sep 27 20:42:48 violet-regb kernel: pci 0000:04:00.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:04:00.1: Dev MPS 128 MPSS 256
MRRS 512
Sep 27 20:42:48 violet-regb kernel: pci 0000:04:00.1: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:01:00.3: Dev MPS 256 MPSS 256
MRRS 256
Sep 27 20:42:48 violet-regb kernel: pci 0000:01:00.3: Dev MPS 256 MPSS 256
MRRS 256
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:03.0: Dev MPS 128 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:03.0: Dev MPS 256 MPSS 256
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:1c.0: Dev MPS 128 MPSS 128
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci 0000:00:1c.0: Dev MPS 128 MPSS 128
MRRS 128
Sep 27 20:42:48 violet-regb kernel: pci0000:00: Requesting ACPI _OSC control
(0x1d)
Sep 27 20:42:48 violet-regb kernel: pci0000:00: ACPI _OSC request failed
(AE_NOT_FOUND), returned control mask: 0x1d
Sep 27 20:42:48 violet-regb kernel: ACPI _OSC control for PCIe not granted,
disabling ASPM
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3
4 5 6 *7 10 11 14 15)
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3
4 *5 6 7 10 11 14 15)
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3
4 5 6 7 *10 11 14 15)
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3
4 5 6 7 10 *11 14 15)
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3
4 5 6 7 10 11 14 15) *0, disabled.
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4
5 6 7 10 11 14 15) *0, disabled.
Sep 27 20:42:48 violet-regb kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3

4 5 6 7 10 11 14 15) *0, disabled.

Quoted 2 lines

 and lspci -vvv on
the host?

# lspci -vvv
00:00.0 Host bridge: Intel Corporation 5000V Chipset Memory Controller Hub
(rev 92)
   Subsystem: Tyan Computer Device 5372
   Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Interrupt: pin A routed to IRQ 0
   Capabilities: [50] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [58] MSI: Enable- Count=1/2 Maskable- 64bit-
       Address: fee00000  Data: 0000
   Capabilities: [6c] Express (v1) Root Port (Slot-), MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited,
L1 unlimited
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
           RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
           MaxPayload 128 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0
unlimited, L1 unlimited
           ClockPM- Surprise+ LLActRep+ BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
           ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+
BWMgmt- ABWMgmt-
       RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna-
CRSVisible-
       RootCap: CRSVisible-
       RootSta: PME ReqID 0000, PMEStatus- PMEPending-
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol+
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
MalfTLP+ ECRC- UnsupReq- ACSViol+
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 15, GenCap- CGenEn- ChkCap- ChkEn-

00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 2-3 (rev 92) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 00002000-00002fff
   Memory behind bridge: df100000-df2fffff
   Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [50] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [58] MSI: Enable- Count=1/2 Maskable- 64bit-
       Address: fee00000  Data: 0000
   Capabilities: [6c] Express (v1) Root Port (Slot-), MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:    Port #2, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
           ClockPM- Surprise+ LLActRep+ BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
       RootCtl: ErrCorrectable+ ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
       RootCap: CRSVisible-
       RootSta: PME ReqID 0000, PMEStatus- PMEPending-
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:    DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport

00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev 92) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 0000f000-00000fff
   Memory behind bridge: fff00000-000fffff
   Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [50] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [58] MSI: Enable- Count=1/2 Maskable- 64bit-
       Address: fee00000  Data: 0000
   Capabilities: [6c] Express (v1) Root Port (Slot-), MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:    Port #3, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited
           ClockPM- Surprise+ LLActRep+ BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       RootCtl: ErrCorrectable+ ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
       RootCap: CRSVisible-
       RootSta: PME ReqID 0000, PMEStatus- PMEPending-
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:    DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport

00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA Engine (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin A routed to IRQ 0
   Region 0: Memory at fe700000 (64-bit, non-prefetchable) [size=1K]
   Capabilities: [50] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [58] MSI: Enable- Count=1/1 Maskable- 64bit-
       Address: fee00000  Data: 0000
   Capabilities: [6c] Express (v1) Root Complex Integrated Endpoint, MSI 00
       DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:    Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 92)
   Subsystem: Intel Corporation Device 8086
   Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=00, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 00004000-00004fff
   Memory behind bridge: d0100000-d02fffff
   Prefetchable memory behind bridge: 00000000d0300000-00000000d04fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [40] Express (v1) Root Port (Slot+), MSI 00
       DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
           ExtTag+ RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
       LnkCap:    Port #1, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <256ns, L1 <4us
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       SltCap:    AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
       SltCtl:    Enable: AttnBtn+ PwrFlt- MRL- PresDet+ CmdCplt- HPIrq- LinkChg-
           Control: AttnInd Off, PwrInd Off, Power- Interlock-
       SltSta:    Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
           Changed: MRL- PresDet- LinkState-
       RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
       RootCap: CRSVisible-
       RootSta: PME ReqID 0000, PMEStatus- PMEPending-
   Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
       Address: fee0300c  Data: 4151
   Capabilities: [90] Subsystem: Tyan Computer Device 5372
   Capabilities: [a0] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [100 v1] Virtual Channel
       Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1
       Arb:    Fixed+ WRR32- WRR64- WRR128-
       Ctrl:    ArbSelect=Fixed
       Status:    InProgress-
       VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
           Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
           Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
           Status:    NegoPending- InProgress-
       VC1:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
           Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
           Ctrl:    Enable- ID=0 ArbSelect=Fixed TC/VC=00
           Status:    NegoPending- InProgress-
   Capabilities: [180 v1] Root Complex Link
       Desc:    PortNumber=01 ComponentID=02 EltType=Config
       Link0:    Desc:    TargetPort=00 TargetComponent=02 AssocRCRB- LinkType=MemMapped LinkValid+
           Addr:    00000000fed1c001
   Kernel driver in use: pcieport

00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) (prog-if 00 [UHCI])
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin A routed to IRQ 17
   Region 4: I/O ports at 1800 [size=32]
   Kernel driver in use: uhci_hcd

00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) (prog-if 00 [UHCI])
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin B routed to IRQ 19
   Region 4: I/O ports at 1820 [size=32]
   Kernel driver in use: uhci_hcd

00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) (prog-if 00 [UHCI])
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin C routed to IRQ 18
   Region 4: I/O ports at 1840 [size=32]
   Kernel driver in use: uhci_hcd

00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09) (prog-if 00 [UHCI])
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin D routed to IRQ 16
   Region 4: I/O ports at 1860 [size=32]
   Kernel driver in use: uhci_hcd

00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) (prog-if 20 [EHCI])
   Subsystem: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller
   Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin A routed to IRQ 17
   Region 0: Memory at df500000 (32-bit, non-prefetchable) [size=1K]
   Capabilities: [50] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [58] Debug port: BAR=1 offset=00a0
   Kernel driver in use: ehci_hcd

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) (prog-if 01 [Subtractive decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Bus: primary=00, secondary=08, subordinate=08, sec-latency=32 I/O behind bridge: 00003000-00003fff
   Memory behind bridge: df000000-df0fffff
   Prefetchable memory behind bridge: 00000000de000000-00000000deffffff Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [50] Subsystem: Tyan Computer Device 5372

00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0

00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller (rev 09) (prog-if 80 [Master])
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0
   Interrupt: pin B routed to IRQ 19
   Region 0: I/O ports at 01f0 [size=8]
   Region 1: I/O ports at 03f4 [size=1]
   Region 2: I/O ports at 0170 [size=8]
   Region 3: I/O ports at 0374 [size=1]
   Region 4: I/O ports at 18b0 [size=16]
   Capabilities: [70] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: ata_piix
   Kernel modules: ata_generic, pata_acpi

00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller (rev 09)
   Subsystem: Tyan Computer Device 5372
   Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Interrupt: pin C routed to IRQ 18
   Region 4: I/O ports at 1100 [size=32]
   Kernel driver in use: i801_smbus
   Kernel modules: i2c-i801

01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=01, secondary=02, subordinate=04, sec-latency=0 I/O behind bridge: 00002000-00002fff
   Memory behind bridge: df200000-df2fffff
   Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [44] Express (v1) Upstream Port, MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-SlotPowerLimit 0.000W
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
   Capabilities: [70] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [80] Subsystem: Tyan Computer Device 5372
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport

01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 64 bytes
   Bus: primary=01, secondary=05, subordinate=05, sec-latency=64 I/O behind bridge: 0000f000-00000fff
   Memory behind bridge: fff00000-000fffff
   Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry- MaxPayload 256 bytes, MaxReadReq 256 bytes
       DevSta:    CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
   Capabilities: [6c] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [80] Subsystem: Tyan Computer Device 5372
   Capabilities: [d8] PCI-X bridge device
       Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=133MHz
       Status: Dev=00:00.3 64bit- 133MHz- SCD- USC- SCO- SRD-
       Upstream: Capacity=65535 CommitmentLimit=65535
       Downstream: Capacity=65535 CommitmentLimit=65535
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-

02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000f000-00000fff
   Memory behind bridge: fff00000-000fffff
   Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [44] Express (v1) Downstream Port (Slot-), MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 <512ns, L1 unlimited
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
   Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
       Address: 0000000000000000  Data: 0000
   Capabilities: [70] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [80] Subsystem: Tyan Computer Device 5372
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport

02:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3 (rev 01) (prog-if 00 [Normal decode])
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 00002000-00002fff
   Memory behind bridge: df200000-df2fffff
   Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
   BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
   Capabilities: [44] Express (v1) Downstream Port (Slot-), MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
           ExtTag- RBE- FLReset-
       DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
   Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
       Address: 0000000000000000  Data: 0000
   Capabilities: [70] Power Management version 2
       Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [80] Subsystem: Tyan Computer Device 5372
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport

04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
   Subsystem: Tyan Computer Device 5300
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Interrupt: pin A routed to IRQ 65
   Region 0: Memory at df220000 (32-bit, non-prefetchable) [size=128K] Region 1: Memory at df200000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 2000 [size=32]
   [virtual] Expansion ROM at d0000000 [disabled] [size=64K]
   Capabilities: [c8] Power Management version 2
       Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
   Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
       Address: 00000000fee0100c  Data: 4179
   Capabilities: [e0] Express (v1) Endpoint, MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
       DevCtl:    Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <128ns, L1 <64us
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
   Capabilities: [140 v1] Device Serial Number 00-e0-81-ff-ff-44-3f-4e Kernel driver in use: e1000e
   Kernel modules: e1000e

04:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
   Subsystem: Tyan Computer Device 5300
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 32 bytes
   Interrupt: pin B routed to IRQ 66
   Region 0: Memory at df260000 (32-bit, non-prefetchable) [size=128K] Region 1: Memory at df240000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 2020 [size=32]
   [virtual] Expansion ROM at d0010000 [disabled] [size=64K]
   Capabilities: [c8] Power Management version 2
       Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
   Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
       Address: 00000000fee0200c  Data: 4189
   Capabilities: [e0] Express (v1) Endpoint, MSI 00
       DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
       DevCtl:    Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes
       DevSta:    CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
       LnkCap:    Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <128ns, L1 <64us
           ClockPM- Surprise- LLActRep- BwNot-
       LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:    Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
       UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:    DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:    RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       AERCap:    First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
   Capabilities: [140 v1] Device Serial Number 00-e0-81-ff-ff-44-3f-4e Kernel driver in use: e1000e
   Kernel modules: e1000e

08:01.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics Innovation) Z7/Z9 (XG20 core) (prog-if 00 [VGA controller])
   Subsystem: XGI Technology Inc. (eXtreme Graphics Innovation) Z7/Z9 (XG20 core)
   Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   BIST result: 00
   Region 0: Memory at de000000 (32-bit, prefetchable) [size=16M] Region 1: Memory at df000000 (32-bit, non-prefetchable) [size=256K] Region 2: I/O ports at 3000 [size=128]
   Expansion ROM at <unassigned> [disabled]
   Capabilities: [40] Power Management version 2
       Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

08:02.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 10)
   Subsystem: Intel Corporation EtherExpress PRO/100 S Server Adapter
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
   Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
   Interrupt: pin A routed to IRQ 10
   Region 0: Memory at df060000 (32-bit, non-prefetchable) [size=4K] Region 1: I/O ports at 3080 [size=64]
   Region 2: Memory at df040000 (32-bit, non-prefetchable) [size=128K]
   Capabilities: [dc] Power Management version 2
       Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
       Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=2 PME-

Quoted 1 line

Is this an e1000e 4port adapter?

Dual port.

--
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.

Booting with "pci=pcie_bus_perf" should provide you with the same behavior as before (e.g., with the patch reverted). This should enable you work around this issue until I can get it resolved. On the positive side, your system should be 10% faster than it was in 3.0 (due to the PCI bus payloads being 2x larger).

If you can provide the same output with the patch not reverted, it would be very beneficial. Based on your description, the lockup only occurs with traffic. You can gather this before any traffic goes out (perhaps booting without networking enabled).

Quoted 6 lines

Booting with "pci=pcie_bus_perf" should provide you with the same behavior as before (e.g., with the patch reverted).  This should enable you work around this issue until I can get it resolved.  On the positive side, your system should be 10% faster than it was in 3.0 (due to the PCI bus payloads being 2x larger).

Ugh, that's assuming your fixes for the "perf" mode are in, which I thought wasn't the case yet ... without them, "perf" does very wrong things..

Quoted 4 lines

If you can provide the same output with the patch not reverted, it would be very beneficial.  Based on your description, the lockup only occurs with traffic.  You can gather this before any traffic goes out (perhaps booting without networking enabled).

Right, the output he provided has the MRRS clearly changing in addition to the MPS.

Ideally, we'd need the output with current upstream vs. with before your very first patch went in and compare that.

Also, Avi, can you send these outputs like patches ? IE without wrapping them ? That would be handy :-)

Cheers,
Ben.

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

+       /* Disable read completion coalescing to allow an MPS of 256 */
+       if (mps == 256) {
+               int err;
+               u16 rcc;

drivers/pci/probe.c +7 -0

+
+               /* Intel errata specifies bits to change but does not say what
+                * they are.  Keeping them magical until such time as the
+                * registers and values can be explained.
+                */
+               err = pci_read_config_word(dev, 0x48, &rcc);
+               if (err) {

+                               "completion coalescing register.\n");
+                       return;
+               }

Quoted 8 lines · Jon Mason

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

   PCI: Workaround for Intel MPS errata

Quoted 4 lines

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

Hey Jon, glad I could help out by pointing this erratum out on IRC today.  The patch looks mostly fine, with one nit, see below.

Quoted 15 lines

+       /* Disable read completion coalescing to allow an MPS of 256 */
+       if (mps == 256) {
+               int err;
+               u16 rcc;
+
+               /* Intel errata specifies bits to change but does not say what
+                * they are.  Keeping them magical until such time as the
+                * registers and values can be explained.
+                */
+               err = pci_read_config_word(dev, 0x48, &rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "
+                               "completion coalescing register.\n");
+                       return;
+               }

drivers/pci/probe.c static int pcie_find_smpss(struct pci_dev *dev, void *data) · line 1429

Quoted 6 lines

drivers/pci/probe.c +6 -0

+
+               rcc &= ~(1 << 10);
+
+               err = pci_write_config_word(dev, 0x48, rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "

this should be "to write the read "

Quoted 4 lines

+                               "completion coalescing register.\n");
+                       return;
+               }

do you have to do anything to change the bit back to 1 if something sets the mps back to 128?

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

drivers/pci/probe.c +77 -2

index a919db2..13c733a 100644
@@ -1361,6 +1361,80 @@ static int pcie_find_smpss(struct pci_dev *dev, void *data)
        return 0;
  }
 
+static void pcie_errata_check(int mps)
+{
+       struct pci_dev *dev = NULL;
+       static bool done = false;
+
+       if (done)
+               return;
+
+       /* Intel 5000 and 5100 Memory controllers have an errata with read
+        * completion coalescing (which is enabled by default) and MPS of 256B.
+        */
+       /* 5000X Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25C0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000Z Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000V Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D4, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000P Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D8, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5100 Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x65C0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* No Intel 5000 or 5100 Memory controller in the system, no need to
+        * check again
+        */
+       if (!dev) {
+               done = true;
+               return;
+       }
+
+fixup:
+       /* Disable read completion coalescing to allow an MPS of 256 */
+       if (mps == 256) {
+               int err;
+               u16 rcc;
+
+               /* Intel errata specifies bits to change but does not say what
+                * they are.  Keeping them magical until such time as the
+                * registers and values can be explained.
+                */
+               err = pci_read_config_word(dev, 0x48, &rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "
+                               "completion coalescing register.\n");
+                       return;
+               }
+
+               rcc &= ~(1 << 10);
+
+               err = pci_write_config_word(dev, 0x48, rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "
+                               "completion coalescing register.\n");
+                       return;
+               }
+
+               done = true;
+       }
+}
+
  static void pcie_write_mps(struct pci_dev *dev, int mps)
  {
        int rc;

@@ -1384,6 +1458,8 @@ static void pcie_write_mps(struct pci_dev *dev, int mps)
                        mps = min(mps, pcie_get_mps(dev->bus->self));
        }
 
+       pcie_errata_check(mps);
+
        rc = pcie_set_mps(dev, mps);
        if (rc)
                dev_err(&dev->dev, "Failed attempting to set the MPS\n");

@@ -1445,7 +1521,7 @@ static int pcie_bus_configure_set(struct pci_dev *dev, void *data)
        return 0;
  }
 
-/* pcie_bus_configure_mps requires that pci_walk_bus work in a top-down,
+/* pcie_bus_configure_settings requires that pci_walk_bus work in a top-down,
  * parents then children fashion.  If this changes, then this code will not
  * work as designed.
  */
--

To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Jon Mason

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

Quoted 6 lines

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

I'd much rather see this done as an early quirk so it doesn't clutter probe.c.

I don't know how you decide whether
   - no coalescing with MPS=256, or
   - coalescing with MPS=128
is better.  I suspect that having a quirk that doesn't change the setting, but merely limits MPS to 128 if the BIOS enabled coalescing, would be simplest and would stay in the best-tested chipset configuration.

If you do end up changing the coalescing setting, I think you should check the current setting, then log something in dmesg if you change it.  If the quirk limits the max MPS, I think dmesg should reflect that, too.

I think you're missing a pci_dev_put().

Even if we work out these issues and Avi reports that it fixes his hang, I'm still concerned about MPS configuration being enabled by default in 3.1.  It feels like we happened to trip over a few issues and fix them, but I'm worried that with wider testing, we'll find more.

I'd feel more comfortable if everything were disabled in 3.1 (it'd be OK to have a command-line flag to enable it), then turned on by default early in the 3.2 merge window.

Bjorn

Quoted 108 lines

drivers/pci/probe.c +77 -2

index a919db2..13c733a 100644
@@ -1361,6 +1361,80 @@ static int pcie_find_smpss(struct pci_dev *dev, void *data)
        return 0;
  }
 
+static void pcie_errata_check(int mps)
+{
+       struct pci_dev *dev = NULL;
+       static bool done = false;
+
+       if (done)
+               return;
+
+       /* Intel 5000 and 5100 Memory controllers have an errata with read
+        * completion coalescing (which is enabled by default) and MPS of 256B.
+        */
+       /* 5000X Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25C0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000Z Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000V Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D4, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5000P Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x25D8, NULL);
+       if (dev)
+               goto fixup;
+
+       /* 5100 Chipset Memory Controller Hub */
+       dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x65C0, NULL);
+       if (dev)
+               goto fixup;
+
+       /* No Intel 5000 or 5100 Memory controller in the system, no need to
+        * check again
+        */
+       if (!dev) {
+               done = true;
+               return;
+       }
+
+fixup:
+       /* Disable read completion coalescing to allow an MPS of 256 */
+       if (mps == 256) {
+               int err;
+               u16 rcc;
+
+               /* Intel errata specifies bits to change but does not say what
+                * they are.  Keeping them magical until such time as the
+                * registers and values can be explained.
+                */
+               err = pci_read_config_word(dev, 0x48, &rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "
+                               "completion coalescing register.\n");
+                       return;
+               }
+
+               rcc &= ~(1 << 10);
+
+               err = pci_write_config_word(dev, 0x48, rcc);
+               if (err) {
+                       dev_err(&dev->dev, "Error attempting to read the read "
+                               "completion coalescing register.\n");
+                       return;
+               }
+
+               done = true;
+       }
+}
+
  static void pcie_write_mps(struct pci_dev *dev, int mps)
  {
        int rc;

@@ -1384,6 +1458,8 @@ static void pcie_write_mps(struct pci_dev *dev, int mps)
                        mps = min(mps, pcie_get_mps(dev->bus->self));
        }
 
+       pcie_errata_check(mps);
+
        rc = pcie_set_mps(dev, mps);
        if (rc)
                dev_err(&dev->dev, "Failed attempting to set the MPS\n");

@@ -1445,7 +1521,7 @@ static int pcie_bus_configure_set(struct pci_dev *dev, void *data)
        return 0;
  }
 
-/* pcie_bus_configure_mps requires that pci_walk_bus work in a top-down,
+/* pcie_bus_configure_settings requires that pci_walk_bus work in a top-down,
  * parents then children fashion.  If this changes, then this code will not
  * work as designed.
  */
--

To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 16 lines · Bjorn Helgaas

Quoted 6 lines · Jon Mason

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

Quoted 6 lines

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

I'd much rather see this done as an early quirk so it doesn't clutter probe.c.

Quoted 7 lines

I don't know how you decide whether
   - no coalescing with MPS=256, or
   - coalescing with MPS=128
is better.  I suspect that having a quirk that doesn't change the setting, but merely limits MPS to 128 if the BIOS enabled coalescing, would be simplest and would stay in the best-tested chipset configuration.

This is what I was debating yesterday.  Is it better to disable coalescing and get better throughput (which could be a net negative if the MPS isn't 256) or never allow it to be greater than 128?  There is no way of knowing at quirk time if the disable is necessary or not, only when setting the MPS is it known (which is why I did it this way).  I could, as you suggest, simply read the bit and see if it is enabled by the BIOS (which I'd bet it is every single time), and then limit the MPS to 128 as a quirk.  This would be fairly simple to do. However, the errata from Intel says Windows 2008 always disables the coalescing and sets the MPS to 256B.  With this known, Linux's I/O performance would be less than Windows on these systems. ...

Quoted 26 lines · Jon Mason

Quoted 16 lines · Bjorn Helgaas

Quoted 6 lines · Jon Mason

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

Quoted 6 lines

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

I'd much rather see this done as an early quirk so it doesn't clutter probe.c.

Quoted 7 lines

I don't know how you decide whether
   - no coalescing with MPS=256, or
   - coalescing with MPS=128
is better.  I suspect that having a quirk that doesn't change the setting, but merely limits MPS to 128 if the BIOS enabled coalescing, would be simplest and would stay in the best-tested chipset configuration.

Quoted 11 lines

This is what I was debating yesterday.  Is it better to disable coalescing and get better throughput (which could be a net negative if the MPS isn't 256) or never allow it to be greater than 128?  There is no way of knowing at quirk time if the disable is necessary or not, only when setting the MPS is it known (which is why I did it this way).  I could, as you suggest, simply read the bit and see if it is enabled by the BIOS (which I'd bet it is every single time), and then limit the MPS to 128 as a quirk.  This would be fairly simple to do. However, the errata from Intel says Windows 2008 always disables the coalescing and sets the MPS to 256B.  With this known, Linux's I/O performance would be less than Windows on these systems. ...

Presumably coalescing improves performance, too, and I don't have the evidence that says "no coalescing with MPS=256" performs better than "coalescing with MPS=128."

But the fact that Windows 2008 disables coalescing is worth a lot (if this is in a public erratum, a URL would be good).  Given that, I'd

probably go with "no coalescing and MPS=256" just as you did.

Maybe the quirk could be moved out of the generic code by making pcie_set_mps() a weak function, so x86 could supply a version that disables coalescing if MPS=256?

No news from Avi?  Were you able to reproduce the problem and verify

that the quirk fixes it?  I wish the kernel.org bugzilla were back.

Since it's not, maybe we should include the LKML URL
(https://lkml.org/lkml/2011/9/27/274) in the changelog.

Bjorn

Quoted 43 lines · Bjorn Helgaas

Quoted 26 lines · Jon Mason

Quoted 16 lines · Bjorn Helgaas

Quoted 6 lines · Jon Mason

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

Quoted 6 lines

   PCI: Workaround for Intel MPS errata

   Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable read completion coalescing if the MPS is 256B.

I'd much rather see this done as an early quirk so it doesn't clutter probe.c.

Quoted 7 lines

I don't know how you decide whether
   - no coalescing with MPS=256, or
   - coalescing with MPS=128
is better.  I suspect that having a quirk that doesn't change the setting, but merely limits MPS to 128 if the BIOS enabled coalescing, would be simplest and would stay in the best-tested chipset configuration.

Quoted 11 lines

This is what I was debating yesterday.  Is it better to disable coalescing and get better throughput (which could be a net negative if the MPS isn't 256) or never allow it to be greater than 128?  There is no way of knowing at quirk time if the disable is necessary or not, only when setting the MPS is it known (which is why I did it this way).  I could, as you suggest, simply read the bit and see if it is enabled by the BIOS (which I'd bet it is every single time), and then limit the MPS to 128 as a quirk.  This would be fairly simple to do. However, the errata from Intel says Windows 2008 always disables the coalescing and sets the MPS to 256B.  With this known, Linux's I/O performance would be less than Windows on these systems. ...

Presumably coalescing improves performance, too, and I don't have the evidence that says "no coalescing with MPS=256" performs better than "coalescing with MPS=128."

Quoted 2 lines

But the fact that Windows 2008 disables coalescing is worth a lot (if this is in a public erratum, a URL would be good).  Given that, I'd

The URLs were in the top of the patch, but perhaps they weren't obvious. ... Search for "MPS", as they are several pages into the errata chapter.

Quoted 3 lines

Maybe the quirk could be moved out of the generic code by making pcie_set_mps() a weak function, so x86 could supply a version that disables coalescing if MPS=256?

Not sure what you mean here.  Are you saying to make the function defined differently on each arch?

Hey Avi,
Can you try this patch? It should resolve the issue you are seeing.

Thanks,
Jon

PCI: Workaround for Intel MPS errata

Intel 5000 and 5100 series memory controllers have a known issue if
read
completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B. To work around this issue,
disable
read completion coalescing if the MPS is 256B.

http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf
http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

Reported-by Avi Kivity <[email protected]>

Signed-off-by Jon Mason <[email protected]>

drivers/pci/probe.c +0 -0

index a919db2..13c733a 100644

Quoted 30 lines

Hey Avi,
Can you try this patch?  It should resolve the issue you are seeing.

Thanks,
Jon

    PCI: Workaround for Intel MPS errata

    Intel 5000 and 5100 series memory controllers have a known issue if read
    completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable
    read completion coalescing if the MPS is 256B.

    http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf
    http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

Reported-by Avi Kivity <[email protected]>

Signed-off-by Jon Mason <[email protected]>

drivers/pci/probe.c +0 -0

index a919db2..13c733a 100644
@@ -1361,6 +1361,80 @@ static int pcie_find_smpss(struct pci_dev *dev,
 void *data)
       return 0;
  }
 

drivers/pci/probe.c static int pcie_find_smpss(struct pci_dev *dev, void *data) · line 1365

Quoted 2 lines

drivers/pci/probe.c +2 -0

+static void pcie_errata_check(int mps)
+{

While this whole function is about Intel 5x00 devices the name of it is very generic (and in a very generic file). Maybe this can be changed, e.g.

"fixup:" renamed to "fixup_intel_5x00:". Also I wonder if that shouldn't be a nop when CONFIG_PCI_QUIRKS is not set.

Eike

Hey Avi,
Can you try this patch? It should resolve the issue you are seeing.

Quoted 2 lines · Jon Mason

Hey Avi,
Can you try this patch? It should resolve the issue you are seeing.

It doesn't; the fixup: label is not reached (though I do have an 0x25d4 device).

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 10 lines · Avi Kivity

Quoted 2 lines · Jon Mason

Hey Avi,
Can you try this patch? It should resolve the issue you are seeing.

It doesn't; the fixup: label is not reached (though I do have an 0x25d4 device).

--
error compiling committee.c: too many arguments to function

I found a system with a 5000X Memory controller (which should have the same errata). It doesn't have the faulty bit (perhaps better BIOS). I was able to findout why the code in the previous patch wasn't working, but wasn't able to cause the crash by setting the bit from the errata. The reworked version of the previous patch found below should resolve the issue. Please test it if you can.

Quoted 12 lines · Jon Mason

Quoted 8 lines · Avi Kivity

Quoted 2 lines · Jon Mason

Hey Avi,
Can you try this patch? It should resolve the issue you are seeing.

It doesn't; the fixup: label is not reached (though I do have an 0x25d4 device).

-- > error compiling committee.c: too many arguments to

function

Quoted 1 line

Quoted 6 lines

I found a system with a 5000X Memory controller (which should have the same errata). It doesn't have the faulty bit (perhaps better BIOS). I was able to findout why the code in the previous patch wasn't working, but wasn't able to cause the crash by setting the bit from the errata. The reworked version of the previous patch found below should resolve the issue. Please test it if you can.

Will be happy to test, but patch appears to be against a different tree?

$ git apply -C2 .git/rebase-apply/patch
.git/rebase-apply/patch:75: trailing whitespace, shock horror.
     *
Context reduced to (2/2) to apply fragment at 1362
Context reduced to (2/2) to apply fragment at 1475
error: patch failed: drivers/pci/probe.c:1433
error: drivers/pci/probe.c: patch does not apply

PCI: Workaround for Intel MPS errata

Intel 5000 and 5100 series memory controllers have a known issue if read completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B. To work around this issue, disable read completion coalescing if the MPS is 256B.

It is worth noting that there is no function to undo the disable of read completion coalescing, and the performance benefit of read completion coalescing will be lost if the MPS is set from 256B to 128B. It is only possible to have this issue via hotplug removing the only 256B MPS device in the system (thus making all of the other devices in the system have a performance degradation without the benefit of any 256B transfers). Therefore, this trade off is acceptable.

http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf
http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

Thanks to Jesse Brandeburg and Ben Hutchings for providing insight into the problem.

Reported-by Avi Kivity<[email protected]>

Signed-off-by Jon Mason<[email protected]>

drivers/pci/probe.c +14 -0

+		err = pci_bus_write_config_word(bus, 0, 0x48, val);
+		if (err) {
+			dev_err(&bus->dev, "Error attempting to write the read "
+				"completion coalescing register.\n");
+			return;
+		}
+
+		dev_info(&bus->dev, "Read completion coalescing disabled due "
+			 "to hardware errata relating to 256B MPS.\n");
+
+		done = true;
+	}
+}
+

Quoted 34 lines · Jon Mason

PCI: Workaround for Intel MPS errata

Intel 5000 and 5100 series memory controllers have a known issue
if read
completion coalescing is enabled (the default setting) and the
PCI-E
Maximum Payload Size is set to 256B. To work around this issue,
disable
read completion coalescing if the MPS is 256B.

It is worth noting that there is no function to undo the disable
of read
completion coalescing, and the performance benefit of read
completion
coalescing will be lost if the MPS is set from 256B to 128B. It
is only
possible to have this issue via hotplug removing the only 256B MPS device in the system (thus making all of the other devices in
the system
have a performance degradation without the benefit of any 256B transfers). Therefore, this trade off is acceptable.

http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf

http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

Thanks to Jesse Brandeburg and Ben Hutchings for providing
insight into
the problem.

Reported-by Avi Kivity<[email protected]>

Signed-off-by Jon Mason<[email protected]>

drivers/pci/probe.c static int pcie_find_smpss(struct pci_dev *dev, void *data) · line 1388

Quoted 5 lines

drivers/pci/probe.c +5 -0

+
+        if (!(val&  (1<<  10))) {
+            done = true;
+            return;
+        }

Here, you bail out if bit 10 is clear. So if we're here, it's set.

Review

Quoted 2 lines

(quoted diff) +2 -0

+
+        val |= (1<<  10);

Now it's even more set?

Quoted 43 lines · Avi Kivity

Quoted 30 lines · Jon Mason

    PCI: Workaround for Intel MPS errata

    Intel 5000 and 5100 series memory controllers have a known issue if read
    completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable
    read completion coalescing if the MPS is 256B.

    It is worth noting that there is no function to undo the disable of read
    completion coalescing, and the performance benefit of read completion coalescing will be lost if the MPS is set from 256B to 128B.  It is only
    possible to have this issue via hotplug removing the only 256B MPS device in the system (thus making all of the other devices in the system
    have a performance degradation without the benefit of any 256B transfers).  Therefore, this trade off is acceptable.

http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf

http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

    Thanks to Jesse Brandeburg and Ben Hutchings for providing insight into
    the problem.

Quoted 8 lines

Reported-by Avi Kivity<[email protected]>

Signed-off-by Jon Mason<[email protected]>

+
+        if (!(val&  (1<<  10))) {
+            done = true;
+            return;
+        }

Here, you bail out if bit 10 is clear.  So if we're here, it's set.

Quoted 5 lines

Quoted 2 lines

+
+        val |= (1<<  10);

Now it's even more set?

Even with this line changed to clear bit 10, I still get a hard lockup.  Do

we need to clear this bit on the other 5000 devices?  I notice they have similar values in word 0x48, with bits 10 set in them.

What does "Device 7-2,0" refer to in the workaround description?  Seems to me we need to apply the workaround to the PCIe ports as well.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 41 lines · Avi Kivity

Quoted 30 lines · Jon Mason

    PCI: Workaround for Intel MPS errata

    Intel 5000 and 5100 series memory controllers have a known issue if read
    completion coalescing is enabled (the default setting) and the PCI-E Maximum Payload Size is set to 256B.  To work around this issue, disable
    read completion coalescing if the MPS is 256B.

    It is worth noting that there is no function to undo the disable of read
    completion coalescing, and the performance benefit of read completion coalescing will be lost if the MPS is set from 256B to 128B.  It is only
    possible to have this issue via hotplug removing the only 256B MPS device in the system (thus making all of the other devices in the system
    have a performance degradation without the benefit of any 256B transfers).  Therefore, this trade off is acceptable.

http://www.intel.com/content/dam/doc/specification-update/5000-chipset-memory-controller-hub-specification-update.pdf

http://www.intel.com/content/dam/doc/specification-update/5100-memory-controller-hub-chipset-specification-update.pdf

    Thanks to Jesse Brandeburg and Ben Hutchings for providing insight into
    the problem.

Quoted 8 lines

Reported-by Avi Kivity<[email protected]>

Signed-off-by Jon Mason<[email protected]>

+
+        if (!(val&  (1<<  10))) {
+            done = true;
+            return;
+        }

Quoted 7 lines

Here, you bail out if bit 10 is clear.  So if we're here, it's set.

Quoted 2 lines

+
+        val |= (1<<  10);

Now it's even more set?

Even with this line changed to clear bit 10, I still get a hard lockup.  Do we need to clear this bit on the other 5000 devices?  I notice they have similar values in word 0x48, with bits 10 set in them.

What does "Device 7-2,0" refer to in the workaround description?  Seems to me we need to apply the workaround to the PCIe ports as well.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hey Avi,
I believe this will fix the issue (assuming the errata is the issue in the first place). You'll need to apply the patch on top of Linus' latest code and re-enable the MPS tuning (as it is now off by default). This can be done by adding "pci=pcie_bus_safe" to your boot args.

After thinking about it some more, a PCI quark is the correct way of doing things. We must always disable read completion coalescing due to the possibility of hotplugging a device with a MPS of 256B. Also, I believe everyone will think this is much cleaner.

Let me know how it goes and thanks again for testing this for me.