Skip to contentSkip to conversation

e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e)

DISCUSSION Ben Greear 7 participants 17 messages Apr 19, 2012 – May 1, 2012

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Ben Greear

author Acked-by Eric Dumazet
#0 lore raw

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce) reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256 dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out
Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: nf_nat]
Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted 3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel: <IRQ>  [<ffffffff81042902>] warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>] warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>] dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>] run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ? netif_tx_unlock+0x57/0x57
Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>] __do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ? hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>] call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>] smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>] apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel: <EOI>  [<ffffffff81010b8c>] ? mwait_idle+0x6e/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ? mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>] start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: 1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump directories
Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0' creation detected
Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt
Apr 19 16:12:57 localhost abrtd: Can't open file '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR: /var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of /var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 74
Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000feeff00c Data: 41a3
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e
Kernel driver in use: e1000e
Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert <[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert <[email protected]>
Acked-by Eric Dumazet <[email protected]>
Signed-off-by David S. Miller <[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe 51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

Signature · 14 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired
#1 lore raw

Thanks, will try to reproduce.

Tom

Quoted 182 lines · Ben Greear

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce)
reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link
becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at
/home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit
queue 0 timed out
Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q

garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded:
nf_nat]

Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted
3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel: <IRQ>  [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>]
warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>]
dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>]
run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
netif_tx_unlock+0x57/0x57
Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>]
__do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ?
hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>]
call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel: <EOI>  [<ffffffff81010b8c>] ?
mwait_idle+0x6e/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ?
mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>]
start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses:
1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump
directories
Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0'
creation detected
Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt
Apr 19 16:12:57 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or
directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory
oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0),
sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link

becomes ready
....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
       Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
       Latency: 0, Cache Line Size: 32 bytes
       Interrupt: pin A routed to IRQ 74
       Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32]
       [virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
       Capabilities: [c8] Power Management version 2
               Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
               Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
       Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
               Address: 00000000feeff00c  Data: 41a3
       Capabilities: [e0] Express (v1) Endpoint, MSI 00
               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
               DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
               DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
               LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency L0 <4us, L1 <64us
                       ClockPM- Surprise- LLActRep- BwNot-
               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       Capabilities: [100 v1] Advanced Error Reporting
               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
               UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
               AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
       Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e Kernel driver in use: e1000e
       Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c
Author: Tom Herbert <[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

   e1000e: Support for byte queue limits

   Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert <[email protected]>
Acked-by Eric Dumazet <[email protected]>
Signed-off-by David S. Miller <[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe 51914e221547b95a989b5c7e9b037c9370fd734e M      drivers

Thanks,
Ben

--
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

#2 lore raw

No, we have seen it in our test. I just checked all our Ilium machines' /var/log/messages back to earlier to month. We got numerous "tg3 transmit timed out" on Ilium GLAG machines only, no e1000e transmit queue timed out.

The "tg3 transmit queue timed out" does not seem to be caused by BQL, since I can see it happen even in 420 kernel!

Apr 13 18:36:08 lpb77 kernel: [   51.684212] ------------[ cut here
]------------
Apr 13 18:36:08 lpb77 kernel: [   51.684223] WARNING: at
net/sched/sch_generic.c:263 dev_watchdog+0x1c8/0x1f0()
Apr 13 18:36:08 lpb77 kernel: [   51.684226] Hardware name: Greencreek,ESB2
Apr 13 18:36:08 lpb77 kernel: [   51.684228] NETDEV WATCHDOG: eth4 (tg3):
transmit queue 0 timed out
Apr 13 18:36:08 lpb77 kernel: [   51.684230] Modules linked in: bonding
sata_mv i2c_imch pca954x i2c_virtual uhci_hcd msr cpuid i2c_dev i2c_i801
i2c_core i2c_debug tg3 e1000e ipv6 genrtc
Apr 13 18:36:08 lpb77 kernel: [   51.684246] Pid: 0, comm: swapper Not
tainted 2.6.34-smp-420.16 #1
Apr 13 18:36:08 lpb77 kernel: [   51.684248] Call Trace:
Apr 13 18:36:08 lpb77 kernel: [   51.684251]  <IRQ>  [<ffffffff811dc939>]
warn_slowpath_common+0x7c/0xce
Apr 13 18:36:08 lpb77 kernel: [   51.684261]  [<ffffffff811dc9e2>]
warn_slowpath_fmt+0x41/0x43
Apr 13 18:36:08 lpb77 kernel: [   51.684265]  [<ffffffff811494d8>]
dev_watchdog+0x1c8/0x1f0
Apr 13 18:36:08 lpb77 kernel: [   51.684270]  [<ffffffff811f14a5>] ?
sched_clock_local+0x1c/0x82
Apr 13 18:36:08 lpb77 kernel: [   51.684275]  [<ffffffff8101826c>]
run_timer_softirq+0x34c/0x360
Apr 13 18:36:08 lpb77 kernel: [   51.684278]  [<ffffffff81149310>] ?
dev_watchdog+0x0/0x1f0
Apr 13 18:36:08 lpb77 kernel: [   51.684282]  [<ffffffff8101780e>]
__do_softirq+0x39e/0x490
Apr 13 18:36:08 lpb77 kernel: [   51.684316]  [<ffffffff81020360>] ?
tick_program_event+0x60/0x120
Apr 13 18:36:08 lpb77 kernel: [   51.684319]  [<ffffffff811f14a5>] ?
sched_clock_local+0x1c/0x82
Apr 13 18:36:08 lpb77 kernel: [   51.684323]  [<ffffffff8107770c>]
call_softirq+0x1c/0x30
Apr 13 18:36:08 lpb77 kernel: [   51.684327]  [<ffffffff810009a2>]
do_softirq+0x42/0x80
Apr 13 18:36:08 lpb77 kernel: [   51.684329]  [<ffffffff81017159>]
irq_exit+0x49/0xa0
Apr 13 18:36:08 lpb77 kernel: [   51.684334]  [<ffffffff814ac184>]
smp_apic_timer_interrupt+0x84/0xf4
Apr 13 18:36:08 lpb77 kernel: [   51.684338]  [<ffffffff810771d3>]
apic_timer_interrupt+0x13/0x20
Apr 13 18:36:08 lpb77 kernel: [   51.684339]  <EOI>  [<ffffffff811b5709>] ?
mwait_idle+0x7a/0x81
Apr 13 18:36:08 lpb77 kernel: [   51.684355]  [<ffffffff811b56bb>] ?
mwait_idle+0x2c/0x81
Apr 13 18:36:08 lpb77 kernel: [   51.684358]  [<ffffffff810005c2>]
cpu_idle+0x92/0x140
Apr 13 18:36:08 lpb77 kernel: [   51.684363]  [<ffffffff81b80a53>]
start_secondary+0x1c3/0x1c7
Apr 13 18:36:08 lpb77 kernel: [   51.684373] ---[ end trace

fc8464e86676a1b7 ]---

Quoted 203 lines · Tom Herbert

Thanks, will try to reproduce.

Tom

Quoted 11 lines · Ben Greear

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce)
reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset

adapter

Quoted 2 lines
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps

Full

Quoted 4 lines

Duplex, Flow Control: Rx/Tx

Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps

Full

Quoted 11 lines

Duplex, Flow Control: Rx/Tx

Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link
becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at
/home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e):

transmit

Quoted 2 lines

queue 0 timed out

Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw

8021q

Quoted 1 line

garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen

iscsi_tcp

Quoted 1 line

libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf

hwmon_vid

Quoted 17 lines

coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded:
nf_nat]

Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted
3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel: <IRQ>  [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>]
warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>]
dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>]
run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
Quoted 8 lines

netif_tx_unlock+0x57/0x57

Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>]
__do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ?
hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>]
call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>]
do_softirq+0x41/0x7e
> Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Quoted 5 lines
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel: <EOI>  [<ffffffff81010b8c>] ?
Quoted 8 lines

mwait_idle+0x6e/0x8c

Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ?
mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>]
start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset

adapter

Quoted 1 line
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found

oopses:

Quoted 2 lines

1

Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating

dump

Quoted 2 lines

directories

Apr 19 16:12:57 localhost abrtd: Directory

'oops-2012-04-19-16:12:57-898-0'

Quoted 2 lines

creation detected

Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to

Abrt

Quoted 11 lines
Apr 19 16:12:57 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or
directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory
oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0),
sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps

Full

Quoted 6 lines

Duplex, Flow Control: Rx/Tx

Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset

adapter

Quoted 2 lines
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps

Full

Quoted 11 lines

Duplex, Flow Control: Rx/Tx

Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link

becomes ready
....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-

ParErr+

Quoted 6 lines

Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 74
Region 0: Memory at d8300000 (32-bit, non-prefetchable)

[size=128K]

Quoted 32 lines

Region 2: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000feeff00c Data: 41a3
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-

               LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1,

Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-

Quoted 8 lines

RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap-
ChkEn-
Capabilities: [140 v1] Device Serial Number

00-e0-ed-ff-ff-0c-11-6e

Quoted 24 lines

Kernel driver in use: e1000e
Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert <[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert <[email protected]>
Acked-by Eric Dumazet <[email protected]>
Signed-off-by David S. Miller <[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe
51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

Quoted 9 lines
Quoted 4 lines

--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2

_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

author
#3 lore raw
In reply to Tom Herbert
Quoted 1 line · Tom Herbert

Thanks, will try to reproduce.

I am seeing something similar with the 'igb' driver, though this NIC also involves a side-driver that does bypass. When I enable/disable bypass, the links bounce (as expected), and igb reports the same timeout that I was seeing with e1000e.

If I revert the igb BQL patch, then it works fine (kernel 3.3.2+).

I did not do an exhaustive bisect, nor have tested non-bypass igb hardware, but since reverting the patch fixes the problem....

Maybe there is some fundamental issue with BQL when a NIC resets itself (in this case, due to remote port doing a reset) Maybe we are not properly accounting pkts cleared from the xmit queue on reset or something of that nature?

I'm happy to test patches of someone has suggestions...

Thanks,
Ben

Quoted 186 lines

Tom

Quoted 182 lines · Ben Greear

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce)
reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link
becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at
/home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e): transmit
queue 0 timed out
Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q

garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded:
nf_nat]

Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted
3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel:<IRQ>    [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>]
warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>]
dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>]
run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
netif_tx_unlock+0x57/0x57
Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>]
__do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ?
hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>]
call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel:<EOI>    [<ffffffff81010b8c>] ?
mwait_idle+0x6e/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ?
mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>]
start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses:
1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating dump
directories
Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04-19-16:12:57-898-0'
creation detected
Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt
Apr 19 16:12:57 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or
directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory
oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0),
sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link

becomes ready
....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort-
<TAbort-<MAbort->SERR-<PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 74
Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000feeff00c Data: 41a3
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<512ns, L1<64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-

                LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1,

Latency L0<4us, L1<64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap-
ChkEn-
Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e Kernel driver in use: e1000e
Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert<[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert<[email protected]>
Acked-by Eric Dumazet<[email protected]>
Signed-off-by David S. Miller<[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe
51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

--
Ben Greear<[email protected]>
Candela Technologies Inc http://www.candelatech.com

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#4 lore raw
Quoted 5 lines

I am seeing something similar with the 'igb' driver, though this NIC also involves a side-driver that does bypass.  When I enable/disable bypass, the links bounce (as expected), and igb reports the same timeout that I was seeing with e1000e.

Hi Ben,

Are the circumstances the same between igb and e1000e for the issue? That is in both cases are links being bounced or reset?

Thanks,
Tom

Quoted 225 lines

If I revert the igb BQL patch, then it works fine (kernel 3.3.2+).

I did not do an exhaustive bisect, nor have tested non-bypass igb hardware, but since reverting the patch fixes the problem....

Maybe there is some fundamental issue with BQL when a NIC resets itself (in this case, due to remote port doing a reset)  Maybe we are not properly accounting pkts cleared from the xmit queue on reset or something of that nature?

I'm happy to test patches of someone has suggestions...

Thanks,
Ben

Quoted 207 lines

Tom

Quoted 202 lines · Ben Greear

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce)
reset peer interface
---->  tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link
becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at
/home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e):
transmit
queue 0 timed out
Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw

8021q
garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid
coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded:
nf_nat]

Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted
3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel:<IRQ>    [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>]
warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>]
dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>]
run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
netif_tx_unlock+0x57/0x57
Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>]
__do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ?
hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>]
call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>]
do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel:<EOI>    [<ffffffff81010b8c>] ?
mwait_idle+0x6e/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ?
mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>]
start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found
oopses:
1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating
dump
directories
Apr 19 16:12:57 localhost abrtd: Directory
'oops-2012-04-19-16:12:57-898-0'
creation detected
Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to
Abrt
Apr 19 16:12:57 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or
directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory
oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0),
sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link

becomes ready
....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
       Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort- <TAbort-<MAbort->SERR-<PERR- INTx-
       Latency: 0, Cache Line Size: 32 bytes
       Interrupt: pin A routed to IRQ 74
       Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K]
       Region 2: I/O ports at 3000 [size=32]
       [virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
       Capabilities: [c8] Power Management version 2
               Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
               Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
       Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
               Address: 00000000feeff00c  Data: 41a3
       Capabilities: [e0] Express (v1) Endpoint, MSI 00
               DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1<64us
                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
               DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
               DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
               LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1, Latency L0<4us, L1<64us
                       ClockPM- Surprise- LLActRep- BwNot-
               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
               LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       Capabilities: [100 v1] Advanced Error Reporting
               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
               UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
               AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
       Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e
       Kernel driver in use: e1000e
       Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c
Author: Tom Herbert<[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

   e1000e: Support for byte queue limits

   Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert<[email protected]>
Acked-by Eric Dumazet<[email protected]>
Signed-off-by David S. Miller<[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe 51914e221547b95a989b5c7e9b037c9370fd734e M      drivers

Thanks,
Ben

--
Ben Greear<[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Quoted 4 lines

--
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

author
#5 lore raw
Quoted 10 lines · Tom Herbert
Quoted 5 lines

I am seeing something similar with the 'igb' driver, though this NIC also involves a side-driver that does bypass. When I enable/disable bypass, the links bounce (as expected), and igb reports the same timeout that I was seeing with e1000e.

Hi Ben,

Are the circumstances the same between igb and e1000e for the issue? That is in both cases are links being bounced or reset?

Yes, peer interface is being bounced in both cases. The igb NIC is in a different machine, running Fedora 14 instead of F16 for the e1000e.

I have the bypass NIC cabled to itself..2 ports sending/receiving traffic, and 2 doing bypass (in hardware, or bridging in software).

When I twiddle to/from hardware/software bridging on the two bypass ports, link is re-negotiated on the traffic-generating ports, and the traffic generating ports have the tx queue timeout issue.

If you have an igb or e1000e system that can generate traffic while resetting the peer interface, I think you should easily see the problem. If you DON'T see the problem, please let me know and I will try to provide a more concise test case using open-source traffic generators (we're using our own traffic generator, but at least on the e1000e bisect, I was using standard upstream kernels and pure user-space API to generate traffic).

Thanks,
Ben

Quoted 237 lines

Thanks,
Tom

Quoted 225 lines

If I revert the igb BQL patch, then it works fine (kernel 3.3.2+).

I did not do an exhaustive bisect, nor have tested non-bypass igb hardware, but since reverting the patch fixes the problem....

Maybe there is some fundamental issue with BQL when a NIC resets itself (in this case, due to remote port doing a reset) Maybe we are not properly accounting pkts cleared from the xmit queue on reset or something of that nature?

I'm happy to test patches of someone has suggestions...

Thanks,
Ben

Quoted 207 lines

Tom

Quoted 202 lines · Ben Greear

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce)
reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down
Apr 19 16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down
Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link
becomes ready
Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:12:55 localhost kernel: ------------[ cut here ]------------
Apr 19 16:12:55 localhost kernel: WARNING: at
/home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154()
Apr 19 16:12:55 localhost kernel: Hardware name: X7DBU
Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e):
transmit
queue 0 timed out
Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw

8021q
garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid
coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801
i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded:
nf_nat]

Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted
3.2.0-rc2+ #36
Apr 19 16:12:55 localhost kernel: Call Trace:
Apr 19 16:12:55 localhost kernel:<IRQ>      [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98
Apr 19 16:12:55 localhost kernel: [<ffffffff810429ae>]
warn_slowpath_fmt+0x41/0x43
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f8a3>]
dev_watchdog+0xf4/0x154
Apr 19 16:12:55 localhost kernel: [<ffffffff8104d371>]
run_timer_softirq+0x16f/0x201
Apr 19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
netif_tx_unlock+0x57/0x57
Apr 19 16:12:55 localhost kernel: [<ffffffff81047e47>]
__do_softirq+0x86/0x12f
Apr 19 16:12:55 localhost kernel: [<ffffffff8105d54e>] ?
hrtimer_interrupt+0x12b/0x1bd
Apr 19 16:12:55 localhost kernel: [<ffffffff8144296c>]
call_softirq+0x1c/0x30
Apr 19 16:12:55 localhost kernel: [<ffffffff8100bb75>]
do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80
Apr 19 16:12:55 localhost kernel:<EOI>      [<ffffffff81010b8c>] ?
mwait_idle+0x6e/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81010b7f>] ?
mwait_idle+0x61/0x8c
Apr 19 16:12:55 localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe
Apr 19 16:12:55 localhost kernel: [<ffffffff81435477>]
start_secondary+0x194/0x199
Apr 19 16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]---
Apr 19 16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found
oopses:
1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating
dump
directories
Apr 19 16:12:57 localhost abrtd: Directory
'oops-2012-04-19-16:12:57-898-0'
creation detected
Apr 19 16:12:57 localhost abrt-dump-oops: Reported 1 kernel oopses to
Abrt
Apr 19 16:12:57 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or
directory
Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory
oops-2012-04-19-16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0),
sending dbus signal
Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:12:58 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link
becomes ready
Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing,
re-enabling all cpus for irq balancing
Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset
adapter
Apr 19 16:13:05 localhost chronyd[1003]: Selected source 108.59.2.194
Apr 19 16:13:07 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps
Full
Duplex, Flow Control: Rx/Tx
Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link

becomes ready
....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast>TAbort-
<TAbort-<MAbort->SERR-<PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 74
Region 0: Memory at d8300000 (32-bit, non-prefetchable)
[size=128K]
Region 2: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000feeff00c Data: 41a3
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<512ns, L1<64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-

                LnkCap: Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1,

Latency L0<4us, L1<64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap-
ChkEn-
Capabilities: [140 v1] Device Serial Number
00-e0-ed-ff-ff-0c-11-6e
Kernel driver in use: e1000e
Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert<[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert<[email protected]>
Acked-by Eric Dumazet<[email protected]>
Signed-off-by David S. Miller<[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe
51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

--
Ben Greear<[email protected]>
Candela Technologies Inc http://www.candelatech.com

Quoted 4 lines

--
Ben Greear<[email protected]>
Candela Technologies Inc http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 14 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired
Tested-by Stephen Ko Tested-by Stephen Ko
#6 lore raw
Quoted 7 lines · Ben Greear
Quoted 6 lines · Tom Herbert
Quoted 5 lines

I am seeing something similar with the 'igb' driver, though this NIC also involves a side-driver that does bypass. When I enable/disable bypass, the links bounce (as expected), and igb reports the same timeout that I was seeing with e1000e.

[...]

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert<[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert<[email protected]>
Acked-by Eric Dumazet<[email protected]>
Signed-off-by David S. Miller<[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe
51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

Tom, did you see these two patches? Maybe this is resolved by the second patch.

We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3.

commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

Author: Alexander Duyck <[email protected]>
Date:   Tue Feb 7 02:29:06 2012 +0000

net: Add memory barriers to prevent possible race in byte queue limits

This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic.

Signed-off-by Alexander Duyck <[email protected]>
Tested-by Stephen Ko <[email protected]>
Signed-off-by Jeff Kirsher <[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

commit 5c4903549c05bbb373479e0ce2992573c120654a

Author: Alexander Duyck <[email protected]>
Date:   Tue Feb 7 02:29:01 2012 +0000

net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state

We are seeing dev_watchdog hangs on several drivers. I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue. This change corrects that.

In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests. We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting.

Signed-off-by Alexander Duyck <[email protected]>
Tested-by Stephen Ko <[email protected]>
Signed-off-by Jeff Kirsher <[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a

#7 lore raw

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

Tom

Quoted 49 lines

Tom, did you see these two patches? Maybe this is resolved by the second patch.

We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3.

commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635
Author: Alexander Duyck <[email protected]>
Date:   Tue Feb 7 02:29:06 2012 +0000

   net: Add memory barriers to prevent possible race in byte queue limits

   This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic.

Signed-off-by Alexander Duyck <[email protected]>
Tested-by Stephen Ko <[email protected]>
Signed-off-by Jeff Kirsher <[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

commit 5c4903549c05bbb373479e0ce2992573c120654a
Author: Alexander Duyck <[email protected]>
Date:   Tue Feb 7 02:29:01 2012 +0000

   net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state

   We are seeing dev_watchdog hangs on several drivers.  I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue.  This change corrects that.

   In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests.  We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting.

Signed-off-by Alexander Duyck <[email protected]>
Tested-by Stephen Ko <[email protected]>
Signed-off-by Jeff Kirsher <[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a

author
#8 lore raw
Quoted 2 lines · Tom Herbert

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

I don't think so. I'll add them and re-test.

Thanks,
Ben

Quoted 52 lines

Tom

Quoted 49 lines

Tom, did you see these two patches? Maybe this is resolved by the second patch.

We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3.

commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:06 2012 +0000

net: Add memory barriers to prevent possible race in byte queue limits

This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

commit 5c4903549c05bbb373479e0ce2992573c120654a

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:01 2012 +0000

net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state

We are seeing dev_watchdog hangs on several drivers. I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue. This change corrects that.

In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests. We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a

Signature · 14 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired
author
#9 lore raw
In reply to Tom Herbert
Quoted 2 lines · Tom Herbert

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

I just tested this on my e1000e and igb machine. With these patches, I can no longer reproduce the problem.

So, please make sure those are queued up for 3.3 stable!

Thanks,
Ben

Quoted 52 lines

Tom

Quoted 49 lines

Tom, did you see these two patches? Maybe this is resolved by the second patch.

We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3.

commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:06 2012 +0000

net: Add memory barriers to prevent possible race in byte queue limits

This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

commit 5c4903549c05bbb373479e0ce2992573c120654a

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:01 2012 +0000

net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state

We are seeing dev_watchdog hangs on several drivers. I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue. This change corrects that.

In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests. We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a

Signature · 14 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired
author
#10 lore raw
Quoted 8 lines · Ben Greear
Quoted 2 lines · Tom Herbert

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

I just tested this on my e1000e and igb machine. With these patches, I can no longer reproduce the problem.

So, please make sure those are queued up for 3.3 stable!

Dave: I think these patches below should go to 3.3 stable.

They are not queued for stable yet as far as I can tell.

Thanks,
Ben

Quoted 58 lines

Thanks,
Ben

Quoted 52 lines

Tom

Quoted 49 lines

Tom, did you see these two patches? Maybe this is resolved by the second patch.

We needed these to fixup ixgbe and igb (i didn't test e1000e) looks like we might want to push these at stable. I don't believe they are in 3.3.

commit b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:06 2012 +0000

net: Add memory barriers to prevent possible race in byte queue limits

This change adds a memory barrier to the byte queue limit code to address a possible race as has been seen in the past with the netif_stop_queue/netif_wake_queue logic.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b37c0fbe3f6dfba1f8ad2aed47fb40578a254635

commit 5c4903549c05bbb373479e0ce2992573c120654a

Author: Alexander Duyck<[email protected]>
Date:   Tue Feb 7 02:29:01 2012 +0000

net: Fix issue with netdev_tx_reset_queue not resetting queue from XOFF state

We are seeing dev_watchdog hangs on several drivers. I suspect this is due to the __QUEUE_STATE_STACK_XOFF bit being set prior to a reset for link change, and then not being cleared by netdev_tx_reset_queue. This change corrects that.

In addition we were seeing dev_watchdog hangs on igb after running the ethtool tests. We found this to be due to the fact that the ethtool test runs the same logic as ndo_start_xmit, but we were never clearing the XOFF flag since the loopback test in ethtool does not do byte queue accounting.

Signed-off-by Alexander Duyck<[email protected]>
Tested-by Stephen Ko<[email protected]>
Signed-off-by Jeff Kirsher<[email protected]>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=5c4903549c05bbb373479e0ce2992573c120654a

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
maintainer
#11 lore raw

From: Ben Greear <[email protected]>
Date: Tue, 01 May 2012 14:10:43 -0700

Quoted 5 lines · Ben Greear
Quoted 4 lines · Tom Herbert
Quoted 2 lines

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

Quoted 8 lines
Quoted 4 lines

I just tested this on my e1000e and igb machine. With these patches, I can no longer reproduce the problem.

So, please make sure those are queued up for 3.3 stable!

Dave: I think these patches below should go to 3.3 stable.

They are not queued for stable yet as far as I can tell.

I let the Intel developers handle -stable submissions for their drivers.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

author
#12 lore raw
Quoted 11 lines · David Miller

From: Ben Greear<[email protected]>
Date: Tue, 01 May 2012 14:10:43 -0700

Quoted 8 lines · Ben Greear
Quoted 7 lines · Tom Herbert
Quoted 2 lines

Thanks John for pointers to those. Ben, are you running a kernel with these patches?

I just tested this on my e1000e and igb machine. With these patches, I can no longer reproduce the problem.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

So, please make sure those are queued up for 3.3 stable!

Dave: I think these patches below should go to 3.3 stable.

They are not queued for stable yet as far as I can tell.

I let the Intel developers handle -stable submissions for their drivers.

I don't think this is specific to their drivers.

But, as long as _someone_ is pushing it to stable, it is fine by me.

Thanks,
Ben

Signature · 15 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired
#13 lore raw
Quoted 23 lines · Ben Greear
Quoted 6 lines · David Miller

From: Ben Greear<[email protected]>
Date: Tue, 01 May 2012 14:10:43 -0700

Quoted 3 lines · Ben Greear
Quoted 2 lines · Tom Herbert
Quoted 1 line

Thanks John for pointers to those. Ben, are you running a kernel

with

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

these patches?

I just tested this on my e1000e and igb machine. With these

patches,

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

I can no longer reproduce the problem.

So, please make sure those are queued up for 3.3 stable!

Dave: I think these patches below should go to 3.3 stable.

They are not queued for stable yet as far as I can tell.

I let the Intel developers handle -stable submissions for their drivers.

Quoted 7 lines

I don't think this is specific to their drivers.

But, as long as _someone_ is pushing it to stable, it is fine by me.

Thanks,
Ben

I will take care of the stable submission's Ben.

Cheers,
Jeff

maintainer
#14 lore raw

From: Jeff Kirsher <[email protected]>
Date: Tue, 01 May 2012 15:42:28 -0700

Quoted 1 line

I will take care of the stable submission's Ben.

Actually, Jeff, hold off on that.

I misread Ben's email and didn't see that these were not driver-specific changes.

Therefore, I'll take care of the -stable submissions and you don't need to worry about it.

Thanks everyone.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

#15 lore raw
Quoted 7 lines · David Miller

From: Jeff Kirsher <[email protected]>
Date: Tue, 01 May 2012 15:42:28 -0700

Quoted 1 line

I will take care of the stable submission's Ben.

Actually, Jeff, hold off on that.

Quoted 7 lines

I misread Ben's email and didn't see that these were not driver-specific changes.

Therefore, I'll take care of the -stable submissions and you don't need to worry about it.

Thanks everyone.

Ok, sounds good Dave.

#16 lore raw
In reply to Ben Greear

I had done some work on this and to me it looks like this can only happen if driver does not report bytes_compl and pkts_compl stats correctly.

I will experiment more tomorrow.

-Tushar

Quoted 166 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ben Greear
Sent: Thursday, April 19, 2012 4:27 PM
To: netdev; e1000-devel list; [email protected]
Subject: e1000e tx queue timeout in 3.3.0 (bisected to BQL support for e1000e)

Test case:

Run full duplex traffic (900Mbps rx, 400Mbps tx) UDP traffic (moderate speeds of traffic has issues as well, maybe not as easy to reproduce) reset peer interface
----> tx queue timeout

Apr 19 16:12:48 localhost kernel: e1000e: eth2 NIC Link is Down Apr 19

16:12:48 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 16:12:48 localhost kernel: e1000e: eth3 NIC Link is Down Apr 19 16:12:50 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: Rx/Tx Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE):
eth2: link becomes ready Apr 19 16:12:50 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:50 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready Apr 19 16:12:54 localhost /usr/sbin/irqbalance: Load average increasing, re- enabling all cpus for irq balancing Apr 19 16:12:55 localhost kernel: ---- --------[ cut here ]------------ Apr 19 16:12:55 localhost kernel:

WARNING: at /home/greearb/git/linux-3.3.dev.y/net/sched/sch_generic.c:256
dev_watchdog+0xf4/0x154() Apr 19 16:12:55 localhost kernel: Hardware name:

X7DBU Apr 19 16:12:55 localhost kernel: NETDEV WATCHDOG: eth2 (e1000e):
transmit queue 0 timed out Apr 19 16:12:55 localhost kernel: Modules linked in: xt_CT iptable_raw 8021q garp stp llc veth ppdev parport_pc lp parport fuse macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi lockd w83793 w83627hf hwmon_vid coretemp iTCO_wdt microcode iTCO_vendor_support pcspkr i5k_amb ioatdma i2c_i801 i5000_edac dca edac_core e1000e shpchp uinput sunrpc ipv6 autofs4 floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: nf_nat] Apr 19 16:12:55 localhost kernel: Pid: 0, comm: kworker/0:1 Not tainted 3.2.0- rc2+ #36 Apr 19 16:12:55 localhost kernel: Call Trace:

Apr 19 16:12:55 localhost kernel: <IRQ>  [<ffffffff81042902>]
warn_slowpath_common+0x80/0x98 Apr 19 16:12:55 localhost kernel:
[<ffffffff810429ae>] warn_slowpath_fmt+0x41/0x43 Apr 19 16:12:55 localhost
kernel: [<ffffffff8139f8a3>] dev_watchdog+0xf4/0x154 Apr 19 16:12:55
localhost kernel: [<ffffffff8104d371>] run_timer_softirq+0x16f/0x201 Apr
19 16:12:55 localhost kernel: [<ffffffff8139f7af>] ?
netif_tx_unlock+0x57/0x57 Apr 19 16:12:55 localhost kernel:
[<ffffffff81047e47>] __do_softirq+0x86/0x12f Apr 19 16:12:55 localhost
kernel: [<ffffffff8105d54e>] ? hrtimer_interrupt+0x12b/0x1bd Apr 19
16:12:55 localhost kernel: [<ffffffff8144296c>] call_softirq+0x1c/0x30 Apr
19 16:12:55 localhost kernel: [<ffffffff8100bb75>] do_softirq+0x41/0x7e
Apr 19 16:12:55 localhost kernel: [<ffffffff81047c26>] irq_exit+0x3f/0xbb
Apr 19 16:12:55 localhost kernel: [<ffffffff81021df5>]
smp_apic_timer_interrupt+0x85/0x93
Apr 19 16:12:55 localhost kernel: [<ffffffff814411de>]
apic_timer_interrupt+0x6e/0x80 Apr 19 16:12:55 localhost kernel: <EOI>
[<ffffffff81010b8c>] ? mwait_idle+0x6e/0x8c Apr 19 16:12:55 localhost
kernel: [<ffffffff81010b7f>] ? mwait_idle+0x61/0x8c Apr 19 16:12:55
localhost kernel: [<ffffffff81009e72>] cpu_idle+0x67/0xbe Apr 19 16:12:55
localhost kernel: [<ffffffff81435477>] start_secondary+0x194/0x199 Apr 19
16:12:55 localhost kernel: ---[ end trace e3ca12fc1a8b85da ]--- Apr 19
16:12:55 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19
16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Found oopses: 1
Apr 19 16:12:57 localhost abrt-dump-oops[898]: abrt-dump-oops: Creating

dump directories Apr 19 16:12:57 localhost abrtd: Directory 'oops-2012-04- 19-16:12:57-898-0' creation detected Apr 19 16:12:57 localhost abrt-dump-
oops: Reported 1 kernel oopses to Abrt Apr 19 16:12:57 localhost abrtd:
Can't open file '/var/spool/abrt/oops-2012-04-19-16:12:57-898-0/uid': No such file or directory Apr 19 16:12:57 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0

Apr 19 16:12:57 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-04-19-15:02:13-862-0
Apr 19 16:12:57 localhost abrtd: Deleting dump directory oops-2012-04-19-

16:12:57-898-0 (dup of oops-2012-04-19-15:02:13-862-0), sending dbus signal Apr 19 16:12:58 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:12:58 localhost kernel:
ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Apr 19 16:13:03 localhost /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing Apr 19 16:13:04 localhost kernel: e1000e 0000:08:00.0: eth2: Reset adapter Apr 19 16:13:05 localhost chronyd[1003]:
Selected source 108.59.2.194 Apr 19 16:13:07 localhost kernel: e1000e:
eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Apr 19 16:13:07 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready ....

lspci:

08:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 74
	Region 0: Memory at d8300000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at 3000 [size=32]
	[virtual] Expansion ROM at d8d00000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-

,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000feeff00c Data: 41a3
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-

		LnkCap:	Port #1, Speed 2.5GT/s, Width x2, ASPM L0s L1,

Latency L0 <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap-
ChkEn-
Capabilities: [140 v1] Device Serial Number 00-e0-ed-ff-ff-0c-11-6e Kernel driver in use: e1000e
Kernel modules: e1000e

3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c is the first bad commit commit 3f0cfa3bc11e7f00c9994e0f469cbc0e7da7b00c

Author: Tom Herbert <[email protected]>
Date:   Mon Nov 28 16:33:16 2011 +0000

e1000e: Support for byte queue limits

Changes to e1000e to use byte queue limits.

Signed-off-by Tom Herbert <[email protected]>
Acked-by Eric Dumazet <[email protected]>
Signed-off-by David S. Miller <[email protected]>

:040000 040000 bf3e2ec64fd74253563e1ab39797b27a5f2df3fe
51914e221547b95a989b5c7e9b037c9370fd734e M drivers

Thanks,
Ben

--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html