sch_generic warn_on (timed out)
Subsystems (2)
We've recieved quite a few bug reports in Fedora recently concerning this warning in sch_generic..
WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev, drivername, 64), i);
https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?
note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for sch_generic, it doesn't seem that there's anything obvious that needs backporting.
thanks,
Dave
David Miller
maintainerFrom: Dave Jones <[email protected]>
Date: Mon, 11 Jul 2011 16:48:34 -0400
Quoted 6 lines
We've recieved quite a few bug reports in Fedora recently concerning this warning in
sch_generic..WARN_ONCE(1,
KERN_INFO"NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name,netdev_drivername(dev, drivername, 64), i);
Quoted 8 lines
https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?
note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for
sch_generic, it doesn't seem that there's anything obvious that needs backporting.
It means the transmitter stopped sending packets for several seconds.
I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.
Quoted 14 lines · David Miller
From: Dave Jones <[email protected]>
Date: Mon, 11 Jul 2011 16:48:34 -0400Quoted 11 lines
We've recieved quite a few bug reports in Fedora recently concerning this warning in
sch_generic..WARN_ONCE(1,
KERN_INFO"NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name,netdev_drivername(dev, drivername, 64), i);https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?
Quoted 8 lines
Quoted 3 lines
note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for
sch_generic, it doesn't seem that there's anything obvious that needs backporting.It means the transmitter stopped sending packets for several seconds.
I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.
Last time I took a look (on one r8169 NIC), it wasnt clear if this could be a PAUSE problem.
David Miller <[email protected]> :
[...]
Quoted 3 lines
Quoted 3 lines
duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?
The r8169 driver has seen driver bugs, poorly supported hardware, "interesting" hardware, evolving hardware, false alarm reports, crappy maintainership and almost anything one can think of (as long as you do not think about public datasheets or errata).
[...]
Quoted 2 lines
I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.
+1 +1 +1
One could almost split the r8169 problems further by XID, then eventually by platform.
Signature · 1 line
Ueimor
Quoted 7 lines · Dave Jones
WARN_ONCE(1,
KERN_INFO"NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name,netdev_drivername(dev, drivername, 64), i);https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?
neither, probably
Quoted 3 lines
note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for
sch_generic, it doesn't seem that there's anything obvious that needs backporting.
it used to just be a KERN_ERR printk, then it changed it to be a WARN_ONCE in order to trigger kerneloops reports so we knew how many people were getting tx hangs from their hardware.
The bad news is there is never anything useful in the backtrace, besides what driver it is. Users have been trained to send backtrace for panic messages, and in this case it doesn't help very much to identify what the problem was.
If the reports within each driver were able to be traced back to a specific *model* of hardware then that might be useful (particularly for Intel hardware). Maybe the WARN_ONCE should print vendor/device pair so we would at least know the hardware from the panic trace.
If this is happening more frequently on F15 than F14 across multiple pieces of hardware, it may indicate that a kernel/stack change is starting to (ab)use a changed working model that is causing an issue, or that there is an actual kernel issue with locks or interrupts or tx completions that is causing an excessive delay in completion of transmits.
Dave can you query for F14 reports and/or isolate what kernel this started with?
Quoted 3 lines · Jesse
it used to just be a
KERN_ERRprintk, then it changed it to be aWARN_ONCEin order to trigger kerneloops reports so we knew how many people were getting tx hangs from their hardware.
That explains the uptick in reports we've got, as most of them seem to be coming in through abrt (Fedora's kerneloops-like thing).
Quoted 8 lines
If this is happening more frequently on F15 than F14 across multiple pieces of hardware, it may indicate that a kernel/stack change is starting to (ab)use a changed working model that is causing an issue, or that there is an actual kernel issue with locks or interrupts or tx completions that is causing an excessive delay in completion of transmits.
Dave can you query for F14 reports and/or isolate what kernel this started with?
Unfortunatly, F14 lagged behind rebasing for unrelated reasons, and is still stuck on 2.6.35, so we've got quite a delta between that and f15 right now.
A quick search of the F14 bugs didn't show up any similar bugs, but due to the 4 versions we jumped, that's probably not too helpful, as it's probably still just a printk in that old codebase.
We have plans in progress to rebase both releases to something more current, so I'll keep an eye on this to see if it gets worse/better.
thanks,
Dave