Skip to contentSkip to conversation

sch_generic warn_on (timed out)

DISCUSSION Dave Jones 5 participants 6 messages Jul 11, 2011 – Jul 21, 2011

Subsystems (2)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

We've recieved quite a few bug reports in Fedora recently concerning this warning in sch_generic..

WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev, drivername, 64), i);

https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?

note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for sch_generic, it doesn't seem that there's anything obvious that needs backporting.

thanks,

Dave

maintainer
#1 lore raw

From: Dave Jones <[email protected]>
Date: Mon, 11 Jul 2011 16:48:34 -0400

Quoted 6 lines

We've recieved quite a few bug reports in Fedora recently concerning this warning in sch_generic..

WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev, drivername, 64), i);

Quoted 8 lines

https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?

note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for sch_generic, it doesn't seem that there's anything obvious that needs backporting.

It means the transmitter stopped sending packets for several seconds.

I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.

#2 lore raw
Quoted 14 lines · David Miller

From: Dave Jones <[email protected]>
Date: Mon, 11 Jul 2011 16:48:34 -0400

Quoted 11 lines

We've recieved quite a few bug reports in Fedora recently concerning this warning in sch_generic..

WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev, drivername, 64), i);

https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?

Quoted 8 lines
Quoted 3 lines

note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for sch_generic, it doesn't seem that there's anything obvious that needs backporting.

It means the transmitter stopped sending packets for several seconds.

I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.

Last time I took a look (on one r8169 NIC), it wasnt clear if this could be a PAUSE problem.

#4 lore raw
In reply to Dave Jones
Quoted 7 lines · Dave Jones

WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev, drivername, 64), i);

https://bugzilla.redhat.com/show_bug.cgi?id=702723 is our 'master bug' that we're duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?

neither, probably

Quoted 3 lines

note that I've only been looking through fedora 15 bugs so far (which is still on 2.6.38), but looking at the commit log for sch_generic, it doesn't seem that there's anything obvious that needs backporting.

it used to just be a KERN_ERR printk, then it changed it to be a WARN_ONCE in order to trigger kerneloops reports so we knew how many people were getting tx hangs from their hardware.

The bad news is there is never anything useful in the backtrace, besides what driver it is. Users have been trained to send backtrace for panic messages, and in this case it doesn't help very much to identify what the problem was.

If the reports within each driver were able to be traced back to a specific *model* of hardware then that might be useful (particularly for Intel hardware). Maybe the WARN_ONCE should print vendor/device pair so we would at least know the hardware from the panic trace.

If this is happening more frequently on F15 than F14 across multiple pieces of hardware, it may indicate that a kernel/stack change is starting to (ab)use a changed working model that is causing an issue, or that there is an actual kernel issue with locks or interrupts or tx completions that is causing an excessive delay in completion of transmits.

Dave can you query for F14 reports and/or isolate what kernel this started with?

#3 lore raw
In reply to David Miller

David Miller <[email protected]> :
[...]

Quoted 3 lines
Quoted 3 lines

duping all others against. It seems to be showing up on a variety of different hardware (r8169, atl1c, ipheth, e1000e, 8139too). Do all these drivers need fixing ? or is it just 'crap hardware' ?

The r8169 driver has seen driver bugs, poorly supported hardware, "interesting" hardware, evolving hardware, false alarm reports, crappy maintainership and almost anything one can think of (as long as you do not think about public datasheets or errata).

[...]

Quoted 2 lines

I would track this on a per-device basis if I were you, instead of combining them all into one super-bug.

+1 +1 +1

One could almost split the r8169 problems further by XID, then eventually by platform.

Signature · 1 line
Ueimor
author
#5 lore raw
In reply to Brandeburg, Jesse
Quoted 3 lines · Jesse

it used to just be a KERN_ERR printk, then it changed it to be a WARN_ONCE in order to trigger kerneloops reports so we knew how many people were getting tx hangs from their hardware.

That explains the uptick in reports we've got, as most of them seem to be coming in through abrt (Fedora's kerneloops-like thing).

Quoted 8 lines

If this is happening more frequently on F15 than F14 across multiple pieces of hardware, it may indicate that a kernel/stack change is starting to (ab)use a changed working model that is causing an issue, or that there is an actual kernel issue with locks or interrupts or tx completions that is causing an excessive delay in completion of transmits.

Dave can you query for F14 reports and/or isolate what kernel this started with?

Unfortunatly, F14 lagged behind rebasing for unrelated reasons, and is still stuck on 2.6.35, so we've got quite a delta between that and f15 right now.

A quick search of the F14 bugs didn't show up any similar bugs, but due to the 4 versions we jumped, that's probably not too helpful, as it's probably still just a printk in that old codebase.

We have plans in progress to rebase both releases to something more current, so I'll keep an eye on this to see if it gets worse/better.

thanks,

Dave