NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out
Subsystems (2)
We continue to see warnings like this reported against the Fedora kernel for a number of different NICs. I just hit this one myself for the first time on that hardware iirc.
Anything else I can provide ?
Dave
[22347.960097] ------------[ cut here ]------------ [22347.965388] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x258/0x270() [22347.973764] Hardware name: [22347.977748] NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out [22347.985212] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_LOG xt_limit ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 ip6table_filter xt_conntrack ip6_tables nf_conntrack microcode pcspkr r8169 mii nfsd auth_rpcgss nfs_acl lockd sunrpc i915 video backlight i2c_algo_bit drm_kms_helper drm [22348.020948] Pid: 0, comm: swapper/3 Not tainted 3.7.0-rc8+ #5 [22348.027483] Call Trace: [22348.030420] <IRQ> [<ffffffff8104928f>] warn_slowpath_common+0x7f/0xc0 [22348.038029] [<ffffffff81049386>] warn_slowpath_fmt+0x46/0x50 [22348.044710] [<ffffffff814d35a8>] dev_watchdog+0x258/0x270 [22348.051096] [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80 [22348.059408] [<ffffffff8105afea>] call_timer_fn+0x8a/0x330 [22348.065791] [<ffffffff8105af65>] ? call_timer_fn+0x5/0x330 [22348.072276] [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80 [22348.080721] [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80 [22348.089032] [<ffffffff8105b4d4>] run_timer_softirq+0x244/0x380 [22348.095913] [<ffffffff81052a00>] __do_softirq+0xe0/0x3c0 [22348.102204] [<ffffffff810a71d4>] ? tick_program_event+0x24/0x30 [22348.109045] [<ffffffff815e36cc>] call_softirq+0x1c/0x26 [22348.115236] [<ffffffff810043cd>] do_softirq+0x8d/0xc0 [22348.121226] [<ffffffff81052eb5>] irq_exit+0xd5/0xe0 [22348.126882] [<ffffffff815e381b>] smp_apic_timer_interrupt+0x6b/0x98 [22348.134253] [<ffffffff815e302f>] apic_timer_interrupt+0x6f/0x80 [22348.141293] <EOI> [<ffffffff8108bad5>] ? sched_clock_local+0x25/0xa0 [22348.148819] [<ffffffff81343d3d>] ? intel_idle+0xfd/0x160 [22348.155110] [<ffffffff81343d36>] ? intel_idle+0xf6/0x160 [22348.161399] [<ffffffff81474dc9>] cpuidle_enter+0x19/0x20 [22348.167562] [<ffffffff81475462>] cpuidle_idle_call+0xa2/0x5f0 [22348.174346] [<ffffffff8100c59a>] cpu_idle+0xca/0x160 [22348.180235] [<ffffffff81ceead4>] start_secondary+0x251/0x258 [22348.186776] ---[ end trace 62a349a839e1b608 ]--- [22348.261360] r8169 0000:05:00.0 eth1: link up
Quoted 5 lines · Dave Jones
We continue to see warnings like this reported against the Fedora kernel for a number of different NICs. I just hit this one myself for the first time on that hardware iirc.
Anything else I can provide ?
[...]
In general, useful information might include:
- was this preceded by any interface reconfiguration or link changes?
- extended network stats (ethtool -S)
- MDIO register dump (mii-tool -vv) (if the interface has an MDIO PHY)
Having seen this error many times with different causes, I wrote a short summary for the support team here, which (with some references removed) may be generally useful:
---
The watchdog will fire if all these conditions are met:
1. The interface is up
2. A TX queue is stopped (normally because it is full)
3. No packets have been added to the queue in the last 5 seconds
4. The driver has not told the kernel that the device is unable to transmit now (e.g. link is down).
Conditions 2 and 3 together normally mean that the TX queue has been stopped for 5 seconds and therefore that few packets (not necessarily none at all) have been completed in that time. The time taken for individual packets to be completed is *not* considered.
This can happen due to:
a. Driver bug causing conditions 2 and 4 to be true during reconfiguration
b. MAC blocked by a pause frame flood
c. IRQ handling is delayed by a long time (can happen due to excessive serial logging)
d. Firmware bug causes driver to see link as up when it's not e. Hardware fault (always a possibility)
---
Item d should really be expanded to hardware/firmware/software bug.
Ben.
Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Dave Jones <[email protected]> :
Quoted 10 lines
Mezigues :
Quoted 4 lines
Dave Jones <[email protected]> :
[...]Quoted 1 line
Anything else I can provide ?
Please send the XID of the device.
[ 55.576012] r8169 0000:01:00.0 eth0: RTL8168d/8111d at 0xffffc9000037a000, 70:71:bc:63:19:05, XID 081000c0 IRQ 46 [ 55.805446] r8169 0000:05:00.0 eth1: RTL8110s at 0xffffc9000037e000, 00:0f:b5:fb:c3:a6, XID 04000000 IRQ 21The one that seems to be complaining is the 2nd one.
I'll plug one and see if it went more touchy than usual.
I do not see anything obvious at first sight. Reverting aee77e4accbeb2c86b1d294cd84fec4a12dde3bd maybe.
Signature · 1 line
Ueimor