Skip to contentSkip to conversation

i.Mx6Quad - eth0: tx queue full!

DISCUSSION Vikram Narayanan 5 participants 13 messages Jan 28, 2013 – Feb 21, 2013

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Vikram Narayanan

author
#0 lore raw

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
[ 2555.026733] Modules linked in:
[ 2555.030598] Backtrace:
[ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c)
[ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
[ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c)
[ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
[ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000
r3:00000009
[ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8)
[ 2555.095535]  r3:8f977000 r2:8049f6d4
[ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c)
[ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204)
[ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
[ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180)
[ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90)
[ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98)
[ 2555.160112]  r4:804ffde0 r3:00000220
[ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64)
[ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
[ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50)
[ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
[ 2555.198639] 3f20:                   8052f150 a0000093 00000000 00000000 80502000 8052ed08
[ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70
[ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
[ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
[ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114)
[ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c)
[ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298)
[ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078)
[ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
[ 2555.277270] eth0: tx queue full!.
[ 2555.288776] eth0: tx queue full!.
[ 2555.293594] eth0: tx queue full!.
[ 2555.297944] eth0: tx queue full!.
[ 2555.302229] eth0: tx queue full!.

...... continues indefinitely and needs a reboot to recover the system. == End log ==

i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface.

I found a similar thread [1], but that solution didn't work for me. Any ideas on why the fec driver is unhappy?

Thanks,
Vikram

[1] https://community.freescale.com/thread/281457

#1 lore raw
Quoted 7 lines · Vikram Narayanan

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out

I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never
happened. The controller should have been reset here, but perhaps a bug with the reset code.
Are you using the mainline kernel, or a version Freescale's kernel.

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

to fec_restart, though it would be better to call fec_enet_tx in fec_timeout and skip the call to fec_restart if it returns some packets.

Quoted 27 lines
[ 2555.026733] Modules linked in:
[ 2555.030598] Backtrace:
[ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from [<803b8494>] (dump_stack+0x18/0x1c)
[ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
[ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>] (warn_slowpath_common+0x54/0x6c)
[ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from [<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
[ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4 r4:8f977000
r3:00000009
[ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from [<80302f64>] (dev_watchdog+0x298/0x2b8)
[ 2555.095535]  r3:8f977000 r2:8049f6d4
[ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>] (call_timer_fn.isra.33+0x2c/0x8c)
[ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from [<8002af48>] (run_timer_softirq+0x1f0/0x204)
[ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
[ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from [<80025750>] (__do_softirq+0xc8/0x180)
[ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>] (irq_exit+0x88/0x90)
[ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>] (handle_IRQ+0x44/0x98)
[ 2555.160112]  r4:804ffde0 r3:00000220
[ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>] (gic_handle_irq+0x30/0x64)
[ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
[ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from [<8000df80>] (__irq_svc+0x40/0x50)
[ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
[ 2555.198639] 3f20:                   8052f150 a0000093 00000000 00000000 80502000 8052ed08
[ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000 80503f7c 80503f80 80503f70
[ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
[ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
[ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>] (cpu_idle+0xcc/0x114)
[ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>] (rest_init+0x64/0x7c)
Quoted 8 lines
[ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>] (start_kernel+0x258/0x298)
[ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>] (0x10008078)
[ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
[ 2555.277270] eth0: tx queue full!.
[ 2555.288776] eth0: tx queue full!.
[ 2555.293594] eth0: tx queue full!.
[ 2555.297944] eth0: tx queue full!.
[ 2555.302229] eth0: tx queue full!.

All the packet have been transmitted but the transmit queue is full, so no more
tx interrupts can happen to replace the previously lost tx interrupt.

I've seen MII interrupts clear the TX interrupt status bit.

Quoted 18 lines

...... continues indefinitely and needs a reboot to recover the system. == End log ==

i.Mx6Quad's MAC is connected to SMSC LAN88730 PHY via an MII interface.

I found a similar thread [1], but that solution didn't work for me. Any ideas on why the fec driver is unhappy?

Thanks,
Vikram

[1] https://community.freescale.com/thread/281457

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author
#2 lore raw
Quoted 10 lines · Troy Kisky
Quoted 8 lines · Vikram Narayanan

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254
dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
Quoted 5 lines

I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never
happened. The controller should have been reset here, but perhaps a bug with the reset code.
Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the same error.

Quoted 4 lines

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

With this there was no improvement.

Quoted 3 lines

to fec_restart, though it would be better to call fec_enet_tx in fec_timeout
and skip the call to fec_restart if it returns some packets.

fec_enet_tx returns void, how do I check for the return packets. I'm sorry, I couldn't get your point here.

Quoted 54 lines
Quoted 46 lines
[ 2555.026733] Modules linked in:
[ 2555.030598] Backtrace:
[ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from
[<803b8494>] (dump_stack+0x18/0x1c)
[ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
[ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>]
(warn_slowpath_common+0x54/0x6c)
[ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from
[<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
[ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4
r4:8f977000
r3:00000009
[ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from
[<80302f64>] (dev_watchdog+0x298/0x2b8)
[ 2555.095535]  r3:8f977000 r2:8049f6d4
[ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>]
(call_timer_fn.isra.33+0x2c/0x8c)
[ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from
[<8002af48>] (run_timer_softirq+0x1f0/0x204)
[ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
[ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from
[<80025750>] (__do_softirq+0xc8/0x180)
[ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>]
(irq_exit+0x88/0x90)
[ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>]
(handle_IRQ+0x44/0x98)
[ 2555.160112]  r4:804ffde0 r3:00000220
[ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>]
(gic_handle_irq+0x30/0x64)
[ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
[ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from
[<8000df80>] (__irq_svc+0x40/0x50)
[ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
[ 2555.198639] 3f20:                   8052f150 a0000093 00000000
00000000 80502000 8052ed08
[ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000
80503f7c 80503f80 80503f70
[ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
[ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
[ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>]
(cpu_idle+0xcc/0x114)
[ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>]
(rest_init+0x64/0x7c)
[ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>]

(start_kernel+0x258/0x298)

Quoted 8 lines
[ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>]
(0x10008078)
[ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
[ 2555.277270] eth0: tx queue full!.
[ 2555.288776] eth0: tx queue full!.
[ 2555.293594] eth0: tx queue full!.
[ 2555.297944] eth0: tx queue full!.
[ 2555.302229] eth0: tx queue full!.
Quoted 6 lines

All the packet have been transmitted but the transmit queue is full, so no more
tx interrupts can happen to replace the previously lost tx interrupt.

I've seen MII interrupts clear the TX interrupt status bit.

Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface.

Regards,
Vikram

#3 lore raw
Quoted 2 lines · Vikram Narayanan

I tried with both the kernels. Freescale's and mainline results in the same error.

Do you also see this issue with a 3.6 or 3.7 stable kernels?

author
#4 lore raw
Quoted 7 lines · Fabio Estevam
Quoted 2 lines · Vikram Narayanan

I tried with both the kernels. Freescale's and mainline results in the same error.

Do you also see this issue with a 3.6 or 3.7 stable kernels?

I didn't try with the 3.6 or 3.7 stable.

author
#5 lore raw
In reply to Troy Kisky
Quoted 10 lines · Troy Kisky
Quoted 8 lines · Vikram Narayanan

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254
dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
Quoted 5 lines

I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never
happened. The controller should have been reset here, but perhaps a bug with the reset code.
Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the same error.

Quoted 4 lines

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

With this there was no improvement.

Quoted 3 lines

to fec_restart, though it would be better to call fec_enet_tx in fec_timeout
and skip the call to fec_restart if it returns some packets.

fec_enet_tx returns void, how do I check for the return packets. I'm sorry, I couldn't get your point here.

Quoted 54 lines
Quoted 46 lines
[ 2555.026733] Modules linked in:
[ 2555.030598] Backtrace:
[ 2555.034252] [<800119c8>] (dump_backtrace+0x0/0x10c) from
[<803b8494>] (dump_stack+0x18/0x1c)
[ 2555.044438]  r6:000000fe r5:80302f64 r4:80503dd0 r3:80510e80
[ 2555.052019] [<803b847c>] (dump_stack+0x0/0x1c) from [<8001df08>]
(warn_slowpath_common+0x54/0x6c)
[ 2555.062679] [<8001deb4>] (warn_slowpath_common+0x0/0x6c) from
[<8001dfc4>] (warn_slowpath_fmt+0x38/0x40)
[ 2555.073936]  r8:8052ebf1 r7:805040c0 r6:00000000 r5:8f9771d4
r4:8f977000
r3:00000009
[ 2555.084816] [<8001df8c>] (warn_slowpath_fmt+0x0/0x40) from
[<80302f64>] (dev_watchdog+0x298/0x2b8)
[ 2555.095535]  r3:8f977000 r2:8049f6d4
[ 2555.099868] [<80302ccc>] (dev_watchdog+0x0/0x2b8) from [<8002acf8>]
(call_timer_fn.isra.33+0x2c/0x8c)
[ 2555.110794] [<8002accc>] (call_timer_fn.isra.33+0x0/0x8c) from
[<8002af48>] (run_timer_softirq+0x1f0/0x204)
[ 2555.122240]  r7:80571114 r6:805040c0 r5:00000000 r4:80570900
[ 2555.129894] [<8002ad58>] (run_timer_softirq+0x0/0x204) from
[<80025750>] (__do_softirq+0xc8/0x180)
[ 2555.140599] [<80025688>] (__do_softirq+0x0/0x180) from [<80025b40>]
(irq_exit+0x88/0x90)
[ 2555.150492] [<80025ab8>] (irq_exit+0x0/0x90) from [<8000ec58>]
(handle_IRQ+0x44/0x98)
[ 2555.160112]  r4:804ffde0 r3:00000220
[ 2555.164848] [<8000ec14>] (handle_IRQ+0x0/0x98) from [<80008540>]
(gic_handle_irq+0x30/0x64)
[ 2555.174956]  r6:80503f28 r5:8050a518 r4:f400010c r3:00000000
[ 2555.182694] [<80008510>] (gic_handle_irq+0x0/0x64) from
[<8000df80>] (__irq_svc+0x40/0x50)
[ 2555.192737] Exception stack(0x80503f28 to 0x80503f70)
[ 2555.198639] 3f20:                   8052f150 a0000093 00000000
00000000 80502000 8052ed08
[ 2555.208600] 3f40: 8050a4f4 803bfaec 8050df00 412fc09a 80502000
80503f7c 80503f80 80503f70
[ 2555.218584] 3f60: 8000eee4 8000eee8 60000013 ffffffff
[ 2555.224730]  r7:80503f5c r6:ffffffff r5:60000013 r4:8000eee8
[ 2555.232292] [<8000eeb8>] (default_idle+0x0/0x38) from [<8000f0d8>]
(cpu_idle+0xcc/0x114)
[ 2555.242204] [<8000f00c>] (cpu_idle+0x0/0x114) from [<803b3718>]
(rest_init+0x64/0x7c)
[ 2555.251858] [<803b36b4>] (rest_init+0x0/0x7c) from [<804cc7dc>]

(start_kernel+0x258/0x298)

Quoted 8 lines
[ 2555.261963] [<804cc584>] (start_kernel+0x0/0x298) from [<10008078>]
(0x10008078)
[ 2555.271167] ---[ end trace 3d2ffb53e6fe41f3 ]---
[ 2555.277270] eth0: tx queue full!.
[ 2555.288776] eth0: tx queue full!.
[ 2555.293594] eth0: tx queue full!.
[ 2555.297944] eth0: tx queue full!.
[ 2555.302229] eth0: tx queue full!.
Quoted 6 lines

All the packet have been transmitted but the transmit queue is full, so no more
tx interrupts can happen to replace the previously lost tx interrupt.

I've seen MII interrupts clear the TX interrupt status bit.

Do you suspect the MAC <-> PHY interconnect? I'm having an MII interface.

Regards,
Vikram

#6 lore raw
Quoted 20 lines · Vikram Narayanan
Quoted 10 lines · Troy Kisky
Quoted 8 lines · Vikram Narayanan

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------
[ 2555.009740] WARNING: at net/sched/sch_generic.c:254
dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
Quoted 5 lines

I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never
happened. The controller should have been reset here, but perhaps a bug with the reset code.
Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the same error.

Quoted 6 lines
Quoted 4 lines

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

With this there was no improvement.

I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline.

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

git://github.com/boundarydevices/linux-imx6.git ethernet_test

Please let me know results.
Thanks
Troy

#7 lore raw

Hi, all

The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU) ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git
branch name: imx_3.0.35

The patch as below:

Quoted 1 line

From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001

From: Fugang Duan <[email protected]>
Date: Thu, 9 Aug 2012 17:59:44 +0800
Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0

Kernel dump when do wifi stress test with suspend and resume as below:

        eth0: tx queue full!.
        remove wake up source irq 103 
        PM: resume of devices complete after 348.934 msecs
        Restarting tasks ... done.
        ------------[ cut here ]------------
        WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x284/0x2a8()
        NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out 
        Modules linked in: ar6000
        [<8004482c>] (unwind_backtrace+0x0/0xf8) from
        [<80068cd0>] (warn_slowpath_common+0x4c/0x64)
        [<80068cd0>] (warn_slowpath_common+0x4c/0x64)from
        [<80068d7c>] (warn_slowpath_fmt+0x30/0x40)
        [<80068d7c>] (warn_slowpath_fmt+0x30/0x40) from
        [<803f0c50>] (dev_watchdog+0x284/0x2a8)
        [<803f0c50>] (dev_watchdog+0x284/0x2a8) from
        [<80074430>] (run_timer_softirq+0xec/0x214)
        [<80074430>] (run_timer_softirq+0xec/0x214) from
        [<8006e524>] (__do_softirq+0xac/0x140)
        [<8006e524>] (__do_softirq+0xac/0x140) from
        [<8006ea60>] (irq_exit+0x94/0x9c)
        [<8006ea60>] (irq_exit+0x94/0x9c) from
        [<80039240>] (do_local_timer+0x54/0x70)
        [<80039240>] (do_local_timer+0x54/0x70) from
        [<8003ea0c>] (__irq_svc+0x4c/0xe8)
        Exception stack(0x80a2bf68 to 0x80a2bfb0)
        bf60:                   0000001f 80a3babc 80a2bfb0 00000000 80a2a000 80a7b8e4
        bf80: 804befcc 80a3ee7c 1000406a 412fc09a 00000000 00000000 80a81440 80a2bfb0
        bfa0: 8003fa64 8003fa68 60000013 ffffffff
        [<8003ea0c>] (__irq_svc+0x4c/0xe8) from [<8003fa68>] (default_idle+0x24/0x28)
        [<8003fa68>] (default_idle+0x24/0x28) from [<8003fc60>] (cpu_idle+0xbc/0xfc)
        [<8003fc60>] (cpu_idle+0xbc/0xfc) from [<80008878>] (start_kernel+0x258/0x29c)
        [<80008878>] (start_kernel+0x258/0x29c) from [<10008040>] (0x10008040)
        ---[ end trace 30671ac42e272c2d ]---

But ethernet and system still be alive. In sometime,the issue will cause system hang like "nfs: server 10.192.242.179 not responding, still trying".

The root cause is tx buffer descriptors are not cleaned when ethernet resume back.

Signed-off-by Fugang Duan <[email protected]>

drivers/net/fec.c | 39 ++++++++++++++++++++++++++-------------

 1 files changed, 26 insertions(+), 13 deletions(-)
drivers/net/fec.c +26 -13
index f007bf0..b1fa464 100755
@@ -1456,6 +1456,28 @@ static const struct net_device_ops fec_netdev_ops = {
 #endif
 };
 
+/* Init TX buffer descriptors
+ */
+static void fec_enet_txbd_init(struct net_device *dev)
+{
+       struct fec_enet_private *fep = netdev_priv(dev);
+       struct bufdesc *bdp;
+       int i;
+
+       /* ...and the same for transmit */
+       bdp = fep->tx_bd_base;
+       for (i = 0; i < TX_RING_SIZE; i++) {
+
+               /* Initialize the BD for every fragment in the page. */
+               bdp->cbd_sc = 0;
+               bdp++;
+       }
+
+       /* Set the last buffer to wrap */
+       bdp--;
+       bdp->cbd_sc |= BD_SC_WRAP;
+}
+
  /*
   * XXX:  We need to clean up on failure exits here.
   *

@@ -1512,19 +1534,8 @@ static int fec_enet_init(struct net_device *ndev)
        bdp--;
        bdp->cbd_sc |= BD_SC_WRAP;
 
-       /* ...and the same for transmit */
-       bdp = fep->tx_bd_base;
-       for (i = 0; i < TX_RING_SIZE; i++) {
-
-               /* Initialize the BD for every fragment in the page. */
-               bdp->cbd_sc = 0;
-               bdp->cbd_bufaddr = 0;
-               bdp++;
-       }
-
-       /* Set the last buffer to wrap */
-       bdp--;
-       bdp->cbd_sc |= BD_SC_WRAP;
+       /* Init transmit descriptors */
+       fec_enet_txbd_init(ndev);
 
        fec_restart(ndev, 0);
 

@@ -1575,6 +1586,8 @@ fec_restart(struct net_device *dev, int duplex)
        writel(fep->bd_dma, fep->hwp + FEC_R_DES_START);
        writel((unsigned long)fep->bd_dma + sizeof(struct bufdesc) * RX_RING_SIZE,
                        fep->hwp + FEC_X_DES_START);
+       /* Reinit transmit descriptors */
+       fec_enet_txbd_init(dev);
 
        fep->dirty_tx = fep->cur_tx = fep->tx_bd_base;
        fep->cur_rx = fep->rx_bd_base;
Signature · 4 lines
1.7.0.4

Thanks,
Andy

-----Original Message-----
From: [email protected] [mailto:netdev-owner at vger.kernel.org] On Behalf Of Troy Kisky
Sent: Wednesday, January 30, 2013 2:47 AM
To: Vikram Narayanan
Cc: netdev at vger.kernel.org; Greg Ungerer; shawn.guo at linaro.org; LAK; Uwe Kleine-K?nig; Fabio Estevam
Subject: Re: i.Mx6Quad - eth0: tx queue full!

Quoted 20 lines · Vikram Narayanan
Quoted 11 lines · Troy Kisky
Quoted 9 lines · Vikram Narayanan

Running the latest head <linux-2.6.git> on an i.Mx6Quad based platform gives me the below error when flooded with ping requests.

== Start log ==

[ 2555.004031] ------------[ cut here ]------------ [ 2555.009740] 
WARNING: at net/sched/sch_generic.c:254
dev_watchdog+0x298/0x2b8()
[ 2555.018721] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed

out

Quoted 4 lines

I think the tx interrupt status bit was lost. The packets were transmitted, but the interrupt never happened. The controller should have been reset here, but perhaps a bug with the reset code. Are you using the mainline kernel, or a version Freescale's kernel.

I tried with both the kernels. Freescale's and mainline results in the same error.

Quoted 6 lines
Quoted 4 lines

mainline fec_restart does not reset tx_full

You can try adding
fep->tx_full = 0;

With this there was no improvement.

I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline.

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

git://github.com/boundarydevices/linux-imx6.git ethernet_test

Please let me know results.
Thanks
Troy

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#8 lore raw

Hello Duan,

Quoted 4 lines · On 1/30/2013 2:07 PM, Duan Fugang-B38611 wrote:

Hi, all

The issue cannot be found kernel 3.0.35 branch: (even run stress test with IPU and VPU) ssh://sw-git01-tx30/git/sw_git/repos/linux-2.6-imx.git

Where is this git located?

Quoted 1 line

branch name: imx_3.0.35

Troy's branch (located in github) works fine for me. I've kept it for a long run with heavy network load. Should I encounter some errors, I'll post back.

Quoted 6 lines

The patch as below:

From 91a0c892263e57ecde9e9ff38be3acdb7f66a17f Mon Sep 17 00:00:00 2001
From: Fugang Duan <[email protected]>
Date: Thu, 9 Aug 2012 17:59:44 +0800
Subject: [PATCH] ENGR00180288 - FEC : Fix kernel dump about eth0

Thanks, I'll give a try with this patch as well.

Regards,
Vikram

author
#9 lore raw
In reply to Troy Kisky
Quoted 3 lines · Troy Kisky

I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline.

Thanks a lot for the branch. I saw a lot of differences between your driver and the mainline. I should spend some time If I had to port this to mainline.

Quoted 5 lines

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

Please let me know results.

When NAPI is disabled I saw a severe packet loss (around 40% in a flood ping) which spitted out this,

[  470.390928] net eth0: missed rxf 2000000 1c000000 0
[  470.419098] net eth0: missed rxf 2000000 1c000000 0
[  470.443800] net eth0: missed rxf 2000000 1c000000 0
[  470.450315] net eth0: missed rxf 2000000 1c000000 0

When NAPI is enabled, it was working good. Though it resulted in the same error, it recovered by itself, which is perfectly fine for me. :)

[  507.359857] ------------[ cut here ]------------
[  507.365841] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x298/0x2c0()
[  507.375655] NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
[  507.384521] Modules linked in:
[  507.388907] [<80033e6c>] (unwind_backtrace+0x0/0xf8) from [<800558d0>] (warn_slowpath_common+0x4c/0x64)
[  507.401045] [<800558d0>] (warn_slowpath_common+0x4c/0x64) from [<8005597c>] (warn_slowpath_fmt+0x30/0x40)
[  507.413387] [<8005597c>] (warn_slowpath_fmt+0x30/0x40) from [<802db42c>] (dev_watchdog+0x298/0x2c0)
[  507.425223] [<802db42c>] (dev_watchdog+0x298/0x2c0) from [<80060ea4>] (run_timer_softirq+0xf4/0x23c)
[  507.437170] [<80060ea4>] (run_timer_softirq+0xf4/0x23c) from [<8005aea0>] (__do_softirq+0x98/0x130)
[  507.449063] [<8005aea0>] (__do_softirq+0x98/0x130) from [<8005b354>] (irq_exit+0x8c/0x94)
[  507.460155] [<8005b354>] (irq_exit+0x8c/0x94) from [<8002f3f0>] (handle_IRQ+0x34/0x84)
[  507.471019] [<8002f3f0>] (handle_IRQ+0x34/0x84) from [<8002e70c>] (__irq_svc+0x4c/0xa4)
[  507.482010] [<8002e70c>] (__irq_svc+0x4c/0xa4) from [<8002f500>] (default_idle+0x24/0x28)
[  507.493222] [<8002f500>] (default_idle+0x24/0x28) from [<8002f6a8>] (cpu_idle+0x84/0xb8)
[  507.504384] [<8002f6a8>] (cpu_idle+0x84/0xb8) from [<80008824>] (start_kernel+0x22c/0x26c)
[  507.515724] [<80008824>] (start_kernel+0x22c/0x26c) from [<1000803c>] (0x1000803c)
[  507.526419] ---[ end trace 1b75b31a2719ed1e ]---
[  507.532613] net eth0: fec_timeout tx_full=1 tx_insert=5a tx_remove=5a 9c00
[  507.542567] net eth0: 12000000 8c00000, 12000000
[  507.548754] net eth0: calling restart
[  509.250736] PHY: 1:00 - Link is Up - 100/Full

Thank you so much for your time and help.

Regards,
Vikram

#10 lore raw
Quoted 16 lines · Vikram Narayanan
Quoted 3 lines · Troy Kisky

I have fixed this bug (and more) on Freescale's kernel (imx-3.0.35_1.1.0). I created a branch you can try. Feel free to port to mainline.

Thanks a lot for the branch. I saw a lot of differences between your driver and the mainline. I should spend some time If I had to port this to mainline.

Quoted 5 lines

This is the patch that should fix your problem
fec: clear TX_FULL in fec_restart

Please let me know results.

When NAPI is disabled I saw a severe packet loss (around 40% in a flood ping) which spitted out this,

Quoted 9 lines
[  470.390928] net eth0: missed rxf 2000000 1c000000 0
[  470.419098] net eth0: missed rxf 2000000 1c000000 0
[  470.443800] net eth0: missed rxf 2000000 1c000000 0
[  470.450315] net eth0: missed rxf 2000000 1c000000 0

When NAPI is enabled, it was working good. Though it resulted in the same error, it recovered by itself, which is perfectly fine for me. :)

You'll also have better performance if you pass "enable_wait_mode=off" in bootargs.

#11 lore raw

Do you have a way to duplicate this issue consistently?

best regards
Frank Li

author
#12 lore raw
Quoted 1 line · Frank Li

Do you have a way to duplicate this issue consistently?

I think you meant "reproduce".

Yes. Almost everytime in my hardware.
Any network load would trigger this.

On the NFS, I did something like
for i in {1..10000};do;ls -lR *;done

and did a flood ping to the target as well.

Hope this helps,
Vikram