sky2: flow control off
Subsystems (2)
PATCH 1/1 sky2: flow control off
Turn flow control off for sky2. When flow control is on, the transmitter may get randomly stuck. Perhaps there is hardware problem, but until Marvell provides errata information for workaround, it should default to off.
--- drivers/net/sky2.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/sky2.c +1 -1
@@ -3263,7 +3263,7 @@ #endif/* Auto speed and flow control */ sky2->autoneg =AUTONEG_ENABLE ; - sky2->flow_mode =FC_BOTH ; + sky2->flow_mode =FC_NONE ; sky2->duplex =-1 ; sky2->speed =-1 ;
Signature · 1 line
1.4.1
Hi Stephen,
Quoted 3 lines · Stephen Hemminger
Turn flow control off for sky2. When flow control is on, the transmitter may get randomly stuck. Perhaps there is hardware problem, but until Marvell provides errata information for workaround, it should default to off.
Are you aware of any way to reproduce or at least to increases chances of occurrences of this problem ? It looks much like a problem I've been experiencing with Marvell's driver with a 88E8053 chip on 2.4. Doing "ethtool -r" proved useful to reset the transceiver in production.
When trying to reproduce the problem, I noticed corrupted and duplicated frames in tcpdump captures when packets between 147 and 496 bytes were received then re-routed via the same NIC towards a 100 Mbps machine on the same switch, a Gig Dlink doing flow control. Yes, I know, there are a lot of variables ! But changing the target machine, or the packet sizes was enough to stop the corruption. I could not get the transmitter stuck as I sometimes have in production, but I think that both problems are related. The fact that a comparable problem is encountered with your driver really makes me think about a hardware bug :-/
BTW, switching to your driver (sky2-1.5), I could not reproduce the corruption problem anymore, but I've not yet put it in production to check if it fixes the transceiver problem. Anyway, it might be interesting that users who encounter this problem try an old version of the driver just in case they detect a difference.
regards,
Willy
Here is what I saw.
The transmitter on the Marvell Yukon II (88e8053) hangs when doing transmit flow control under load. There appears to be a bug or race condition that causes the MAC to stop transmitting data.
There are two drivers for the Yukon II device on Linux. SysKonnect/Marvell has one called sk98lin it is downloadable from syskonnect.def, and I wrote one called sky2 that is part of the standard Linux kernel. This problem is reproducible with the sky2 driver only; the sk98lin driver has a watchdog routine that resets the hardware perodically, so it masks the problem.
When the failure mode occurs only after several minutes of sustained activity and a situation where PAUSE frames would be received. In my testing I used
server == 1000mbit ===> switch --- 100mbit ---> client
Server was Mac Mini (88E8053) running Linux 2.6.20-rc7 and client was a Sony Vaio (88e8036) laptop. The server was running NFS in kernel and client was doing a large copy. The server was using UDP to cause large amounts of 802 pause frames. The problem is not as reproducible with TCP tests because TCP congestion control avoids over running the switch.
When failure occurs:
* packets continue to be received and passed up the stack
* GMAC status register is the pause state
* transmit packets continue transferred by the DMA into the RAM buffer
* when the the RAM buffer fills no more packets are DMA'd
* when transmit queue in driver fills, it gets a watch dog timeout
* switch appears to get confused and other ports hang as well.
During development of the sky2 driver a similar problem was observed on receive if the receive DMA buffer was not 8 byte aligned. For performance reasons, Linux drivers usually offset the Rx buffer by 2 bytes so that the TCP/IP headers are aligned for faster CPU access. If the sky2 Rx buffer was offset, then the receiver DMA would occasionally hung. The workaround for receive was to align the receive buffer on a quad word boundary.
This problem appears to be flow control related because after disabling flow control, no errors occurred in a 48 hour test run.
There probably are other races and hangs that are related. I don't consider all the hangs eliminated yet.
Signature · 1 line
Stephen Hemminger <[email protected]>
Hi Stephen,
First, thanks for this detailed explanation.
Quoted 15 lines · Stephen Hemminger
Here is what I saw.
The transmitter on the Marvell Yukon II (88e8053) hangs when doing transmit flow control under load. There appears to be a bug or race condition that causes the MAC to stop transmitting data.
There are two drivers for the Yukon II device on Linux. SysKonnect/Marvell has one called sk98lin it is downloadable from syskonnect.def, and I wrote one called sky2 that is part of the standard Linux kernel. This problem is reproducible with the sky2 driver only; the sk98lin driver has a watchdog routine that resets the hardware perodically, so it masks the problem.
When the failure mode occurs only after several minutes of sustained activity and a situation where PAUSE frames would be received. In my testing I used
Quoted 7 lines
server == 1000mbit ===> switch --- 100mbit ---> client
Server was Mac Mini (88E8053) running Linux 2.6.20-rc7 and client was a Sony Vaio (88e8036) laptop. The server was running NFS in kernel and client was doing a large copy. The server was using UDP to cause large amounts of 802 pause frames. The problem is not as reproducible with TCP tests because TCP congestion control avoids over running the switch.
I encountered *exactly* this problem with a one-leg firewall equipped with a 88E8053 attached to a 1000 Mbps switch, itself hosting 100 Mbps stations, but with sk98lin (2.4). Running tcpdump on the firewall, I noticed duplicated and corrupted frames. I could only reproduce the duplicated and corrupted frames on a lab setup, not the Tx hangs, by sending high UDP traffic on the port to a 100 Mbps host. Sending to 1000 Mbps hosts never triggered the problem, hence my conclusions about flow control too. What I found interesting is that using a very old version of the sky2 driver which I had with me (sky2 v0.5), I could not trigger the problem anymore. But right now, I realize that this version of the driver did not support flow control yet, which might converge with your observations :
# ethtool -i eth0 driver: sky2 version: 0.5 firmware-version: N/A bus-info: 01:00.0
# ethtool -a eth0 Pause parameters for eth0: Autonegotiate: on RX: off TX: off
Quoted 18 lines
When failure occurs:
* packets continue to be received and passed up the stack* GMAC status register is the pause state
* transmit packets continue transferred by the DMA into the RAM buffer
* when the the RAM buffer fills no more packets are DMA'd
* when transmit queue in driver fills, it gets a watch dog timeout* switch appears to get confused and other ports hang as well.
During development of the sky2 driver a similar problem was observed on receive if the receive DMA buffer was not 8 byte aligned. For performance reasons, Linux drivers usually offset the Rx buffer by 2 bytes so that the TCP/IP headers are aligned for faster CPU access. If the sky2 Rx buffer was offset, then the receiver DMA would occasionally hung. The workaround for receive was to align the receive buffer on a quad word boundary.
Quoted 2 lines
This problem appears to be flow control related because after disabling flow control, no errors occurred in a 48 hour test run.
No problem here with the old driver without flow control either. I can try to disable it right here on my setup with sk98lin, and test again. I did not know that the sk98lin had a watchdog, it could explain why sometimes the system entered a strange state (packets taking *seconds* to be forwarded).
Anyway, I'm more and more convinced that there are hardware bugs. It is not normal at all that both the original syskonnect driver and your fresh new code show such similar problems !
Quoted 2 lines
There probably are other races and hangs that are related. I don't consider all the hangs eliminated yet.
Well, at least you have a more maintainable driver than what was the previous one, so you will eventually manage to fix all problems ;-)
Best regards,
Willy
Quoted 14 lines · Stephen Hemminger
Turn flow control off for sky2. When flow control is on, the transmitter may get randomly stuck. Perhaps there is hardware problem, but until Marvell provides errata information for workaround, it should default to off.
Signed-off-by Stephen Hemminger <[email protected]>--- drivers/net/sky2.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)drivers/net/sky2.c +0
-0
drivers/net/sky2.c #endif · line 3266
Quoted 5 lines
drivers/net/sky2.c +1
-1@@ -3263,7 +3263,7 @@ #endif/* Auto speed and flow control */ sky2->autoneg =AUTONEG_ENABLE ; - sky2->flow_mode =FC_BOTH ; + sky2->flow_mode =FC_NONE ;
I ACK the patch... conditional on some -mm style testing and user ACKs.
Logic: if there were no downsides to disabling flow control globally, the world's networks would have already done so. Flow control can be quite helpful, so I while I understand the errata argument, I also want to understand the full effect of this tiny patch.
Jeff
Quoted 23 lines · On Tue, 06 Feb 2007 19:18:07 -0500 Jeff Garzik <[email protected]> wrote:
Quoted 14 lines · Stephen Hemminger
Turn flow control off for sky2. When flow control is on, the transmitter may get randomly stuck. Perhaps there is hardware problem, but until Marvell provides errata information for workaround, it should default to off.
Signed-off-by Stephen Hemminger <[email protected]>--- drivers/net/sky2.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)drivers/net/sky2.c +0
-0drivers/net/sky2.c #endif · line 3266
Quoted 5 lines
drivers/net/sky2.c +1
-1@@ -3263,7 +3263,7 @@ #endif/* Auto speed and flow control */ sky2->autoneg =AUTONEG_ENABLE ; - sky2->flow_mode =FC_BOTH ; + sky2->flow_mode =FC_NONE ;I ACK the patch... conditional on some -mm style testing and user ACKs.
Quoted 5 lines
Logic: if there were no downsides to disabling flow control globally, the world's networks would have already done so. Flow control can be quite helpful, so I while I understand the errata argument, I also want to understand the full effect of this tiny patch.
Actually, the E1000 had it off until recently. The downside is that if a system is connected on a switch with a gigabit to 100mbit port and using a stupid protocol like NFS over UDP, then the packet burst is sure to get truncated so the 8K fragmented UDP never gets through.