2.6.18 forcedeth GSO panic on send

Hello,

I am using an AMD64 box with 32bit userspace / 64bit kernel.

Kernels 2.6.18 and 2.6.18.1 semi-randomly hang when I upload stuff over the net - for example, "svn commit", scp are affected. 2.6.17.11 does not seem to be affected.

Unfortunately even 60-line screen is not big enough to catch whole trace. There are at least two traces, and first scrolls off. I have a photo at
http://busybox.net/~vda/gso_panic/forcedeth_gso_panic.jpg

Something bad is happening here, when kernel tries to send some data:

...
error_exit
skb_over_panic
skb_over_panic
skb_segment
tcp_tso_segment
inet_gso_segment
skb_gso_segment
dev_hard_start_xmit
dev_queue_xmit
...

Looks like it is related to hardware accel in forcedeth. I will try disabling all hw accel.

Please find in attached tarball:

.config
dmesg
ethtool-k
lspci
lspci-v
--
vda

Looks like a network stack bug rather than a driver problem. However, I'd really like to see the first oops including the print out from skb_over_panic.

Could you try booting with pause_on_oops=1 or perhaps use a serial console?

Thanks,

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Sorry for the delay.

With "echo 1 >/proc/sys/kernel/panic_on_oops" I've got what you're requested. See screenshot:

http://busybox.net/~vda/gso_panic/forcedeth_gso_panic2.jpg

--
vda

Thanks!

Please let me know if this patch fixes it:

[NET]: Fix segmentation of linear packets

skb_segment fails to segment linear packets correctly because it tries to write all linear parts of the original skb into each segment. This will always panic as each segment only contains enough space for one MSS.

This was not detected earlier because linear packets should be rare for GSO. In fact it still remains to be seen what exactly created the linear packets that triggered this bug. Basically the only time this should happen is if someone enables GSO emulation on an interface that does not support SG.

The fact that you're triggering this bug at all means that something else has gone wrong. First of all your picture shows a device setting of "lo". This does not tally with the fact that the BUG was triggered by ssh. Do you have any idea why this is the case? Do you have any netfilter rules that might cause this?

If it is indeed lo, could you please check the ethtool -k setting on it? Also what is the ethtool -k setting on the interface where you expect ssh to go out?

Cheers,

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--

index 3c23760..f735455 100644
@@ -1946,7 +1946,7 @@ struct sk_buff *skb_segment(struct sk_bu
 	do {
 		struct sk_buff *nskb;
 		skb_frag_t *frag;
-		int hsize, nsize;
+		int hsize;
 		int k;
 		int size;
 

@@ -1957,11 +1957,10 @@ struct sk_buff *skb_segment(struct sk_bu
 		hsize = skb_headlen(skb) - offset;
 		if (hsize < 0)
 			hsize = 0;
-		nsize = hsize + doffset;
-		if (nsize > len + doffset || !sg)
-			nsize = len + doffset;
+		if (hsize > len || !sg)
+			hsize = len;
 
-		nskb = alloc_skb(nsize + headroom, GFP_ATOMIC);
+		nskb = alloc_skb(hsize + doffset + headroom, GFP_ATOMIC);
 		if (unlikely(!nskb))
 			goto err;
 

Okay, will test now.

Yes, I have netfilter rules which redirect all outgoing port 22 connections to local port 2222. I run tcpserver on 2222 which spawns hose for each connection.

All this ugliness is required to get ssh working across buggy D-Link router. It chokes on nagle-disabled sessions, I think.

The rules:

# iptables -t nat -N redir22
# iptables -t nat -A redir22 -d 192.168.1.111 -j RETURN
# iptables -t nat -A redir22 -d 127.0.0.1 -j RETURN
# iptables -t nat -A redir22 --match owner --uid-owner daemon -j RETURN
# iptables -t nat -A redir22 -p tcp -j REDIRECT --to 2222
# iptables -t nat -A OUTPUT -p tcp --dport 22 -j redir22

---FILTER--
Chain INPUT (policy ACCEPT 20 packets, 1360 bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 2 packets, 100 bytes) ---NAT-----
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) Chain POSTROUTING (policy ACCEPT 1 packets, 60 bytes) Chain OUTPUT (policy ACCEPT 1 packets, 60 bytes)

       0        0 redir22    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22
Chain redir22 (1 references)
       0        0 RETURN     all  --  *      *       0.0.0.0/0            192.168.1.111
       0        0 RETURN     all  --  *      *       0.0.0.0/0            127.0.0.1
       0        0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           OWNER UID match 50
       0        0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           redir ports 2222

---MANGLE--
Chain PREROUTING (policy ACCEPT 2 packets, 100 bytes) Chain INPUT (policy ACCEPT 2 packets, 100 bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 2 packets, 100 bytes) Chain POSTROUTING (policy ACCEPT 2 packets, 100 bytes)

On port 2222 I have the following running:

run:
====
ip=0.0.0.0
port=2222
user=daemon
maxconn=100

# envuidgid $user tcpserver -U:
# run tcpserver as $user
exec \
env - \
softlimit \
envuidgid $user \
tcpserver \
    -U \
    -v -R -H -l 0 -c $maxconn \
    $ip $port \
./startproxy22

startproxy22
============
targetip="$TCPORIGDSTIP"
targetport=22
# Loop prevention
if test "$TCPLOCALIP" = "$targetip"; then
if test "$TCPLOCALPORT" = "$targetport"; then
env >loop.detected
exit 0
fi
fi

# netcat rulez... however hose of netpipes fame
# is still better because --netslave properly closes
# connection on input EOF.
exec env - hose "$targetip" "$targetport" --netslave

# ethtool -k lo
Offload parameters for lo:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
--
vda

Patch seems to fix the issue. Thanks.

# ethtool -k if
Offload parameters for if:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
--
vda