Skip to contentSkip to conversation

forcedeth driver hangs under heavy load

DISCUSSION Ben Hutchings 7 participants 29 messages Apr 10, 2010 – Apr 14, 2010 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#5 lore raw
In reply to a message not in the archive

Stephen Mulcahy reported a regression in forcedeth at
<http://bugs.debian.org/572201>. The system information and some diagnostic information can be found there. Anyone able to help?

Ben.

Quoted 13 lines · stephen mulcahy

When running linux-image-2.6.32-trunk-amd64, the network stops responding if large amounts of traffic are transmitted/received. Running ifdown eth0 followed by ifup eth0 restores operation of the network. There are no errors relating to this failure logged in /var/log that I could see.

Downgrading to linux-image-2.6.30-2-amd64 results in a stable network. Not sure if this is a forcedeth specific problem or a general problem in the newer kernel (I have seen problems with forcedeth on other distro/kernel combinations).

Happy to run further diagnostics to tie this down if you let me know what to run.

Signature · 2 lines
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
#6 lore raw

Bug#572201: forcedeth driver hangs under heavy load

Quoted 3 lines · Ben Hutchings

Stephen Mulcahy reported a regression in forcedeth at
<http://bugs.debian.org/572201>. The system information and some diagnostic information can be found there. Anyone able to help?

Incidentally, I also tried the 2.6.33.2 kernel with CONFIG_FORCEDETH_NAPI set to "y" to see if that made a difference.

It doesn't - further testing over the weekend saw 6 of 45 machines drop off the network with this problem. Nothing in dmesg or system logs. Happy to run more tests if someone can advise on what should be run.

-stephen

Signature · 2 lines
Stephen Mulcahy     Atlantic Linux         http://www.atlanticlinux.ie
Registered in Ireland, no. 376591 (144 Ros Caoin, Roscam, Galway)
#7 lore raw

Bug#572201: forcedeth driver hangs under heavy load

Quoted 1 line · stephen mulcahy

It doesn't - further testing over the weekend saw 6 of 45 machines drop off the network with this problem. Nothing in dmesg or system logs. Happy to run more tests if someone can advise on what should be run.

I also just tried using the 2.6.30-2-amd64 (Debian) forcedeth kernel module while running the 2.6.32-3-amd64 (Debian) kernel and experienced the same symptoms.

Not sure if thats any help.

-stephen

#8 lore raw

Bug#572201: forcedeth driver hangs under heavy load

Quoted 11 lines · stephen mulcahy
Quoted 3 lines · stephen mulcahy

It doesn't - further testing over the weekend saw 6 of 45 machines drop off the network with this problem. Nothing in dmesg or system logs. Happy to run more tests if someone can advise on what should be run.

I also just tried using the 2.6.30-2-amd64 (Debian) forcedeth kernel module while running the 2.6.32-3-amd64 (Debian) kernel and experienced the same symptoms.

Not sure if thats any help.

I am not sure I understand. Are you saying that using 2.6.30-2-amd64 kernel also makes your forcedeth adapter being not functional ?

Are both way non functional (RX and TX), or only one side ?

Signature · 2 lines
To UNSUBSCRIBE, email to [email protected]
with a subject of "unsubscribe". Trouble? Contact [email protected]
#9 lore raw
Quoted 3 lines · Eric Dumazet

Le lundi 12 avril 2010 à 13:39 +0100, stephen mulcahy a écrit :
I am not sure I understand. Are you saying that using 2.6.30-2-amd64 kernel also makes your forcedeth adapter being not functional ?

Hi Eric,

If I run my tests with the 2.6.30-2-amd64 kernel the network doesn't malfunction.

If I run my tests with the 2.6.32-3-amd64 kernel the network does malfunction.

If I take the forcedeth.ko module from the 2.6.30-2-amd64 kernel and drop that into /lib/modules/2.6.32-3-amd64/kernel/drivers/net/ and then reboot to 2.6.32-3-amd64 and rerun my tests - the network does malfunction.

Quoted 1 line

Are both way non functional (RX and TX), or only one side ?

Whats the best way of testing this? (tcpdump listening on both hosts and then running pings between the systems?)

-stephen

#10 lore raw
Quoted 3 lines · stephen mulcahy
Quoted 1 line

Are both way non functional (RX and TX), or only one side ?

Whats the best way of testing this? (tcpdump listening on both hosts and then running pings between the systems?)

Quoted 4 lines · stephen mulcahy
Quoted 1 line

Are both way non functional (RX and TX), or only one side ?

Whats the best way of testing this? (tcpdump listening on both hosts and then running pings between the systems?)

On one of the nodes that is in the malfunctioning state (node05), I ran

ssh node20

and grabbed the following output from running tcpdump on node20

root@node20:~# tcpdump host node20 and node05
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:12:59.612626 IP node05.webstar.cnet.36295 > node20.ssh: Flags [S], seq 3677858646, win 5840, options [mss 1460,sackOK,TS val 1599534 ecr 0,nop,wscale 7], length 0
14:12:59.612656 IP node20.ssh > node05.webstar.cnet.36295: Flags [S.], seq 3610575850, ack 3677858647, win 5792, options [mss 1460,sackOK,TS val 1598775 ecr 1599534,nop,wscale 7], length 0
14:12:59.612718 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 1, win 46, options [nop,nop,TS val 1599534 ecr 1598775], length 0
14:12:59.617434 IP node20.ssh > node05.webstar.cnet.36295: Flags [P.], seq 1:33, ack 1, win 46, options [nop,nop,TS val 1598776 ecr 1599534], length 32
14:12:59.617522 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 33, win 46, options [nop,nop,TS val 1599535 ecr 1598776], length 0
14:12:59.617609 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1599535 ecr 1598776], length 32
14:12:59.820434 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294936586:4294936618, ack 2620194849, win 46, options [nop,nop,TS val 1599586 ecr 1598776], length 32
14:13:00.229069 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294961734:4294961766, ack 3928358945, win 46, options [nop,nop,TS val 1599688 ecr 1598776], length 32
14:13:01.044396 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294964167:4294964199, ack 410320929, win 46, options [nop,nop,TS val 1599892 ecr 1598776], length 32
14:13:02.676308 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1600300 ecr 1598776], length 32
14:13:05.940804 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 17294:17326, ack 3045851169, win 46, options [nop,nop,TS val 1601116 ecr 1598776], length 32
14:13:12.468484 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 17294:17326, ack 3045851169, win 46, options [nop,nop,TS val 1602748 ecr 1598776], length 32
14:13:23.846891 IP node20.ssh > node05.webstar.cnet.36084: Flags [F.], seq 2093054475, ack 2175389538, win 46, options [nop,nop,TS val 1604834 ecr 1575591], length 0
14:13:23.847278 IP node05.webstar.cnet.36084 > node20.ssh: Flags [R], seq 2175389538, win 0, length 0
14:13:25.523850 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1606012 ecr 1598776], length 32
14:13:50.127509 IP node20.ssh > node05.webstar.cnet.36143: Flags [F.], seq 2526196657, ack 2590340885, win 46, options [nop,nop,TS val 1611404 ecr 1582161], length 0
14:13:50.127879 IP node05.webstar.cnet.36143 > node20.ssh: Flags [R], seq 2590340885, win 0, length 0
14:13:51.633934 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294963190:4294963222, ack 9830433, win 46, options [nop,nop,TS val 1612540 ecr 1598776], length 32
14:13:55.125525 ARP, Request who-has node05.webstar.cnet tell node20, length 28
14:13:55.125886 ARP, Reply node05.webstar.cnet is-at 00:30:48:ce:dc:02 (oui Unknown), length 46
14:14:43.855380 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1625596 ecr 1598776], length 32
14:14:48.855143 ARP, Request who-has node20 tell node05.webstar.cnet, length 46
14:14:48.855469 ARP, Reply node20 is-at 00:30:48:ce:de:34 (oui Unknown), length 28
14:14:59.617675 IP node20.ssh > node05.webstar.cnet.36295: Flags [F.], seq 33, ack 1, win 46, options [nop,nop,TS val 1628777 ecr 1599535], length 0
14:14:59.618202 IP node05.webstar.cnet.36295 > node20.ssh: Flags [FP.], seq 4294959654:4294960446, ack 3930456098, win 46, options [nop,nop,TS val 1629536 ecr 1628777], length 792
14:14:59.821527 IP node20.ssh > node05.webstar.cnet.36295: Flags [F.], seq 33, ack 1, win 46, options [nop,nop,TS val 1628828 ecr 1599535], length 0
14:14:59.821598 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 34, win 46, options [nop,nop,TS val 1629587 ecr 1628828,nop,nop,sack 1 {33:34}], length 0
^C^
27 packets captured
31 packets received by filter
0 packets dropped by kernel

I then did ifdown and ifup on node05 and again ran

ssh node20

and grabbed the following output from running tcpdump on node20

root@node20:~# tcpdump host node20 and node05
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:15:50.626410 IP node05.webstar.cnet.36690 > node20.ssh: Flags [S], seq 2044900531, win 5840, options [mss 1460,sackOK,TS val 1642289 ecr 0,nop,wscale 7], length 0
14:15:50.626441 IP node20.ssh > node05.webstar.cnet.36690: Flags [S.], seq 1976694445, ack 2044900532, win 5792, options [mss 1460,sackOK,TS val 1641529 ecr 1642289,nop,wscale 7], length 0
14:15:50.626482 IP node05.webstar.cnet.36690 > node20.ssh: Flags [.], ack 1, win 46, options [nop,nop,TS val 1642289 ecr 1641529], length 0
14:15:50.631138 IP node20.ssh > node05.webstar.cnet.36690: Flags [P.], seq 1:33, ack 1, win 46, options [nop,nop,TS val 1641530 ecr 1642289], length 32
14:15:50.631218 IP node05.webstar.cnet.36690 > node20.ssh: Flags [.], ack 33, win 46, options [nop,nop,TS val 1642290 ecr 1641530], length 0
14:15:50.631267 IP node05.webstar.cnet.36690 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1642290 ecr 1641530], length 32
14:15:50.631281 IP node20.ssh > node05.webstar.cnet.36690: Flags [.], ack 33, win 46, options [nop,nop,TS val 1641530 ecr 1642290], length 0
14:15:50.631367 IP node05.webstar.cnet.36690 > node20.ssh: Flags [P.], seq 33:825, ack 33, win 46, options [nop,nop,TS val 1642290 ecr 1641530], length 792
14:15:50.631376 IP node20.ssh > node05.webstar.cnet.36690: Flags [.], ack 825, win 58, options [nop,nop,TS val 1641530 ecr 1642290], length 0
14:15:50.631808 IP node20.ssh > node05.webstar.cnet.36690: Flags [P.], seq 33:817, ack 825, win 58, options [nop,nop,TS val 1641530 ecr 1642290], length 784
14:15:50.631950 IP node05.webstar.cnet.36690 > node20.ssh: Flags [P.], seq 825:849, ack 817, win 58, options [nop,nop,TS val 1642290 ecr 1641530], length 24
14:15:50.633353 IP node20.ssh > node05.webstar.cnet.36690: Flags [P.], seq 817:969, ack 849, win 58, options [nop,nop,TS val 1641530 ecr 1642290], length 152
14:15:50.633932 IP node05.webstar.cnet.36690 > node20.ssh: Flags [P.], seq 849:993, ack 969, win 71, options [nop,nop,TS val 1642291 ecr 1641530], length 144
14:15:50.637998 IP node20.ssh > node05.webstar.cnet.36690: Flags [P.], seq 969:1689, ack 993, win 70, options [nop,nop,TS val 1641532 ecr 1642291], length 720
14:15:50.676465 IP node05.webstar.cnet.36690 > node20.ssh: Flags [.], ack 1689, win 83, options [nop,nop,TS val 1642302 ecr 1641532], length 0
14:16:09.776134 IP node05.webstar.cnet.49671 > node20.50060: Flags [S], seq 2348078217, win 5840, options [mss 1460,sackOK,TS val 1647077 ecr 0,nop,wscale 7], length 0
14:16:09.776498 IP node20.50060 > node05.webstar.cnet.49671: Flags [R.], seq 0, ack 2348078218, win 0, length 0
^C
17 packets captured
21 packets received by filter
0 packets dropped by kernel

Does that help?

#11 lore raw
Quoted 1 line · stephen mulcahy

Does that help?

Well, yes, because it seems a TCP problem.

root@node20:~# tcpdump host node20 and node05
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
14:12:59.612626 IP node05.webstar.cnet.36295 > node20.ssh: Flags [S], seq 3677858646, win 5840, options [mss 1460,sackOK,TS val 1599534 ecr 0,nop,wscale 7], length 0
14:12:59.612656 IP node20.ssh > node05.webstar.cnet.36295: Flags [S.], seq 3610575850, ack 3677858647, win 5792, options [mss 1460,sackOK,TS val 1598775 ecr 1599534,nop,wscale 7], length 0
14:12:59.612718 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 1, win 46, options [nop,nop,TS val 1599534 ecr 1598775], length 0
14:12:59.617434 IP node20.ssh > node05.webstar.cnet.36295: Flags [P.], seq 1:33, ack 1, win 46, options [nop,nop,TS val 1598776 ecr 1599534], length 32
14:12:59.617522 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 33, win 46, options [nop,nop,TS val 1599535 ecr 1598776], length 0
14:12:59.617609 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1599535 ecr 1598776], length 32

All following xmitted frames are completely out of sync, this makes no sense.

Sequence number went backward.

14:12:59.820434 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294936586:4294936618, ack 2620194849, win 46, options [nop,nop,TS val 1599586 ecr 1598776], length 32 14:13:00.229069 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294961734:4294961766, ack 3928358945, win 46, options [nop,nop,TS val 1599688 ecr 1598776], length 32 14:13:01.044396 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294964167:4294964199, ack 410320929, win 46, options [nop,nop,TS val 1599892 ecr 1598776], length 32

14:13:02.676308 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1600300 ecr 1598776], length 32 14:13:05.940804 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 17294:17326, ack 3045851169, win 46, options [nop,nop,TS val 1601116 ecr 1598776], length 32 14:13:12.468484 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 17294:17326, ack 3045851169, win 46, options [nop,nop,TS val 1602748 ecr 1598776], length 32 14:13:25.523850 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1606012 ecr 1598776], length 32 14:13:51.633934 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 4294963190:4294963222, ack 9830433, win 46, options [nop,nop,TS val 1612540 ecr 1598776], length 32 14:14:43.855380 IP node05.webstar.cnet.36295 > node20.ssh: Flags [P.], seq 1:33, ack 33, win 46, options [nop,nop,TS val 1625596 ecr 1598776], length 32 14:14:59.617675 IP node20.ssh > node05.webstar.cnet.36295: Flags [F.], seq 33, ack 1, win 46, options [nop,nop,TS val 1628777 ecr 1599535], length 0 14:14:59.618202 IP node05.webstar.cnet.36295 > node20.ssh: Flags [FP.], seq 4294959654:4294960446, ack 3930456098, win 46, options [nop,nop,TS val 1629536 ecr 1628777], length 792 14:14:59.821527 IP node20.ssh > node05.webstar.cnet.36295: Flags [F.], seq 33, ack 1, win 46, options [nop,nop,TS val 1628828 ecr 1599535], length 0 14:14:59.821598 IP node05.webstar.cnet.36295 > node20.ssh: Flags [.], ack 34, win 46, options [nop,nop,TS val 1629587 ecr 1628828,nop,nop,sack 1 {33:34}], length 0

Do you have some netfilters rules ?

#12 lore raw
Quoted 4 lines · Eric Dumazet

Le lundi 12 avril 2010 à 14:19 +0100, stephen mulcahy a écrit :

Do you have some netfilters rules ?

Hi Eric,

I don't have any netfilters rules:

root@node34:~# for table in filter nat mangle raw; do iptables -t $table -L; done
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

I re-ran this on the 2.6.32 kernel (with the 2.6.32 forcedeth module) just in case that was screwing something up.

node33 is in the unresponsive state this time. I'm running tcpdump on node34. on node33 I try to ssh to node34 (using ip address of node34). I note that I can ping between node33 and node34.

root@node34:~# tcpdump -v host node34 and node33
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
17:05:19.622384 IP (tos 0x0, ttl 64, id 21435, offset 0, flags [DF], proto TCP (6), length 60)
    node33.webstar.cnet.43653 > node34.ssh: Flags [S], cksum 0xb994 
(correct), seq 1675314077, win 5840, options [mss 1460,sackOK,TS val 331814 ecr 0,nop,wscale 7], length 0
17:05:19.622754 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    node34.ssh > node33.webstar.cnet.43653: Flags [S.], cksum 0x9d81 
(correct), seq 1669769379, ack 1675314078, win 5792, options [mss 1460,sackOK,TS val 331779 ecr 331814,nop,wscale 7], length 0
17:05:19.622813 IP (tos 0x0, ttl 64, id 21436, offset 0, flags [DF], proto TCP (6), length 52)
    node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xe2bf 
(correct), ack 1, win 46, options [nop,nop,TS val 331814 ecr 331779], length 0
17:05:19.627666 IP (tos 0x0, ttl 64, id 47271, offset 0, flags [DF], proto TCP (6), length 84)
    node34.ssh > node33.webstar.cnet.43653: Flags [P.], seq 1:33, ack 
1, win 46, options [nop,nop,TS val 331780 ecr 331814], length 32
17:05:19.627748 IP (tos 0x0, ttl 64, id 21437, offset 0, flags [DF], proto TCP (6), length 52)
    node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xe29c 
(correct), ack 33, win 46, options [nop,nop,TS val 331816 ecr 331780], length 0
17:05:19.627833 IP (tos 0x0, ttl 64, id 21438, offset 0, flags [DF], proto TCP (6), length 84, bad cksum 1f8a (->d189)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
23413:23445, ack 2749038625, win 46, options [nop,nop,TS val 331816 ecr 331780], length 32
17:05:19.831634 IP (tos 0x0, ttl 64, id 21439, offset 0, flags [DF], proto TCP (6), length 84, bad cksum d189 (->d188)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 331867 ecr 331780], length 32
17:05:20.239603 IP (tos 0x0, ttl 64, id 21440, offset 0, flags [DF], proto TCP (6), length 84, bad cksum 15c6 (->d187)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
30492:30524, ack 809893921, win 46, options [nop,nop,TS val 331969 ecr 331780], length 32
17:05:21.055534 IP (tos 0x0, ttl 64, id 21441, offset 0, flags [DF], proto TCP (6), length 84, bad cksum d187 (->d186)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 332173 ecr 331780], length 32
17:05:22.687386 IP (tos 0x0, ttl 64, id 21442, offset 0, flags [DF], proto TCP (6), length 84, bad cksum d186 (->d185)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 332581 ecr 331780], length 32
17:05:25.950935 IP (tos 0x0, ttl 64, id 21443, offset 0, flags [DF], proto TCP (6), length 84, bad cksum 15c4 (->d184)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
30492:30524, ack 809893921, win 46, options [nop,nop,TS val 333397 ecr 331780], length 32
17:05:32.478527 IP (tos 0x0, ttl 64, id 21444, offset 0, flags [DF], proto TCP (6), length 84, bad cksum c01 (->d183)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
43997:44029, ack 1311047713, win 46, options [nop,nop,TS val 335029 ecr 331780], length 32
17:05:45.533370 IP (tos 0x0, ttl 64, id 21445, offset 0, flags [DF], proto TCP (6), length 84, bad cksum 23d (->d182)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 3348:3380, 
ack 4054450209, win 46, options [nop,nop,TS val 338293 ecr 331780], length 32
17:06:08.719187 IP (tos 0x0, ttl 64, id 27660, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 5360 (->b3b3)!)
    node33.webstar.cnet.50060 > node34.35725: Flags [.], seq 
1203473738:1203475186, ack 1191452767, win 54, options [nop,nop,TS val 344089 ecr 256770], length 1448
17:06:11.643080 IP (tos 0x0, ttl 64, id 21446, offset 0, flags [DF], proto TCP (6), length 84, bad cksum e4f2 (->d181)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
47331:47363, ack 4110811169, win 46, options [nop,nop,TS val 344821 ecr 331780], length 32
17:06:13.715233 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has node34 tell node33.webstar.cnet, length 46
17:06:13.715257 ARP, Ethernet (len 6), IPv4 (len 4), Reply node34 is-at 00:30:48:f0:06:72 (oui Unknown), length 28
17:07:03.866492 IP (tos 0x0, ttl 64, id 21447, offset 0, flags [DF], proto TCP (6), length 84, bad cksum b413 (->d180)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
28939:28971, ack 1913782305, win 46, options [nop,nop,TS val 357877 ecr 331780], length 32
17:07:08.862055 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has node34 tell node33.webstar.cnet, length 46
17:07:08.862370 ARP, Ethernet (len 6), IPv4 (len 4), Reply node34 is-at 00:30:48:f0:06:72 (oui Unknown), length 28
17:07:19.627910 IP (tos 0x0, ttl 64, id 47272, offset 0, flags [DF], proto TCP (6), length 52)
    node34.ssh > node33.webstar.cnet.43653: Flags [F.], cksum 0x6d6b 
(correct), seq 33, ack 1, win 46, options [nop,nop,TS val 361780 ecr 331816], length 0
17:07:19.628403 IP (tos 0x0, ttl 64, id 21448, offset 0, flags [DF], proto TCP (6), length 844, bad cksum aa4d (->ce87)!)
    node33.webstar.cnet.43653 > node34.ssh: Flags [FP.], seq 
20399:21191, ack 2356871202, win 46, options [nop,nop,TS val 361818 ecr 361780], length 792
17:07:19.833456 IP (tos 0x0, ttl 64, id 47273, offset 0, flags [DF], proto TCP (6), length 52)
    node34.ssh > node33.webstar.cnet.43653: Flags [F.], cksum 0x6d37 
(correct), seq 33, ack 1, win 46, options [nop,nop,TS val 361832 ecr 331816], length 0
17:07:19.833517 IP (tos 0x0, ttl 64, id 21449, offset 0, flags [DF], proto TCP (6), length 64)
    node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xa5e9 
(correct), ack 34, win 46, options [nop,nop,TS val 361870 ecr 361832,nop,nop,sack 1 {33:34}], length 0

At this point, I see a "Connection closed by 10.141.0.34" message on node33 (from where I am attempting to ssh).

Again, if I ifdown on node33 and ifup again - I can then see from node33 to node34 without problems.

-stephen

#13 lore raw
Quoted 164 lines · stephen mulcahy
Quoted 4 lines · Eric Dumazet

Le lundi 12 avril 2010 à 14:19 +0100, stephen mulcahy a écrit :

Do you have some netfilters rules ?

Hi Eric,

I don't have any netfilters rules:

root@node34:~# for table in filter nat mangle raw; do iptables -t $table 
-L; done
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

I re-ran this on the 2.6.32 kernel (with the 2.6.32 forcedeth module) just in case that was screwing something up.

node33 is in the unresponsive state this time. I'm running tcpdump on node34. on node33 I try to ssh to node34 (using ip address of node34). I note that I can ping between node33 and node34.

root@node34:~# tcpdump -v host node34 and node33
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 
bytes
17:05:19.622384 IP (tos 0x0, ttl 64, id 21435, offset 0, flags [DF], 
proto TCP (6), length 60)
     node33.webstar.cnet.43653 > node34.ssh: Flags [S], cksum 0xb994 
(correct), seq 1675314077, win 5840, options [mss 1460,sackOK,TS val 
331814 ecr 0,nop,wscale 7], length 0
17:05:19.622754 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 
TCP (6), length 60)
     node34.ssh > node33.webstar.cnet.43653: Flags [S.], cksum 0x9d81 
(correct), seq 1669769379, ack 1675314078, win 5792, options [mss 
1460,sackOK,TS val 331779 ecr 331814,nop,wscale 7], length 0
17:05:19.622813 IP (tos 0x0, ttl 64, id 21436, offset 0, flags [DF], 
proto TCP (6), length 52)
     node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xe2bf 
(correct), ack 1, win 46, options [nop,nop,TS val 331814 ecr 331779], 
length 0
17:05:19.627666 IP (tos 0x0, ttl 64, id 47271, offset 0, flags [DF], 
proto TCP (6), length 84)
     node34.ssh > node33.webstar.cnet.43653: Flags [P.], seq 1:33, ack 
1, win 46, options [nop,nop,TS val 331780 ecr 331814], length 32
17:05:19.627748 IP (tos 0x0, ttl 64, id 21437, offset 0, flags [DF], 
proto TCP (6), length 52)
     node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xe29c 
(correct), ack 33, win 46, options [nop,nop,TS val 331816 ecr 331780], 
length 0
17:05:19.627833 IP (tos 0x0, ttl 64, id 21438, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum 1f8a (->d189)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
23413:23445, ack 2749038625, win 46, options [nop,nop,TS val 331816 ecr 
331780], length 32
17:05:19.831634 IP (tos 0x0, ttl 64, id 21439, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum d189 (->d188)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 331867 ecr 331780], length 32
17:05:20.239603 IP (tos 0x0, ttl 64, id 21440, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum 15c6 (->d187)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
30492:30524, ack 809893921, win 46, options [nop,nop,TS val 331969 ecr 
331780], length 32
17:05:21.055534 IP (tos 0x0, ttl 64, id 21441, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum d187 (->d186)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 332173 ecr 331780], length 32
17:05:22.687386 IP (tos 0x0, ttl 64, id 21442, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum d186 (->d185)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 1:33, ack 
33, win 46, options [nop,nop,TS val 332581 ecr 331780], length 32
17:05:25.950935 IP (tos 0x0, ttl 64, id 21443, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum 15c4 (->d184)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
30492:30524, ack 809893921, win 46, options [nop,nop,TS val 333397 ecr 
331780], length 32
17:05:32.478527 IP (tos 0x0, ttl 64, id 21444, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum c01 (->d183)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
43997:44029, ack 1311047713, win 46, options [nop,nop,TS val 335029 ecr 
331780], length 32
17:05:45.533370 IP (tos 0x0, ttl 64, id 21445, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum 23d (->d182)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 3348:3380, 
ack 4054450209, win 46, options [nop,nop,TS val 338293 ecr 331780], 
length 32
17:06:08.719187 IP (tos 0x0, ttl 64, id 27660, offset 0, flags [DF], 
proto TCP (6), length 1500, bad cksum 5360 (->b3b3)!)
     node33.webstar.cnet.50060 > node34.35725: Flags [.], seq 
1203473738:1203475186, ack 1191452767, win 54, options [nop,nop,TS val 
344089 ecr 256770], length 1448
17:06:11.643080 IP (tos 0x0, ttl 64, id 21446, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum e4f2 (->d181)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
47331:47363, ack 4110811169, win 46, options [nop,nop,TS val 344821 ecr 
331780], length 32
17:06:13.715233 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
node34 tell node33.webstar.cnet, length 46
17:06:13.715257 ARP, Ethernet (len 6), IPv4 (len 4), Reply node34 is-at 
00:30:48:f0:06:72 (oui Unknown), length 28
17:07:03.866492 IP (tos 0x0, ttl 64, id 21447, offset 0, flags [DF], 
proto TCP (6), length 84, bad cksum b413 (->d180)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [P.], seq 
28939:28971, ack 1913782305, win 46, options [nop,nop,TS val 357877 ecr 
331780], length 32
17:07:08.862055 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
node34 tell node33.webstar.cnet, length 46
17:07:08.862370 ARP, Ethernet (len 6), IPv4 (len 4), Reply node34 is-at 
00:30:48:f0:06:72 (oui Unknown), length 28
17:07:19.627910 IP (tos 0x0, ttl 64, id 47272, offset 0, flags [DF], 
proto TCP (6), length 52)
     node34.ssh > node33.webstar.cnet.43653: Flags [F.], cksum 0x6d6b 
(correct), seq 33, ack 1, win 46, options [nop,nop,TS val 361780 ecr 
331816], length 0
17:07:19.628403 IP (tos 0x0, ttl 64, id 21448, offset 0, flags [DF], 
proto TCP (6), length 844, bad cksum aa4d (->ce87)!)
     node33.webstar.cnet.43653 > node34.ssh: Flags [FP.], seq 
20399:21191, ack 2356871202, win 46, options [nop,nop,TS val 361818 ecr 
361780], length 792
17:07:19.833456 IP (tos 0x0, ttl 64, id 47273, offset 0, flags [DF], 
proto TCP (6), length 52)
     node34.ssh > node33.webstar.cnet.43653: Flags [F.], cksum 0x6d37 
(correct), seq 33, ack 1, win 46, options [nop,nop,TS val 361832 ecr 
331816], length 0
17:07:19.833517 IP (tos 0x0, ttl 64, id 21449, offset 0, flags [DF], 
proto TCP (6), length 64)
     node33.webstar.cnet.43653 > node34.ssh: Flags [.], cksum 0xa5e9 
(correct), ack 34, win 46, options [nop,nop,TS val 361870 ecr 
361832,nop,nop,sack 1 {33:34}], length 0
Quoted 6 lines

At this point, I see a "Connection closed by 10.141.0.34" message on node33 (from where I am attempting to ssh).

Again, if I ifdown on node33 and ifup again - I can then see from node33 to node34 without problems.

OK it seems forcedeth has problem with checksums ?

Try to change "ethtool -k eth0" settings ?

ethtool -K eth0 tso off tx off

#14 lore raw
Quoted 5 lines · Eric Dumazet

OK it seems forcedeth has problem with checksums ?

Try to change "ethtool -k eth0" settings ?

ethtool -K eth0 tso off tx off

Yes, that makes an unresponsive system responsive again immediately, nice!

Should the driver default to disabling this until we problem is corrected?

-stephen

#15 lore raw
Quoted 5 lines · stephen mulcahy
Quoted 4 lines · Eric Dumazet

OK it seems forcedeth has problem with checksums ?

Try to change "ethtool -k eth0" settings ?

Quoted 7 lines
Quoted 1 line

ethtool -K eth0 tso off tx off

Yes, that makes an unresponsive system responsive again immediately, nice!

Should the driver default to disabling this until we problem is corrected?

-stephen

Both flags need to be disabled, or only one is OK ?

#16 lore raw
Quoted 6 lines · Eric Dumazet
Quoted 5 lines · stephen mulcahy
Quoted 4 lines · Eric Dumazet

OK it seems forcedeth has problem with checksums ?

Try to change "ethtool -k eth0" settings ?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

ethtool -K eth0 tso off tx off

Yes, that makes an unresponsive system responsive again immediately, nice!

Should the driver default to disabling this until we problem is corrected?

-stephen

Both flags need to be disabled, or only one is OK ?

ethtool -K eth0 tx off

fixes the problem (without tso)

but running

ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

seems to indicate that tso is also disabled by this - does that sound correct?

-stephen

#17 lore raw
Quoted 33 lines · stephen mulcahy
Quoted 6 lines · Eric Dumazet
Quoted 5 lines · stephen mulcahy
Quoted 4 lines · Eric Dumazet

OK it seems forcedeth has problem with checksums ?

Try to change "ethtool -k eth0" settings ?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

ethtool -K eth0 tso off tx off

Yes, that makes an unresponsive system responsive again immediately, nice!

Should the driver default to disabling this until we problem is corrected?

-stephen

Both flags need to be disabled, or only one is OK ?

ethtool -K eth0 tx off

fixes the problem (without tso)

but running

ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

Quoted 2 lines

seems to indicate that tso is also disabled by this - does that sound correct?

That's correct - TSO requires TX offload. What happens if you only turn off TSO?

Ben. (wearing another hat)

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#18 lore raw

Ok, I've tried both of the following with my reproducer

1. ethtool -K eth0 tso off

RESULT: reproducer causes multiple hosts to be come unresponsive on first run.

2. ethtool -K eth0 tx off

RESULT: reproducer runs three times without any hosts becoming unresponsive.

-stephen

#19 lore raw
Quoted 4 lines · stephen mulcahy

Ok, I've tried both of the following with my reproducer

1. ethtool -K eth0 tso off

Quoted 8 lines

RESULT: reproducer causes multiple hosts to be come unresponsive on first run.

2. ethtool -K eth0 tx off

RESULT: reproducer runs three times without any hosts becoming unresponsive.

-stephen

Thanks Stephen !

Now some brave fouls to check the 6410 lines of this driver ? ;)

Question of the day : Why TSO is broken in forcedeth ? Is it generically broken or is it broken for specific NICS ?

#20 lore raw
Quoted 11 lines · Eric Dumazet
Quoted 10 lines · stephen mulcahy

Ok, I've tried both of the following with my reproducer

1. ethtool -K eth0 tso off

RESULT: reproducer causes multiple hosts to be come unresponsive on first run.

2. ethtool -K eth0 tx off

RESULT: reproducer runs three times without any hosts becoming unresponsive.

Quoted 9 lines
Quoted 1 line

-stephen

Thanks Stephen !

Now some brave fouls to check the 6410 lines of this driver ? ;)

Question of the day : Why TSO is broken in forcedeth ? Is it generically broken or is it broken for specific NICS ?

Actually, it is only when tx-checksumming is turned off that the problem
doesn't occur (so I'm not sure TSO is the problem).

Additionally, a google also turns up this existing Debian bug http://bugs.debian.org/506419 which seems to be related.

-stephen

#21 lore raw
Quoted 10 lines · stephen mulcahy
Quoted 5 lines

Now some brave fouls to check the 6410 lines of this driver ? ;)

Question of the day : Why TSO is broken in forcedeth ? Is it generically broken or is it broken for specific NICS ?

Actually, it is only when tx-checksumming is turned off that the problem
doesn't occur (so I'm not sure TSO is the problem).

Additionally, a google also turns up this existing Debian bug http://bugs.debian.org/506419 which seems to be related.

As mentioned in the original Debian bug - I can reproduce this by running Hadoop[1] TeraSort[2] but I haven't identified a simpler reproducer. I tried to recreate this with iperf and ping -f but neither helped - it may be that the problem only occurs when systems are passing large amounts of traffic and have very high cpu utilisation (when running the Hadoop TeraSort all 8 cores run at 70-100% utilisation as measure with htop - I plan to instrument the nodes with something like Zabbix or Ganglia but it hasn't happened yet).

-stephen

[1] http://hadoop.apache.org/
[2] http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/examples/terasort/package-summary.html

#22 lore raw
In reply to stephen mulcahy
Quoted 23 lines · stephen mulcahy
Quoted 12 lines · Eric Dumazet
Quoted 11 lines · stephen mulcahy

Ok, I've tried both of the following with my reproducer

1. ethtool -K eth0 tso off

RESULT: reproducer causes multiple hosts to be come unresponsive on first run.

2. ethtool -K eth0 tx off

RESULT: reproducer runs three times without any hosts becoming unresponsive.

Quoted 9 lines
Quoted 1 line

-stephen

Thanks Stephen !

Now some brave fouls to check the 6410 lines of this driver ? ;)

Question of the day : Why TSO is broken in forcedeth ? Is it generically broken or is it broken for specific NICS ?

Quoted 8 lines

Actually, it is only when tx-checksumming is turned off that the problem
doesn't occur (so I'm not sure TSO is the problem).

Additionally, a google also turns up this existing Debian bug
http://bugs.debian.org/506419 which seems to be related.

-stephen

I am scratching my head, but I thought you told me that

ethtool -K eth0 tso off
ethtool -K eth0 tx on

was working ?

And, given that when you perform "ethtool -K eth0 tx off", TSO is automatically off...

tx checksums work, TSO doesnt work, therefore TSO is broken ?

#23 lore raw
Quoted 6 lines · Eric Dumazet

I am scratching my head, but I thought you told me that

ethtool -K eth0 tso off
ethtool -K eth0 tx on

was working ?

No, sorry for the confusion.

ethtool -K eth0 tx off

fixes the problem.

Setting only

ethtool -K eth0 tso off
ethtool -K eth0 tx on

still results in failures.

-stephen

#24 lore raw
Quoted 16 lines · stephen mulcahy
Quoted 6 lines · Eric Dumazet

I am scratching my head, but I thought you told me that

ethtool -K eth0 tso off
ethtool -K eth0 tx on

was working ?

No, sorry for the confusion.

ethtool -K eth0 tx off

fixes the problem.

Quoted 6 lines

Setting only

ethtool -K eth0 tso off
ethtool -K eth0 tx on

still results in failures.

OK, thanks for clarification.

Last question, did you tried a vanilla kernel, aka 2.6.33.2 for example ?

#25 lore raw
Quoted 4 lines · Eric Dumazet

OK, thanks for clarification.

Last question, did you tried a vanilla kernel, aka 2.6.33.2 for example ?

I built a Debian package from the vanilla 2.6.33.2 and installed that on all nodes and tried my reproducer with the same results - nodes becoming unresponsive.

I didn't try changing the tso and tx settings with the 2.6.33.2 kernel though. Let me know if that would be useful (and/or if there is another kernel that you would like me to test with) and I'll try to fit it in.

Thanks again for your help.

-stephen

#26 lore raw
Quoted 6 lines · stephen mulcahy
Quoted 4 lines · Eric Dumazet

OK, thanks for clarification.

Last question, did you tried a vanilla kernel, aka 2.6.33.2 for example ?

Quoted 8 lines

I built a Debian package from the vanilla 2.6.33.2 and installed that on all nodes and tried my reproducer with the same results - nodes becoming unresponsive.

I didn't try changing the tso and tx settings with the 2.6.33.2 kernel though. Let me know if that would be useful (and/or if there is another kernel that you would like me to test with) and I'll try to fit it in.

I tried 2.6.34-rc4 (64bits) on an old machine I had lying at home.

00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
Subsystem: ASUSTeK Computer Inc. K8N4-E or A8N-E Mainboard
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0 (250ns min, 5000ns max)
Interrupt: pin A routed to IRQ 21
Region 0: Memory at d4000000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at b000 [size=8]
Capabilities: [44] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME-
Kernel driver in use: forcedeth
Kernel modules: forcedeth

I could not reproduce the problem you have.

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 31
model name : AMD Athlon(tm) 64 Processor 3200+
stepping : 0
cpu MHz : 1000.000
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good lahf_lm
bogomips : 2010.09
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

RAM : 3 Gbytes

Only strange thing I noticed is ethtool -S results with an insane tx_broadcast

# ethtool -S eth1
NIC statistics:
     tx_bytes: 90388
     tx_zero_rexmt: 348
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 413
     rx_multicast: 22
     rx_broadcast: 2
     rx_packets: 437
     rx_errors_total: 0
     tx_errors_total: 0
     tx_deferral: 718
     tx_packets: 718
     rx_bytes: 718
     tx_pause: 718
     rx_pause: 718
     rx_drop_frame: 718
     tx_unicast: 15748
     tx_multicast: 5552
     tx_broadcast: 115174309658

[root@localhost ~]# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:11:D8:9A:6D:06
inet adr:192.168.99.99 Bcast:192.168.99.255 Masque:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:466 errors:0 dropped:0 overruns:0 frame:0 TX packets:354 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 lg file transmission:1000
RX bytes:50751 (49.5 KiB) TX bytes:92974 (90.7 KiB) Interruption:21 Adresse de base:0x2000

[root@localhost ~]# grep eth1 /proc/interrupts
21: 954 IO-APIC-fasteoi eth1

maintainer
#27 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Tue, 13 Apr 2010 16:42:21 +0200

Quoted 12 lines · stephen mulcahy
Quoted 11 lines

Ok, I've tried both of the following with my reproducer

1. ethtool -K eth0 tso off

RESULT: reproducer causes multiple hosts to be come unresponsive on first run.

2. ethtool -K eth0 tx off

RESULT: reproducer runs three times without any hosts becoming unresponsive.

Quoted 8 lines
Quoted 1 line

-stephen

Thanks Stephen !

Now some brave fouls to check the 6410 lines of this driver ? ;)

Question of the day : Why TSO is broken in forcedeth ? Is it generically broken or is it broken for specific NICS ?

Do you really come to the conclusion that TSO is broken with the above test results?

I would conclude that there is a TX checksumming issue, since merely turning TSO off does not fix the problem whereas turning TX checksumming off does.

#28 lore raw
Quoted 6 lines · David Miller

Do you really come to the conclusion that TSO is broken with the above test results?

I would conclude that there is a TX checksumming issue, since merely turning TSO off does not fix the problem whereas turning TX checksumming off does.

Indeed, we clarified the point and it is a TX checksum issue.

#32 lore raw

Bug#572201: [PATCH] forcedeth: fix tx limit2 flag check

In reply to David Miller

This patch fixes the TX_LIMIT feature flag. The previous logic check for TX_LIMIT2 also took into account a device that only had TX_LIMIT set.

This is a fix for bug 572201 @ bugs.debian.org

drivers/net/forcedeth.c +1 -1
@@ -5901,7 +5901,7 @@
 	/* Limit the number of tx's outstanding for hw bug */
 	if (id->driver_data & DEV_NEED_TX_LIMIT) {
 		np->tx_limit = 1;
-		if ((id->driver_data & DEV_NEED_TX_LIMIT2) &&
+		if (((id->driver_data & DEV_NEED_TX_LIMIT2) == DEV_NEED_TX_LIMIT2) &&
 		    pci_dev->revision >= 0xA2)
 			np->tx_limit = 0;
 	}
#29 lore raw

Bug#572201: forcedeth driver hangs under heavy load

In reply to Eric Dumazet

Attached fix has been submitted to netdev.

Ayaz

Quoted 5 lines · Eric Dumazet
Quoted 3 lines · David Miller

Do you really come to the conclusion that TSO is broken with the above test results?

Quoted 8 lines
Quoted 3 lines

I would conclude that there is a TX checksumming issue, since merely turning TSO off does not fix the problem whereas turning TX checksumming off does.

Indeed, we clarified the point and it is a TX checksum issue.

drivers/net/forcedeth.c +1 -1
@@ -5901,7 +5901,7 @@
 	/* Limit the number of tx's outstanding for hw bug */
 	if (id->driver_data & DEV_NEED_TX_LIMIT) {
 		np->tx_limit = 1;
-		if ((id->driver_data & DEV_NEED_TX_LIMIT2) &&
+		if (((id->driver_data & DEV_NEED_TX_LIMIT2) == DEV_NEED_TX_LIMIT2) &&
 		    pci_dev->revision >= 0xA2)
 			np->tx_limit = 0;
 	}
#33 lore raw

Re: [PATCH] forcedeth: fix tx limit2 flag check

In reply to Ayaz Abdulla
Quoted 5 lines · Ayaz Abdulla

This patch fixes the TX_LIMIT feature flag. The previous logic check for TX_LIMIT2 also took into account a device that only had TX_LIMIT set.

This is a fix for bug 572201 @ bugs.debian.org

Hi,

Thanks! I'll rebuild my Debian kernel with this and run a test today.

-stephen

#31 lore raw
In reply to Ayaz Abdulla
Quoted 1 line · Ayaz Abdulla

Attached fix has been submitted to netdev.

I've run my reproducer with this patch applied to be Debian 2.6.32 kernel and so far the problem with nodes becoming unresponsive hasn't occurred.

NIC settings were left the default so this looks positive

root@node23:~# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

Thanks!

-stephen