Skip to contentSkip to conversation

help in TCP checksum offload , TSO and zero copy

DISCUSSION MP M 4 participants 4 messages Feb 26, 2004 – Feb 27, 2004

lore
Share & tools
lore

MP M

author
#0 lore raw

Hello ,

I am working on TCP checksum offload , TSO and zero copy with linux 2.6.1 kernel .

IMHO I find that the code for TCP checksum offload and TSO are already supported by the linux 2.6 kernel . I arrived at this conclusion on seeing the presence of flag CHECKSUM_HW and the #defines for NETIF_F_IP_CSUM , NETIF_F_NO_CSUM and
NETIF_F_HW_CSUM . By default , it seems that CHECKSUM_HW is enabled by default so that the TSO supported driver will do the processing on the ethernet card.

Please correct me if I am wrong .

In the driver for e1000 and tg3 , support for TSO is already seen .

But when I was testing the performance using ttcp utility , I found some weird results.
I just want to share to obtain some feedback from some experienced guys in this area who has already worked in TSO ,TCP checksum offload .

On the server machine I had my linux 2.6 kernel running and it had e1000 Intel pro ethercard support.Initially with ethtool utility I ensured that the Tx and Rx checksum setting on e1000 is set to on . I started ttcp utiltity in receive mode on the server machine listening on my specified port .

Next I pumped in data from my client machine using ttcp utility in transmit mode to the server .

I measured the time duration for data transfer to happen . say x milliseconds.

Next I set the tx and rx checksum on e1000 card using ethtool , and repeated the above test with ttcp utility .Since the content size is same and with tx/rx checksum off on e1000 , I expected the time duaration of data transfer from server to client to be x+some delta . But surprisingly I am noticing the data transfer at lesser time than x .(ie faster than before with tx/rx checksum off on e1000 ) .

I would appreciate if anyone could shed some light on this odd behaviour .

I have pasted below what I encountered :-

setup
-----
aniket-linux is client m/c
tc105 is server m/c which has linux 2.6.1 kernel running on it .

eth2 is e1000 NIC on tc105 which supports checksum offload and segmentation offloading

[root@tc105 mpm]# ethtool -k eth2
RX/TX checksumming parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
[root@tc105 mpm]#

Pumping data to server tc105 when rx-checksum and
---------------------------------------------------
tx-checksum are enabled on the server's NIC interface
-----------------------------------------------------
eth2
-----
[magesh@aniket-linux current]$ ls
rpm_topdir
[magesh@aniket-linux current]$ tar of - rpm_topdir | ttcp -t tc105 -p 4001
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=4001 tcp -> tc105
ttcp-t: socket
ttcp-t: connect
ttcp-t: 216442880 bytes in 299.34 real seconds = 706.13 KB/sec +++
ttcp-t: 40574 I/O calls, msec/call = 7.55, calls/sec = 135.55
ttcp-t: 0.0user 2.2sys 4:59real 0% 0i+0d 0maxrss 0+2pf 0+0csw
[magesh@aniket-linux current]$

[root@tc105 mpm]# ethtool -k eth2
RX/TX checksumming parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: on
[root@tc105 mpm]#

Pumping data to server tc105 when rx-checksum and
---------------------------------------------------
tx-checksum are disabled on the server's NIC interface
-----------------------------------------------------
eth2
-----

[magesh@aniket-linux current]$ tar cf - rpm_topdir | ttcp -t tc105 -p 4001
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=4001 tcp -> tc105
ttcp-t: socket
ttcp-t: connect
ttcp-t: 216442880 bytes in 62.26 real seconds = 3394.84 KB/sec +++
ttcp-t: 36257 I/O calls, msec/call = 1.76, calls/sec = 582.33
ttcp-t: 0.1user 1.8sys 1:02real 3% 0i+0d 0maxrss 0+2pf 0+0csw
[magesh@aniket-linux current]$

I would also appreciate if any one could shed more light on zero copy functionality support in linux 2.6.1

Thanks
Magesh

__________________________________
Do you Yahoo!?
Get better spam protection with Yahoo! Mail.
http://antispam.yahoo.com/tools

#1 lore raw
Quoted 11 lines · MP M

Next I set the tx and rx checksum on e1000 card using ethtool , and repeated the above test with ttcp utility .Since the content size is same and with tx/rx checksum off on e1000 , I expected the time duaration of data transfer from server to client to be x+some delta . But surprisingly I am noticing the data transfer at lesser time than x .(ie faster than before with tx/rx checksum off on e1000 ) .

I would appreciate if anyone could shed some light on this odd behaviour .

If you only ran the test once, could this be due to disk access and caching of the original command by the block layer?

You'd be better off getting rid of tar from the equation, by creating the tar output on disk, and then copying that to null to prime the disk cache before you perform your tests. Avoid any other disk activity on the box, as it could flush the disk cache and change the results.

#2 lore raw
In reply to MP M
Quoted 2 lines · MP M

ttcp-t: 216442880 bytes in 299.34 real seconds =
706.13 KB/sec +++

^^^^^^^^^

It's obvious that you have other problems, because lack of hardware checksum won't lead to such abysmal performance. You should start with *real* traffic, and not with I/O bound tools such as tar. For instance, you'd better use "dd if=/dev/zero bs=64k count=4000" (or use /dev/urandom) to generate some data if you absolutely need some. But here, I'm fairly certain that your disk is *THE* problem. I could easily reach the Gb/s with a P3/1G on a crappy dl2k, so I don't see why you could not to it on better hardware.

Oh, and BTW, check your cable (netstat -i) to see if you get drops or errors. You can also have poor wires which lead to frequent retransmits. GigE uses all 4 pairs (8 wires), so a cable which works well on FastEth is not necessarily OK on GigE.

Cheers,
Willy

#3 lore raw
In reply to MP M
Quoted 48 lines · MP M

Hello ,

I am working on TCP checksum offload , TSO and zero copy with linux 2.6.1 kernel .

IMHO I find that the code for TCP checksum offload and TSO are already supported by the linux 2.6 kernel . I arrived at this conclusion on seeing the presence of flag CHECKSUM_HW and the #defines for NETIF_F_IP_CSUM , NETIF_F_NO_CSUM and
NETIF_F_HW_CSUM . By default , it seems that CHECKSUM_HW is enabled by default so that the TSO supported driver will do the processing on the ethernet card.

Please correct me if I am wrong .

In the driver for e1000 and tg3 , support for TSO is already seen .

But when I was testing the performance using ttcp utility , I found some weird results.
I just want to share to obtain some feedback from some experienced guys in this area who has already worked in TSO ,TCP checksum offload .

On the server machine I had my linux 2.6 kernel running and it had e1000 Intel pro ethercard support.Initially with ethtool utility I ensured that the Tx and Rx checksum setting on e1000 is set to on . I started ttcp utiltity in receive mode on the server machine listening on my specified port .

Next I pumped in data from my client machine using ttcp utility in transmit mode to the server .

I measured the time duration for data transfer to happen . say x milliseconds.

Next I set the tx and rx checksum on e1000 card using ethtool , and repeated the above test with ttcp utility .Since the content size is same and with tx/rx checksum off on e1000 , I expected the time duaration of data transfer from server to client to be x+some delta . But surprisingly I am noticing the data transfer at lesser time than x .(ie faster than before with tx/rx checksum off on e1000 ) .

Quoted 2 lines

I would appreciate if anyone could shed some light on this odd behaviour .

Not that odd - the local CPU is better at computing checksums than the interface.

Note - there are some optimizations in the network stack that reduce the effort of actually computing checksums. In some cases, all that is done is to subtract/add those values that are changed in making replies/ACKs. This eliminates having to make a complete checksum evaluation. In others, the checksum is done during some other operation that requires a pass through the packet - making the checksum cost negligable.

I'm not the person for a detailed, low level explaination, but I think this is what you are seeing. The hardware cannot take advantage of global optimizations - and what you are seeing is that difference.