MTU and TCP transmit offload.
Subsystems (2)
We saw something interesting while doing some testing on 3.0.4.
We configured 2 Ethernet NICs with standard 1500 MTU, and added a mac-vlan on each, with MTU of 300. The goal was to generate as many ~300 byte TCP packets as possible, for load testing purposes. We configured our tool to open sockets on the mac-vlans and send/receive TCP (IPv4) traffic.
This actually seems to work quite nicely, allowing user-space to do large writes (24k in our case), and it appears have lots of small packets on the wire. We still need to sniff with external system to verify this..but packets-per-second counters look good.
Evidently this all works because macvlans know that the NIC can do TSO, and the '300' MTU is passed in the big packet given to the NIC.
This got me thinking...at least for my purposes, it would be nice to have a per-socket 'MTU' setting. The idea is that you could ask the NIC to do the TSO at whatever 'mtu' you wanted, without having to resort to mac-vlans with artificially small MTU.
So, is there any interest in supporting such a socket option?
I can't think of any use besides TCP traffic load testing, but perhaps someone else can think of one? Or, is load-testing enough?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 8 lines · Ben Greear
We saw something interesting while doing some testing on 3.0.4.
We configured 2 Ethernet NICs with standard 1500 MTU, and added a mac-vlan on each, with MTU of 300. The goal was to generate as many ~300 byte TCP packets as possible, for load testing purposes. We configured our tool to open sockets on the mac-vlans and send/receive TCP (IPv4) traffic.
Presumably one could instead set static PathMTU entries in the routing tables and accomplish the same thing as you did with the mac-vlans?
Quoted 15 lines
This actually seems to work quite nicely, allowing user-space to do large writes (24k in our case), and it appears have lots of small packets on the wire. We still need to sniff with external system to verify this..but packets-per-second counters look good.
Evidently this all works because macvlans know that the NIC can do TSO, and the '300' MTU is passed in the big packet given to the NIC.
This got me thinking...at least for my purposes, it would be nice to have a per-socket 'MTU' setting. The idea is that you could ask the NIC to do the TSO at whatever 'mtu' you wanted, without having to resort to mac-vlans with artificially small MTU.
Quoted 5 lines
So, is there any interest in supporting such a socket option?
I can't think of any use besides TCP traffic load testing, but perhaps someone else can think of one? Or, is load-testing enough?
Isn't that covered by setsockopt() support for TCP_MAX_SEG? With TSO what gets passed to the NIC isn't the MTU, but the connection's MSS derived (in part at least) from the MTU of the egress interface. If one had made a setsockopt(TCP_MAX_SEG) call prior to the connect() or listen() call, presumably that would have influenced the MSS exchange at connection establishment.
rick jones
Quoted 29 lines · Rick Jones
Quoted 8 lines · Ben Greear
We saw something interesting while doing some testing on 3.0.4.
We configured 2 Ethernet NICs with standard 1500 MTU, and added a mac-vlan on each, with MTU of 300. The goal was to generate as many ~300 byte TCP packets as possible, for load testing purposes. We configured our tool to open sockets on the mac-vlans and send/receive TCP (IPv4) traffic.
Presumably one could instead set static PathMTU entries in the routing tables and accomplish the same thing as you did with the mac-vlans?
Quoted 17 lines
This actually seems to work quite nicely, allowing user-space to do large writes (24k in our case), and it appears have lots of small packets on the wire. We still need to sniff with external system to verify this..but packets-per-second counters look good.
Evidently this all works because macvlans know that the NIC can do TSO, and the '300' MTU is passed in the big packet given to the NIC.
This got me thinking...at least for my purposes, it would be nice to have a per-socket 'MTU' setting. The idea is that you could ask the NIC to do the TSO at whatever 'mtu' you wanted, without having to resort to mac-vlans with artificially small MTU.
So, is there any interest in supporting such a socket option?
Quoted 7 lines
Quoted 3 lines
I can't think of any use besides TCP traffic load testing, but perhaps someone else can think of one? Or, is load-testing enough?
Isn't that covered by
setsockopt()support forTCP_MAX_SEG? With TSO what gets passed to the NIC isn't the MTU, but the connection's MSS derived (in part at least) from the MTU of the egress interface. If one had made asetsockopt(TCP_MAX_SEG)call prior to theconnect()orlisten()call, presumably that would have influenced the MSS exchange at connection establishment.
Ohh, that looks promising!
I'll give that a try.
Thanks,
Ben
Quoted 2 lines
rick jones
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 34 lines · Ben Greear
Quoted 33 lines · Rick Jones
Quoted 8 lines · Ben Greear
We saw something interesting while doing some testing on 3.0.4.
We configured 2 Ethernet NICs with standard 1500 MTU, and added a mac-vlan on each, with MTU of 300. The goal was to generate as many ~300 byte TCP packets as possible, for load testing purposes. We configured our tool to open sockets on the mac-vlans and send/receive TCP (IPv4) traffic.
Presumably one could instead set static PathMTU entries in the routing tables and accomplish the same thing as you did with the mac-vlans?
Quoted 20 lines
This actually seems to work quite nicely, allowing user-space to do large writes (24k in our case), and it appears have lots of small packets on the wire. We still need to sniff with external system to verify this..but packets-per-second counters look good.
Evidently this all works because macvlans know that the NIC can do TSO, and the '300' MTU is passed in the big packet given to the NIC.
This got me thinking...at least for my purposes, it would be nice to have a per-socket 'MTU' setting. The idea is that you could ask the NIC to do the TSO at whatever 'mtu' you wanted, without having to resort to mac-vlans with artificially small MTU.
So, is there any interest in supporting such a socket option?
I can't think of any use besides TCP traffic load testing, but perhaps someone else can think of one? Or, is load-testing enough?
Quoted 7 lines
Quoted 3 lines
Isn't that covered by
setsockopt()support forTCP_MAX_SEG? With TSO what gets passed to the NIC isn't the MTU, but the connection's MSS derived (in part at least) from the MTU of the egress interface. If one had made asetsockopt(TCP_MAX_SEG)call prior to theconnect()orlisten()call, presumably that would have influenced the MSS exchange at connection establishment.Ohh, that looks promising!
I'll give that a try.
This works like a charm. I'm so glad I don't need to hack a new sockopt!
Thanks,
Ben
Quoted 8 lines
Thanks,
BenQuoted 2 lines
rick jones
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Isn't that covered by
setsockopt()support forTCP_MAX_SEG? With TSO what gets passed to the NIC isn't the MTU, but the connection's MSS derived (in part at least) from the MTU of the egress interface. If one had made asetsockopt(TCP_MAX_SEG)call prior to theconnect()orlisten()call, presumably that would have influenced the MSS exchange at connection establishment.
Quoted 6 lines
Quoted 3 lines
Ohh, that looks promising!
I'll give that a try.
This works like a charm. I'm so glad I don't need to hack a new sockopt!
Glad it works for you.
happy benchmarking,
rick jones
One of these days I'll have to decide if I add that as an option to netperf, which presently only does the getsockopt(TCP_MAX_SEG). Folks with a strong preference one way or t'other should feel free to contact me on the side or via netperf-talk at netperf.org.