Why is LRO off by default on ixgbe?
Subsystems (2)
I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
Why is LRO disabled by default?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines · On Wed, 23 Sep 2009 09:29:59 -0700 Ben Greear <[email protected]> wrote:
I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
Why is LRO disabled by default?
Thanks,
Ben
LRO is turned off if bridging or routing because of End to End requirements.
Quoted 6 lines · Stephen Hemminger
Quoted 3 lines · On Wed, 23 Sep 2009 09:29:59 -0700 Ben Greear<[email protected]> wrote:
I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
Quoted 6 lines
Quoted 4 lines
Why is LRO disabled by default?
Thanks,
BenLRO is turned off if bridging or routing because of End to End requirements.
That makes sense.
If I know that all interfaces in question can handle TSO and LRO, I could manually enable LRO w/out risk, right?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 12 lines · On Wed, 23 Sep 2009 09:57:51 -0700 Ben Greear <[email protected]> wrote:
Quoted 11 lines · Stephen Hemminger
Quoted 7 lines · On Wed, 23 Sep 2009 09:29:59 -0700 Ben Greear<[email protected]> wrote:
I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
Why is LRO disabled by default?
Thanks,
Ben
Quoted 7 lines
Quoted 1 line
LRO is turned off if bridging or routing because of End to End requirements.
That makes sense.
If I know that all interfaces in question can handle TSO and LRO, I could manually enable LRO w/out risk, right?
The problem is that LRO merges TCP packets, this breaks the end-to-end ack clocking and checksumming, and therefore is not enabled. That is why GRO is the replacement solution (preserves packet boundaries)
Quoted 19 lines · Stephen Hemminger
Quoted 16 lines · On Wed, 23 Sep 2009 09:57:51 -0700 Ben Greear<[email protected]> wrote:
Quoted 6 lines · Stephen Hemminger
Quoted 3 lines · On Wed, 23 Sep 2009 09:29:59 -0700 Ben Greear<[email protected]> wrote:
I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU).
Quoted 6 lines
Quoted 4 lines
Why is LRO disabled by default?
Thanks,
BenLRO is turned off if bridging or routing because of End to End requirements.
That makes sense.
Quoted 7 lines
Quoted 3 lines
If I know that all interfaces in question can handle TSO and LRO, I could manually enable LRO w/out risk, right?
The problem is that LRO merges TCP packets, this breaks the end-to-end ack clocking and checksumming, and therefore is not enabled. That is why GRO is the replacement solution (preserves packet boundaries)
Ok. It seems GRO was enabled the whole time, but LRO is what gave me the extra performance boost.
In this particular case, I'm not actually routing, though I do have ip-forward enabled, so I guess LRO will be OK as long as I'm careful...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 6 lines · Ben Greear
Ok. It seems GRO was enabled the whole time, but LRO is what gave me the extra performance boost.
In this particular case, I'm not actually routing, though I do have ip-forward enabled, so I guess LRO will be OK as long as I'm careful...
What? A driver is either GRO or LRO, it can't be both. What kernel version was this?
Thanks,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 9 lines · Herbert Xu
Quoted 6 lines
Ok. It seems GRO was enabled the whole time, but LRO is what gave me the extra performance boost.
In this particular case, I'm not actually routing, though I do have ip-forward enabled, so I guess LRO will be OK as long as I'm careful...
What? A driver is either GRO or LRO, it can't be both. What kernel version was this?
Oh perhaps you mean LRO performed by hardware? That would make sense indeed. It's also one of the reasons why we hope the hardware folks would switch over to GRO so we can enable it for everyone :)
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 8 lines · Herbert Xu
Quoted 7 lines · Herbert Xu
Quoted 6 lines
Ok. It seems GRO was enabled the whole time, but LRO is what gave me the extra performance boost.
In this particular case, I'm not actually routing, though I do have ip-forward enabled, so I guess LRO will be OK as long as I'm careful...
Quoted 6 lines
Quoted 2 lines
What? A driver is either GRO or LRO, it can't be both. What kernel version was this?
Oh perhaps you mean LRO performed by hardware? That would make sense indeed. It's also one of the reasons why we hope the hardware folks would switch over to GRO so we can enable it for everyone :)
I assume it's in hardware. I am using ixgbe driver, 82599 chipset, 2.6.31 kernel.
GRO was showing as enabled the whole time in ethtool. Turning on LRO gave me an extra 6Gbps (from 12Gbps -> 18Gbps) receive throughput, so it certainly had some affect!
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com