drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?
Subsystems (2)
RFC PATCH 1/1 drivers/net/tg3.c: Raise Jumbo Frame MTU to 9216?
The TG3 apparently supports 9K frame sizes.
http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
Is exactly 9000 a hardware limit?
Should the jumbo frame MTU be raised to 9216 or 9216 less the size of MAC, VLAN, IP and TCP headers?
drivers/net/tg3.c +1 -1
@@ -95,7 +95,7 @@/* hardware minimum and maximum for a single frame's data payload */ #define TG3_MIN_MTU 60#define TG3_MAX_MTU (tp) \ - ((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9000 : 1500) + ((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9216 : 1500)/* These numbers seem to be hard coded in the NIC firmware somehow. * You can't change the ring sizes, but you can change where you place
Quoted 8 lines · Joe Perches
The TG3 apparently supports 9K frame sizes.
http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
Is exactly 9000 a hardware limit?
Should the jumbo frame MTU be raised to 9216 or 9216 less the size of MAC, VLAN, IP and TCP headers?
9000 has been the de facto standard, has it been changed recently?
Anyway, we've never done any testing on 9216. As it uses up to 2 more internal mbufs per packet, there may not be sufficient buffers inside the chip for optimal operations. At best, some water marks will need to be tweaked. The hardware statistics counters (ethtool -S) also may not work for packets bigger than 9022 bytes.
Quoted 17 lines
drivers/net/tg3.c +1
-1@@ -95,7 +95,7 @@/* hardware minimum and maximum for a single frame's data payload */ #define TG3_MIN_MTU 60#define TG3_MAX_MTU (tp) \ - ((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9000 : 1500) + ((tp->tg3_flags & TG3_FLAG_JUMBO_CAPABLE) ? 9216 : 1500)/* These numbers seem to be hard coded in the NIC firmware somehow. * You can't change the ring sizes, but you can change where you place
Quoted 7 lines · Michael Chan
Quoted 5 lines · Joe Perches
The TG3 apparently supports 9K frame sizes.
http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
Is exactly 9000 a hardware limit?
Should the jumbo frame MTU be raised to 9216 or 9216 less the size of MAC, VLAN, IP and TCP headers?9000 has been the de facto standard, has it been changed recently?
I know of a performance lab that's trying to use 9216 as a "standard" jumbo frame length.
Unrelated to the performance lab:
http://www.uoregon.edu/~joe/jumbo-clean-gear.html
9216 seems popular, especially with Cisco gear. Contrary to that link, the Cisco 3750 does work with 9216 length jumbo frames.
Quoted 5 lines
Anyway, we've never done any testing on 9216. As it uses up to 2 more internal mbufs per packet, there may not be sufficient buffers inside the chip for optimal operations. At best, some water marks will need to be tweaked. The hardware statistics counters (ethtool -S) also may not work for packets bigger than 9022 bytes.
Thanks, do you have pointers to where the tweaking needs to be done?
Is this define a hardware upper bound?
#define TG3_RX_JMB_DMA_SZ 9046
cheers, Joe
Quoted 11 lines · Joe Perches
Quoted 7 lines · Michael Chan
Quoted 5 lines · Joe Perches
The TG3 apparently supports 9K frame sizes.
http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
Is exactly 9000 a hardware limit?
Should the jumbo frame MTU be raised to 9216 or 9216 less the size of MAC, VLAN, IP and TCP headers?9000 has been the de facto standard, has it been changed recently?
I know of a performance lab that's trying to use 9216 as a "standard" jumbo frame length.
[...]
We also use 9000 as the 'normal' jumbo MTU, though hard limit in sfc is a bit higher than this.
The frame length limit is not defined only by the DMA engine and FIFOs but also by the MAC and PHY. On the receive side, the MAC and/or PHY need to handle skew between the remote and local clocks which can grow proportionally with frame length. For a controller which is used with several different PHYs, I would be hesitant to raise the driver's limit.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Quoted 9 lines · Joe Perches
Quoted 7 lines · Michael Chan
Quoted 5 lines · Joe Perches
The TG3 apparently supports 9K frame sizes.
http://www.broadcom.com/collateral/pb/5704C-PB05-R.pdf
Is exactly 9000 a hardware limit?
Should the jumbo frame MTU be raised to 9216 or 9216 less the size of MAC, VLAN, IP and TCP headers?9000 has been the de facto standard, has it been changed recently?
Quoted 10 lines
I know of a performance lab that's trying to use 9216 as a "standard" jumbo frame length.
Unrelated to the performance lab:
http://www.uoregon.edu/~joe/jumbo-clean-gear.html
9216 seems popular, especially with Cisco gear. Contrary to that link, the Cisco 3750 does work with 9216 length jumbo frames.Quoted 1 line
Anyway, we've never done any testing on 9216.
9216 is commonly used in Ethernet switch testing [1]. Most data center switches support it, including some built around Broadcom silicon (e.g., Dell PowerConnect 8024, Extreme Summit X650).
There is no IEEE standard for jumbo lengths. The IETF's RFC 5180 recommends 9216 for switch performance testing.
Although 5180 specs this length as a SHOULD and not a MUST, its use does represent widespread industry practice in switch performance benchmarking.
Regards,
David Newman
Network Test
[1] Length from first byte of DA to last byte of CRC absent any VLAN header(s).