myri10ge Large Receive Offload
Subsystems (2)
cover myri10ge Large Receive Offload
This is a complete rework of the myri10ge receive path. The first patch converts skb allocation to use physical pages. The second one adds a software implementation of Large Receive Offload. The third one updates the driver version to 1.1.0.
The complete driver code in our CVS actually also supports high-order allocations instead of single physical pages since it significantly increase the performance. Order=2 allows us to receive standard frames at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot about memory fragmentation if the performance is better.
But, since high-order allocations are generally considered a bad idea, we do not include the relevant code in the following patch for inclusion in Linux. Here, we simply pass order=0 to all page allocation routines. If necessary, I could drop the remaining reference to high-order (especially replace alloc_pages() with alloc_page()) but I'd rather keep it as is.
If high-order allocations are ever considered OK under some circum- stances, we could send an additional patch (a module parameter would be used to switch from single physical pages to high-order pages).
Thanks,
Brice
Quoted 4 lines · Brice Goglin
This is a complete rework of the myri10ge receive path. The first patch converts skb allocation to use physical pages. The second one adds a software implementation of Large Receive Offload. The third one updates the driver version to 1.1.0.
If this LRO is completely in software it might be time to consider something more generic since this would be the second software LRO implementation (the other being s2io).
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 19 lines · Brice Goglin
This is a complete rework of the myri10ge receive path. The first patch converts skb allocation to use physical pages. The second one adds a software implementation of Large Receive Offload. The third one updates the driver version to 1.1.0.
The complete driver code in our CVS actually also supports high-order allocations instead of single physical pages since it significantly increase the performance. Order=2 allows us to receive standard frames at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot about memory fragmentation if the performance is better.
But, since high-order allocations are generally considered a bad idea, we do not include the relevant code in the following patch for inclusion in Linux. Here, we simply pass order=0 to all page allocation routines. If necessary, I could drop the remaining reference to high-order (especially replace
alloc_pages()withalloc_page()) but I'd rather keep it as is.
Quoted 3 lines
If high-order allocations are ever considered OK under some circum- stances, we could send an additional patch (a module parameter would be used to switch from single physical pages to high-order pages).
As Herbert's already done, I would rather let the net core people comment on this. The code implementation doesn't look scary, but we may want to be smarter about this in the core net stack, rather than implementing it inside multiple drivers.
Jeff
Quoted 20 lines · Jeff Garzik
Quoted 19 lines · Brice Goglin
This is a complete rework of the myri10ge receive path. The first patch converts skb allocation to use physical pages. The second one adds a software implementation of Large Receive Offload. The third one updates the driver version to 1.1.0.
The complete driver code in our CVS actually also supports high-order allocations instead of single physical pages since it significantly increase the performance. Order=2 allows us to receive standard frames at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot about memory fragmentation if the performance is better.
But, since high-order allocations are generally considered a bad idea, we do not include the relevant code in the following patch for inclusion in Linux. Here, we simply pass order=0 to all page allocation routines. If necessary, I could drop the remaining reference to high-order (especially replace
alloc_pages()withalloc_page()) but I'd rather keep it as is.
Quoted 8 lines
Quoted 3 lines
If high-order allocations are ever considered OK under some circum- stances, we could send an additional patch (a module parameter would be used to switch from single physical pages to high-order pages).
As Herbert's already done, I would rather let the net core people comment on this. The code implementation doesn't look scary, but we may want to be smarter about this in the core net stack, rather than implementing it inside multiple drivers.
Ok, makes sense. We look forward to see this.
Could we get patch #1 merged anyway (page-based skb allocation)?
Any comments about what I was saying about high-order allocations above?
thanks,
Brice
Quoted 30 lines · Brice Goglin
Quoted 20 lines · Jeff Garzik
Quoted 19 lines · Brice Goglin
This is a complete rework of the myri10ge receive path. The first patch converts skb allocation to use physical pages. The second one adds a software implementation of Large Receive Offload. The third one updates the driver version to 1.1.0.
The complete driver code in our CVS actually also supports high-order allocations instead of single physical pages since it significantly increase the performance. Order=2 allows us to receive standard frames at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot about memory fragmentation if the performance is better.
But, since high-order allocations are generally considered a bad idea, we do not include the relevant code in the following patch for inclusion in Linux. Here, we simply pass order=0 to all page allocation routines. If necessary, I could drop the remaining reference to high-order (especially replace
alloc_pages()withalloc_page()) but I'd rather keep it as is.Quoted 8 lines
Quoted 3 lines
If high-order allocations are ever considered OK under some circum- stances, we could send an additional patch (a module parameter would be used to switch from single physical pages to high-order pages).
As Herbert's already done, I would rather let the net core people comment on this. The code implementation doesn't look scary, but we may want to be smarter about this in the core net stack, rather than implementing it inside multiple drivers.
Quoted 5 lines
Ok, makes sense. We look forward to see this.
Could we get patch #1 merged anyway (page-based skb allocation)?
Any comments about what I was saying about high-order allocations above?
It is quite strnage that you see very noticeble speed degradation after switching from higher order to 0 order allocations, could specify where observed bottleneck in network stack is?
Quoted 2 lines
thanks,
Brice
Signature · 1 line
Evgeniy Polyakov
Quoted 24 lines · Evgeniy Polyakov
Quoted 22 lines · Brice Goglin
Quoted 20 lines · Jeff Garzik
Quoted 18 lines · Brice Goglin
The complete driver code in our CVS actually also supports high-order allocations instead of single physical pages since it significantly increase the performance. Order=2 allows us to receive standard frames at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot about memory fragmentation if the performance is better.
But, since high-order allocations are generally considered a bad idea, we do not include the relevant code in the following patch for inclusion in Linux. Here, we simply pass order=0 to all page allocation routines. If necessary, I could drop the remaining reference to high-order (especially replace
alloc_pages()withalloc_page()) but I'd rather keep it as is.If high-order allocations are ever considered OK under some circum- stances, we could send an additional patch (a module parameter would be used to switch from single physical pages to high-order pages).
Quoted 7 lines
Quoted 2 lines
Any comments about what I was saying about high-order allocations above?
It is quite strnage that you see very noticeble speed degradation after switching from higher order to 0 order allocations, could specify where observed bottleneck in network stack is?
The bottleneck is not in the network stack, it is simply related to the number of page allocations that are required. Since we store multiple fragments in a same page, if MTU=1500, we need one 0-order allocation every 2 fragments, while we need one 2-order allocation every 8 fragments. IIRC, we observed about 20% higher throughput on the receive side when switching from order=0 to order=2 (7.5Gbit/s -> 9.3Gbit/s with roughly the same CPU usage).
Brice
Quoted 2 lines · Brice Goglin
Could we get patch #1 merged anyway (page-based skb allocation)?
Oops, actually, one hunk has to be dropped (it reverts const'ification of ethtool_ops by mistake). If it is ok to merge the patch apart from this hunk, will resend an updated patch.
Brice