gro vs vlan in myri10ge
Subsystems (2)
Hi,
I've wanted to convert myri10ge from LRO to GRO for quite a while. The problem I'm facing is that the NIC cannot perform hardware vlan tag offload, so GRO performance is far below LRO performance when receiving vlan tagged TCP traffic.
If a vlan tagged frame is passed to lro_receive_frags(), inet_lro will look at the encapsulated IPv4 frame and TCP aggregation will succeed. However, it appears that GRO will not do this. When I patch the driver to use GRO, and configure a vlan interface, I see high CPU utilization and poor bandwidth when I'm receiving a netperf TCP stream on the vlan interface. If I use LRO in an unpatched driver, then I see good receive performance in the same scenario.
What is the best way to "fix" this?
Unless I'm just using GRO wrong, it seems that the simplest thing for me to do is to claim NETIF_F_HW_VLAN_RX, but pop the tags in the driver so as to allow myri10ge to pass up a non-encapsulated frame the same way that (nearly?) every other 10GbE NIC does. I've got a quick and dirty patch that confirms doing the vtag pop in the driver gives me roughly the same performance with GRO as I used to have with LRO.
Is this (popping vlan tags in the driver) acceptable, or is it too much of a layering violation?
Thanks,
Drew
Quoted 24 lines · Andrew Gallatin
Hi,
I've wanted to convert myri10ge from LRO to GRO for quite a while. The problem I'm facing is that the NIC cannot perform hardware vlan tag offload, so GRO performance is far below LRO performance when receiving vlan tagged TCP traffic.
If a vlan tagged frame is passed to
lro_receive_frags(),inet_lrowill look at the encapsulated IPv4 frame and TCP aggregation will succeed. However, it appears that GRO will not do this. When I patch the driver to use GRO, and configure a vlan interface, I see high CPU utilization and poor bandwidth when I'm receiving a netperf TCP stream on the vlan interface. If I use LRO in an unpatched driver, then I see good receive performance in the same scenario.What is the best way to "fix" this?
Unless I'm just using GRO wrong, it seems that the simplest thing for me to do is to claim
NETIF_F_HW_VLAN_RX, but pop the tags in the driver so as to allow myri10ge to pass up a non-encapsulated frame the same way that (nearly?) every other 10GbE NIC does. I've got a quick and dirty patch that confirms doing the vtag pop in the driver gives me roughly the same performance with GRO as I used to have with LRO.
Quoted 2 lines
Is this (popping vlan tags in the driver) acceptable, or is it too much of a layering violation?
Given GRO assumes NIC does hardware vlan offloading, I guess I would chose to do that.
It seems unfortunate to add vlan decap in GRO path, already very complex.
Quoted 1 line · Eric Dumazet
On Thu, 2012-11-08 at 21:20 -0500, Andrew Gallatin wrote:
<...>
Quoted 8 lines
Quoted 2 lines
Is this (popping vlan tags in the driver) acceptable, or is it too much of a layering violation?
Given GRO assumes NIC does hardware vlan offloading, I guess I would chose to do that.
It seems unfortunate to add vlan decap in GRO path, already very complex.
OK, then I will prepare & run through q/a a real patch that does this.
Thanks again,
Drew
Quoted 7 lines · Eric Dumazet
Quoted 6 lines · Andrew Gallatin
Hi,
I've wanted to convert myri10ge from LRO to GRO for quite a while. The problem I'm facing is that the NIC cannot perform hardware vlan tag offload, so GRO performance is far below LRO performance when receiving vlan tagged TCP traffic.
Thanks for the reminder; I need to deal with this in sfc as well.
Quoted 17 lines
Quoted 17 lines
If a vlan tagged frame is passed to
lro_receive_frags(),inet_lrowill look at the encapsulated IPv4 frame and TCP aggregation will succeed. However, it appears that GRO will not do this. When I patch the driver to use GRO, and configure a vlan interface, I see high CPU utilization and poor bandwidth when I'm receiving a netperf TCP stream on the vlan interface. If I use LRO in an unpatched driver, then I see good receive performance in the same scenario.What is the best way to "fix" this?
Unless I'm just using GRO wrong, it seems that the simplest thing for me to do is to claim
NETIF_F_HW_VLAN_RX, but pop the tags in the driver so as to allow myri10ge to pass up a non-encapsulated frame the same way that (nearly?) every other 10GbE NIC does. I've got a quick and dirty patch that confirms doing the vtag pop in the driver gives me roughly the same performance with GRO as I used to have with LRO.
Quoted 5 lines
Quoted 2 lines
Is this (popping vlan tags in the driver) acceptable, or is it too much of a layering violation?
Given GRO assumes NIC does hardware vlan offloading, I guess I would chose to do that.
Really, after all the changes in 2.6.37 to unify behaviour between the offloaded and non-offloaded paths?
Quoted 2 lines
It seems unfortunate to add vlan decap in GRO path, already very complex.
True but can't this be done at the top?
I'll post a patch for this shortly.
Ben.
Signature · 10 lines
Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.