TSO/GRO/LRO/somethingO breaks LVS on 2.6.36
Subsystems (4)
Hello!
We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...
LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:
IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.
Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.
Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653
So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.
Simon-
Quoted 6 lines · Simon Kirby
Hello!
We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...
Quoted 8 lines
LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:
IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
Hi Simon
This is a tcpdump on A ?
Could you take it also on B ?
tcpdump displays large buffers, but they should be split (of course) when sent on wire.
Quoted 10 lines
This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.
Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.
Quoted 8 lines
Possibly related to commit
7fe876af921d1d2bc8353e0062c10ff35e902653So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.
Simon-
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
Simon Horman
maintainerQuoted 41 lines · Eric Dumazet
Quoted 6 lines · Simon Kirby
Hello!
We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...
Quoted 8 lines
LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:
IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
Hi Simon
This is a tcpdump on A ?
Could you take it also on B ?tcpdump displays large buffers, but they should be split (of course) when sent on wire.
Quoted 17 lines
This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.
Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.
Possibly related to commit
7fe876af921d1d2bc8353e0062c10ff35e902653So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.
Quoted 8 lines
Quoted 1 line
Simon-
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.
The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.
Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug
I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.
Simon Horman
maintainer[ CCed lvs-devel, Julian Anastasov and Herbert Xu ]
Quoted 60 lines · Simon Horman
Quoted 41 lines · Eric Dumazet
Quoted 6 lines · Simon Kirby
Hello!
We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...
Quoted 8 lines
LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:
IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
Hi Simon
This is a tcpdump on A ?
Could you take it also on B ?tcpdump displays large buffers, but they should be split (of course) when sent on wire.
Quoted 17 lines
This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.
Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.
Possibly related to commit
7fe876af921d1d2bc8353e0062c10ff35e902653So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.
Quoted 8 lines
Quoted 1 line
Simon-
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.
The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.
Quoted 7 lines
Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug
I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.
Here is the patch for v2.6.36.
From: Simon Horman <[email protected]>
ipvs: allow transmit of GRO aggregated skbs
Attempt at allowing LVS to transmit skbs of greater than MTU length that have been aggregated by GRO and can thus be deaggregated by GSO.
--- net/netfilter/ipvs/ip_vs_xmit.c | 25 +++++++++++++++---------- 1 files changed, 15 insertions(+), 10 deletions(-)
net/netfilter/ipvs/ip_vs_xmit.c +15 -10
@@ -247,7 +247,8 @@ ip_vs_bypass_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if ((skb->len > mtu) && (iph->frag_off &htons (IP_DF ))) { +if ((skb->len > mtu) && (iph->frag_off &htons (IP_DF )) && + !skb_is_gso (skb)) {ip_rt_put (rt);icmp_send (skb,ICMP_DEST_UNREACH ,ICMP_FRAG_NEEDED ,htonl (mtu));IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -311,7 +312,7 @@ ip_vs_bypass_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if (skb->len > mtu) { +if (skb->len > mtu && !skb_is_gso (skb)) {dst_release (&rt->dst);icmpv6_send (skb,ICMPV6_PKT_TOOBIG ,0 , mtu);IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -408,7 +409,8 @@ ip_vs_nat_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if ((skb->len > mtu) && (iph->frag_off &htons (IP_DF ))) { +if ((skb->len > mtu) && (iph->frag_off &htons (IP_DF )) && + !skb_is_gso (skb)) {ip_rt_put (rt);icmp_send (skb,ICMP_DEST_UNREACH ,ICMP_FRAG_NEEDED ,htonl (mtu));IP_VS_DBG_RL_PKT (0 , pp, skb,0 ,"ip_vs_nat_xmit(): frag needed for" );
@@ -486,7 +488,7 @@ ip_vs_nat_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if (skb->len > mtu) { +if (skb->len > mtu && !skb_is_gso (skb)) {dst_release (&rt->dst);icmpv6_send (skb,ICMPV6_PKT_TOOBIG ,0 , mtu); IP_VS_DBG_RL_PKT(0, pp, skb, 0,
@@ -597,8 +599,8 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, df |= (old_iph->frag_off &htons (IP_DF )); -if ((old_iph->frag_off &htons (IP_DF )) - && mtu <ntohs (old_iph->tot_len)) { +if ((old_iph->frag_off &htons (IP_DF ) && + mtu <ntohs (old_iph->tot_len) && !skb_is_gso (skb))) {icmp_send (skb,ICMP_DEST_UNREACH ,ICMP_FRAG_NEEDED ,htonl (mtu));ip_rt_put (rt);IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -707,7 +709,8 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,if (skb_dst (skb))skb_dst (skb)->ops->update_pmtu (skb_dst (skb), mtu); -if (mtu <ntohs (old_iph->payload_len) +sizeof (struct ipv6hdr )) { +if (mtu <ntohs (old_iph->payload_len) +sizeof (struct ipv6hdr ) && + !skb_is_gso (skb)) {icmpv6_send (skb,ICMPV6_PKT_TOOBIG ,0 , mtu);dst_release (&rt->dst);IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -796,7 +799,8 @@ ip_vs_dr_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if ((iph->frag_off &htons (IP_DF )) && skb->len > mtu) { +if ((iph->frag_off &htons (IP_DF )) && skb->len > mtu && + !skb_is_gso (skb)) {icmp_send (skb,ICMP_DEST_UNREACH ,ICMP_FRAG_NEEDED ,htonl (mtu));ip_rt_put (rt);IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -924,7 +928,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if ((skb->len > mtu) && (ip_hdr (skb)->frag_off &htons (IP_DF ))) { +if ((skb->len > mtu) && (ip_hdr (skb)->frag_off &htons (IP_DF )) && + !skb_is_gso (skb)) {ip_rt_put (rt);icmp_send (skb,ICMP_DEST_UNREACH ,ICMP_FRAG_NEEDED ,htonl (mtu));IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
@@ -999,7 +1004,7 @@ ip_vs_icmp_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,/* MTU checking */ mtu =dst_mtu (&rt->dst); -if (skb->len > mtu) { +if (skb->len > mtu && !skb_is_gso (skb)) {dst_release (&rt->dst);icmpv6_send (skb,ICMPV6_PKT_TOOBIG ,0 , mtu);IP_VS_DBG_RL ("%s(): frag needed\n" , __func__);
Signature · 1 line
1.7.2.3
Quoted 1 line · Simon Horman
On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
[...]
Quoted 6 lines
Quoted 5 lines
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
Quoted 8 lines
There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.
The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.
[...]
I think it is desirable, and we already do it for bridging and forwarding - we call dev_disable_lro() whenever a device is added to a bridge or has forwarding enabled. For IPVS it looks like you would have to resolve the server IP address to a device first. Though if the administrator assigns the server address to a different interface later then the failure mode will be very hard to understand.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Simon Horman
maintainerQuoted 17 lines · Ben Hutchings
Quoted 1 line · Simon Horman
On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:
[...]
Quoted 6 lines
Quoted 5 lines
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
Quoted 8 lines
There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.
The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.
Quoted 8 lines
[...]
I think it is desirable, and we already do it for bridging and forwarding - we call
dev_disable_lro()whenever a device is added to a bridge or has forwarding enabled. For IPVS it looks like you would have to resolve the server IP address to a device first. Though if the administrator assigns the server address to a different interface later then the failure mode will be very hard to understand.
Thanks, I'll try and make it so.
Quoted 17 lines · Simon Horman
Quoted 16 lines · Eric Dumazet
Le vendredi 03 d??cembre 2010 ?? 02:34 -0800, Simon Kirby a ??crit :
Quoted 14 lines
Hello!
We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...
LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:
IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556
Quoted 7 lines
Quoted 7 lines
Hi Simon
This is a tcpdump on A ?
Could you take it also on B ?tcpdump displays large buffers, but they should be split (of course) when sent on wire.
I assume you don't need this anymore since the problem was figured out? Otherwise, let me know.
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.
Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.
Possibly related to commit
7fe876af921d1d2bc8353e0062c10ff35e902653So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.
Quoted 8 lines
Quoted 1 line
Simon-
I believe Simon Horman has some patches for GRO and LVS.
Please send the results of "ethtool -k eth0" on all your nics / vlans ?
For TSO, I am not sure why and where it could matter...
Quoted 8 lines
There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.
The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.
Yeah, things will still break if the interfaces have mismatched GRO support, etc.
It would be nice if it could be all smart about it, but I don't see how this could be done sanely, as it doesn't know if the destination interface could support GRO transmission until it has gone through the LVS processing. Would software GRO be faster on transmission be faster than no GRO at all?
Quoted 3 lines
Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug
Right. So, on 2.6.35, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off
# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off
On 2.6.36, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off
# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off
And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'.
Quoted 3 lines
I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.
I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!
Simon-
Quoted 44 lines · Simon Kirby
Right. So, on 2.6.35, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: offOn 2.6.36, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off
Quoted 4 lines
And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc <
/dev/zeroand tcpdump -i any -n 'length > 1500'.
Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)
Quoted 10 lines
Quoted 3 lines
I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.
I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!
Simon-
Quoted 46 lines · Eric Dumazet
Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :
Quoted 44 lines
Right. So, on 2.6.35, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: offOn 2.6.36, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off
Quoted 5 lines
Quoted 3 lines
And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc <
/dev/zeroand tcpdump -i any -n 'length > 1500'.Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)
and turning it off makes no difference, either. Even if I down/up the interface after. I still see merged packets on receive, even if I try to use ethtool to turn off everything that is "on". It's as if the new code to enable it on vlans works, but the disable from ethtool does not.
In other words, I cannot find a way to disable it (and thus make LVS work) on 2.6.36 or 2.6.37-rc6.
One interesting note is that if I destroy and recreate the VLAN interface, it comes back with gso enabled:
# ethtool -K eth1 gso off # ethtool -K eth1.39 gso off # ethtool -K eth1 tso off # ethtool -K eth1.39 tso off # ifdown eth1.39 Removed VLAN -:eth1.39:- # ifconfig eth1 down # ifup eth1.39 Added VLAN with VID == 39 to IF -:eth1:- # ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: on large receive offload: off # ethtool -K eth1.39 gso off # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: off # ifdown eth1.39; ifup eth1.39 Removed VLAN -:eth1.39:- Added VLAN with VID == 39 to IF -:eth1:- # ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: on large receive offload: off
So, the bug seems to related using it on VLANs.
Simon-
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines
I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.
I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!
Simon-
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Simon Horman
maintainerQuoted 54 lines · Simon Kirby
Quoted 46 lines · Eric Dumazet
Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :
Quoted 44 lines
Right. So, on 2.6.35, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off large receive offload: offOn 2.6.36, ethtool shows:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: on large receive offload: offQuoted 5 lines
Quoted 3 lines
And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc <
/dev/zeroand tcpdump -i any -n 'length > 1500'.Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)
Quoted 7 lines
and turning it off makes no difference, either. Even if I down/up the interface after. I still see merged packets on receive, even if I try to use ethtool to turn off everything that is "on". It's as if the new code to enable it on vlans works, but the disable from ethtool does not.
In other words, I cannot find a way to disable it (and thus make LVS work) on 2.6.36 or 2.6.37-rc6.
Hi Simon,
thanks for prodding me to respond to this post offline and sorry for not responding earlier.
Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.
My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.
* LRO (large receive offload)
You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.
* GRO (generic receive offload)
This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?
In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.
N.B: I'm not sure that I have any hardware whose driver supports LRO
so my tests were made without LRO support.
Quoted 2 lines
One interesting note is that if I destroy and recreate the VLAN interface, it comes back with gso enabled:
I guess that relate to how the default is set.
I don't imagine that it is the cause of the problem that you are seeing.
Quoted 14 lines · Simon Horman
Hi Simon,
thanks for prodding me to respond to this post offline and sorry for not responding earlier.
Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.
My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.
* LRO (large receive offload)
Quoted 7 lines
You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.
* GRO (generic receive offload)
This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?
So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).
Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?
Quoted 3 lines
In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.
It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.
Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: off
# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off
If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.
Cheers,
Simon-
Simon Horman
maintainerQuoted 24 lines · Simon Kirby
Quoted 14 lines · Simon Horman
Hi Simon,
thanks for prodding me to respond to this post offline and sorry for not responding earlier.
Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.
My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.
* LRO (large receive offload)
Quoted 7 lines
You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.
* GRO (generic receive offload)
This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?
Quoted 6 lines
So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).
Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?
The patch for IPVS GRO support has been merged and should appear in 2.6.39. This is somewhat later than I previously anticipated due to a merge mix-up on my part.
Quoted 7 lines
Quoted 3 lines
In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.
It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.
I will recheck my results, but in general I think it is a bit of an open question as to how ethtool settings should be propagated between related devices.
Quoted 29 lines
Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off
Quoted 2 lines
If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.
Yes that is a weird.
There has been quite a lot of work on VLANs recently and I suspect that the behaviour that you are observing with 2.6.37 is a regression that occurred during that work. It would be good to fix things to restore the 2.6.35 behaviour.
Quoted 68 lines · Simon Kirby
Quoted 14 lines · Simon Horman
Hi Simon,
thanks for prodding me to respond to this post offline and sorry for not responding earlier.
Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.
My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.
* LRO (large receive offload)
Quoted 7 lines
You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.
* GRO (generic receive offload)
This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?
So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).
Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?
Quoted 3 lines
In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.
It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.
Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off
Quoted 3 lines
If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.
Weird maybe, but GRO check/handling is done in dev_gro_receive(), on eth1 receive path.
Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO machine state) before being delivered to eth1.39
It would be useless/expensive to add another GRO layer on eth1.39
We might not report GRO state on vlan/bonding (or reflect real device GRO state)
Simon Horman
maintainerQuoted 76 lines · Eric Dumazet
Quoted 68 lines · Simon Kirby
Quoted 14 lines · Simon Horman
Hi Simon,
thanks for prodding me to respond to this post offline and sorry for not responding earlier.
Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.
My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.
* LRO (large receive offload)
Quoted 7 lines
You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.
* GRO (generic receive offload)
This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?
So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).
Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?
Quoted 3 lines
In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.
It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.
Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:
# ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off ntuple-filters: off receive-hashing: off# ethtool -k eth1.39 Offload parameters for eth1.39: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: offQuoted 3 lines
If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.
Weird maybe, but GRO check/handling is done in
dev_gro_receive(), on eth1 receive path.
Quoted 7 lines
Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO machine state) before being delivered to eth1.39
It would be useless/expensive to add another GRO layer on eth1.39
We might not report GRO state on vlan/bonding (or reflect real device GRO state)
That makes sense. So to clarify, this is the expected behaviour?