Skip to contentSkip to conversation

TSO/GRO/LRO/somethingO breaks LVS on 2.6.36

DISCUSSION Simon Kirby 4 participants 14 messages Dec 3, 2010 – Jan 27, 2011

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...

LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.

Simon-

#1 lore raw
Quoted 6 lines · Simon Kirby

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...

Quoted 8 lines

LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course) when sent on wire.

Quoted 10 lines

This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.

Quoted 8 lines

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.

Simon-

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

maintainer
#2 lore raw
Quoted 41 lines · Eric Dumazet
Quoted 6 lines · Simon Kirby

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...

Quoted 8 lines

LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course) when sent on wire.

Quoted 17 lines

This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.

Quoted 8 lines
Quoted 1 line

Simon-

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.

Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.

maintainer
#3 lore raw

[ CCed lvs-devel, Julian Anastasov and Herbert Xu ]

Quoted 60 lines · Simon Horman
Quoted 41 lines · Eric Dumazet
Quoted 6 lines · Simon Kirby

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...

Quoted 8 lines

LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course) when sent on wire.

Quoted 17 lines

This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.

Quoted 8 lines
Quoted 1 line

Simon-

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.

Quoted 7 lines

Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.

Here is the patch for v2.6.36.

From: Simon Horman <[email protected]>

ipvs: allow transmit of GRO aggregated skbs

Attempt at allowing LVS to transmit skbs of greater than MTU length that have been aggregated by GRO and can thus be deaggregated by GSO.

Cc Julian Anastasov <[email protected]>
Cc Herbert Xu <[email protected]>
---
 net/netfilter/ipvs/ip_vs_xmit.c |   25 +++++++++++++++----------
 1 files changed, 15 insertions(+), 10 deletions(-)
net/netfilter/ipvs/ip_vs_xmit.c +15 -10
index 49df6be..577f502 100644
@@ -247,7 +247,8 @@ ip_vs_bypass_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -311,7 +312,7 @@ ip_vs_bypass_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -408,7 +409,8 @@ ip_vs_nat_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (iph->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL_PKT(0, pp, skb, 0, "ip_vs_nat_xmit(): frag needed for");
@@ -486,7 +488,7 @@ ip_vs_nat_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL_PKT(0, pp, skb, 0,
@@ -597,8 +599,8 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	df |= (old_iph->frag_off & htons(IP_DF));
 
-	if ((old_iph->frag_off & htons(IP_DF))
-	    && mtu < ntohs(old_iph->tot_len)) {
+	if ((old_iph->frag_off & htons(IP_DF) &&
+	    mtu < ntohs(old_iph->tot_len) && !skb_is_gso(skb))) {
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		ip_rt_put(rt);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -707,7 +709,8 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 	if (skb_dst(skb))
 		skb_dst(skb)->ops->update_pmtu(skb_dst(skb), mtu);
 
-	if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr)) {
+	if (mtu < ntohs(old_iph->payload_len) + sizeof(struct ipv6hdr) &&
+	    !skb_is_gso(skb)) {
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		dst_release(&rt->dst);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -796,7 +799,8 @@ ip_vs_dr_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu) {
+	if ((iph->frag_off & htons(IP_DF)) && skb->len > mtu &&
+	    !skb_is_gso(skb)) {
 		icmp_send(skb, ICMP_DEST_UNREACH,ICMP_FRAG_NEEDED, htonl(mtu));
 		ip_rt_put(rt);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -924,7 +928,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF))) {
+	if ((skb->len > mtu) && (ip_hdr(skb)->frag_off & htons(IP_DF)) &&
+	    !skb_is_gso(skb)) {
 		ip_rt_put(rt);
 		icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED, htonl(mtu));
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
@@ -999,7 +1004,7 @@ ip_vs_icmp_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
 
 	/* MTU checking */
 	mtu = dst_mtu(&rt->dst);
-	if (skb->len > mtu) {
+	if (skb->len > mtu && !skb_is_gso(skb)) {
 		dst_release(&rt->dst);
 		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
 		IP_VS_DBG_RL("%s(): frag needed\n", __func__);
Signature · 1 line
1.7.2.3
#4 lore raw
In reply to Simon Horman
Quoted 1 line · Simon Horman

On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:

[...]

Quoted 6 lines
Quoted 5 lines

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

Quoted 8 lines

There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.

[...]

I think it is desirable, and we already do it for bridging and forwarding - we call dev_disable_lro() whenever a device is added to a bridge or has forwarding enabled. For IPVS it looks like you would have to resolve the server IP address to a device first. Though if the administrator assigns the server address to a different interface later then the failure mode will be very hard to understand.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
maintainer
#5 lore raw
Quoted 17 lines · Ben Hutchings
Quoted 1 line · Simon Horman

On Fri, Dec 03, 2010 at 12:29:03PM +0100, Eric Dumazet wrote:

[...]

Quoted 6 lines
Quoted 5 lines

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

Quoted 8 lines

There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.

Quoted 8 lines

[...]

I think it is desirable, and we already do it for bridging and forwarding - we call dev_disable_lro() whenever a device is added to a bridge or has forwarding enabled. For IPVS it looks like you would have to resolve the server IP address to a device first. Though if the administrator assigns the server address to a different interface later then the failure mode will be very hard to understand.

Thanks, I'll try and make it so.

#6 lore raw
In reply to Simon Horman
Quoted 17 lines · Simon Horman
Quoted 16 lines · Eric Dumazet

Le vendredi 03 d??cembre 2010 ?? 02:34 -0800, Simon Kirby a ??crit :

Quoted 14 lines

Hello!

We upgraded some LVS (DR) servers from 2.6.35 to 2.6.36 on tg3 cards (partno(BCM95721) rev 4201) with VLAN tags in use, to think that everything looked great, but in fact...

LVS was receiving magically-merged TCP packets which it tried to forward on to the real server, only to get annoyed at itself for trying to forward a packet bigger than the device MTU:

IP A.47376 > B.529: . 175488:176936(1448) ack 1 win 92 <nop,nop,timestamp 52737308 29656081> IP A.47376 > B.529: . 176936:179832(2896) ack 1 win 92 <nop,nop,timestamp 52737325 29656098> IP B > A: ICMP B unreachable - need to frag (mtu 1500), length 556

Quoted 7 lines
Quoted 7 lines

Hi Simon

This is a tcpdump on A ?
Could you take it also on B ?

tcpdump displays large buffers, but they should be split (of course) when sent on wire.

I assume you don't need this anymore since the problem was figured out? Otherwise, let me know.

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines

This caused packet loss for any merged frames, which caused abysmal performance for uploads via the LVS server. Local performance to or from the box is still fine, because the stack doesn't care, only the forwarding part of LVS is running into the problem.

Furthermore, disabling _everything_ reported by ethtool -k doesn't seem to change the result, even if I down/up the interface after, and even if I try on every single interface including the VLANned ones. This seems to be another bug. Reverting to 2.6.35 makes it all work again.

Possibly related to commit 7fe876af921d1d2bc8353e0062c10ff35e902653

So how should this be fixed? Should LVS be taught to fragment, or must we disable the merging in this case? It seems like it would work well if the sending side could do the same offload in reverse, but I'm not sure if that would be possible.

Quoted 8 lines
Quoted 1 line

Simon-

I believe Simon Horman has some patches for GRO and LVS.

Please send the results of "ethtool -k eth0" on all your nics / vlans ?

For TSO, I am not sure why and where it could matter...

Quoted 8 lines

There is a patch to teach LVS how to cope with GRO in nf-next-2.6 and I expect it to be included in 2.6.38. The patch is "ipvs: allow transmit of GRO aggregated skbs" and perhaps it should be considered for 2.6.37 and stable. In general the work around is to disable GRO.

The patch does not resolve the incompatibility of LVS with LRO. The work around there is to disable LRO. I'm not entirely sure how to teach LVS to disable LRO automatically, or if its desirable.

Yeah, things will still break if the interfaces have mismatched GRO support, etc.

It would be nice if it could be all smart about it, but I don't see how this could be done sanely, as it doesn't know if the destination interface could support GRO transmission until it has gone through the LVS processing. Would software GRO be faster on transmission be faster than no GRO at all?

Quoted 3 lines

Simon, you mention that you disabled everything with ethtool, but the tcpdump above shows a 2896 byte packet, which seems that GRO (or LRO?) is active. So perhaps as you speculate that is a bug

Right. So, on 2.6.35, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

On 2.6.36, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'.

Quoted 3 lines

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.

I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!

Simon-

#7 lore raw
Quoted 44 lines · Simon Kirby

Right. So, on 2.6.35, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

On 2.6.36, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
Quoted 4 lines

And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'.

Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)

Quoted 10 lines
Quoted 3 lines

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.

I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!

Simon-

#8 lore raw
Quoted 46 lines · Eric Dumazet

Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :

Quoted 44 lines

Right. So, on 2.6.35, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

On 2.6.36, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
Quoted 5 lines
Quoted 3 lines

And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'.

Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)

and turning it off makes no difference, either. Even if I down/up the interface after. I still see merged packets on receive, even if I try to use ethtool to turn off everything that is "on". It's as if the new code to enable it on vlans works, but the disable from ethtool does not.

In other words, I cannot find a way to disable it (and thus make LVS work) on 2.6.36 or 2.6.37-rc6.

One interesting note is that if I destroy and recreate the VLAN interface, it comes back with gso enabled:

# ethtool -K eth1 gso off
# ethtool -K eth1.39 gso off
# ethtool -K eth1 tso off
# ethtool -K eth1.39 tso off
# ifdown eth1.39
Removed VLAN -:eth1.39:-
# ifconfig eth1 down
# ifup eth1.39
Added VLAN with VID == 39 to IF -:eth1:-
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -K eth1.39 gso off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off
# ifdown eth1.39; ifup eth1.39
Removed VLAN -:eth1.39:-
Added VLAN with VID == 39 to IF -:eth1:-
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off

So, the bug seems to related using it on VLANs.

Simon-

Quoted 16 lines
Quoted 10 lines
Quoted 3 lines

I will prepare a backport of the "ipvs: allow transmit of GRO aggregated skbs" patch to v2.6.36 and post it shortly. Testing to see if that resolves the problem that you are seeing would probably be a good start.

I can test this in our case, and it should specifically work on the servers we were seeing the problem on, but it still needs some complicated logic to be safe for all cases (like when LVS decides to route out an interface without GRO). Anyway, thanks!

Simon-

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#9 lore raw
Quoted 54 lines · Simon Kirby
Quoted 46 lines · Eric Dumazet

Le mercredi 08 d??cembre 2010 ?? 18:35 -0800, Simon Kirby a ??crit :

Quoted 44 lines

Right. So, on 2.6.35, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
large receive offload: off

On 2.6.36, ethtool shows:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
large receive offload: off
Quoted 5 lines
Quoted 3 lines

And if I set it with ethtool -K eth1.39 gso off; ethtool -K eth1 gso off, it says "off", but I still see merged frames, as verified with an nc < /dev/zero and tcpdump -i any -n 'length > 1500'.

Yep, but you also have "tso on" by default on 2.6.36 and eth1.39 (vlan)

Quoted 7 lines

and turning it off makes no difference, either. Even if I down/up the interface after. I still see merged packets on receive, even if I try to use ethtool to turn off everything that is "on". It's as if the new code to enable it on vlans works, but the disable from ethtool does not.

In other words, I cannot find a way to disable it (and thus make LVS work) on 2.6.36 or 2.6.37-rc6.

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.

My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.

* GRO (generic receive offload)

This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?

In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.

N.B: I'm not sure that I have any hardware whose driver supports LRO
so my tests were made without LRO support.

Quoted 2 lines

One interesting note is that if I destroy and recreate the VLAN interface, it comes back with gso enabled:

I guess that relate to how the default is set.
I don't imagine that it is the cause of the problem that you are seeing.

#10 lore raw
Quoted 14 lines · Simon Horman

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.

My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

Quoted 7 lines

You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.

* GRO (generic receive offload)

This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?

So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).

Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?

Quoted 3 lines

In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.

It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.

Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.

Cheers,

Simon-

maintainer
#11 lore raw
Quoted 24 lines · Simon Kirby
Quoted 14 lines · Simon Horman

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.

My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

Quoted 7 lines

You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.

* GRO (generic receive offload)

This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?

Quoted 6 lines

So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).

Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?

The patch for IPVS GRO support has been merged and should appear in 2.6.39. This is somewhat later than I previously anticipated due to a merge mix-up on my part.

Quoted 7 lines
Quoted 3 lines

In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.

It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.

I will recheck my results, but in general I think it is a bit of an open question as to how ethtool settings should be propagated between related devices.

Quoted 29 lines

Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 2 lines

If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.

Yes that is a weird.

There has been quite a lot of work on VLANs recently and I suspect that the behaviour that you are observing with 2.6.37 is a regression that occurred during that work. It would be good to fix things to restore the 2.6.35 behaviour.

#12 lore raw
In reply to Simon Kirby
Quoted 68 lines · Simon Kirby
Quoted 14 lines · Simon Horman

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.

My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

Quoted 7 lines

You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.

* GRO (generic receive offload)

This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?

So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).

Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?

Quoted 3 lines

In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.

It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.

Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 3 lines

If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.

Weird maybe, but GRO check/handling is done in dev_gro_receive(), on eth1 receive path.

Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO machine state) before being delivered to eth1.39

It would be useless/expensive to add another GRO layer on eth1.39

We might not report GRO state on vlan/bonding (or reflect real device GRO state)

maintainer
#13 lore raw
Quoted 76 lines · Eric Dumazet
Quoted 68 lines · Simon Kirby
Quoted 14 lines · Simon Horman

Hi Simon,

thanks for prodding me to respond to this post offline and sorry for not responding earlier.

Firstly, I think that this is a receive-side problem so I don't believe that GSO (generic segmentation offload) or other transmit-side options are likely to have any affect.

My understanding is that on the receive-side there are two options which when enabled can result in the behaviour that you describe.

* LRO (large receive offload)

Quoted 7 lines

You have this disabled, and assuming it really is disabled it shouldn't be causing a problem.

* GRO (generic receive offload)

This does not seem to be in the output of your ethtool commands at all. So I wonder if your ethtool is too old to support this option?

So, this was the case. Our ethtool (lenny) was too old to see the GRO option, only GSO. Disabling GRO on eth1.39 has no effect, but disabling it on eth1 caused it to stop receiving the merged frames, fixing the LVS packet loss (due to no sending GSO support from LVS/IPVS).

Speaking of this, did your patch for LVS/IPVS GSO support go anywhere?

Quoted 3 lines

In any case, I was able to reproduce the problem that you describe (or at least something very similar) using 2.6.36 with GRO enabled on eth1.1 and the problem did not manifest when I disabled GRO on eth1.1.

It worked for you to do ethtool -K eth1.1 gro off, then? For me on 2.6.37, it seemed to be that "ethtool -K eth1 gro off" was needed, even though packets arrive on eth1.39.

Also, strangely, 2.6.35.4's default state (with no received merged frames) has GRO on for eth1 but off for eth1.39:

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
# ethtool -k eth1.39
Offload parameters for eth1.39:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
Quoted 3 lines

If I set 2.6.37 to have all of the same options, I still see GRO frames on 2.6.37 (tg3), which is weird.

Weird maybe, but GRO check/handling is done in dev_gro_receive(), on eth1 receive path.

Quoted 7 lines

Frames are assembled by GRO layer using tg3 NAPI structure (holding GRO machine state) before being delivered to eth1.39

It would be useless/expensive to add another GRO layer on eth1.39

We might not report GRO state on vlan/bonding (or reflect real device GRO state)

That makes sense. So to clarify, this is the expected behaviour?