Skip to contentSkip to conversation

GRO issue with kernel 3.4.94 (icmp fragmentation needed)

DISCUSSION Thomas Jarosch 4 participants 6 messages Jul 11, 2014 – Aug 5, 2014

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Thomas Jarosch

author
#0 lore raw

Hi Greg,

I don't have the original patch email, but please include the mentioned commit

------------------------------------------
commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier
------------------------------------------

in the 3.4.x stable tree.

After updating from 3.4.91 to 3.4.96, TCP connections from Windows 7 clients slowed down to a crawl. (to be precise: totally unusable)

The commit recommended by Florian Westphal fixes it immediately.

Best regards,
Thomas

---------- Forwarded Message ----------

Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
Date: Friday, 27. June 2014, 11:05:25
From: Florian Westphal <[email protected]>
To: jungwon park <[email protected]>
CC: [email protected]

Quoted 1 line · jungwon park

When using the linux router is turned on GRO, router send the

'fragmentation

Quoted 1 line

needed' packets to the sender.

Indeed 8-(

Quoted 1 line

When I turned off GRO, the router operate normally, and there is no

problem.

Quoted 5 lines

and with 3.4.91 kernel, the router has no problem.

I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
(http://patchwork.ozlabs.org/patch/345509/)
When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of 3.4.y tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine the size of the individual packets. To do this, we rely on the transport header.

Unfortunately the transport header is not set for the forward path in 3.4, so we look at the network header instead.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
-----------------------------------------

#1 lore raw
Quoted 55 lines · Thomas Jarosch

Hi Greg,

I don't have the original patch email, but please include the mentioned commit

------------------------------------------
commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier
------------------------------------------

in the 3.4.x stable tree.

After updating from 3.4.91 to 3.4.96, TCP connections from Windows 7 clients slowed down to a crawl. (to be precise: totally unusable)

The commit recommended by Florian Westphal fixes it immediately.

Best regards,
Thomas

---------- Forwarded Message ----------

Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
Date: Friday, 27. June 2014, 11:05:25
From: Florian Westphal <[email protected]>
To: jungwon park <[email protected]>
CC: [email protected]

Quoted 1 line · jungwon park

When using the linux router is turned on GRO, router send the

'fragmentation

Quoted 1 line

needed' packets to the sender.

Indeed 8-(

Quoted 1 line

When I turned off GRO, the router operate normally, and there is no

problem.

Quoted 5 lines

and with 3.4.91 kernel, the router has no problem.

I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
(http://patchwork.ozlabs.org/patch/345509/)
When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of 3.4.y tree.

Quoted 8 lines

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine the size of the individual packets. To do this, we rely on the transport header.

Unfortunately the transport header is not set for the forward path in 3.4, so we look at the network header instead.

Why would this patch matter, as commit 45f00f99d6e, which this patch fixes, is not in the 3.4-stable tree at all.

Also, network stable patches need to go through the network maintainer, not me directly.

thanks,

greg k-h

#2 lore raw

Greg KH <[email protected]> wrote:

[ CCing David. If I understood him correctly, he recommended to revert
the GRO forwarding changes for oldstable 3.2/3.4.y series. I second this. But see below for answer to your question ]

Quoted 38 lines · Thomas Jarosch
Quoted 37 lines

Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
Date: Friday, 27. June 2014, 11:05:25
From: Florian Westphal <[email protected]>
To: jungwon park <[email protected]>
CC: [email protected]

Quoted 1 line · jungwon park

When using the linux router is turned on GRO, router send the

'fragmentation

Quoted 1 line

needed' packets to the sender.

Indeed 8-(

Quoted 1 line

When I turned off GRO, the router operate normally, and there is no

problem.

Quoted 5 lines

and with 3.4.91 kernel, the router has no problem.

I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
(http://patchwork.ozlabs.org/patch/345509/)
When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of 3.4.y tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine the size of the individual packets. To do this, we rely on the transport header.

Quoted 5 lines
Quoted 2 lines

Unfortunately the transport header is not set for the forward path in 3.4, so we look at the network header instead.

Why would this patch matter, as commit 45f00f99d6e, which this patch fixes, is not in the 3.4-stable tree at all.

It helps, because I underestimated the changes the network stacki went through since 3.2 and 3.4 had been released.

The GRO forwarding change, (fe6cc55f3a9a053482a76f5a6b2257cee51b4663 upstream) depends on having the transport header available in the forward path, and not just in the input/local delivery path.

The Patch from Eric Dumazet that I referenced above made this change. So, while its unrelated to the GRO forward patch, it is a dependency.

However, given the numerous problems reported with the GRO changes in older stable kernels I think the commit 29a3cd46644ec8098dbe1c12f89643b5c11831a9 in 3.4.y should be reverted.

[ In 3.2.y its caa5344994778a2b4725b2d75c74430f76925e4a ]

Quoted 2 lines

Also, network stable patches need to go through the network maintainer, not me directly.

Greg, is above explanation enough for you to revert? If not, what can I do to help?

David, in case I remember wrong and you did not recommend the revert after all, do you want me to submit 3.2/3.4.y backport of Erics 'set transport header earlier' patch?

Thanks,
Florian

maintainer
#3 lore raw

From: Florian Westphal <[email protected]>
Date: Wed, 30 Jul 2014 08:37:03 +0200

Quoted 5 lines

However, given the numerous problems reported with the GRO changes in older stable kernels I think the commit 29a3cd46644ec8098dbe1c12f89643b5c11831a9 in 3.4.y should be reverted.

[ In 3.2.y its caa5344994778a2b4725b2d75c74430f76925e4a ]

...

Quoted 3 lines

David, in case I remember wrong and you did not recommend the revert after all, do you want me to submit 3.2/3.4.y backport of Erics 'set transport header earlier' patch?

Yes, the above two patches above should be reverted.

Thanks Florian.

#4 lore raw
In reply to Florian Westphal
Quoted 64 lines · Florian Westphal

Greg KH <[email protected]> wrote:

[ CCing David. If I understood him correctly, he recommended to revert
the GRO forwarding changes for oldstable 3.2/3.4.y series. I second this. But see below for answer to your question ]

Quoted 38 lines · Thomas Jarosch
Quoted 37 lines

Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
Date: Friday, 27. June 2014, 11:05:25
From: Florian Westphal <[email protected]>
To: jungwon park <[email protected]>
CC: [email protected]

Quoted 1 line · jungwon park

When using the linux router is turned on GRO, router send the

'fragmentation

Quoted 1 line

needed' packets to the sender.

Indeed 8-(

Quoted 1 line

When I turned off GRO, the router operate normally, and there is no

problem.

Quoted 5 lines

and with 3.4.91 kernel, the router has no problem.

I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
(http://patchwork.ozlabs.org/patch/345509/)
When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <[email protected]>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of 3.4.y tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine the size of the individual packets. To do this, we rely on the transport header.

Quoted 5 lines
Quoted 2 lines

Unfortunately the transport header is not set for the forward path in 3.4, so we look at the network header instead.

Why would this patch matter, as commit 45f00f99d6e, which this patch fixes, is not in the 3.4-stable tree at all.

It helps, because I underestimated the changes the network stacki went through since 3.2 and 3.4 had been released.

The GRO forwarding change, (fe6cc55f3a9a053482a76f5a6b2257cee51b4663 upstream) depends on having the transport header available in the forward path, and not just in the input/local delivery path.

The Patch from Eric Dumazet that I referenced above made this change. So, while its unrelated to the GRO forward patch, it is a dependency.

However, given the numerous problems reported with the GRO changes in older stable kernels I think the commit 29a3cd46644ec8098dbe1c12f89643b5c11831a9 in 3.4.y should be reverted.

Quoted 6 lines

[ In 3.2.y its caa5344994778a2b4725b2d75c74430f76925e4a ]

Quoted 2 lines

Also, network stable patches need to go through the network maintainer, not me directly.

Greg, is above explanation enough for you to revert?

Yup, that's good enough, I had to do the revert by hand, a verification would be nice to have.

thanks,

greg k-h

author
#5 lore raw
In reply to Greg KH

Hi Greg,

Quoted 2 lines · On Tuesday, 29. July 2014 17:22:55 Greg KH wrote:

Also, network stable patches need to go through the network maintainer, not me directly.

alright, I'll try to keep this in mind for the next time *fingers crossed*

@Florian: In your follow up message, the CC: for Davis was missing. Though netdev was on the list, it should be fine.

Thomas