LRO restructuring?
Subsystems (2)
Hi,
You mentioned in the recent "Re: [RFC][PATCH 1/1] cxgb3i: cxgb3 iSCSI initiator" thread that you were planning to restructure LRO to preserve headers so as to make forwarding possible without totally disabling LRO.
For lro_receive_frags() based LRO, it would be ideal to locate the header in place in the frag via the mac_hdr argument to the get_frag_header() callback. Eg, I'm hoping that neither the driver nor the LRO module will need to allocate extra memory per frame and copy the headers to it in the common case when forwarding is not enabled. That would add quite a bit of overhead.
With respect to hardware LRO and headers: Would it be possible to notify the driver via some sort of callback whether the headers are required? I think most hardware LRO implementations are going to collapse the headers, and having the option to fallback to software LRO for forwarding might be needed for those devices which will throw away the intermediate headers.
Last, have you considered simply allowing "inexact" forwarding, where the ingress NIC is doing LRO and the egress nic is doing TSO? You loose exact framing information (eg, what you emit might not be framed exactly as you receive it), but you can still do filtering, and the host overhead is very low.
Thanks,
Drew
David Miller
maintainerFrom: Andrew Gallatin <[email protected]>
Date: Mon, 11 Aug 2008 09:30:33 -0400
Quoted 5 lines
Last, have you considered simply allowing "inexact" forwarding, where the ingress NIC is doing LRO and the egress nic is doing TSO? You loose exact framing information (eg, what you emit might not be framed exactly as you receive it), but you can still do filtering, and the host overhead is very low.
Intermediate nodes are not supposed to change the transport layer checksum if at all possible, especially on routers.
Otherwise it is much more difficult to diagnose checksum errors, and figure out what caused such an error.
When the router doesn't modify the checksum, we know it's an end-node. Even a firewall only "adjusts" checksums based upon packet modifications for NAT and such, which will preserve end-node created errors.
So no this isn't really an option.
This is why Herbert wants to preserve the original headers, we're not supposed to change them.
Quoted 11 lines · David Miller
From: Andrew Gallatin <[email protected]>
Date: Mon, 11 Aug 2008 09:30:33 -0400Quoted 5 lines
Last, have you considered simply allowing "inexact" forwarding, where the ingress NIC is doing LRO and the egress nic is doing TSO? You loose exact framing information (eg, what you emit might not be framed exactly as you receive it), but you can still do filtering, and the host overhead is very low.
Intermediate nodes are not supposed to change the transport layer checksum if at all possible, especially on routers.
Indeed. Nor should they change lengths, or anything else. Everything about this "inexact" forwarding is illegal as hell. However, you have to admit that it is an interesting hack :)
Drew
Quoted 4 lines · Andrew Gallatin
Indeed. Nor should they change lengths, or anything else. Everything about this "inexact" forwarding is illegal as hell. However, you have to admit that it is an interesting hack :)
Solutions like this have been deployed. For instance, many satellite networks use transparent TCP proxies to mitigate the effect of large latencies on older TCP stacks that don't have modern congestion control algorithms.
Surprisingly there are actually very few problems. The biggest one (apart from scalability) is with non-TCP traffic masquerading as TCP such as Cisco's VPN solution.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 7 lines · Andrew Gallatin
For
lro_receive_frags()based LRO, it would be ideal to locate the header in place in the frag via themac_hdrargument to theget_frag_header()callback. Eg, I'm hoping that neither the driver nor the LRO module will need to allocate extra memory per frame and copy the headers to it in the common case when forwarding is not enabled. That would add quite a bit of overhead.
You don't have to save the whole thing, just save enough so we can easily/exactly reconstruct it on output, i.e., save the lengths.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtDavid Miller
maintainerFrom: Herbert Xu <[email protected]>
Date: Tue, 12 Aug 2008 08:50:33 +0800
Quoted 11 lines · Andrew Gallatin
Quoted 7 lines
For
lro_receive_frags()based LRO, it would be ideal to locate the header in place in the frag via themac_hdrargument to theget_frag_header()callback. Eg, I'm hoping that neither the driver nor the LRO module will need to allocate extra memory per frame and copy the headers to it in the common case when forwarding is not enabled. That would add quite a bit of overhead.You don't have to save the whole thing, just save enough so we can easily/exactly reconstruct it on output, i.e., save the lengths.
And the checksums :-) As an intermediate node we don't want to touch the checksum.
The length and the checksum is two u16 values, which would be able to fit in a single 32-bit descriptor or something like that.
Quoted 3 lines · David Miller
And the checksums :-) As an intermediate node we don't want to touch the checksum.
Yeah if it wasn't verified then we must store this as well.
Quoted 3 lines
The length and the checksum is two u16 values, which would be able to fit in a single 32-bit descriptor or something like that.
Yep.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 7 lines · Herbert Xu
Quoted 2 lines · David Miller
And the checksums :-) As an intermediate node we don't want to touch the checksum.
Yeah if it wasn't verified then we must store this as well.
Even if it was verified I think you want to keep the checksums from the header. Since an intermediate device isn't supposed to be peeking at the TCP part anyway, it wouldn't do to drop the segment ourselves, pass it along to be dropped by the ultimate reciever. And if there is something amis in the verification or the regeneration, we don't want to
introduce silent data corruption.
Likely that also goes for the IP header checksum...
rick jones
David Miller
maintainerFrom: Rick Jones <[email protected]>
Date: Mon, 11 Aug 2008 18:30:11 -0700
Quoted 8 lines
Even if it was verified I think you want to keep the checksums from the
header. Since an intermediate device isn't supposed to be peeking at
the TCP part anyway, it wouldn't do to drop the segment ourselves, pass it along to be dropped by the ultimate reciever. And if there is something amis in the verification or the regeneration, we don't want to
introduce silent data corruption.Likely that also goes for the IP header checksum...
IP header is a little different, intermediate nodes should verify it (and we do adjust it when decrementing TTL).
Quoted 9 lines · David Miller
From: Rick Jones <[email protected]>
Date: Mon, 11 Aug 2008 18:30:11 -0700Quoted 6 lines
Even if it was verified I think you want to keep the checksums from the
header. Since an intermediate device isn't supposed to be peeking at
the TCP part anyway, it wouldn't do to drop the segment ourselves, pass it along to be dropped by the ultimate reciever. And if there is something amis in the verification or the regeneration, we don't want to
introduce silent data corruption.
Well I wasn't suggesting that it be dropped, but simply skip LRO if the inbound packet fails the checksum check.
But yeah, it's only two bytes so we might as well always have it.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi Herbert,
Any idea when this LRO restructuring work will be done?
Making LRO available even when ip forwarding is enabled will significantly improve performace of network appliances in the data path.
I have some questions on this:
(1) Based on the emails in this thread, I suppose you are going to keep the original length of each segment you coalesced into the big packet and use that info to segment the big packet on the output path. In case the packet was modified by an appliance in the path and the total length is changed (e.g. NAT on ftp control packets), should the corresponding segment length info also get updated? This same question also applies to the checksums.
(2) Do you make sure all of the segments to be coalesced have the same DF bit?
(3) I think bridged packets should not be LROed. Whether a packet is bridged or not can be based on the L2 MAC destination address. Is this how it is done?
(4) Does LRO work only for IPv4? Any plan to extend it to support IPv6?
Thanks,
James Huang
Quoted 5 lines · James Huang
Hi Herbert,
Any idea when this LRO restructuring work will be done?
Making LRO available even when ip forwarding is enabled will significantly improve performace of network appliances in the data path.
Herbert has added "GRO" rather than immediately replacing the inet_lro code. An early version of this is in 2.6.29 and there is more in net-next-2.6 destined for 2.6.30.
Quoted 1 line
I have some questions on this:
[...]
Quoted 2 lines
(3) I think bridged packets should not be LROed. Whether a packet is bridged or not can be based on the L2 MAC destination address. Is this how it is done?
GRO preserves enough information to reconstruct the original frames on output, so there is no specific check for bridging. Presumably it would be cheaper not to do use GRO if the frames are not going to hit the TCP/IP stack though.
Quoted 1 line
(4) Does LRO work only for IPv4? Any plan to extend it to support IPv6?
IPv6 is covered.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
Quoted 5 lines · On Wed, 18 Feb 2009 19:25:49 +0000 (UTC) James Huang <[email protected]>…
Hi Herbert,
Any idea when this LRO restructuring work will be done?
Making LRO available even when ip forwarding is enabled will significantly improve performace of network appliances in the data path.
LRO with ip forwarding is a bad idea since it violates the end-to-end principle. It might even be a DO NOT in host requirements RFC.
Quoted 1 line · James Huang
Hi Herbert,
Please cc me if you're going to address the email to me :)
Quoted 1 line
Any idea when this LRO restructuring work will be done?
The target is to get GRO into 2.6.30, and have all LRO drivers converted by 2.6.31.
Quoted 2 lines
Making LRO available even when ip forwarding is enabled will significantly improve performace of network appliances in the data path.
I'd like to see more performance data on the forwarding side. The act of aggregation isn't exactly free so it very much depends on what you're forwarding.
The driver for all this work is bridging used for virtualisation where the data is very likely to be mergeable.
Quoted 7 lines
I have some questions on this:
(1) Based on the emails in this thread, I suppose you are going to keep the original length of each segment you coalesced into the big packet and use that info to segment the big packet on the output path. In case the packet was modified by an appliance in the path and the total length is changed (e.g. NAT on ftp control packets), should the corresponding segment length info also get updated? This same question also applies to the checksums.
If packet length changes due to NAT then we're no loner bound by the end-to-end restriction so we can do whatever we want.
For GRO we treat it in exactly the same way as a GSO packet that undergoes NAT. That is, NAT just sees a very large IP packet and does what it has to do, and on final output we resegment the packet.
Quoted 1 line
(2) Do you make sure all of the segments to be coalesced have the same DF bit?
Yes.
Quoted 2 lines
(3) I think bridged packets should not be LROed. Whether a packet is bridged or not can be based on the L2 MAC destination address. Is this how it is done?
We currently require the Ethernet header to be identical.
Quoted 1 line
(4) Does LRO work only for IPv4? Any plan to extend it to support IPv6?
IPv6 is supported.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi all,
One question about inet_lro.c (linux 2.6.28.6)
In lro_proc_segment(), it does not check the number of available frags entries in lro_desc->parent against the number of new fragments to be added before calling lro_add_frags(). Is that a bug? I see that lro_proc_segment() does some type of checking at the end, but could that be too late (i.e. overflow the frags[])??
Also I think some obvious optimization in lro_get_desc() can be done by
combining the two for loops into one.
-- James Huang
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 11 lines · James Huang
Hi all,
One question about inet_lro.c (linux 2.6.28.6)In
lro_proc_segment(), it does not check the number of available frags entries inlro_desc->parentagainst the number of new fragments to be added before callinglro_add_frags(). Is that a bug? I see thatlro_proc_segment()does some type of checking at the end, but could that be too late (i.e. overflow the frags[])??Also I think some obvious optimization in
lro_get_desc()can be done by
combining the two for loops into one.
That file is obsolete. The GRO code that will replace it lives in various places:
net/core/dev.c net/core/skbuff.c net/ipv4/af_inet.c net/ipv6/af_inet6.c net/ipv4/tcp_ipv4.c net/ipv6/tcp_ipv6.c net/ipv4/tcp.c net/8021q/vlan_core.c
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt