Poor Performance for ethernet bonding
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Bernhard Busch
Hi
I have tried to use ethernet network interfaces bonding to increase peformance.
Quoted 6 lines
Bonding is working fine, but the performance is rather poor. FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet card)
results in a transfer rate of 3MB/s).Any Hints?
Bonding reorders packets, which causes frequent retransmits and stalls in TCP. One setup that doesn't is multipath routing (ip route .. with multiple nexthops over different interfaces). It'll only load balance (srcip, dstip,tos) tuples though, not individual flows, but then it has the advantage that it actually works.
-Andi
Quoted 14 lines · Andi Kleen
Quoted 11 lines · Bernhard Busch
Hi
I have tried to use ethernet network interfaces bonding to increase peformance.
Bonding is working fine, but the performance is rather poor. FTP between 2 machines ( kernel 2.4.4 and 4 port DLink 100Mbit ethernet card)
results in a transfer rate of 3MB/s).
Quoted 7 lines
Quoted 1 line
Any Hints?
Bonding reorders packets, which causes frequent retransmits and stalls in TCP. One setup that doesn't is multipath routing (ip route .. with multiple nexthops over different interfaces). It'll only load balance (srcip, dstip,tos) tuples though, not individual flows, but then it has the advantage that it actually works.
Couldn't the bonding code be made to distribute pkts to one interface or another based on a hash of the sending IP port or something? Seems like that would fix the reordering problem for IP packets.... It wouldn't help for a single stream, but I'm guessing the real world problem involves many streams, which on average should hash such that the load is balanced...
Ben
Quoted 7 lines
(quoted diff) +0
-2-Andi -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 3 lines
Ben Greear <[email protected]> <[email protected]> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 5 lines · Ben Greear
Couldn't the bonding code be made to distribute pkts to one interface or another based on a hash of the sending IP port or something? Seems like that would fix the reordering problem for IP packets.... It wouldn't help for a single stream, but I'm guessing the real world problem involves many streams, which on average should hash such that the load is balanced...
It could, but then it is already implemented in a better way in multipath routing and I see no reason to duplicate the functionality.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 5 lines · Ben Greear
Couldn't the bonding code be made to distribute pkts to one interface or another based on a hash of the sending IP port or something? Seems like that would fix the reordering problem for IP packets.... It wouldn't help for a single stream, but I'm guessing the real world problem involves many streams, which on average should hash such that the load is balanced...
It could, but then it is already implemented in a better way in multipath routing and I see no reason to duplicate the functionality.
On the surface, multi-path routing sounds complicated to me, while layer-2 bonding seems relatively trivial to set up/administer. Since we do support bonding, if it's a simple fix to make it better, we might as well do that, eh?
I haven't used either, so this is just idle supposition on my part...
Ben
Signature · 3 lines
Ben Greear <[email protected]> <[email protected]> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 4 lines · Ben Greear
On the surface, multi-path routing sounds complicated to me, while layer-2 bonding seems relatively trivial to set up/administer. Since we do support bonding, if it's a simple fix to make it better, we might as well do that, eh?
multipath routing is really not complicated; I don't know why it "sounds" complicated to you. Of course you could always add new features to the kernel because the existing ones which do the same thing in a better way "sound complicated" to someone; I doubt it is a good use of developer time however.
BTW when you would teach bonding about flows it wouldn't be layer-2 anymore.
To kill the "sounds complicated" myth:
ip route add 10.0.0.0/8 nexthop dev eth0 nexthop dev eth1
gives you a multipath route with eth0 and eth1 with the same weight for 10.0.0.0 netmask 255.0.0.0. If you replace 10.0.0.0/8 with default it'll be your default route. The kernel does the rest.
-Andi
Quoted 7 lines · Ben Greear
Couldn't the bonding code be made to distribute pkts to one interface or another based on a hash of the sending IP port or something? Seems like that would fix the reordering problem for IP packets.... It wouldn't help for a single stream, but I'm guessing the real world problem involves many streams, which on average should hash such that the load is balanced...
Cisco etherchannel does this, by XOR'ing the dest address with the source address, AND'ing with # of interfaces (limiting you to a power of 2), and then using the number to determine what channel to use.
Now, you end up in a 4 way Etherchannel, all the traffic going down one channel, and the none going down the other three. Does that sound like a balanced solution?
Most bonding problems are either the card driver is busted, the card is busted (el-cheapo NE2000 PCI clones are really bad..) or the switch can't handle it. Most cheap, dumb switches break big time when using Bonding with them.
Signature · 4 lines
------------------------+-------------------------------------------------- Thomas Davis | ASG Cluster guy [email protected] | (510) 486-4524 | "80 nodes and chugging Captain!"
Quoted 10 lines · Thomas Davis
Quoted 7 lines · Ben Greear
Couldn't the bonding code be made to distribute pkts to one interface or another based on a hash of the sending IP port or something? Seems like that would fix the reordering problem for IP packets.... It wouldn't help for a single stream, but I'm guessing the real world problem involves many streams, which on average should hash such that the load is balanced...
Quoted 7 lines
Cisco etherchannel does this, by XOR'ing the dest address with the source address, AND'ing with # of interfaces (limiting you to a power of 2), and then using the number to determine what channel to use.
Now, you end up in a 4 way Etherchannel, all the traffic going down one channel, and the none going down the other three. Does that sound like a balanced solution?
If Cisco can't do a balanced hash, that doesn't mean the idea is bad, it just means they implemented it poorly. I would think that something as simple as: src_ip_port % num_devices would give a fairly even spread. I'm sure you could get fancier and take other fields (dst_ip_port) into account in your hash.
Signature · 3 lines
Ben Greear <[email protected]> <[email protected]> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 11 lines · Andi Kleen
Quoted 4 lines · Ben Greear
On the surface, multi-path routing sounds complicated to me, while layer-2 bonding seems relatively trivial to set up/administer. Since we do support bonding, if it's a simple fix to make it better, we might as well do that, eh?
multipath routing is really not complicated; I don't know why it "sounds" complicated to you. Of course you could always add new features to the kernel because the existing ones which do the same thing in a better way "sound complicated" to someone; I doubt it is a good use of developer time however.
But it is still the wrong layer.
Ethernet Bonding ist "below" IP. Very handy if you want to run something "not ip".
Regards
Henning
Signature · 5 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen -- Geschaeftsfuehrer INTERMETA - Gesellschaft fuer Mehrwertdienste mbH [email protected] Am Schwabachgrund 22 Fon.: 09131 / 50654-0 [email protected] D-91054 Buckenhof Fax.: 09131 / 50654-20