remove claim balance_rr won't reorder on many to one
Subsystems (3)
PATCH 1/1 remove claim balance_rr won't reorder on many to one
Remove the text which suggests that many balance_rr links feeding into a single uplink will not experience packet reordering.
More up-to-date tests, with 1G links feeding into a switch with a 10G uplink, using a 2.6.23-rc8 kernel on the system on which the 1G links were bonded with balance_rr (mode=0) shows that even a many to one link configuration will experience packet reordering and the attendant TCP issues involving spurrious retransmissions and the congestion window. This happens even with a single, simple bulk transfer such as a netperf TCP_STREAM test. A more complete description of the tests and results, including tcptrace analysis of packet traces showing the degree of reordering and such can be found at:
http://marc.info/?l=linux-netdev&m=119101513406349&w=2
Also, note that some switches use the term "trunking" in a context other than link aggregation.
---
Documentation/networking/bonding.txt +3 -18
@@ -1696,23 +1696,6 @@ balance-rr: This mode is the only mode t interface's worth of throughput, even after adjusting tcp_reordering. - Note that this out of order delivery occurs when both the - sending and receiving systems are utilizing a multiple - interface bond. Consider a configuration in which a - balance-rr bond feeds into a single higher capacity network - channel (e.g., multiple 100Mb/sec ethernets feeding a single - gigabit ethernet via an etherchannel capable switch). In this - configuration, traffic sent from the multiple 100Mb devices to - a destination connected to the gigabit device will not see - packets out of order. However, traffic sent from the gigabit - device to the multiple 100Mb devices may or may not see - traffic out of order, depending upon the balance policy of the - switch. Many switches do not support any modes that stripe - traffic (instead choosing a port based upon IP or MAC level - addresses); for those devices, traffic flowing from the - gigabit device to the many 100Mb devices will only utilize one - interface. - If you are utilizing protocols other than TCP/IP, UDP for example, and your application can tolerate out of order delivery, then this mode can allow for single stream datagram
@@ -1720,7 +1703,9 @@ balance-rr: This mode is the only mode t to the bond. This mode requires the switch to have the appropriate ports - configured for "etherchannel" or "trunking." + configured for "etherchannel" or "aggregation." N.B. some + switches might use the term "trunking" for something other + than link aggregation. active-backup: There is not much advantage in this network topology to the active-backup mode, as the inactive backup devices are all
Rick Jones <[email protected]> wrote:
[...]
Quoted 6 lines
Documentation/networking/bonding.txt +0
-6- Note that this out of order delivery occurs when both the - sending and receiving systems are utilizing a multiple - interface bond. Consider a configuration in which a - balance-rr bond feeds into a single higher capacity network - channel (e.g., multiple 100Mb/sec ethernets feeding a single - gigabit ethernet via an etherchannel capable switch). In this
Documentation/networking/bonding.txt balance-rr: This mode is the only mode t · line 1714
Quoted 10 lines
Documentation/networking/bonding.txt +0
-10- configuration, traffic sent from the multiple 100Mb devices to - a destination connected to the gigabit device will not see - packets out of order. However, traffic sent from the gigabit - device to the multiple 100Mb devices may or may not see - traffic out of order, depending upon the balance policy of the - switch. Many switches do not support any modes that stripe - traffic (instead choosing a port based upon IP or MAC level - addresses); for those devices, traffic flowing from the - gigabit device to the many 100Mb devices will only utilize one - interface.
Rather than simply removing this entirely (because I do think
there is value in discussion of the reordering aspects of balance-rr), I'd rather see something that makes the following points:
1- the worst reordering is balance-rr to balance-rr, back to
back. The reordering rate here depends upon (a) the number of slaves involved and (b) packet reception scheduling behaviors (packet coalescing, NAPI, etc), and thus will vary signficantly, but won't be better than case #2.
2- next worst is "balance-rr many slow" to "single fast", with
the reordering rate generally being substantially lower than case #1 (it looked like your test showed about a 1% reordering rate, if I'm reading your data correctly).
3- For the "single fast" to "balance-rr many" case, going
through a switch configured for etherchannel "may or may not see traffic out of order, depending upon the balance policy of the switch. Many switches do not support any modes that stripe traffic (instead choosing a port based upon IP or MAC level addresses); for those devices, traffic flowing from the [single fast] device to the [balance-rr many] devices will only utilize one interface."
[...]
Documentation/networking/bonding.txt balance-rr: This mode is the only mode t · line 1708
Quoted 5 lines
This mode requires the switch to have the appropriate ports
Documentation/networking/bonding.txt +3
-1- configured for "etherchannel" or "trunking." + configured for "etherchannel" or "aggregation." N.B. some + switches might use the term "trunking" for something other + than link aggregation.
If memory serves, Sun uses the term "trunking" to refer to
"etherchannel" compatible behavior.
I'm also hearing "aggregation" used to described 802.3ad
specifically.
Perhaps text of the form:
This mode requires the switch to have the appropriate ports
configured for "Etherchannel." Some switches use different terms, so the configuration may be called "trunking" or "aggregation." Note that both of these terms also have other meanings. For example, "trunking" is also used to describe a type of switch port, and "aggregation" or "link aggregation" is often used to refer to 802.3ad link aggregation, which is compatible with bonding's 802.3ad mode, but not balance-rr.
Thoughts? -J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 36 lines · Jay Vosburgh
Rick Jones <[email protected]> wrote:
[...]Quoted 6 lines
Documentation/networking/bonding.txt +0
-6- Note that this out of order delivery occurs when both the - sending and receiving systems are utilizing a multiple - interface bond. Consider a configuration in which a - balance-rr bond feeds into a single higher capacity network - channel (e.g., multiple 100Mb/sec ethernets feeding a single - gigabit ethernet via an etherchannel capable switch). In thisDocumentation/networking/bonding.txt balance-rr: This mode is the only mode t · line 1714
Quoted 10 lines
Documentation/networking/bonding.txt +0
-10- configuration, traffic sent from the multiple 100Mb devices to - a destination connected to the gigabit device will not see - packets out of order. However, traffic sent from the gigabit - device to the multiple 100Mb devices may or may not see - traffic out of order, depending upon the balance policy of the - switch. Many switches do not support any modes that stripe - traffic (instead choosing a port based upon IP or MAC level - addresses); for those devices, traffic flowing from the - gigabit device to the many 100Mb devices will only utilize one - interface.Rather than simply removing this entirely (because I do think
there is value in discussion of the reordering aspects of balance-rr), I'd rather see something that makes the following points:1- the worst reordering is balance-rr to balance-rr, back to
back. The reordering rate here depends upon (a) the number of slaves involved and (b) packet reception scheduling behaviors (packet coalescing, NAPI, etc), and thus will vary signficantly, but won't be better than case #2.2- next worst is "balance-rr many slow" to "single fast", with
the reordering rate generally being substantially lower than case #1 (it looked like your test showed about a 1% reordering rate, if I'm reading your data correctly).
Quoted 7 lines
3- For the "single fast" to "balance-rr many" case, going
through a switch configured for etherchannel "may or may not see traffic out of order, depending upon the balance policy of the switch. Many switches do not support any modes that stripe traffic (instead choosing a port based upon IP or MAC level addresses); for those devices, traffic flowing from the [single fast] device to the [balance-rr many] devices will only utilize one interface."
I have to wonder if the full description of the different versions of being a little bit pregnant is worth it. Just saying that using balance-rr will result in reordering seems much more simple to comprehend. Also, since balance-rr is strictly an outbound policy, does case three even enter into it - as you say, that will be up to the switch, which will be doing whatever it was told or felt like doing regardless of balance-rr on the bond in the host.
Quoted 12 lines
[...]
Documentation/networking/bonding.txt balance-rr: This mode is the only mode t · line 1708
Quoted 5 lines
This mode requires the switch to have the appropriate ports
Documentation/networking/bonding.txt +3
-1- configured for "etherchannel" or "trunking." + configured for "etherchannel" or "aggregation." N.B. some + switches might use the term "trunking" for something other + than link aggregation.If memory serves, Sun uses the term "trunking" to refer to
"etherchannel" compatible behavior.
I'm not really all that tied to that part of the change - it is there because I noticed in one of the HP ITRC forums someone talking about a switch (Cisco?) where trunking meant something with vlans rather than aggregation.
Quoted 14 lines
I'm also hearing "aggregation" used to described 802.3ad
specifically.Perhaps text of the form:
This mode requires the switch to have the appropriate ports
configured for "Etherchannel." Some switches use different terms, so the configuration may be called "trunking" or "aggregation." Note that both of these terms also have other meanings. For example, "trunking" is also used to describe a type of switch port, and "aggregation" or "link aggregation" is often used to refer to 802.3ad link aggregation, which is compatible with bonding's 802.3ad mode, but not balance-rr.
Quoted 1 line
Thoughts?
Even better would be to be able to start to move away from "etherchannel" towards the de jure standard's terms, whatever the heck they are :)
rick jones
Quoted 3 lines · Rick Jones
I have to wonder if the full description of the different versions of being a little bit pregnant is worth it. Just saying that using balance-rr will result in reordering seems much more simple to comprehend.
True, but the different configurations produce very different
levels of reordering.
There seem to be users out there trying to use balance-rr to
maximize single stream TCP throughput (even with reordering), so I think the relative badness information is worthwhile.
Quoted 4 lines
Also, since balance-rr is strictly an outbound policy, does case three even enter into it - as you say, that will be up to the switch, which will be doing whatever it was told or felt like doing regardless of balance-rr on the bond in the host.
Point three provides an answer to a question I've been asked
pretty regularly by customers, so I think it's good information.
[...]
Quoted 4 lines
I'm not really all that tied to that part of the change - it is there because I noticed in one of the HP ITRC forums someone talking about a switch (Cisco?) where trunking meant something with vlans rather than aggregation.
In Ciscoville, switch ports can be configured as either "access"
or "trunk." A trunk port accepts all VLANs, an access port is tied to a specific VLAN (simplifying some here). The Cisco documentation uses the term EtherChannel to descibe the link aggregation system we're talking about here in reference to bonding's balance-rr mode.
Quoted 2 lines
Even better would be to be able to start to move away from "etherchannel" towards the de jure standard's terms, whatever the heck they are :)
I believe that EtherChannel is the standard term for what we're
talking about here, but it's a Cisco trademark. I'd guess that most switch vendors don't come right out and call their "EtherChannel(tm) compatible" mode exactly that; they call it something else, but it's still meant to be compatible with EtherChannel.
For bonding, this applies to the balance-rr and balance-xor
modes.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 8 lines · Jay Vosburgh
Quoted 3 lines · Rick Jones
I have to wonder if the full description of the different versions of being a little bit pregnant is worth it. Just saying that using balance-rr will result in reordering seems much more simple to comprehend.
Quoted 6 lines
True, but the different configurations produce very different
levels of reordering.There seem to be users out there trying to use balance-rr to
maximize single stream TCP throughput (even with reordering), so I think the relative badness information is worthwhile.
I will admit to coming from an "if you want a single stream to go faster buy the next higher speed NIC" point of view, which means I pretty much lump all the degrees of reordering badness together.
The relative badness though is likely a _very_ broad space which couldn't be covered adequately in just a paragraph or two. Notice how long my email describing my one experiment ended-up.
For example, I suspect, but have not verified that the one way one might get minimal reordering with many to one would be to have a sender with the many interfaces slow enough to not be able to get ahead of the sum of the NICs in the bond, so the transmit queues all remain at 0, coupled perhaps with NICs all in equal-speed and equal-feed I/O slots.
Start to be able to keep ahead of one or more of the NICs and soon we are starting along a rather long continuum which includes whether there are other concurrent connections, the distribution of send() sized by the applications, whether or not various offloads are enabled etc etc etc...
Quoted 8 lines
Quoted 4 lines
Also, since balance-rr is strictly an outbound policy, does case three even enter into it - as you say, that will be up to the switch, which will be doing whatever it was told or felt like doing regardless of balance-rr on the bond in the host.
Point three provides an answer to a question I've been asked
pretty regularly by customers, so I think it's good information.
But since it isn't specific to balance_rr it would seem better placed in a "Switch Considerations" or "Inbound Considerations" section?
Quoted 9 lines
Quoted 2 lines
Even better would be to be able to start to move away from "etherchannel" towards the de jure standard's terms, whatever the heck they are :)
I believe that EtherChannel is the standard term for what we're
talking about here, but it's a Cisco trademark. I'd guess that most switch vendors don't come right out and call their "EtherChannel(tm) compatible" mode exactly that; they call it something else, but it's still meant to be compatible with EtherChannel.
Well, that assumes that many switch vendors are still including EtherChannel. I know of at least one non-trivial switch vendor which has consolidated on LACP/802.3ad. When that vendor was supporting EtherChannel, they called it such.
rick jones
Quoted 4 lines
2- next worst is "balance-rr many slow" to "single fast", with
the reordering rate generally being substantially lower than case #1 (it looked like your test showed about a 1% reordering rate, if I'm reading your data correctly).
The percentage of reordering for TCP is likely capped by its effect on the congestion window, limiting the number of outstanding segments.
rick jones
Jay -
So, where do you and I stand wrt the proposed changes to bonding.txt? Are we at an impass?
sincerely,
rick jones
Quoted 2 lines · Rick Jones
So, where do you and I stand wrt the proposed changes to bonding.txt? Are we at an impass?
Nope, I'm doing a doc update next to incorporate several things,
the reordering stuff included (which I plan to change to describe the levels of badness, as it were). Needed to fix a couple "no workee" things first.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 10 lines · Jay Vosburgh
Quoted 2 lines · Rick Jones
So, where do you and I stand wrt the proposed changes to bonding.txt? Are we at an impass?
Nope, I'm doing a doc update next to incorporate several things,
the reordering stuff included (which I plan to change to describe the levels of badness, as it were). Needed to fix a couple "no workee" things first.
OK. I look forward to applying my critical eye to that section :)
rick jones