Bonding can't show correct speed if lower interface is bond 802.3ad
Subsystems (2)
Hi Jay,
A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.
This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.
Here is a reproducer:
```
#!/bin/bash
s_ns="s"
c_ns="c"
ip netns del ${c_ns} &> /dev/null
ip netns del ${s_ns} &> /dev/null
sleep 1
ip netns add ${c_ns}
ip netns add ${s_ns}
ip -n ${c_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond1 type bond mode active-backup miimon 100
for i in $(seq 0 2); do
ip -n ${c_ns} link add eth${i} type veth peer name eth${i} netns ${s_ns}
[ $i -eq 2 ] && break
ip -n ${c_ns} link set eth${i} master bond0
ip -n ${s_ns} link set eth${i} master bond0
done
ip -n ${c_ns} link set eth2 up
ip -n ${c_ns} link set bond0 up
ip -n ${s_ns} link set bond0 master bond1
ip -n ${s_ns} link set bond1 up
sleep 5
ip netns exec ${s_ns} ethtool bond0 | grep Speed
ip netns exec ${s_ns} ethtool bond1 | grep Speed
```When run the reproducer directly, you will see:
# ./bond_topo_lacp.sh
Speed: 20000Mb/s
Speed: 10000Mb/sSo do you have any thoughts about how to fix it?
Thanks
Hangbin
Quoted 10 lines · Hangbin Liu
A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.
This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via
bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.
Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.
In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.
Is there a reason the user in your case doesn't use 802.3ad mode
directly?
Quoted 36 lines
Here is a reproducer:
``` #!/bin/bash s_ns="s" c_ns="c" ip netns del ${c_ns} &> /dev/null ip netns del ${s_ns} &> /dev/null sleep 1 ip netns add ${c_ns} ip netns add ${s_ns} ip -n ${c_ns} link add bond0 type bond mode 802.3ad miimon 100 ip -n ${s_ns} link add bond0 type bond mode 802.3ad miimon 100 ip -n ${s_ns} link add bond1 type bond mode active-backup miimon 100 for i in $(seq 0 2); do ip -n ${c_ns} link add eth${i} type veth peer name eth${i} netns ${s_ns} [ $i -eq 2 ] && break ip -n ${c_ns} link set eth${i} master bond0 ip -n ${s_ns} link set eth${i} master bond0 done ip -n ${c_ns} link set eth2 up ip -n ${c_ns} link set bond0 up ip -n ${s_ns} link set bond0 master bond1 ip -n ${s_ns} link set bond1 up sleep 5 ip netns exec ${s_ns} ethtool bond0 | grep Speed ip netns exec ${s_ns} ethtool bond1 | grep Speed ```
Quoted 6 lines
When run the reproducer directly, you will see:
# ./bond_topo_lacp.sh Speed: 20000Mb/s Speed: 10000Mb/sSo do you have any thoughts about how to fix it?
Maybe it's time to disable nesting of bonds, update the
documentation to note that it's disabled and that 802.3ad mode is smart enough to do multiple aggregators, and then see if anyone has some other use case and complains.
In the past, I've been against doing this, but only because it
might break existing configurations. If nested configurations are going to misbehave and require complicated shenanigans to fix, then perhaps it's time to push users into a configuration that works without the nesting.
The only thing I can think of that active-backup over 802.3ad
gets is the gratuitous ARP / NS on failover. If that's the key feature for nesting, then I'd rather add the grat ARP to 802.3ad aggregator selection and disable nesting.
-J
---
-Jay Vosburgh, [email protected]
Quoted 20 lines · Jay Vosburgh
Quoted 10 lines · Hangbin Liu
A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.
This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via
bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.
Quoted 8 lines
In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.Is there a reason the user in your case doesn't use 802.3ad mode
directly?
Hi Jay,
I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.
What do you think?
Thanks
Hangbin
Quoted 30 lines · Hangbin Liu
Quoted 20 lines · Jay Vosburgh
Quoted 10 lines · Hangbin Liu
A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.
This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via
bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.Quoted 8 lines
In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.Is there a reason the user in your case doesn't use 802.3ad mode
directly?
Quoted 8 lines
Hi Jay,
I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.
What do you think?
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().
-J
---
-Jay Vosburgh, [email protected]
Quoted 14 lines · Jay Vosburgh
Quoted 8 lines
Hi Jay,
I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.
What do you think?
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.
Quoted 8 lines
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary withinad_agg_selection_test().
Thanks for your explanation. I didn't know this before. Now I have learned.
Regards
Hangbin
Quoted 14 lines · Jay Vosburgh
Quoted 8 lines
Hi Jay,
I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.
What do you think?
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.
Quoted 8 lines
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary withinad_agg_selection_test().
cc Andrew, He add active-backup bond over LACP bond because he want to use arp_ip_target to ensure that the target network is reachable...
Hangbin
Hi Hangbin & Jay,
Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.Quoted 8 lines
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary withinad_agg_selection_test().
Quoted 2 lines
cc Andrew, He add active-backup bond over LACP bond because he want to use
arp_ip_targetto ensure that the target network is reachable...
That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.
I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in drivers/net/bonding/bond_3ad.c to set is_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only in ad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?
But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.
If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.
Best regards,
Andy
Sorry -- resending from a different email address to fix a problem with gmail rejecting it.
Quoted 48 lines · Andrew J. Schorr
Hi Hangbin & Jay,
Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.Quoted 8 lines
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary withinad_agg_selection_test().Quoted 2 lines
cc Andrew, He add active-backup bond over LACP bond because he want to use
arp_ip_targetto ensure that the target network is reachable...That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.
I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in
drivers/net/bonding/bond_3ad.cto setis_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only inad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.
If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.
Best regards,
Andy
Quoted 39 lines · Andrew J. Schorr
Sorry -- resending from a different email address to fix a problem with gmail rejecting it.
Quoted 35 lines · Andrew J. Schorr
Hi Hangbin & Jay,
Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh
That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.Quoted 8 lines
If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary withinad_agg_selection_test().Quoted 2 lines
cc Andrew, He add active-backup bond over LACP bond because he want to use
arp_ip_targetto ensure that the target network is reachable...That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.
I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in
drivers/net/bonding/bond_3ad.cto setis_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only inad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?
The individual port behavior is part of the LACP standard (IEEE
802.1AX, recent editions call this "Solitary"), and is done automatically by the kernel. One of the reasons for it is to permit exactly the situation you mention: to enable PXE or "fallback" communication to work even if LACP negotiation fails or is not configured or implemented at one end. This is called out explicitly in 802.1AX, 6.1.1.j.
The duplex test is only part of the "individual" logic; it comes
up because LACP negotiation requires the peers to be point-to-point links, i.e., full duplex (IEEE 802.1AX-2014, 6.4.8). That's the norm for most everything now, but historically a port in half duplex could be on a multiple access topology, e.g., 802.3 CSMA/CD 10BASE2 on a coax cable, which is incompatible with LACP aggregation. This situation doesn't come up a lot these days.
The important part of the "individual" logic is whether or not
the port successfully completes LACP negotiation with a link partner. If not, the port is an individual port, which acts essentially like an aggregator with just one port in it. This is separate from "is_individual" in the bonding code, and happens in ad_port_selection_logic(), after the comment "check if current aggregator suits us". "is_individual" is one element of this test, the remaining tests compare the various keys and whether the partner MAC address has been populated.
As far as documentation goes, the bonding docs[0] describe some
of the parameters, but doesn't describe the specifics of bonding's ability to manage multiple aggregators; I should write that up, since this comes up periodically. The IEEE standard (to which the bonding implementation conforms) describes how the whole system works, but doesn't really have a simple overview.
[0] https://www.kernel.org/doc/Documentation/networking/bonding.rst
Quoted 9 lines
Quoted 9 lines
But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.
If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.
I'll have to give this some thought. The best long term
solution would be to decouple the link monitoring stuff from the mode, and thus allow ARP and MII in a wider variety of modes. I've prototyped that out in the past, along with changing the MII monitor to respond to carrier state changes in real time instead of polling, and it's fairly complicated.
In any event, this does sound like a valid use case for nesting
the bonds, so simply disabling that facility seems to be off the table.
-J
---
-Jay Vosburgh, [email protected]
Hi Jay,
Quoted 16 lines · Jay Vosburgh
The individual port behavior is part of the LACP standard (IEEE
802.1AX, recent editions call this "Solitary"), and is done automatically by the kernel. One of the reasons for it is to permit exactly the situation you mention: to enable PXE or "fallback" communication to work even if LACP negotiation fails or is not configured or implemented at one end. This is called out explicitly in 802.1AX, 6.1.1.j.The duplex test is only part of the "individual" logic; it comes
up because LACP negotiation requires the peers to be point-to-point links, i.e., full duplex (IEEE 802.1AX-2014, 6.4.8). That's the norm for most everything now, but historically a port in half duplex could be on a multiple access topology, e.g., 802.3 CSMA/CD 10BASE2 on a coax cable, which is incompatible with LACP aggregation. This situation doesn't come up a lot these days.
Quoted 9 lines
The important part of the "individual" logic is whether or not
the port successfully completes LACP negotiation with a link partner. If not, the port is an individual port, which acts essentially like an aggregator with just one port in it. This is separate from "is_individual" in the bonding code, and happens inad_port_selection_logic(), after the comment "check if current aggregator suits us". "is_individual" is one element of this test, the remaining tests compare the various keys and whether the partner MAC address has been populated.
OK. So it sounds like this should just work automatically with no configuration required to identify which slaves are running in individual mode. Thanks for clarifying.
Quoted 8 lines
As far as documentation goes, the bonding docs[0] describe some
of the parameters, but doesn't describe the specifics of bonding's ability to manage multiple aggregators; I should write that up, since this comes up periodically. The IEEE standard (to which the bonding implementation conforms) describes how the whole system works, but doesn't really have a simple overview.[0] https://www.kernel.org/doc/Documentation/networking/bonding.rst
I noticed the parameters related to this and did do some google searching to learn about having multiple aggregators, but as you say, it would be helpful to have a few more clues about how this works in the Bonding Howto, as well as a mention of this individual port capability.
Quoted 9 lines
I'll have to give this some thought. The best long term
solution would be to decouple the link monitoring stuff from the mode, and thus allow ARP and MII in a wider variety of modes. I've prototyped that out in the past, along with changing the MII monitor to respond to carrier state changes in real time instead of polling, and it's fairly complicated.In any event, this does sound like a valid use case for nesting
the bonds, so simply disabling that facility seems to be off the table.
OK, great. Then I'll stick with this config for now, even though NetworkManager has some brain damage in this area, since it tries to bring up both bonds before the MAC addresses have gotten sorted out, which can leave everything with a random MAC address. I've managed to kludge a solution to this by setting ONBOOT=no for the active-backup bond, which convinces NetworkManager to start it a bit later and somehow fixes the race condition.
Regards,
Andy
Hi,
Quoted 3 lines · Andrew J. Schorr
OK. So it sounds like this should just work automatically with no configuration required to identify which slaves are running in individual mode. Thanks for clarifying.
Just to follow up on this -- for test purposes, I booted the system with the 802.3ad bond containing both the 20 Gbps port-channel and the individual 1 Gbps port on the other switch, and it worked as expected. The only drawback to this configuration is the lack of ARP monitoring, so I will stick with the active-backup bond on top of the 802.3ad bond.
Regards,
Andy