Skip to contentSkip to conversation

Bonding can't show correct speed if lower interface is bond 802.3ad

Issue Hangbin Liu 4 participants 11 messages Apr 28, 2023 – May 16, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Jay,

A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.

This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.

Here is a reproducer:

```
#!/bin/bash
s_ns="s"
c_ns="c"

ip netns del ${c_ns} &> /dev/null
ip netns del ${s_ns} &> /dev/null
sleep 1
ip netns add ${c_ns}
ip netns add ${s_ns}

ip -n ${c_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond1 type bond mode active-backup miimon 100

for i in $(seq 0 2); do
        ip -n ${c_ns} link add eth${i} type veth peer name eth${i} netns ${s_ns}
        [ $i -eq 2 ] && break
        ip -n ${c_ns} link set eth${i} master bond0
        ip -n ${s_ns} link set eth${i} master bond0
done

ip -n ${c_ns} link set eth2 up
ip -n ${c_ns} link set bond0 up

ip -n ${s_ns} link set bond0 master bond1
ip -n ${s_ns} link set bond1 up

sleep 5

ip netns exec ${s_ns} ethtool bond0 | grep Speed
ip netns exec ${s_ns} ethtool bond1 | grep Speed
```

When run the reproducer directly, you will see:

# ./bond_topo_lacp.sh
        Speed: 20000Mb/s
        Speed: 10000Mb/s

So do you have any thoughts about how to fix it?

Thanks
Hangbin

#1 lore raw
Quoted 10 lines · Hangbin Liu

A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.

This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.

Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.

In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.

Is there a reason the user in your case doesn't use 802.3ad mode
directly?

Quoted 36 lines

Here is a reproducer:

```
#!/bin/bash
s_ns="s"
c_ns="c"

ip netns del ${c_ns} &> /dev/null
ip netns del ${s_ns} &> /dev/null
sleep 1
ip netns add ${c_ns}
ip netns add ${s_ns}

ip -n ${c_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond0 type bond mode 802.3ad miimon 100
ip -n ${s_ns} link add bond1 type bond mode active-backup miimon 100

for i in $(seq 0 2); do
       ip -n ${c_ns} link add eth${i} type veth peer name eth${i} netns ${s_ns}
       [ $i -eq 2 ] && break
       ip -n ${c_ns} link set eth${i} master bond0
       ip -n ${s_ns} link set eth${i} master bond0
done

ip -n ${c_ns} link set eth2 up
ip -n ${c_ns} link set bond0 up

ip -n ${s_ns} link set bond0 master bond1
ip -n ${s_ns} link set bond1 up

sleep 5

ip netns exec ${s_ns} ethtool bond0 | grep Speed
ip netns exec ${s_ns} ethtool bond1 | grep Speed
```
Quoted 6 lines

When run the reproducer directly, you will see:

# ./bond_topo_lacp.sh
       Speed: 20000Mb/s
       Speed: 10000Mb/s

So do you have any thoughts about how to fix it?

Maybe it's time to disable nesting of bonds, update the
documentation to note that it's disabled and that 802.3ad mode is smart enough to do multiple aggregators, and then see if anyone has some other use case and complains.

In the past, I've been against doing this, but only because it
might break existing configurations. If nested configurations are going to misbehave and require complicated shenanigans to fix, then perhaps it's time to push users into a configuration that works without the nesting.

The only thing I can think of that active-backup over 802.3ad
gets is the gratuitous ARP / NS on failover. If that's the key feature for nesting, then I'd rather add the grat ARP to 802.3ad aggregator selection and disable nesting.

	-J

---
-Jay Vosburgh, [email protected]

#2 lore raw
Quoted 20 lines · Jay Vosburgh
Quoted 10 lines · Hangbin Liu

A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.

This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.

Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.

Quoted 8 lines

In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.

Is there a reason the user in your case doesn't use 802.3ad mode
directly?

Hi Jay,

I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.

What do you think?

Thanks
Hangbin

#3 lore raw
Quoted 30 lines · Hangbin Liu
Quoted 20 lines · Jay Vosburgh
Quoted 10 lines · Hangbin Liu

A user reported a bonding issue that if we put an active-back bond on top of a 802.3ad bond interface. When the 802.3ad bond's speed/duplex changed dynamically. The upper bonding interface's speed/duplex can't be changed at the same time.

This seems not easy to fix since we update the speed/duplex only when there is a failover(except 802.3ad mode) or slave netdev change. But the lower bonding interface doesn't trigger netdev change when the speed changed as ethtool get bonding speed via bond_ethtool_get_link_ksettings(), which not affect bonding interface itself.

Well, this gets back into the intermittent discussion on whether
or not being able to nest bonds is useful or not, and thus whether it should be allowed or not. It's at best a niche use case (I don't recall the example configurations ever being anything other than 802.3ad under active-backup), and was broken for a number of years without much uproar.

Quoted 8 lines

In this particular case, nesting two LACP (802.3ad) bonds inside
an active-backup bond provides no functional benefit as far as I'm aware (maybe gratuitous ARP?), as 802.3ad mode will correctly handle switching between multiple aggregators. The "ad_select" option provides a few choices on the criteria for choosing the active aggregator.

Is there a reason the user in your case doesn't use 802.3ad mode
directly?

Quoted 8 lines

Hi Jay,

I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.

What do you think?

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

	-J

---
-Jay Vosburgh, [email protected]

#4 lore raw
Quoted 14 lines · Jay Vosburgh
Quoted 8 lines

Hi Jay,

I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.

What do you think?

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

Quoted 8 lines

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

Thanks for your explanation. I didn't know this before. Now I have learned.

Regards
Hangbin

#5 lore raw
In reply to Jay Vosburgh
Quoted 14 lines · Jay Vosburgh
Quoted 8 lines

Hi Jay,

I just back from holiday and re-read you reply. The user doesn't add 2 LACP bonds inside an active-backup bond. He add 1 LACP bond and 1 normal NIC in to an active-backup bond. This seems reasonable. e.g. The LACP bond in a switch and the normal NIC in another switch.

What do you think?

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

Quoted 8 lines

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

cc Andrew, He add active-backup bond over LACP bond because he want to use arp_ip_target to ensure that the target network is reachable...

Hangbin

#6 lore raw

Hi Hangbin & Jay,

Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

Quoted 8 lines

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

Quoted 2 lines

cc Andrew, He add active-backup bond over LACP bond because he want to use arp_ip_target to ensure that the target network is reachable...

That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.

I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in drivers/net/bonding/bond_3ad.c to set is_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only in ad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?

But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.

If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.

Best regards,
Andy

#7 lore raw

Sorry -- resending from a different email address to fix a problem with gmail rejecting it.

Quoted 48 lines · Andrew J. Schorr

Hi Hangbin & Jay,

Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

Quoted 8 lines

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

Quoted 2 lines

cc Andrew, He add active-backup bond over LACP bond because he want to use arp_ip_target to ensure that the target network is reachable...

That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.

I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in drivers/net/bonding/bond_3ad.c to set is_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only in ad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?

But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.

If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.

Best regards,
Andy

#8 lore raw
Quoted 39 lines · Andrew J. Schorr

Sorry -- resending from a different email address to fix a problem with gmail rejecting it.

Quoted 35 lines · Andrew J. Schorr

Hi Hangbin & Jay,

Quoted 15 lines · Hangbin Liu
Quoted 5 lines · Jay Vosburgh

That case should work fine without the active-backup. LACP has
a concept of an "individual" port, which (in this context) would be the "normal NIC," presuming that that means its link peer isn't running LACP.

Quoted 8 lines

If all of the ports (N that are LACP to a single switch, plus 1
that's the non-LACP "normal NIC") were attached to a single bond, it would create one aggregator with the LACP enabled ports, and then a separate aggregator for the indvidual port that's not. The aggregator selection logic prefers the LACP enabled aggregator over the individual port aggregator. The precise criteria is in the commentary within ad_agg_selection_test().

Quoted 2 lines

cc Andrew, He add active-backup bond over LACP bond because he want to use arp_ip_target to ensure that the target network is reachable...

That's correct. I prefer the ARP monitoring to ensure that the needed connectivity is actually there instead of relying on MII monitoring.

I also confess that I was unaware of the possibility of using an individual port inside an 802.3ad bond without having to stick that individual port into a port-channel group with LACP enabled. I want to avoid enabling LACP on that link because I'd like to be able to PXE boot over it, not to mention the switch configuration hassle. Is that individual port configuration without LACP detected automatically by the kernel, or do I need to configure something to do that? I see the logic in drivers/net/bonding/bond_3ad.c to set is_individual, but it appears to depend on whether duplex is enabled. At that point, I got lost, since I see duplex mentioned only in ad_user_port_key, and that seems to be a property of the bond master, not the slaves. Is there any documentation of how this configuration works?

The individual port behavior is part of the LACP standard (IEEE
802.1AX, recent editions call this "Solitary"), and is done automatically by the kernel. One of the reasons for it is to permit exactly the situation you mention: to enable PXE or "fallback" communication to work even if LACP negotiation fails or is not configured or implemented at one end. This is called out explicitly in 802.1AX, 6.1.1.j.

The duplex test is only part of the "individual" logic; it comes
up because LACP negotiation requires the peers to be point-to-point links, i.e., full duplex (IEEE 802.1AX-2014, 6.4.8). That's the norm for most everything now, but historically a port in half duplex could be on a multiple access topology, e.g., 802.3 CSMA/CD 10BASE2 on a coax cable, which is incompatible with LACP aggregation. This situation doesn't come up a lot these days.

The important part of the "individual" logic is whether or not
the port successfully completes LACP negotiation with a link partner. If not, the port is an individual port, which acts essentially like an aggregator with just one port in it. This is separate from "is_individual" in the bonding code, and happens in ad_port_selection_logic(), after the comment "check if current aggregator suits us". "is_individual" is one element of this test, the remaining tests compare the various keys and whether the partner MAC address has been populated.

As far as documentation goes, the bonding docs[0] describe some
of the parameters, but doesn't describe the specifics of bonding's ability to manage multiple aggregators; I should write that up, since this comes up periodically. The IEEE standard (to which the bonding implementation conforms) describes how the whole system works, but doesn't really have a simple overview.

[0] https://www.kernel.org/doc/Documentation/networking/bonding.rst

Quoted 9 lines
Quoted 9 lines

But in any case, I still prefer active-backup on top of 802.3ad so that I can have the ARP monitoring.

If it's too much trouble to get the top-level bond to report duplex/speed correctly when the underlying bond speed changes, then I think it would be an improvement to set duplex/speed to N/A (or -1) for a bond of bonds configuration instead of potentially having incorrect information. I imagine such a fix might be much easier than updating dynamically when the lower-level 802.3ad bond changes speed.

I'll have to give this some thought. The best long term
solution would be to decouple the link monitoring stuff from the mode, and thus allow ARP and MII in a wider variety of modes. I've prototyped that out in the past, along with changing the MII monitor to respond to carrier state changes in real time instead of polling, and it's fairly complicated.

In any event, this does sound like a valid use case for nesting
the bonds, so simply disabling that facility seems to be off the table.

	-J

---
-Jay Vosburgh, [email protected]

#9 lore raw

Hi Jay,

Quoted 16 lines · Jay Vosburgh

The individual port behavior is part of the LACP standard (IEEE
802.1AX, recent editions call this "Solitary"), and is done automatically by the kernel. One of the reasons for it is to permit exactly the situation you mention: to enable PXE or "fallback" communication to work even if LACP negotiation fails or is not configured or implemented at one end. This is called out explicitly in 802.1AX, 6.1.1.j.

The duplex test is only part of the "individual" logic; it comes
up because LACP negotiation requires the peers to be point-to-point links, i.e., full duplex (IEEE 802.1AX-2014, 6.4.8). That's the norm for most everything now, but historically a port in half duplex could be on a multiple access topology, e.g., 802.3 CSMA/CD 10BASE2 on a coax cable, which is incompatible with LACP aggregation. This situation doesn't come up a lot these days.

Quoted 9 lines

The important part of the "individual" logic is whether or not
the port successfully completes LACP negotiation with a link partner. If not, the port is an individual port, which acts essentially like an aggregator with just one port in it. This is separate from "is_individual" in the bonding code, and happens in ad_port_selection_logic(), after the comment "check if current aggregator suits us". "is_individual" is one element of this test, the remaining tests compare the various keys and whether the partner MAC address has been populated.

OK. So it sounds like this should just work automatically with no configuration required to identify which slaves are running in individual mode. Thanks for clarifying.

Quoted 8 lines

As far as documentation goes, the bonding docs[0] describe some
of the parameters, but doesn't describe the specifics of bonding's ability to manage multiple aggregators; I should write that up, since this comes up periodically. The IEEE standard (to which the bonding implementation conforms) describes how the whole system works, but doesn't really have a simple overview.

[0] https://www.kernel.org/doc/Documentation/networking/bonding.rst

I noticed the parameters related to this and did do some google searching to learn about having multiple aggregators, but as you say, it would be helpful to have a few more clues about how this works in the Bonding Howto, as well as a mention of this individual port capability.

Quoted 9 lines

I'll have to give this some thought. The best long term
solution would be to decouple the link monitoring stuff from the mode, and thus allow ARP and MII in a wider variety of modes. I've prototyped that out in the past, along with changing the MII monitor to respond to carrier state changes in real time instead of polling, and it's fairly complicated.

In any event, this does sound like a valid use case for nesting
the bonds, so simply disabling that facility seems to be off the table.

OK, great. Then I'll stick with this config for now, even though NetworkManager has some brain damage in this area, since it tries to bring up both bonds before the MAC addresses have gotten sorted out, which can leave everything with a random MAC address. I've managed to kludge a solution to this by setting ONBOOT=no for the active-backup bond, which convinces NetworkManager to start it a bit later and somehow fixes the race condition.

Regards,
Andy

#10 lore raw

Hi,

Quoted 3 lines · Andrew J. Schorr

OK. So it sounds like this should just work automatically with no configuration required to identify which slaves are running in individual mode. Thanks for clarifying.

Just to follow up on this -- for test purposes, I booted the system with the 802.3ad bond containing both the 20 Gbps port-channel and the individual 1 Gbps port on the other switch, and it worked as expected. The only drawback to this configuration is the lack of ARP monitoring, so I will stick with the active-backup bond on top of the 802.3ad bond.

Regards,
Andy