bonding and SR-IOV -- do we need arp_validation for loadbalancing too?
Subsystems (2)
Hi all,
We've been starting to look at bonding VFs from separate physical devices in a guest, but we've run into a problem.
The host is bonding the corresponding PFs, and it uses arp monitoring.
What we have found is that any broadcast traffic from the guest (if
they enable arp monitoring, for example) will be seen by the internal L2 switch of the NIC and sent up into the host, where the bonding driver will count it as incoming packets and use it to mark the link as good.
The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.
2) put all the VMs in VLANs
Anyone have any better ideas?
Chris
Signature · 6 lines
Chris Friesen Software Designer 3500 Carling Avenue Ottawa, Ontario K2H 8E9 www.genband.com
Quoted 12 lines · Tue, Jul 24, 2012 at 05:57:03PM CEST, [email protected] wrote:
Hi all,
We've been starting to look at bonding VFs from separate physical devices in a guest, but we've run into a problem.
The host is bonding the corresponding PFs, and it uses arp monitoring. What we have found is that any broadcast traffic from the guest (if they enable arp monitoring, for example) will be seen by the internal L2 switch of the NIC and sent up into the host, where the bonding driver will count it as incoming packets and use it to mark the link as good.
Quoted 2 lines
The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.
This is my favourite.... No reason to not to turn arp validation on. TEAM device (teamd arpping linkwatch) does arp or NSNA validation always.
Quoted 20 lines
2) put all the VMs in VLANs
Anyone have any better ideas?
Chris
Signature · 11 lines
Chris Friesen Software Designer 3500 Carling Avenue Ottawa, Ontario K2H 8E9 www.genband.com -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 13 lines · Jiri Pirko
Quoted 12 lines · Tue, Jul 24, 2012 at 05:57:03PM CEST, [email protected] wrote:
Hi all,
We've been starting to look at bonding VFs from separate physical devices in a guest, but we've run into a problem.
The host is bonding the corresponding PFs, and it uses arp monitoring. What we have found is that any broadcast traffic from the guest (if they enable arp monitoring, for example) will be seen by the internal L2 switch of the NIC and sent up into the host, where the bonding driver will count it as incoming packets and use it to mark the link as good.
Quoted 6 lines
Quoted 2 lines
The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.This is my favourite.... No reason to not to turn arp validation on. TEAM device (teamd arpping linkwatch) does arp or NSNA validation always.
How does that operate for a load balancing mode?
For arp validate to function (as it's implemented in bonding),
the arp requests (broadcasts) or the arp replies (unicasts) must be seen by each slave at regular intervals. Most load balance systems (etherchannel or 802.3ad, for example) don't flood the broadcast requests to all members of a channel group, and the unicast replies only go to one member.
This generally results in either only one slave staying up, or
slaves going up and down at odd intervals. The arp monitor for the load balance modes is already dependent upon there being a steady stream of traffic to all slaves, and can be unreliable in low traffic conditions (because not all slaves receive traffic with sufficient frequency).
-J
Quoted 3 lines
Quoted 3 lines
2) put all the VMs in VLANs
Anyone have any better ideas?
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Quoted 29 lines · Jay Vosburgh
Quoted 13 lines · Jiri Pirko
Quoted 12 lines · Tue, Jul 24, 2012 at 05:57:03PM CEST, [email protected] wrote:
Hi all,
We've been starting to look at bonding VFs from separate physical devices in a guest, but we've run into a problem.
The host is bonding the corresponding PFs, and it uses arp monitoring. What we have found is that any broadcast traffic from the guest (if they enable arp monitoring, for example) will be seen by the internal L2 switch of the NIC and sent up into the host, where the bonding driver will count it as incoming packets and use it to mark the link as good.
Quoted 5 lines
Quoted 2 lines
The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.This is my favourite.... No reason to not to turn arp validation on. TEAM device (teamd arpping linkwatch) does arp or NSNA validation always.
How does that operate for a load balancing mode?
For arp validate to function (as it's implemented in bonding),
the arp requests (broadcasts) or the arp replies (unicasts) must be seen by each slave at regular intervals. Most load balance systems (etherchannel or 802.3ad, for example) don't flood the broadcast requests to all members of a channel group, and the unicast replies only go to one member.
Quoted 5 lines
This generally results in either only one slave staying up, or
slaves going up and down at odd intervals. The arp monitor for the load balance modes is already dependent upon there being a steady stream of traffic to all slaves, and can be unreliable in low traffic conditions (because not all slaves receive traffic with sufficient frequency).
In loadbalance mode wouldn't it just work similar to active-backup? If it's a reply then verify that it came from the arp target, if it's a request then check to see if it came from one of the other slaves.
In our case we have control over the L2 switches involved so we ensure that the broadcast arp request is sent to all the other slaves, while the reply comes back to the sender. I think we still have a window where you could have a device with a faulty tx but functional rx and never detect the problem in the monitor.
A more general solution might be to have the device driver also track the time of the last incoming packet that came from the external network (rather than a VF) and having the bond driver ignore those packets for the purpose of link health. Doing this efficiently would likely require some kind of hardware support though--as an example the 82599 seems to support this with the "LB" bit in the rx descriptor.
Chris
Quoted 2 lines · Chris Friesen
A more general solution might be to have the device driver also track the time of the last incoming packet that came from the external network (rather than a VF) and having the bond driver ignore those packets for the purpose of link health. Doing this efficiently would likely require some kind of hardware support though--as an example the 82599 seems to support this with the "LB" bit in the rx descriptor.
That should of course be reversed. We want the bond driver to only use the packets from the external network for the purpose of link health.
Does anyone other than bonding actually care about dev->last_rx? If not then we could just change the drivers to only set it for external packets.
Chris
Quoted 36 lines · Chris Friesen
Quoted 29 lines · Jay Vosburgh
Quoted 13 lines · Jiri Pirko
Quoted 12 lines · Tue, Jul 24, 2012 at 05:57:03PM CEST, [email protected] wrote:
Hi all,
We've been starting to look at bonding VFs from separate physical devices in a guest, but we've run into a problem.
The host is bonding the corresponding PFs, and it uses arp monitoring. What we have found is that any broadcast traffic from the guest (if they enable arp monitoring, for example) will be seen by the internal L2 switch of the NIC and sent up into the host, where the bonding driver will count it as incoming packets and use it to mark the link as good.
Quoted 5 lines
Quoted 2 lines
The only solutions I've been able to come up with are:
1) add arp validation for load balancing modes as well as active-backup.This is my favourite.... No reason to not to turn arp validation on. TEAM device (teamd arpping linkwatch) does arp or NSNA validation always.
How does that operate for a load balancing mode?
For arp validate to function (as it's implemented in bonding),
the arp requests (broadcasts) or the arp replies (unicasts) must be seen by each slave at regular intervals. Most load balance systems (etherchannel or 802.3ad, for example) don't flood the broadcast requests to all members of a channel group, and the unicast replies only go to one member.Quoted 5 lines
This generally results in either only one slave staying up, or
slaves going up and down at odd intervals. The arp monitor for the load balance modes is already dependent upon there being a steady stream of traffic to all slaves, and can be unreliable in low traffic conditions (because not all slaves receive traffic with sufficient frequency).
Quoted 3 lines
In loadbalance mode wouldn't it just work similar to active-backup? If it's a reply then verify that it came from the arp target, if it's a request then check to see if it came from one of the other slaves.
The problem isn't verifying the requests or replies, it's that
the ARP packets are not distributed across all slaves (because the switch ports are in a channel group / aggregator), so some slaves do not receive any ARPs.
The bond sends the ARP request as a broadcast. For
active-backup, this ends up at the inactive slaves because the switch sends the broadcast to all ports. For a loadbalance mode, the switch won't send the broadcast ARP to the other slaves, because all the slaves are in a channel group or lacp aggregator, which is treated by the switch as effectively a single switch port for this case.
Similarly, the ARP replies are unicast, and the switch will send
those unicast replies to only one member of the channel group or aggregator. The choice there is usually a hash of some kind, so generally only one slave will receive the replies.
Quoted 5 lines
In our case we have control over the L2 switches involved so we ensure that the broadcast arp request is sent to all the other slaves, while the reply comes back to the sender. I think we still have a window where you could have a device with a faulty tx but functional rx and never detect the problem in the monitor.
You can set up -xor or -rr mode against a switch without setting
up a channel group on the switch, but that has the down side that any incoming broadcast or multicast packet may be received multiple times (one copy per slave). Some switches will also disable ports (due to MAC flapping) or complain about seeing the same MAC address on multiple ports for this case. This also will not load balance incoming traffic to the bond very well.
Quoted 8 lines · Chris Friesen
Quoted 6 lines
A more general solution might be to have the device driver also track the time of the last incoming packet that came from the external network (rather than a VF) and having the bond driver ignore those packets for the purpose of link health. Doing this efficiently would likely require some kind of hardware support though--as an example the 82599 seems to support this with the "LB" bit in the rx descriptor.
Quoted 5 lines
That should of course be reversed. We want the bond driver to only use the packets from the external network for the purpose of link health.
Does anyone other than bonding actually care about
dev->last_rx? If not then we could just change the drivers to only set it for external packets.
I believe bonding is the main user of last_rx (a search shows a
couple of drivers using it internally). For bonding use, in current mainline last_rx is set by bonding itself, not in the network device driver.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
Le 24/07/2012 22:49, Jay Vosburgh a écrit :
[...]
Quoted 16 lines
Quoted 3 lines
In loadbalance mode wouldn't it just work similar to active-backup? If it's a reply then verify that it came from the arp target, if it's a request then check to see if it came from one of the other slaves.
The problem isn't verifying the requests or replies, it's that
the ARP packets are not distributed across all slaves (because the switch ports are in a channel group / aggregator), so some slaves do not receive any ARPs.The bond sends the ARP request as a broadcast. For
active-backup, this ends up at the inactive slaves because the switch sends the broadcast to all ports. For a loadbalance mode, the switch won't send the broadcast ARP to the other slaves, because all the slaves are in a channel group or lacp aggregator, which is treated by the switch as effectively a single switch port for this case.
Quoted 4 lines
Similarly, the ARP replies are unicast, and the switch will send
those unicast replies to only one member of the channel group or aggregator. The choice there is usually a hash of some kind, so generally only one slave will receive the replies.
I assume team should suffer the exact same problem, because most of this is on the switch side and out of the control of the host. Jiri, can you confirm?
[...]
Quoted 4 lines
I believe bonding is the main user of
last_rx(a search shows a
couple of drivers using it internally). For bonding use, in current mainlinelast_rxis set by bonding itself, not in the network device driver.
If last_rx is set and used internally by bonding and mostly unused elsewhere, can't we remove it from net_device and move it into private data for the slaves in bonding?
A comment in netdevice.h even recommends not to set it into drivers:
unsigned long last_rx; /* Time of last Rx
* This should not be set in
* drivers, unless really needed,
* because network stack (bonding)
* use it if/when necessary, to
* avoid dirtying this cache line. */
Nicolas.
Quoted 10 lines · Jay Vosburgh
Quoted 3 lines · Chris Friesen
In loadbalance mode wouldn't it just work similar to active-backup? If it's a reply then verify that it came from the arp target, if it's a request then check to see if it came from one of the other slaves.
The problem isn't verifying the requests or replies, it's that
the ARP packets are not distributed across all slaves (because the switch ports are in a channel group / aggregator), so some slaves do not receive any ARPs.
Yeah, okay. And if we turn on arp validation then we ignore all the other packets and so they looks dead. Got it.
In our environment (ATCA shelf) the switches have been customized to handle some of this stuff so arpmon does work reliably with xor.
In the general case it sounds like the "PF bonding ignores packets from VFs" is a better bet then.
Quoted 13 lines
Quoted 8 lines · Chris Friesen
Quoted 6 lines
A more general solution might be to have the device driver also track the time of the last incoming packet that came from the external network (rather than a VF) and having the bond driver ignore those packets for the purpose of link health. Doing this efficiently would likely require some kind of hardware support though--as an example the 82599 seems to support this with the "LB" bit in the rx descriptor.
Quoted 5 lines
That should of course be reversed. We want the bond driver to only use the packets from the external network for the purpose of link health.
Does anyone other than bonding actually care about
dev->last_rx? If not then we could just change the drivers to only set it for external packets.
Quoted 7 lines
That should of course be reversed. We want the bond driver to only use
Quoted 2 lines
the packets from the external network for the purpose of link health.
I believe bonding is the main user of
last_rx(a search shows a
couple of drivers using it internally). For bonding use, in current mainlinelast_rxis set by bonding itself, not in the network device driver.
Right, I was looking at older code. In that case presumably the driver could set an skb flag (external vs VF loopback) that the bonding code could check?
Chris
Quoted 7 lines · Chris Friesen
Quoted 6 lines · Jay Vosburgh
Quoted 3 lines · Chris Friesen
In loadbalance mode wouldn't it just work similar to active-backup? If it's a reply then verify that it came from the arp target, if it's a request then check to see if it came from one of the other slaves.
Quoted 7 lines
Quoted 4 lines
The problem isn't verifying the requests or replies, it's that
the ARP packets are not distributed across all slaves (because the switch ports are in a channel group / aggregator), so some slaves do not receive any ARPs.Yeah, okay. And if we turn on arp validation then we ignore all the other packets and so they looks dead. Got it.
If you wanted to make this work, I think it could pretty easily be done if it isn't working right now. Did you try arp_validate=all in your setup?
Quoted 2 lines
In our environment (ATCA shelf) the switches have been customized to handle some of this stuff so arpmon does work reliably with xor.
Good.
Quoted 2 lines
In the general case it sounds like the "PF bonding ignores packets from VFs" is a better bet then.
It really might be. There are some registers in the 82599 datasheets that are not used by the ixgbe driver, but might help you in this area.
If you take a look at PFVML2FLT and PFUTA and their current status on your system you might be able to put something together that gives you what you want.
It would likely mean you have to run a custom ixgbe-driver, but that doesn't sound like much of an issue.
Quoted 32 lines
Quoted 15 lines
Quoted 10 lines · Chris Friesen
Quoted 6 lines
A more general solution might be to have the device driver also track the time of the last incoming packet that came from the external network (rather than a VF) and having the bond driver ignore those packets for the purpose of link health. Doing this efficiently would likely require some kind of hardware support though--as an example the 82599 seems to support this with the "LB" bit in the rx descriptor.
Quoted 5 lines
That should of course be reversed. We want the bond driver to only use the packets from the external network for the purpose of link health.
Does anyone other than bonding actually care about
dev->last_rx? If not then we could just change the drivers to only set it for external packets.Quoted 7 lines
That should of course be reversed. We want the bond driver to only use
Quoted 2 lines
the packets from the external network for the purpose of link health.
I believe bonding is the main user of
last_rx(a search shows a
couple of drivers using it internally). For bonding use, in current mainlinelast_rxis set by bonding itself, not in the network device driver.Right, I was looking at older code. In that case presumably the driver could set an skb flag (external vs VF loopback) that the bonding code could check?
Chris
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Andy Gospodarek
Quoted 2 lines · Chris Friesen
In our environment (ATCA shelf) the switches have been customized to handle some of this stuff so arpmon does work reliably with xor.
Good.
Quoted 8 lines
Quoted 2 lines
In the general case it sounds like the "PF bonding ignores packets from VFs" is a better bet then.
It really might be. There are some registers in the 82599 datasheets that are not used by the ixgbe driver, but might help you in this area.
If you take a look at PFVML2FLT and PFUTA and their current status on your system you might be able to put something together that gives you what you want.
I think it's simpler than that. By my reading of the 82599 datasheet, the LB bit in the status field of the rx descriptor will indicate if the packet came from a VM.
Quoted 2 lines
It would likely mean you have to run a custom ixgbe-driver, but that doesn't sound like much of an issue.
While I'd like to avoid it if possible, we've already had to tweak the driver for other things.
The complication is that we've got a few different types of hardware and they're not all running 82599. I'm trying to figure out if the other hardware can do something similar.
Chris
Quoted 12 lines · Chris Friesen
Quoted 11 lines · Andy Gospodarek
Quoted 2 lines · Chris Friesen
In our environment (ATCA shelf) the switches have been customized to handle some of this stuff so arpmon does work reliably with xor.
Good.
Quoted 2 lines
In the general case it sounds like the "PF bonding ignores packets from VFs" is a better bet then.
It really might be. There are some registers in the 82599 datasheets that are not used by the ixgbe driver, but might help you in this area.
Quoted 8 lines
Quoted 3 lines
If you take a look at PFVML2FLT and PFUTA and their current status on your system you might be able to put something together that gives you what you want.
I think it's simpler than that. By my reading of the 82599 datasheet, the LB bit in the status field of the rx descriptor will indicate if the packet came from a VM.
Ah, you are correct. I'm curious if this bit is also set on VF<->VF traffic.
Quoted 6 lines
Quoted 2 lines
It would likely mean you have to run a custom ixgbe-driver, but that doesn't sound like much of an issue.
While I'd like to avoid it if possible, we've already had to tweak the driver for other things.
As long as driver resets are not needed for some of these changes to take effect, there is a chance you can write some of these bits from a userspace program. I haven't done it myself, but I should try it. That might allow you to run a stock driver from a distro and still get what you want.
Quoted 3 lines
The complication is that we've got a few different types of hardware and they're not all running 82599. I'm trying to figure out if the other hardware can do something similar.
That would indicate that this would be something that the stack as well as some of the stacked modules bonding, vlans, bridges, etc. may need to have some special code to handle. I'm not sure I'm a big fan right now, but I'll hold off judgement until we see how this shakes out.