gratuitous arps lost during IB switch failure
Hi All,
We are using dual ported HCAs connected with each port connected to 2 different IB switches so that we can tolerate the failure of any one of those switches and we are trying to cut down the amount of time it takes for traffic (TCP & RDS) to resume when there is an IB switch failure and the hosts failover from one port to the other.
We have the bonding driver configured in active-backup mode and setup to send out 100 gratuitous arps at intervals of 100ms whenever there is a failover. In most cases, traffic resumes within a few seconds after a failover because these gratuitous arps take care of updating all the nodes with the new IP:GID mapping.
The problem we are seeing is that sometimes, one or more of the nodes on the fabric do not receive even 1 of these gratuitous arps and re-establishing communication with these nodes takes a much longer time (over 40 seconds) as it depends on various arp cache timeouts. Does anyone know why all these gratuitous arps might be lost?
Besides the gratuitous arp settings, are there any other tunables to look at to minimize the time it takes for IPoIB traffic to resume?
- Sumeet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
It turns out that this problem was being caused because we had multiple IPs configured on the bonded infiniband interface. It appears that grat. arps are being sent out for only one of those IPs.
For example, if we configure the bonded interfaces as below, and we trigger a failover, we only see grat arps go out for the bond0:1 IP. Can the bonding driver be fixed to send out grat arps for both these IPs?
# cat ifcfg-bond0 DEVICE=bond0 USERCTL=no BOOTPROTO=none ONBOOT=yes IPADDR=192.168.50.118 NETMASK=255.255.254.0 NETWORK=192.168.50.0 BROADCAST=192.168.51.255 BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" MTU=65520
# cat ifcfg-bond0:1 DEVICE=bond0:1 USERCTL=no BOOTPROTO=none ONBOOT=yes IPADDR=192.169.50.118 NETMASK=255.255.254.0 NETWORK=192.169.50.0 BROADCAST=192.169.51.255 BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000" MTU=65520
- Sumeet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Sumeet
It turns out that this problem was being caused because we had multiple IPs configured on the bonded infiniband interface. It appears that grat. arps are being sent out for only one of those IPs. [...] Can the bonding driver be fixed to send out grat arps for both these IPs?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Sumeet
It turns out that this problem was being caused because we had multiple IPs configured on the bonded infiniband interface. It appears that grat. arps are being sent out for only one of those IPs. [...] Can the bonding driver be fixed to send out grat arps for both these IPs?
is there anything that makes you think this issue has something to do with ipoib/bonding? did you check with ethernet? the bonding driver isn't maintained over linux-rdma but rather over netdev.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html