bridge: Propagate NETDEV_NOTIFY_PEERS notifier
Subsystems (3)
PATCH 1/1 bridge: Propagate NETDEV_NOTIFY_PEERS notifier
After adding bridge as upper layer of bond/team, we usually clean up the IP address on bond/team and set it on bridge. When there is a failover, bond/team will not send gratuitous ARP since it has no IP address. Then the down layer(e.g. VM tap dev) of bridge will not able to receive this notification.
Make bridge to be able to handle NETDEV_NOTIFY_PEERS notifier.
--- net/bridge/br.c | 1 + 1 file changed, 1 insertion(+)
net/bridge/br.c +1 -0
@@ -125,6 +125,7 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v/* Forbid underlying device to change its type. */ return NOTIFY_BAD ; +case NETDEV_NOTIFY_PEERS :case NETDEV_RESEND_IGMP :/* Propagate to master device */ call_netdevice_notifiers (event, br->dev);
Signature · 1 line
2.26.2
Quoted 19 lines · Hangbin Liu
After adding bridge as upper layer of bond/team, we usually clean up the IP address on bond/team and set it on bridge. When there is a failover, bond/team will not send gratuitous ARP since it has no IP address. Then the down layer(e.g. VM tap dev) of bridge will not able to receive this notification.
Make bridge to be able to handle
NETDEV_NOTIFY_PEERSnotifier.Signed-off-by Hangbin Liu <[email protected]>--- net/bridge/br.c | 1 + 1 file changed, 1 insertion(+)net/bridge/br.c +0
-0
net/bridge/br.c static int br_device_event(struct notifier_block *unused, unsigned long event, v · line 131
Quoted 7 lines
net/bridge/br.c +1
-0@@ -125,6 +125,7 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v/* Forbid underlying device to change its type. */ return NOTIFY_BAD ; +case NETDEV_NOTIFY_PEERS :case NETDEV_RESEND_IGMP :/* Propagate to master device */ call_netdevice_notifiers (event, br->dev);
I'm not convinced this should be done by the bridge, setups usually have multiple ports which may have link change events and these events are unrelated, i.e. we shouldn't generate a gratuitous arp for all every time, there might be many different devices present. We have setups with hundreds of ports which are mixed types of devices.
That seems inefficient, redundant and can potentially cause problems.
Also it seems this was proposed few years back: https://lkml.org/lkml/2018/1/6/135
Thanks,
Nik
Quoted 22 lines · Nikolay Aleksandrov
Quoted 21 lines · Hangbin Liu
After adding bridge as upper layer of bond/team, we usually clean up the IP address on bond/team and set it on bridge. When there is a failover, bond/team will not send gratuitous ARP since it has no IP address. Then the down layer(e.g. VM tap dev) of bridge will not able to receive this notification.
Make bridge to be able to handle
NETDEV_NOTIFY_PEERSnotifier.Signed-off-by Hangbin Liu <[email protected]>--- net/bridge/br.c | 1 + 1 file changed, 1 insertion(+)net/bridge/br.c +0
-0@@ -125,6 +125,7 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v/* Forbid underlying device to change its type. */ return NOTIFY_BAD ;
Quoted 11 lines
net/bridge/br.c static int br_device_event(struct notifier_block *unused, unsigned long event, v · line 131
Quoted 5 lines
net/bridge/br.c +1
-0+case NETDEV_NOTIFY_PEERS :case NETDEV_RESEND_IGMP :/* Propagate to master device */ call_netdevice_notifiers (event, br->dev);I'm not convinced this should be done by the bridge, setups usually have multiple ports which may have link change events and these events are unrelated, i.e. we shouldn't generate a gratuitous arp for all every time, there might be many different devices present. We have setups with hundreds of ports which are mixed types of devices.
That seems inefficient, redundant and can potentially cause problems.
Hi Nikolay,
Thanks for the reply. There are a few reasons I think the bridge should handle NETDEV_NOTIFY_PEERS:
1. Only a few devices will call NETDEV_NOTIFY_PEERS notifier: bond, team, virtio, xen, 6lowpan. There should have no much notification message.
2. When set bond/team's upper layer to bridge. The bridge's mac will be the same with bond/team. So when the bond/team's mac changed, the bridge's mac will also change. So bridge should send a GARP to notify other's that it's mac has changed.
3. There already has NETDEV_RESEND_IGMP handling in bridge, which is also generated by bond/team and netdev_notify_peers(). So why there is IGMP but no ARP?
4. If bridge doesn't have IP address, it will omit GARP sending. So having or not having IP address on bridge doesn't matters.
4. I don't see why how many ports affect the bridge sending GARP.
Please correct me if I missed something.
Quoted 1 line
Also it seems this was proposed few years back: https://lkml.org/lkml/2018/1/6/135
Thanks for this link, cc Stephen for this discuss.
Hangbin
Quoted 35 lines · Hangbin Liu
Quoted 22 lines · Nikolay Aleksandrov
Quoted 21 lines · Hangbin Liu
After adding bridge as upper layer of bond/team, we usually clean up the IP address on bond/team and set it on bridge. When there is a failover, bond/team will not send gratuitous ARP since it has no IP address. Then the down layer(e.g. VM tap dev) of bridge will not able to receive this notification.
Make bridge to be able to handle
NETDEV_NOTIFY_PEERSnotifier.Signed-off-by Hangbin Liu <[email protected]>--- net/bridge/br.c | 1 + 1 file changed, 1 insertion(+)net/bridge/br.c +0
-0@@ -125,6 +125,7 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v/* Forbid underlying device to change its type. */ return NOTIFY_BAD ;Quoted 11 lines
net/bridge/br.c static int br_device_event(struct notifier_block *unused, unsigned long event, v · line 131
Quoted 5 lines
net/bridge/br.c +1
-0+case NETDEV_NOTIFY_PEERS :case NETDEV_RESEND_IGMP :/* Propagate to master device */ call_netdevice_notifiers (event, br->dev);I'm not convinced this should be done by the bridge, setups usually have multiple ports which may have link change events and these events are unrelated, i.e. we shouldn't generate a gratuitous arp for all every time, there might be many different devices present. We have setups with hundreds of ports which are mixed types of devices.
That seems inefficient, redundant and can potentially cause problems.
Quoted 7 lines
Hi Nikolay,
Thanks for the reply. There are a few reasons I think the bridge should handle
NETDEV_NOTIFY_PEERS:1. Only a few devices will call
NETDEV_NOTIFY_PEERSnotifier: bond, team, virtio, xen, 6lowpan. There should have no much notification message.
You can't send a broadcast to all ports because 1 bond's link status has changed. That makes no sense, the GARP needs to be sent only on that bond. The bond devices are heavily used with bridge setups, and in general the bridge is considered a switch device, it shouldn't be broadcasting GARPs to all ports when one changes link state.
Quoted 4 lines
2. When set bond/team's upper layer to bridge. The bridge's mac will be the same with bond/team. So when the bond/team's mac changed, the bridge's mac will also change. So bridge should send a GARP to notify other's that it's mac has changed.
That is not true, the mac doesn't need to be the same at all. And in many situations isn't.
Quoted 3 lines
3. There already has
NETDEV_RESEND_IGMPhandling in bridge, which is also generated by bond/team andnetdev_notify_peers(). So why there is IGMP but no ARP?
Apples and oranges..
Quoted 3 lines
4. If bridge doesn't have IP address, it will omit GARP sending. So having or not having IP address on bridge doesn't matters.
4. I don't see why how many ports affect the bridge sending GARP.
Bridge broadcasts are notoriously slow, they consider every port. We've seen glean traffic take up 100% CPU with only 10k pps. I have patches that fix the situation for *some* cases (i.e. where not all ports need to be considered), but in general you can't optimize it much, so it's best to avoid sending them altogether.
Just imagine having a hundred SVIs on top of the bridge, that would lead to number if SVIs multipled by the number of ports broadcast packets for each link flap of some bond/team port. Same thing happens if there are macvlans on top, we have setups with thousands of virtual devices and this will just kill them, if it was at all correct behaviour then we might look for a solution but it is not in general. GARPs must be confined only to the bond ports which changed state, and not broadcast to all every time.
Quoted 9 lines
Please correct me if I missed something.
Quoted 1 line
Also it seems this was proposed few years back: https://lkml.org/lkml/2018/1/6/135
Thanks for this link, cc Stephen for this discuss.
Hangbin
Quoted 41 lines · Nikolay Aleksandrov
Quoted 40 lines · Hangbin Liu
Quoted 22 lines · Nikolay Aleksandrov
Quoted 21 lines · Hangbin Liu
After adding bridge as upper layer of bond/team, we usually clean up the IP address on bond/team and set it on bridge. When there is a failover, bond/team will not send gratuitous ARP since it has no IP address. Then the down layer(e.g. VM tap dev) of bridge will not able to receive this notification.
Make bridge to be able to handle
NETDEV_NOTIFY_PEERSnotifier.Signed-off-by Hangbin Liu <[email protected]>--- net/bridge/br.c | 1 + 1 file changed, 1 insertion(+)net/bridge/br.c +0
-0@@ -125,6 +125,7 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v/* Forbid underlying device to change its type. */ return NOTIFY_BAD ;Quoted 11 lines
net/bridge/br.c static int br_device_event(struct notifier_block *unused, unsigned long event, v · line 131
Quoted 5 lines
net/bridge/br.c +1
-0+case NETDEV_NOTIFY_PEERS :case NETDEV_RESEND_IGMP :/* Propagate to master device */ call_netdevice_notifiers (event, br->dev);I'm not convinced this should be done by the bridge, setups usually have multiple ports which may have link change events and these events are unrelated, i.e. we shouldn't generate a gratuitous arp for all every time, there might be many different devices present. We have setups with hundreds of ports which are mixed types of devices.
That seems inefficient, redundant and can potentially cause problems.Hi Nikolay,
Thanks for the reply. There are a few reasons I think the bridge should handle
NETDEV_NOTIFY_PEERS:
Quoted 8 lines
Quoted 2 lines
1. Only a few devices will call
NETDEV_NOTIFY_PEERSnotifier: bond, team, virtio, xen, 6lowpan. There should have no much notification message.You can't send a broadcast to all ports because 1 bond's link status has changed. That makes no sense, the GARP needs to be sent only on that bond. The bond devices are heavily used with bridge setups, and in general the bridge is considered a switch device, it shouldn't be broadcasting GARPs to all ports when one changes link state.
Scratch the last sentence, I guess you're talking about when the bond's mac causes the bridge to change mac address by br_stp_recalculate_bridge_id(). I was wondering at first why would you need to send garp, but in fact, as Ido mentioned privately, it is already handled correctly, but you need to have set arp_notify sysctl. Then if the bridge's mac changes because of the bond flapping a NETDEV_NOTIFY_PEERS will be generated. Check:devinet.c inetdev_event() -> case NETDEV_CHANGEADDR
Alternatively you can always set the bridge mac address manually and then it won't be changed by such events.
Quoted 18 lines
Quoted 4 lines
2. When set bond/team's upper layer to bridge. The bridge's mac will be the same with bond/team. So when the bond/team's mac changed, the bridge's mac will also change. So bridge should send a GARP to notify other's that it's mac has changed.
That is not true, the mac doesn't need to be the same at all. And in many situations isn't.
Quoted 3 lines
3. There already has
NETDEV_RESEND_IGMPhandling in bridge, which is also generated by bond/team andnetdev_notify_peers(). So why there is IGMP but no ARP?Apples and oranges..
Quoted 3 lines
4. If bridge doesn't have IP address, it will omit GARP sending. So having or not having IP address on bridge doesn't matters.
4. I don't see why how many ports affect the bridge sending GARP.
Quoted 10 lines
Bridge broadcasts are notoriously slow, they consider every port. We've seen glean traffic take up 100% CPU with only 10k pps. I have patches that fix the situation for *some* cases (i.e. where not all ports need to be considered), but in general you can't optimize it much, so it's best to avoid sending them altogether.
Just imagine having a hundred SVIs on top of the bridge, that would lead to number if SVIs multipled by the number of ports broadcast packets for each link flap of some bond/team port. Same thing happens if there are macvlans on top, we have setups with thousands of virtual devices and this will just kill them, if it was at all correct behaviour then we might look for a solution but it is not in general. GARPs must be confined only to the bond ports which changed state, and not broadcast to all every time.
Again scratch the last part, I misunderstood why you need garps at first.
Quoted 12 lines
Quoted 10 lines
Please correct me if I missed something.
Quoted 1 line
Also it seems this was proposed few years back: https://lkml.org/lkml/2018/1/6/135
Thanks for this link, cc Stephen for this discuss.
Hangbin
Quoted 6 lines · Nikolay Aleksandrov
Quoted 6 lines
Quoted 5 lines
Thanks for the reply. There are a few reasons I think the bridge should handle
NETDEV_NOTIFY_PEERS:1. Only a few devices will call
NETDEV_NOTIFY_PEERSnotifier: bond, team, virtio, xen, 6lowpan. There should have no much notification message.
Quoted 8 lines
Quoted 5 lines
You can't send a broadcast to all ports because 1 bond's link status has changed. That makes no sense, the GARP needs to be sent only on that bond. The bond devices are heavily used with bridge setups, and in general the bridge is considered a switch device, it shouldn't be broadcasting GARPs to all ports when one changes link state.
Scratch the last sentence, I guess you're talking about when the bond's mac causes the bridge to change mac address by
br_stp_recalculate_bridge_id(). I was wondering
Yes, that's what I mean. Sorry I didn't make it clear in commit description.
Quoted 5 lines
at first why would you need to send garp, but in fact, as Ido mentioned privately, it is already handled correctly, but you need to have set
arp_notifysysctl. Then if the bridge's mac changes because of the bond flapping aNETDEV_NOTIFY_PEERSwill be generated. Check:devinet.cinetdev_event()-> caseNETDEV_CHANGEADDR
Yes, this is a generic work around. It will handle all mac changing instead of failover.
For IGMP, although you said they are different. In my understanding, when bridge mac changed, we need to re-join multicast group, while a gratuitous ARP is also needed. I couldn't find a reason why IGMP message is OK but GARP is not.
Quoted 3 lines
Alternatively you can always set the bridge mac address manually and then it won't be changed by such events.
Thanks for this tips. I'm not sure if administers like this way.
This remind me another issue. Should we resend IGMP when got port NETDEV_RESEND_IGMP notify, Even the bridge mac address may not changed? Shouldn't we only resend IGMP, GARP when bridge mac address changed, e.g.
net/bridge/br.c +4 -6
@@ -80,8 +80,11 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v changed_addr =br_stp_recalculate_bridge_id (br);spin_unlock_bh (&br->lock); -if (changed_addr) +if (changed_addr) {call_netdevice_notifiers (NETDEV_CHANGEADDR , br->dev); +call_netdevice_notifiers (NETDEV_RESEND_IGMP , br->dev); +call_netdevice_notifiers (NETDEV_NOTIFY_PEERS , br->dev); + }break ;
@@ -124,11 +127,6 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, vcase NETDEV_PRE_TYPE_CHANGE :/* Forbid underlaying device to change its type. */ return NOTIFY_BAD ; - -case NETDEV_RESEND_IGMP : -/* Propagate to master device */ -call_netdevice_notifiers (event, br->dev); -break ; }if (event !=NETDEV_UNREGISTER )
Thanks
Hangbin
Quoted 24 lines · Hangbin Liu
Quoted 6 lines · Nikolay Aleksandrov
Quoted 6 lines
Quoted 5 lines
Thanks for the reply. There are a few reasons I think the bridge should handle
NETDEV_NOTIFY_PEERS:1. Only a few devices will call
NETDEV_NOTIFY_PEERSnotifier: bond, team, virtio, xen, 6lowpan. There should have no much notification message.Quoted 8 lines
Quoted 5 lines
You can't send a broadcast to all ports because 1 bond's link status has changed. That makes no sense, the GARP needs to be sent only on that bond. The bond devices are heavily used with bridge setups, and in general the bridge is considered a switch device, it shouldn't be broadcasting GARPs to all ports when one changes link state.
Scratch the last sentence, I guess you're talking about when the bond's mac causes the bridge to change mac address by
br_stp_recalculate_bridge_id(). I was wonderingYes, that's what I mean. Sorry I didn't make it clear in commit description.
Quoted 5 lines
at first why would you need to send garp, but in fact, as Ido mentioned privately, it is already handled correctly, but you need to have set
arp_notifysysctl. Then if the bridge's mac changes because of the bond flapping aNETDEV_NOTIFY_PEERSwill be generated. Check:devinet.cinetdev_event()-> caseNETDEV_CHANGEADDR
Quoted 8 lines
Yes, this is a generic work around. It will handle all mac changing instead of failover.
For IGMP, although you said they are different. In my understanding, when bridge mac changed, we need to re-join multicast group, while a gratuitous ARP is also needed. I couldn't find a reason why IGMP message is OK but GARP is not.
I think that's needed more because of port changing rather than mac changing. Switches need to be updated if the port has changed, all of that is already handled correctly by the bond. And I also meant that mcast is handled very differently in the bridge, usually you'd have snooping enabled.
The patch below isn't correct and will actually break some cases when bonding flaps ports and propagates NETDEV_RESEND_IGMP with a bridge on top.
Quoted 44 lines
Quoted 3 lines
Alternatively you can always set the bridge mac address manually and then it won't be changed by such events.
Thanks for this tips. I'm not sure if administers like this way.
This remind me another issue. Should we resend IGMP when got port
NETDEV_RESEND_IGMPnotify, Even the bridge mac address may not changed? Shouldn't we only resend IGMP, GARP when bridge mac address changed, e.g.net/bridge/br.c +4
-6@@ -80,8 +80,11 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, v changed_addr =br_stp_recalculate_bridge_id (br);spin_unlock_bh (&br->lock); -if (changed_addr) +if (changed_addr) {call_netdevice_notifiers (NETDEV_CHANGEADDR , br->dev); +call_netdevice_notifiers (NETDEV_RESEND_IGMP , br->dev); +call_netdevice_notifiers (NETDEV_NOTIFY_PEERS , br->dev); + }break ;@@ -124,11 +127,6 @@ static int br_device_event(struct notifier_block *unused, unsigned long event, vcase NETDEV_PRE_TYPE_CHANGE :/* Forbid underlaying device to change its type. */ return NOTIFY_BAD ; - -case NETDEV_RESEND_IGMP : -/* Propagate to master device */ -call_netdevice_notifiers (event, br->dev); -break ; }if (event !=NETDEV_UNREGISTER )Thanks
Hangbin
Quoted 6 lines · Nikolay Aleksandrov
Quoted 5 lines
For IGMP, although you said they are different. In my understanding, when bridge mac changed, we need to re-join multicast group, while a gratuitous ARP is also needed. I couldn't find a reason why IGMP message is OK but GARP is not.
Quoted 7 lines
I think that's needed more because of port changing rather than mac changing. Switches need to be updated if the port has changed, all of that is already handled correctly by the bond. And I also meant that mcast is handled very differently in the bridge, usually you'd have snooping enabled.
The patch below isn't correct and will actually break some cases when bonding flaps ports and propagates
NETDEV_RESEND_IGMPwith a bridge on top.
Hi Nikolay,
I'm little curious. bond/team device will resend IGMP as their MAC address changed.
- bond_resend_igmp_join_requests_delayed()
- call_netdevice_notifiers(NETDEV_RESEND_IGMP, bond->dev);
- team_mcast_rejoin_work()
- call_netdevice_notifiers(NETDEV_RESEND_IGMP, team->dev);
What's the purpose that bridge resend IGMP if it's mac address not changed?
I mean, when there is a bridge on top of bond/team, when bond/team flap ports, bond/team will re-send IGMP and bridge just need to forward it. bridge doesn't need to re-send the IGMP itself if it's MAC address not changed.
Thanks
Hangbin