fix ipvlan/macvlan link event handing
Subsystems (3)
cover fix ipvlan/macvlan link event handing
When setting the lower-layer link up/down, the ipvlan/macvlan device synchronizes its state via netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan/macvlan state to become out of sync with the lower-layer link state. Fix this by explicitly changing the IFF_UP flag, similar to how VLAN handles it.
Before the patch:
# ./rtnetlink.sh -t "kci_test_vlan kci_test_ipvlan kci_test_macvlan" PASS: vlan link state correct FAIL: ipvlan link state incorrect FAIL: macvlan link state incorrect
After the patch set:
# ./rtnetlink.sh -t "kci_test_vlan kci_test_ipvlan kci_test_macvlan" PASS: vlan link state correct PASS: ipvlan link state correct PASS: macvlan link state correct
Hangbin Liu (3):
ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
macvlan: fix NETDEV_UP/NETDEV_DOWN event handling selftests/rtnetlink.sh: add vlan/ipvlan/macvlan link state test
drivers/net/ipvlan/ipvlan_main.c | 20 +++++++- drivers/net/macvlan.c | 20 ++++++++ tools/testing/selftests/net/rtnetlink.sh | 64 ++++++++++++++++++++++++ 3 files changed, 103 insertions(+), 1 deletion(-)
Signature · 1 line
2.46.0
PATCH 1/3 ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
When setting the lower-layer link up/down, the ipvlan device synchronizes its state via netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan state to become out of sync with the lower-layer link state.
If the lower link and ipvlan are in the same namespace, this issue is hidden because ip link show checks the link state in IFLA_LINK and has a m_flag to control the state, displaying M-DOWN in the flags. However, if the ipvlan and the lower link are in different namespaces, this information is not available, and the ipvlan link state remains unchanged. For example:
1. Add an ipvlan over bond0.
2. Move the ipvlan to a separate namespace and bring it up.
3. Set bond0 link down.
4. The ipvlan remains up.
This issue affects containers and pods, causing them to display an incorrect link state for ipvlan. Fix this by explicitly changing the IFF_UP flag, similar to how VLAN handles it.
--- drivers/net/ipvlan/ipvlan_main.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
drivers/net/ipvlan/ipvlan_main.c +19 -1
@@ -730,7 +730,7 @@ static int ipvlan_device_event(struct notifier_block *unused,struct ipvl_dev *ipvlan, *next;struct ipvl_port *port;LIST_HEAD (lst_kill); -int err; +int flags, err;if (!netif_is_ipvlan_port (dev))return NOTIFY_DONE ;
@@ -739,7 +739,25 @@ static int ipvlan_device_event(struct notifier_block *unused,switch (event) {case NETDEV_UP : +list_for_each_entry (ipvlan, &port->ipvlans, pnode) { + flags = ipvlan->dev->flags; +if (flags &IFF_UP ) +continue ; +dev_change_flags (ipvlan->dev, flags |IFF_UP , extack); +netif_stacked_transfer_operstate (ipvlan->phy_dev, + ipvlan->dev); + } +break ;case NETDEV_DOWN : +list_for_each_entry (ipvlan, &port->ipvlans, pnode) { + flags = ipvlan->dev->flags; +if (!(flags &IFF_UP )) +continue ; +dev_close (ipvlan->dev); +netif_stacked_transfer_operstate (ipvlan->phy_dev, + ipvlan->dev); + } +break ;case NETDEV_CHANGE :list_for_each_entry (ipvlan, &port->ipvlans, pnode) netif_stacked_transfer_operstate(ipvlan->phy_dev,
Signature · 1 line
2.46.0
Re: [PATCH net 1/3] ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
Hello Hangbin,
Quoted 11 lines · Hangbin Liu
When setting the lower-layer link up/down, the ipvlan device synchronizes its state via
netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan state to become out of sync with the lower-layer link state.If the lower link and ipvlan are in the same namespace, this issue is hidden because ip link show checks the link state in
IFLA_LINKand has am_flagto control the state, displaying M-DOWN in the flags. However, if the ipvlan and the lower link are in different namespaces, this information is not available, and the ipvlan link state remains unchanged.
Is the issue with the actual behavior (sending/receiving packets, etc), or just in how it's displayed by iproute?
Quoted 10 lines
For example:
1. Add an ipvlan over bond0.
2. Move the ipvlan to a separate namespace and bring it up.
3. Set bond0 link down.
4. The ipvlan remains up.This issue affects containers and pods, causing them to display an incorrect link state for ipvlan. Fix this by explicitly changing the
IFF_UPflag, similar to how VLAN handles it.
I'm not sure this change of behavior can be done anymore. And I'm not convinced vlan's behavior is better (commit 5e7565930524 ("vlan:
support "loose binding" to the underlying network device") describes why it's not always wanted). IMO it makes sense to have admin state separate from link state.
If you want a consistent behavior, the admin should also not be allowed to set the link UP again while its lower device is not, like VLAN does:
static int vlan_dev_open(struct net_device *dev)
{
struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
struct net_device *real_dev = vlan->real_dev;
int err;
if (!(real_dev->flags & IFF_UP) &&
!(vlan->flags & VLAN_FLAG_LOOSE_BINDING))
return -ENETDOWN;(but that would almost certainly break someone's scripts)
Signature · 1 line
Sabrina
Re: [PATCH net 1/3] ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
Hi Sabrina,
Quoted 9 lines · Sabrina Dubroca
Hello Hangbin,
Quoted 6 lines · Hangbin Liu
When setting the lower-layer link up/down, the ipvlan device synchronizes its state via
netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan state to become out of sync with the lower-layer link state.
Quoted 8 lines
Quoted 5 lines
If the lower link and ipvlan are in the same namespace, this issue is hidden because ip link show checks the link state in
IFLA_LINKand has am_flagto control the state, displaying M-DOWN in the flags. However, if the ipvlan and the lower link are in different namespaces, this information is not available, and the ipvlan link state remains unchanged.Is the issue with the actual behavior (sending/receiving packets, etc), or just in how it's displayed by iproute?
The upper link in netns up while lower link down will cause the traffic break in the pod.
Quoted 12 lines
Quoted 11 lines
For example:
1. Add an ipvlan over bond0.
2. Move the ipvlan to a separate namespace and bring it up.
3. Set bond0 link down.
4. The ipvlan remains up.This issue affects containers and pods, causing them to display an incorrect link state for ipvlan. Fix this by explicitly changing the
IFF_UPflag, similar to how VLAN handles it.
Quoted 5 lines
I'm not sure this change of behavior can be done anymore. And I'm not convinced vlan's behavior is better (commit
5e7565930524("vlan:
support "loose binding" to the underlying network device") describes why it's not always wanted). IMO it makes sense to have admin state separate from link state.
Thanks for the comments, that's also what I am worried. I have send a question email[1] 2 months ago but not reply yet. So I post this patch and welcome any feedback.
[1]https://lore.kernel.org/netdev/Z67lt5v6vrltiRyG@fedora/
Quoted 16 lines
If you want a consistent behavior, the admin should also not be allowed to set the link UP again while its lower device is not, like VLAN does:
static int
vlan_dev_open(struct net_device *dev){ struct vlan_dev_priv *vlan = vlan_dev_priv(dev); struct net_device *real_dev = vlan->real_dev; int err; if (!(real_dev->flags & IFF_UP) && !(vlan->flags & VLAN_FLAG_LOOSE_BINDING)) return -ENETDOWN;
Quoted 1 line
(but that would almost certainly break someone's scripts)
Yes, so let's wait for others feedback first.
Thanks
Hangbin
Re: [PATCH net 1/3] ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
Quoted 17 lines · Hangbin Liu
Hi Sabrina,
Quoted 15 lines · Sabrina Dubroca
Hello Hangbin,
Quoted 11 lines · Hangbin Liu
When setting the lower-layer link up/down, the ipvlan device synchronizes its state via
netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan state to become out of sync with the lower-layer link state.If the lower link and ipvlan are in the same namespace, this issue is hidden because ip link show checks the link state in
IFLA_LINKand has am_flagto control the state, displaying M-DOWN in the flags. However, if the ipvlan and the lower link are in different namespaces, this information is not available, and the ipvlan link state remains unchanged.
Quoted 5 lines
Quoted 2 lines
Is the issue with the actual behavior (sending/receiving packets, etc), or just in how it's displayed by iproute?
The upper link in netns up while lower link down will cause the traffic break in the pod.
That seems like the correct behavior based on the actual (not displayed) state of the links.
I wonder if netif_stacked_transfer_operstate should consider the admin state of the lower device as well as link state:
@@ -10724,7 +10724,7 @@ void netif_stacked_transfer_operstate(const struct net_device *rootdev,
else
netif_testing_off(dev);
- if (netif_carrier_ok(rootdev))
+ if (netif_carrier_ok(rootdev) && rootdev->flags & IFF_UP)
netif_carrier_on(dev);
else
netif_carrier_off(dev);
but I haven't looked at all the consequences and possible side effects.
Signature · 1 line
Sabrina
Re: [PATCH net 1/3] ipvlan: fix NETDEV_UP/NETDEV_DOWN event handling
Quoted 18 lines · Sabrina Dubroca
Quoted 17 lines · Hangbin Liu
Hi Sabrina,
Quoted 15 lines · Sabrina Dubroca
Hello Hangbin,
Quoted 11 lines · Hangbin Liu
When setting the lower-layer link up/down, the ipvlan device synchronizes its state via
netif_stacked_transfer_operstate(), which only checks the carrier state. However, setting the link down does not necessarily change the carrier state for virtual interfaces like bonding. This causes the ipvlan state to become out of sync with the lower-layer link state.If the lower link and ipvlan are in the same namespace, this issue is hidden because ip link show checks the link state in
IFLA_LINKand has am_flagto control the state, displaying M-DOWN in the flags. However, if the ipvlan and the lower link are in different namespaces, this information is not available, and the ipvlan link state remains unchanged.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Is the issue with the actual behavior (sending/receiving packets, etc), or just in how it's displayed by iproute?
The upper link in netns up while lower link down will cause the traffic break in the pod.
That seems like the correct behavior based on the actual (not displayed) state of the links.
Hmm, since this behavior is controversial, do you think if we should drop this until some users request?
Quoted 16 lines
I wonder if
netif_stacked_transfer_operstateshould consider the admin state of the lower device as well as link state:(quoted diff) +1
-1@@ -10724,7 +10724,7 @@ void netif_stacked_transfer_operstate(const struct net_device *rootdev, else netif_testing_off(dev); - if (netif_carrier_ok(rootdev)) + if (netif_carrier_ok(rootdev) && rootdev->flags & IFF_UP) netif_carrier_on(dev); else netif_carrier_off(dev);
Quoted 2 lines
but I haven't looked at all the consequences and possible side effects.
I'm not sure. Only sync link carrier seems reasonable too.
Thanks
Hangbin
PATCH 2/3 macvlan: fix NETDEV_UP/NETDEV_DOWN event handling
In reply to Hangbin LiuSetting the link down does not necessarily change the carrier state for virtual interfaces like bonding. Therefore, handling the device up/down event via netif_stacked_transfer_operstate() is not sufficient.
If the lower link and macvlan are in different namespaces, the upper link state may become out of sync. Fix this by updating the handling logic to be similar to VLAN.
--- drivers/net/macvlan.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/net/macvlan.c +20 -0
@@ -1781,10 +1781,12 @@ static void update_port_bc_queue_len(struct macvlan_port *port)static int macvlan_device_event (struct notifier_block *unused,unsigned long event,void *ptr) { +struct netlink_ext_ack *extack =netdev_notifier_info_to_extack (ptr);struct net_device *dev =netdev_notifier_info_to_dev (ptr);struct macvlan_dev *vlan, *next;struct macvlan_port *port;LIST_HEAD (list_kill); +int flags;if (!netif_is_macvlan_port (dev))return NOTIFY_DONE ;
@@ -1793,7 +1795,25 @@ static int macvlan_device_event(struct notifier_block *unused,switch (event) {case NETDEV_UP : +list_for_each_entry (vlan, &port->vlans, list) { + flags = vlan->dev->flags; +if (flags &IFF_UP ) +continue ; +dev_change_flags (vlan->dev, flags |IFF_UP , extack); +netif_stacked_transfer_operstate (vlan->lowerdev, + vlan->dev); + } +break ;case NETDEV_DOWN : +list_for_each_entry (vlan, &port->vlans, list) { + flags = vlan->dev->flags; +if (!(flags &IFF_UP )) +continue ; +dev_close (vlan->dev); +netif_stacked_transfer_operstate (vlan->lowerdev, + vlan->dev); + } +break ;case NETDEV_CHANGE :list_for_each_entry (vlan, &port->vlans, list) netif_stacked_transfer_operstate(vlan->lowerdev,
Signature · 1 line
2.46.0
PATCH 3/3 selftests/rtnetlink.sh: add vlan/ipvlan/macvlan link state test
In reply to Hangbin LiuAdd tests to create vlan/ipvlan/macvlan over a bond interface and move them to a separate network namespace. Verify that the upper link state correctly reflects the lower-layer link state.
# ./rtnetlink.sh -t "kci_test_vlan kci_test_ipvlan kci_test_macvlan" PASS: vlan link state correct PASS: ipvlan link state correct PASS: macvlan link state correct
--- tools/testing/selftests/net/rtnetlink.sh | 64 ++++++++++++++++++++++++ 1 file changed, 64 insertions(+)
tools/testing/selftests/net/rtnetlink.sh +64 -0
@@ -30,6 +30,9 @@ ALL_TESTS=" kci_test_address_proto kci_test_enslave_bonding kci_test_mngtmpaddr + kci_test_vlan + kci_test_ipvlan + kci_test_macvlan " devdummy="test-dummy0"
@@ -1334,6 +1337,67 @@ kci_test_mngtmpaddr() return $ret } +kci_test_link_state() +{ + local test_link=$(mktemp -u test_link-XXX) + local bond=$(mktemp -u bond-XXX) + local link_param=$2 + local link_type=$1 + local ret=0 + + setup_ns testns + if [ $? -ne 0 ]; then + end_test "SKIP ${link_type} netns tests: cannot add net namespace $testns" + return $ksft_skip + fi + + # 1. Test link state over bond + run_cmd ip link add dev $bond type bond mode active-backup miimon 10 + run_cmd ip link set dev ${devdummy} down + run_cmd ip link set dev ${devdummy} master $bond + run_cmd ip link set dev ${bond} up + run_cmd ip link add link ${bond} name ${test_link} type ${link_type} ${link_param} + run_cmd ip link set ${test_link} up + run_cmd ip link set ${devdummy} down + run_cmd_grep_fail "LOWER_UP" ip link show dev ${test_link} + run_cmd ip link set ${devdummy} up + run_cmd_grep "LOWER_UP" ip link show dev ${test_link} + + # 2. Test link state over bond in netns + run_cmd ip link set ${test_link} netns ${testns} + run_cmd ip -n ${testns} link set ${test_link} up + run_cmd ip link set ${devdummy} down + run_cmd_grep_fail "LOWER_UP" ip -n ${testns} link show dev ${test_link} + run_cmd ip link set ${devdummy} up + run_cmd_grep "LOWER_UP" ip -n ${testns} link show dev ${test_link} + ip -n ${testns} link del ${test_link} + + if [ $ret -ne 0 ]; then + end_test "FAIL: ${link_type} link state incorrect" + else + end_test "PASS: ${link_type} link state correct" + fi + + ip netns del "$testns" + ip link del dev ${bond} + return $ret +} + +kci_test_vlan() +{ + kci_test_link_state "vlan" "id 2" +} + +kci_test_ipvlan() +{ + kci_test_link_state "ipvlan" "mode l2" +} + +kci_test_macvlan() +{ + kci_test_link_state "macvlan" "mode bridge" +} + kci_test_rtnl() { local current_test
Signature · 1 line
2.46.0