VLANs, DSA switches and multiple bridges
Subsystems (5)
cover VLANs, DSA switches and multiple bridges
Hi,
I've been trying to configure DSA for VLANs and not having much success. The setup is quite simple:
- The main network is untagged
- The wifi network is a vlan tagged with id $VN running over the main network.
I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to setup to provide wifi access to the vlan $VN network, while the switch is also part of the main network.
However, I'm encountering problems:
1) vlan support in DSA has a different behaviour from the Linux software bridge implementation.
# bridge vlan
port vlan ids
lan1 1 PVID Egress Untagged
...shows the default setup - the bridge ports are all configured for vlan 1, untagged egress, and vlan 1 as the port vid. Issuing:
# ip li set dev br0 type bridge vlan_filtering 1
with no other vlan configuration commands on a Linux software bridge continues to allow untagged traffic to flow across the bridge.
This difference in behaviour is because the MV88E6xxx VTU is completely empty - because net/dsa ignores all vlan settings for a port if br_vlan_enabled(dp->bridge_dev) is false - this reflects the vlan filtering state of the bridge, not whether the bridge is vlan aware.
What this means is that attempting to configure the bridge port vlans before enabling vlan filtering works for Linux software bridges, but fails for DSA bridges.
2) Assuming the above is sorted, we move on to the next issue, which is altogether more weird. Let's take a setup where we have a DSA bridge with lan1..6 in a bridge device, br0, with vlan filtering enabled. lan1 is the upstream port, lan2 is a downstream port that also wants to see traffic on vlan id $VN.
Both lan1 and lan2 are configured for that:
# bridge vlan add vid $VN dev lan1
# bridge vlan add vid $VN dev lan2
# ip li set br0 type bridge vlan_filtering 1 Untagged traffic can now pass between all the six lan ports, and vlan $VN between lan1 and lan2 only. The MV88E6xxx 8021q_mode debugfs file shows all lan ports are in mode "secure" - this is important! /sys/class/net/br0/bridge/vlan_filtering contains 1.
tcpdumping from another machine on lan4 shows that no $VN traffic reaches it. Everything seems to be working correctly...
In order to further bridge vlan $VN traffic to hostapd's wifi interface, things get a little more complex - we can't add hostapd's wifi interface to br0 directly, because hostapd will bring up the wifi interface and leak the main, untagged traffic onto the wifi. (hostapd does have vlan support, but only as a dynamic per-client thing, and there's no hooks I can see to allow script-based config of the network setup before hostapd up's the wifi interface.)
So, what I tried was:
# ip li add link br0 name br0.$VN type vlan id $VN
# bridge vlan add vid $VN dev br0 self
# ip li set dev br0.$VN upSo far so good, we get a vlan interface on top of the bridge, and tcpdumping it shows we get traffic. The 8021q_mode file has not changed state. Everything still seems to be correct.
# bridge addbr br1
Still nothing has changed.
# bridge addif br1 br0.$VN
And now the 8021q_mode debugfs file shows that all ports are now in "disabled" mode, but /sys/class/net/br0/bridge/vlan_filtering still contains '1'. In other words, br0 still thinks vlan filtering is enabled, but the hardware has had vlan filtering disabled.
Adding some stack traces to an appropriate point indicates that this is because __switchdev_handle_port_attr_set() recurses down through the tree of interfaces, skipping over the vlan interface, applying br1's configuration to br0's ports.
This surely can not be right - surely __switchdev_handle_port_attr_set() and similar should stop recursing down through another master bridge device? There are probably other network device classes that switchdev shouldn't recurse down too.
I've considered whether switchdev is the right level to do it, and I think it is - as we want the check/set callbacks to be called for the top level device even if it is a master bridge device, but we don't want to recurse through a lower master bridge device.
Some proposed patches are attached.
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
RFC 1/3 net: switchdev: do not propagate bridge updates across bridges
When configuring a tree of independent bridges, propagating changes from the upper bridge across a bridge master to the lower bridge ports brings surprises.
For example, a lower bridge may have vlan filtering enabled. It may have a vlan interface attached to the bridge master, which may then be incorporated into another bridge. As soon as the lower bridge vlan interface is attached to the upper bridge, the lower bridge has vlan filtering disabled.
This occurs because switchdev recursively applies its changes to all lower devices no matter what.
--- net/switchdev/switchdev.c | 9 +++++++++ 1 file changed, 9 insertions(+)
net/switchdev/switchdev.c +9 -0
@@ -475,6 +475,9 @@ static int __switchdev_handle_port_obj_add(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_obj_add (lower_dev, port_obj_info, check_cb, add_cb);if (err && err != -EOPNOTSUPP )
@@ -526,6 +529,9 @@ static int __switchdev_handle_port_obj_del(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_obj_del (lower_dev, port_obj_info, check_cb, del_cb);if (err && err != -EOPNOTSUPP )
@@ -576,6 +582,9 @@ static int __switchdev_handle_port_attr_set(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_attr_set (lower_dev, port_attr_info, check_cb, set_cb);if (err && err != -EOPNOTSUPP )
Signature · 1 line
2.20.1
Re: [RFC 1/3] net: switchdev: do not propagate bridge updates across bridges
Quoted 9 lines · Russell King
When configuring a tree of independent bridges, propagating changes from the upper bridge across a bridge master to the lower bridge ports brings surprises.
For example, a lower bridge may have vlan filtering enabled. It may have a vlan interface attached to the bridge master, which may then be incorporated into another bridge. As soon as the lower bridge vlan interface is attached to the upper bridge, the lower bridge has vlan filtering disabled.
Interesting topology :) The change looks OK to me. I'll add the patch to our internal tree and let it go through regression to make sure I didn't miss anything. Will report the results tomorrow.
Quoted 46 lines
This occurs because switchdev recursively applies its changes to all lower devices no matter what.
Signed-off-by Russell King <[email protected]>--- net/switchdev/switchdev.c | 9 +++++++++ 1 file changed, 9 insertions(+)net/switchdev/switchdev.c +9
-0@@ -475,6 +475,9 @@ static int __switchdev_handle_port_obj_add(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_obj_add (lower_dev, port_obj_info, check_cb, add_cb);if (err && err != -EOPNOTSUPP )@@ -526,6 +529,9 @@ static int __switchdev_handle_port_obj_del(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_obj_del (lower_dev, port_obj_info, check_cb, del_cb);if (err && err != -EOPNOTSUPP )@@ -576,6 +582,9 @@ static int __switchdev_handle_port_attr_set(struct net_device *dev,* necessary to go through this helper. */ netdev_for_each_lower_dev (dev, lower_dev, iter) { +if (netif_is_bridge_master (lower_dev)) +continue ; + err =__switchdev_handle_port_attr_set (lower_dev, port_attr_info, check_cb, set_cb);if (err && err != -EOPNOTSUPP )Signature · 1 line
2.20.1
Re: [RFC 1/3] net: switchdev: do not propagate bridge updates across bridges
Quoted 11 lines · Ido Schimmel
Quoted 9 lines · Russell King
When configuring a tree of independent bridges, propagating changes from the upper bridge across a bridge master to the lower bridge ports brings surprises.
For example, a lower bridge may have vlan filtering enabled. It may have a vlan interface attached to the bridge master, which may then be incorporated into another bridge. As soon as the lower bridge vlan interface is attached to the upper bridge, the lower bridge has vlan filtering disabled.
Quoted 3 lines
Interesting topology :) The change looks OK to me. I'll add the patch to our internal tree and let it go through regression to make sure I didn't miss anything. Will report the results tomorrow.
Thanks!
RFC 2/3 net: dsa: mv88e6xxx: fix duplicate vlan warning
In reply to Russell King - ARM Linux adminWhen setting VLANs on DSA switches, the VLAN is added to both the port concerned as well as the CPU port by dsa_slave_vlan_add(). If multiple ports are configured with the same VLAN ID, this triggers a warning on the CPU port.
Avoid this warning for CPU ports.
--- drivers/net/dsa/mv88e6xxx/chip.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/net/dsa/mv88e6xxx/chip.c +9 -3
@@ -1760,7 +1760,7 @@ static int mv88e6xxx_broadcast_setup(struct mv88e6xxx_chip *chip, u16 vid) }static int mv88e6xxx_port_vlan_join (struct mv88e6xxx_chip *chip,int port, -u16 vid,u8 member) +u16 vid,u8 member,bool warn) {const u8 non_member =MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_NON_MEMBER ;struct mv88e6xxx_vtu_entry vlan;
@@ -1805,7 +1805,7 @@ static int mv88e6xxx_port_vlan_join(struct mv88e6xxx_chip *chip, int port, err =mv88e6xxx_vtu_loadpurge (chip, &vlan);if (err)return err; - }else { + }else if (warn) {dev_info (chip->dev,"p%d: already a member of VLAN %d\n" , port, vid); }
@@ -1819,6 +1819,7 @@ static void mv88e6xxx_port_vlan_add(struct dsa_switch *ds, int port,struct mv88e6xxx_chip *chip = ds->priv;bool untagged = vlan->flags &BRIDGE_VLAN_INFO_UNTAGGED ;bool pvid = vlan->flags &BRIDGE_VLAN_INFO_PVID ; +bool warn;u8 member;u16 vid;
@@ -1832,10 +1833,15 @@ static void mv88e6xxx_port_vlan_add(struct dsa_switch *ds, int port,else member =MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_TAGGED ; +/* net/dsa/slave.c will call dsa_port_vlan_add() for the affected port +* and then the CPU port. Do not warn for duplicates for the CPU port. +*/ + warn = !dsa_is_cpu_port (ds, port); +mv88e6xxx_reg_lock (chip);for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) -if (mv88e6xxx_port_vlan_join (chip, port, vid, member)) +if (mv88e6xxx_port_vlan_join (chip, port, vid, member, warn))dev_err (ds->dev,"p%d: failed to add VLAN %d%c\n" , port, vid, untagged ?'u' :'t' );
Signature · 1 line
2.20.1
Re: [RFC 2/3] net: dsa: mv88e6xxx: fix duplicate vlan warning
Quoted 8 lines · Russell King
When setting VLANs on DSA switches, the VLAN is added to both the port concerned as well as the CPU port by
dsa_slave_vlan_add(). If multiple ports are configured with the same VLAN ID, this triggers a warning on the CPU port.Avoid this warning for CPU ports.
Signed-off-by Russell King <[email protected]>
Signature · 1 line
Florian
RFC 3/3 net: dsa: mv88e6xxx: fix vlan setup
In reply to Russell King - ARM Linux adminDSA assumes that a bridge which has vlan filtering disabled is not vlan aware, and ignores all vlan configuration. However, the kernel software bridge code allows configuration in this state.
This causes the kernel's idea of the bridge vlan state and the hardware state to disagree, so "bridge vlan show" indicates a correct configuration but the hardware lacks all configuration. Even worse, enabling vlan filtering on a DSA bridge immediately blocks all traffic which, given the output of "bridge vlan show", is very confusing.
Provide an option that drivers can set to indicate they want to receive vlan configuration even when vlan filtering is disabled. This is safe for Marvell DSA bridges, which do not look up ingress traffic in the VTU if the port is in 8021Q disabled state. Whether this change is suitable for all DSA bridges is not known.
--- drivers/net/dsa/mv88e6xxx/chip.c | 1 + include/net/dsa.h | 1 + net/dsa/slave.c | 12 ++++++++---- 3 files changed, 10 insertions(+), 4 deletions(-)
drivers/net/dsa/mv88e6xxx/chip.c +1 -0
@@ -2680,6 +2680,7 @@ static int mv88e6xxx_setup(struct dsa_switch *ds) chip->ds = ds; ds->slave_mii_bus =mv88e6xxx_default_mdio_bus (chip); + ds->vlan_bridge_vtu = true;mv88e6xxx_reg_lock (chip);
include/net/dsa.h +1 -0
@@ -269,6 +269,7 @@ struct dsa_switch {* settings on ports if not hardware-supported */ bool vlan_filtering_is_global; +bool vlan_bridge_vtu;/* In case vlan_filtering_is_global is set, the VLAN awareness state * should be retrieved from here and not from the per-port settings.
net/dsa/slave.c +8 -4
@@ -323,7 +323,8 @@ static int dsa_slave_vlan_add(struct net_device *dev,if (obj->orig_dev != dev)return -EOPNOTSUPP ; -if (dp->bridge_dev && !br_vlan_enabled (dp->bridge_dev)) +if (dp->bridge_dev && !dp->ds->vlan_bridge_vtu && + !br_vlan_enabled (dp->bridge_dev))return 0 ; vlan = *SWITCHDEV_OBJ_PORT_VLAN (obj);
@@ -390,7 +391,8 @@ static int dsa_slave_vlan_del(struct net_device *dev,if (obj->orig_dev != dev)return -EOPNOTSUPP ; -if (dp->bridge_dev && !br_vlan_enabled (dp->bridge_dev)) +if (dp->bridge_dev && !dp->ds->vlan_bridge_vtu && + !br_vlan_enabled (dp->bridge_dev))return 0 ;/* Do not deprogram the CPU port as it may be shared with other user
@@ -1122,7 +1124,8 @@ static int dsa_slave_vlan_rx_add_vid(struct net_device *dev, __be16 proto,* need to emulate the switchdev prepare + commit phase. */ if (dp->bridge_dev) { -if (!br_vlan_enabled (dp->bridge_dev)) +if (!dp->ds->vlan_bridge_vtu && + !br_vlan_enabled (dp->bridge_dev))return 0 ;/* br_vlan_get_info() returns -EINVAL or -ENOENT if the
@@ -1156,7 +1159,8 @@ static int dsa_slave_vlan_rx_kill_vid(struct net_device *dev, __be16 proto,* need to emulate the switchdev prepare + commit phase. */ if (dp->bridge_dev) { -if (!br_vlan_enabled (dp->bridge_dev)) +if (!dp->ds->vlan_bridge_vtu && + !br_vlan_enabled (dp->bridge_dev))return 0 ;/* br_vlan_get_info() returns -EINVAL or -ENOENT if the
Signature · 1 line
2.20.1
Re: [RFC 3/3] net: dsa: mv88e6xxx: fix vlan setup
+Ido,
Quoted 10 lines · Russell King
DSA assumes that a bridge which has vlan filtering disabled is not vlan aware, and ignores all vlan configuration. However, the kernel software bridge code allows configuration in this state.
This causes the kernel's idea of the bridge vlan state and the hardware state to disagree, so "bridge vlan show" indicates a correct configuration but the hardware lacks all configuration. Even worse, enabling vlan filtering on a DSA bridge immediately blocks all traffic which, given the output of "bridge vlan show", is very confusing.
Quoted 5 lines
Provide an option that drivers can set to indicate they want to receive vlan configuration even when vlan filtering is disabled. This is safe for Marvell DSA bridges, which do not look up ingress traffic in the VTU if the port is in 8021Q disabled state. Whether this change is suitable for all DSA bridges is not known.
s/DSA bridges/DSA switches/ ?
this is also safe to do with b53 switches in fact this is even desirable because VLAN filtering is a global attribute so if you have at least one bridge that spans one of your switch ports and that bridge requests vlan_filtering=1, you *must* have a valid VID entry for the non-bridged ports, or the bridged ports with vlan_filtering=0 otherwise there is no default VID entry programmed to ingress the frame. Today this is achieved by making sure that the default untagged VID 0 for non bridged ports is always programmed at start-up and the switch is always configured with VLAN awareness.
Quoted 2 lines
Signed-off-by Russell King <[email protected]>
This ties in with this commit:
which I believe is still correct in the sense that with Linux a bridge with vlan_filtering=0 also means that the bridge is not VLAN aware. Ido, Jiri, do you disagree?
This seems to be coming every now and then, so maybe it is time to revisit this documentation patch:
https://github.com/ffainelli/linux/commit/3fe61b1722a3b79d2e317a812c54f3afc902e5b0
Signature · 1 line
Florian
Re: [RFC 3/3] net: dsa: mv88e6xxx: fix vlan setup
Quoted 34 lines · Florian Fainelli
+Ido,
Quoted 10 lines · Russell King
DSA assumes that a bridge which has vlan filtering disabled is not vlan aware, and ignores all vlan configuration. However, the kernel software bridge code allows configuration in this state.
This causes the kernel's idea of the bridge vlan state and the hardware state to disagree, so "bridge vlan show" indicates a correct configuration but the hardware lacks all configuration. Even worse, enabling vlan filtering on a DSA bridge immediately blocks all traffic which, given the output of "bridge vlan show", is very confusing.
Quoted 5 lines
Provide an option that drivers can set to indicate they want to receive vlan configuration even when vlan filtering is disabled. This is safe for Marvell DSA bridges, which do not look up ingress traffic in the VTU if the port is in 8021Q disabled state. Whether this change is suitable for all DSA bridges is not known.
s/DSA bridges/DSA switches/ ?
this is also safe to do with b53 switches in fact this is even desirable because VLAN filtering is a global attribute so if you have at least one bridge that spans one of your switch ports and that bridge requests vlan_filtering=1, you *must* have a valid VID entry for the non-bridged ports, or the bridged ports with vlan_filtering=0 otherwise there is no default VID entry programmed to ingress the frame. Today this is achieved by making sure that the default untagged VID 0 for non bridged ports is always programmed at start-up and the switch is always configured with VLAN awareness.
Quoted 2 lines
Signed-off-by Russell King <[email protected]>
Quoted 7 lines
This ties in with this commit:
which I believe is still correct in the sense that with Linux a bridge with vlan_filtering=0 also means that the bridge is not VLAN aware. Ido, Jiri, do you disagree?
Hi Florian,
Yes, vlan_filtering=0 means that the bridge is not VLAN aware. It's not only about filtering at ingress / egress. The man page says "When disabled, the bridge will not consider the VLAN tag when handling packets." It also affects the VLAN with which FDB entries are learned and the VLAN used for FDB lookup.
It is really problematic for switch drivers to properly support the dynamic toggling of this option. This is why mlxsw forbids this toggling. Either you create the bridge with VLAN filtering disabled or enabled. Assuming I'm reading Cumulus documentation correctly, it seems they enforce the same behavior:
Quoted 4 lines
This seems to be coming every now and then, so maybe it is time to revisit this documentation patch:
https://github.com/ffainelli/linux/commit/3fe61b1722a3b79d2e317a812c54f3afc902e5b0
Yes, I remember reviewing it. Not sure why you didn't send another version :)
Andrew Lunn
maintainerQuoted 3 lines · Russell King - ARM Linux admin
Hi,
I've been trying to configure DSA for VLANs and not having much success.
Hi Russell
I'm hoping Vivien will review these patches. He understands all the VLAN things better than i.
Andrew
Quoted 5 lines · On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
Hi,
I've been trying to configure DSA for VLANs and not having much success. The setup is quite simple:
Quoted 7 lines
- The main network is untagged
- The wifi network is a vlan tagged with id $VN running over the main network.I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to setup to provide wifi access to the vlan $VN network, while the switch is also part of the main network.
Hello, I do not know if it is related, but I have a problem with DSa, VLAN and mv88e6085 on Espressobin board (armada-3720).
My setup/topology is similar:
eth0 --> main interface for mv88e6085 switch wan --> first RJ45 port from eth0
lan0 --> second RJ45 port from eth0
wan.10 --> unpacked VLAN with id 10 packets from wan
Just one note, wan and wan.10 uses different MAC addresses. Also lan0 has another MAC address.
Basically on upstream wan are two different/separated networks. First one is untagged, second one is tagged with vlan id 10 and tagged packets should come on wan interface (linux kernel then pass untagged packets to wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream network and in this configuration Espressobin works as router. So there is no switching between RJ45 ports, all packets should come to CPU and Linux's iptables do all stuff.
And now the problem. All (untagged) traffic for first network on wan works fine (incoming + outgoing). Also all outgoing packets from wan.10 interface are correctly transmitted (other side see on first RJ45 port that packets are properly tagged by id 10). But for unknown reason all incoming packets with vlan id 10 on first RJ45 port are dropped. Even tcpdump on eth0 does not see them.
Could be this problem related to one which Russel described? I tried to debug this problem but I give up 2 days before Russel send this email with patches, so I have not had a chance to test it.
One very strange behavior is that sometimes mv88e6085 starts accepting those vlan id 10 packets and kernel them properly send to wan.10 interface and userspacee applications see them. And once they start appearing it works for 5 minutes, exactly 300s. After 300s they are again silently somehow dropped (tcpdump again does not see them). I was not able to detect anything which could cause that kernel started seeing them. Looks for me it was really random. But exact time 300s is really strange.
I used default Debian Buster kernel (without any custom patches). Also one from Debian Buster backports, but behavior was still same.
Signature · 2 lines
Pali Rohár [email protected]
Quoted 14 lines · Pali Rohár
Quoted 5 lines · On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
Hi,
I've been trying to configure DSA for VLANs and not having much success. The setup is quite simple:
Quoted 7 lines
- The main network is untagged
- The wifi network is a vlan tagged with id $VN running over the main network.I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to setup to provide wifi access to the vlan $VN network, while the switch is also part of the main network.
Quoted 6 lines
Hello, I do not know if it is related, but I have a problem with DSa, VLAN and mv88e6085 on Espressobin board (armada-3720).
My setup/topology is similar:
eth0 --> main interface for mv88e6085 switch
This is the DSA master interface?
Quoted 5 lines
wan --> first RJ45 port from eth0
lan0 --> second RJ45 port from eth0
wan.10 --> unpacked VLAN with id 10 packets from wanJust one note, wan and wan.10 uses different MAC addresses.
Is this something you configured? By default the MAC address of the VLAN device should be inherited from the real device. What happens if they have the same MAC address?
Quoted 17 lines
Also lan0 has another MAC address.
Basically on upstream wan are two different/separated networks. First one is untagged, second one is tagged with vlan id 10 and tagged packets should come on wan interface (linux kernel then pass untagged packets to wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream network and in this configuration Espressobin works as router. So there is no switching between RJ45 ports, all packets should come to CPU and Linux's iptables do all stuff.
And now the problem. All (untagged) traffic for first network on wan works fine (incoming + outgoing). Also all outgoing packets from wan.10 interface are correctly transmitted (other side see on first RJ45 port that packets are properly tagged by id 10). But for unknown reason all incoming packets with vlan id 10 on first RJ45 port are dropped. Even tcpdump on eth0 does not see them.
Quoted 3 lines
Could be this problem related to one which Russel described? I tried to debug this problem but I give up 2 days before Russel send this email with patches, so I have not had a chance to test it.
I'm not sure. I believe Russel was not able to receive tagged packets at all and he was using a bridged setup, unlike you (IIUC). Also, below you write that sometimes you're able to receive packets with VLAN 10.
Quoted 8 lines
One very strange behavior is that sometimes mv88e6085 starts accepting those vlan id 10 packets and kernel them properly send to wan.10 interface and userspacee applications see them. And once they start appearing it works for 5 minutes, exactly 300s. After 300s they are again silently somehow dropped (tcpdump again does not see them). I was not able to detect anything which could cause that kernel started seeing them. Looks for me it was really random. But exact time 300s is really strange.
300 seconds is the default ageing time the Linux bridge uses, so maybe the problem is the hardware FDB table and not the VLAN filter.
Andrew / Vivien, how routed traffic is handled on this platform? Is it possible that the FDB table is consulted first and the expectation is that it would direct the packet towards the CPU port? If so, I guess that the FDB entry with {VID 10, wan.10's MAC} is aged-out and flooding towards the CPU is disabled?
Quoted 7 lines
I used default Debian Buster kernel (without any custom patches). Also one from Debian Buster backports, but behavior was still same.
Signature · 2 lines
Pali Rohár [email protected]
Quoted 15 lines · On Tuesday 31 December 2019 20:06:14 Ido Schimmel wrote:
Quoted 14 lines · Pali Rohár
Quoted 12 lines · On Sunday 22 December 2019 19:22:35 Russell King - ARM Linux admin wrote:
Hi,
I've been trying to configure DSA for VLANs and not having much success. The setup is quite simple:
- The main network is untagged
- The wifi network is a vlan tagged with id $VN running over the main network.I have an Armada 388 Clearfog with a PCIe wifi card which I'm trying to setup to provide wifi access to the vlan $VN network, while the switch is also part of the main network.
Quoted 8 lines
Quoted 6 lines
Hello, I do not know if it is related, but I have a problem with DSa, VLAN and mv88e6085 on Espressobin board (armada-3720).
My setup/topology is similar:
eth0 --> main interface for mv88e6085 switch
This is the DSA master interface?
Yes.
Quoted 7 lines
Quoted 5 lines
wan --> first RJ45 port from eth0
lan0 --> second RJ45 port from eth0
wan.10 --> unpacked VLAN with id 10 packets from wanJust one note, wan and wan.10 uses different MAC addresses.
Is this something you configured?
Yes. My configuration is:
ip link set dev wan address <address1> ip link add link wan name wan.10 type vlan id 10 ip link set dev wan.10 address <address2>
Just to note that same setup on Raspberry PI is working fine. Here on Raspberry PI for second RJ45 port I'm using external USB based network adapter, so DSA on RPI is not used at all. Which means that Linux kernel does not have any problem with setting / using two different MAC addresses on one physical network adapter.
Quoted 3 lines
By default the MAC address of the VLAN
device should be inherited from the real device. What happens if they have the same MAC address?
I do not know. I have not tested this scenario. Currently I have my Espressobin with mv88e6085 out of my hands and is disconnected from internet, so I cannot do this test right now.
Quoted 17 lines
Quoted 17 lines
Also lan0 has another MAC address.
Basically on upstream wan are two different/separated networks. First one is untagged, second one is tagged with vlan id 10 and tagged packets should come on wan interface (linux kernel then pass untagged packets to wan and vlan id 10 tagged as "untagged" to wan.10). lan0 is downstream network and in this configuration Espressobin works as router. So there is no switching between RJ45 ports, all packets should come to CPU and Linux's iptables do all stuff.
And now the problem. All (untagged) traffic for first network on wan works fine (incoming + outgoing). Also all outgoing packets from wan.10 interface are correctly transmitted (other side see on first RJ45 port that packets are properly tagged by id 10). But for unknown reason all incoming packets with vlan id 10 on first RJ45 port are dropped. Even tcpdump on eth0 does not see them.
Quoted 7 lines
Quoted 3 lines
Could be this problem related to one which Russel described? I tried to debug this problem but I give up 2 days before Russel send this email with patches, so I have not had a chance to test it.
I'm not sure. I believe Russel was not able to receive tagged packets at all and he was using a bridged setup, unlike you (IIUC). Also, below you write that sometimes you're able to receive packets with VLAN 10.
Yes, sometimes I was able to receive packets, but I run this setup for about 4 hours (or more) and only few times (for 5 minutes) it worked.
Quoted 9 lines
Quoted 8 lines
One very strange behavior is that sometimes mv88e6085 starts accepting those vlan id 10 packets and kernel them properly send to wan.10 interface and userspacee applications see them. And once they start appearing it works for 5 minutes, exactly 300s. After 300s they are again silently somehow dropped (tcpdump again does not see them). I was not able to detect anything which could cause that kernel started seeing them. Looks for me it was really random. But exact time 300s is really strange.
Quoted 8 lines
300 seconds is the default ageing time the Linux bridge uses, so maybe the problem is the hardware FDB table and not the VLAN filter.
Andrew / Vivien, how routed traffic is handled on this platform? Is it possible that the FDB table is consulted first and the expectation is that it would direct the packet towards the CPU port? If so, I guess that the FDB entry with {VID 10, wan.10's MAC} is aged-out and flooding towards the CPU is disabled?
Hm... that is interesting.
When run was running tcpdump I did not passed -p flag, so tcpdump automatically was listening in promiscuous mode.
And I was trying those tests also with setting "promisc" mode via ip link and nothing was changed.
So if above is truth then also promiscuous mode is broken and filtered.
(I expect that network adapter in promiscuous mode receive also packets which target MAC address is not same as MAC address configured for network interface.)
Quoted 9 lines
Quoted 7 lines
I used default Debian Buster kernel (without any custom patches). Also one from Debian Buster backports, but behavior was still same.
Signature · 2 lines
Pali Rohár [email protected]
Signature · 2 lines
Pali Rohár [email protected]
Hi Pali,
I've just tried this setup here:
# ip ad add 192.168.4.1/24 dev lan1 # ip li set dev lan1 up # ip li add li lan1 name lan1.128 type vlan id 128 # ip ad add 192.168.5.1/24 dev lan1.128 # ip li set dev lan1.128 up
without lan1 being part of at Linux software bridge. lan2..6 are part of a software bridge that has vlan filtering disabled.
On the other end of lan1, I have another machine setup with addresses 192.168.4.2/24 on the raw network interface and 192.168.5.2/24 on vlan 128 - and it works perfectly, without issue.
This is my 5.4.0 kernel, which has a lot of patches on top of 5.4.0, including the patch set that started this thread. The hardware is a SolidRun Clearfog (MV88E6176 DSA switch with mvneta host ethernet).
I think the most important thing to do if you're suffering problems like this is to monitor and analyse packets being received from the DSA switch on the host interface:
# tcpdump -enXXi $host_dsa_interface
Here's an example ping packet received over the vlan with the above configuration, captured from the host DSA interface (ether mac addresses obfuscated):
0x0000: DDDD DDDD DDDD SSSS SSSS SSSS dada 0000 .PC.....[h:.....
^^^^^^^^^
0x0010: c020 0000 8100 0080 0800 4500 0054 ec40 ..........E..T.@
^^^^^^^^^ ^^^^^^^^^ ^^^^
0x0020: 4000 4001 c314 c0a8 0502 c0a8 0501 0800 @.@.............
0x0030: 8784 0c85 0001 32c8 0c5e 0000 0000 59fc ......2..^....Y.
0x0040: 0c00 0000 0000 1011 1213 1415 1617 1819 ................
0x0050: 1a1b 1c1d 1e1f 2021 2223 2425 2627 2829 .......!"#$%&'()
0x0060: 2a2b 2c2d 2e2f 3031 3233 3435 3637 *+,-./01234567dada 0000 c020 0000 - EDSA tag
8100 0080 - VLAN ethertype, vlan id
0800 - IPv4 ethertype, and what follows is the ipv4
packet.
That way it would be possible to know whether the DSA switch is forwarding the packets, and in what manner it's forwarding them.
Another tool that I've found useful is Vivien's debugfs patch, which seems to be way superior for understanding the Marvell DSA switch state than any other tool out there. It's my understanding that DaveM doesn't want that in the mainline kernel, but it's really useful for understanding what's going on. It was key to me discovering why vlan stuff wasn't working for me.
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
Quoted 5 lines · On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
I think the most important thing to do if you're suffering problems like this is to monitor and analyse packets being received from the DSA switch on the host interface:
# tcpdump -enXXi $host_dsa_interface
Hello Russell! Main dsa interface for me is eth0 and it does not see any incoming vlan tagged packets. (Except that sometimes for those 5 minutes periods it sometimes see them. And when tcpdump saw them also they arrived to userspace.)
Signature · 2 lines
Pali Rohár [email protected]
Quoted 11 lines · Pali Rohár
Quoted 5 lines · On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
I think the most important thing to do if you're suffering problems like this is to monitor and analyse packets being received from the DSA switch on the host interface:
# tcpdump -enXXi $host_dsa_interfaceHello Russell! Main dsa interface for me is eth0 and it does not see any incoming vlan tagged packets. (Except that sometimes for those 5 minutes periods it sometimes see them. And when tcpdump saw them also they arrived to userspace.)
I think having Vivien's debugfs patch would be really useful to take this further forward. This patch provides direct access to the atu (address translation unit) entries, vtu (vlan translation unit) entries and all the device registers. The atu takes a long time to dump.
It also allows for some experimentation, by writing these files, entries can be added to or removed from the translation units, and registers written.
I think dumping the on-chip ATU contents with it in the "working" state and the "non-working" state may be revealing.
I'm afraid that I couldn't tell you where to get Vivien's debugfs patch from; I'm using an old version that I've ported forward to subsequent kernels - probably much like anyone else who gets their hands dirty with Marvell DSA hacking. Vivien's copied on this thread already, so might chime in...
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
Quoted 11 lines · Pali Rohár
Quoted 5 lines · On Wednesday 01 January 2020 17:30:14 Russell King - ARM Linux admin wrote:
I think the most important thing to do if you're suffering problems like this is to monitor and analyse packets being received from the DSA switch on the host interface:
# tcpdump -enXXi $host_dsa_interfaceHello Russell! Main dsa interface for me is eth0 and it does not see any incoming vlan tagged packets. (Except that sometimes for those 5 minutes periods it sometimes see them. And when tcpdump saw them also they arrived to userspace.)
Based on your description it sounds like there is possibly an address learning issue, which is why the ATU patches that Russell mentioned (latest I know are available at [1]) could come in handy.
[1]: https://github.com/vivien/linux/tree/dsa/debugfs
Since this is a DSA set-up, you could try a few things to isolate the problems further:
- if you keep the same MAC address for wan and wan.10 does the problem go away? If so, this could suggest that the DSA master network device driver potentially has a problem with programming its receive filter for other unicast MAC addresses than the default one (more on that below)
- if your DSA master supports receive VLAN filter offload (NETIF_F_HW_VLAN_CTAG_FILTER) does it work better if you turn it off?
- does it help if you go back to a kernel before and not including v5.1 which does not have commit 061f6a505ac33659eab007731c0f6374df39ab55 ("net: dsa: Add ndo_vlan_rx_{add, kill}_vid implementation") or if you can change your kernel, try something similar to [2] for mv88e6xxx and see if it helps
Regarding the programing of MAC addresses happening through ndo_set_rx_mode(), we should program static unicast and multicast addresses when dev_{mc,uc}_sync() gets called similar to [3] because those addresses are static in nature and the host stack can add/remove them accordingly. This would help with aging as much as unknown/known multicast traffic. I will try to cook something later this week.
[3]:
https://github.com/ffainelli/linux/commit/f91b53449e0346a8d7157e5a9225faaa810cbeab
--
Florian
Hello, thank you for steps! For sure I will try it, but probably next month. Currently I do not have my board in hands, so I cannot do tests. So I will just comment this part:
Quoted 8 lines · On Wednesday 01 January 2020 20:53:57 Florian Fainelli wrote:
- does it help if you go back to a kernel before and not including v5.1 which does not have commit
061f6a505ac33659eab007731c0f6374df39ab55("net: dsa: Add ndo_vlan_rx_{add, kill}_vid implementation") or if you can change your kernel, try something similar to [2] for mv88e6xxx and see if it helps
Debian Buster has kernel version 4.19 and in backports is 5.3. I tested both versions and there was no difference.
Was above mentioned commit propagated to stable trees (and there is possibility that was backported to stable 4.19 version)?
I can use any kernel version since 4.17 (or 4.18?) as board is supported by mainline kernel.
Signature · 2 lines
Pali Rohár [email protected]
Andrew Lunn
maintainerQuoted 16 lines
# tcpdump -enXXi $host_dsa_interfaceHere's an example ping packet received over the vlan with the above configuration, captured from the host DSA interface (ether mac addresses obfuscated):
0x0000: DDDD DDDD DDDD SSSS SSSS SSSS dada 0000 .PC.....[h:..... ^^^^^^^^^ 0x0010: c020 0000 8100 0080 0800 4500 0054 ec40 ..........E..T.@ ^^^^^^^^^ ^^^^^^^^^ ^^^^ 0x0020: 4000 4001 c314 c0a8 0502 c0a8 0501 0800 @.@............. 0x0030: 8784 0c85 0001 32c8 0c5e 0000 0000 59fc ......2..^....Y. 0x0040: 0c00 0000 0000 1011 1213 1415 1617 1819 ................ 0x0050: 1a1b 1c1d 1e1f 2021 2223 2425 2627 2829 .......!"#$%&'() 0x0060: 2a2b 2c2d 2e2f 3031 3233 3435 3637 *+,-./01234567
Quoted 7 lines
dada 0000 c020 0000 - EDSA tag
8100 0080 - VLAN ethertype, vlan id
0800 - IPv4 ethertype, and what follows is the ipv4
packet.That way it would be possible to know whether the DSA switch is forwarding the packets, and in what manner it's forwarding them.
Hi Russell
Try upgrading your tcpdump and pcap library. It now understands DSA headers, will dump them in a semi human readable way, and also the rest of the frame.
Andrew