Skip to contentSkip to conversation

net: dsa: b53: be VLAN unaware when not filtering

PATCH In review Semih Baskan 4 participants 15 messages Aug 5 – Aug 13

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: dsa: b53: be VLAN unaware when not filtering

b53 keeps the VLAN table enabled at all times, so a tagged frame whose VID is not in the table resolves to an empty member set and is dropped before it reaches the CPU. Documentation/networking/switchdev.rst requires a standalone port to keep every VLAN configured on top of it working, and a port that is not filtering to forward frames whose VID is absent from the table.

Turn the table off in that case. It also selects shared VLAN learning, which b53_arl_rw_op() already ties to the same flag. Switches with no tag protocol keep the CPU port tagged in every VLAN and identify the source port from that tag, so they stay VLAN aware.

b53_configure_vlan() has to stop passing dev->vlan_enabled back in as the requested state. b53_enable_vlan() stores its result there, so the disabled state latches and enabling VLAN filtering later would not re-enable the table.

An 8021q upper on a standalone port is the case that breaks, for example a PPPoE WAN on VLAN 35. The PADO comes back tagged and is dropped, so no session comes up and there is no default route.

Tested on an Asus RT-N18U (BCM53011 rev 5) against a PPPoE concentrator on a VLAN 35 subinterface. pppd timed out waiting for PADO before, and the session establishes after. With VLAN filtering enabled the table is still programmed and still enforces port membership.

Fixes 06cfb2df7eb0 ("net: dsa: don't advertise 'rx-vlan-filter' when not needed")

---
I tried setting ds->needs_standalone_vlan_filtering on hardware first. It is not sufficient on b53: f089652b6b16 ("net: dsa: b53: do not program vlans when vlan filtering is off") makes .port_vlan_add return before the hardware write while filtering is off, so the VID still never reaches the table. rx-vlan-filter flipped to on and tagged frames were still dropped, 0 of 5.

Relaxing that check instead also works, but only together with the opt-in above, since without the feature bit 8021q never calls .port_vlan_add at all. Making the port VLAN unaware follows the same switchdev rule that commit cites.

I also checked the case where another bridge on the same chip has vlan_filtering=1, since b53 sets vlan_filtering_is_global. DSA offloads the uppers on the standalone port too, so rx-vlan-filter goes on there and the tagged frames keep arriving. This patch cannot execute in that state.

 drivers/net/dsa/b53/b53_common.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/dsa/b53/b53_common.c +4 -1
index 3f5b9592794d..76378afe4993 100644
@@ -385,6 +385,9 @@ static void b53_enable_vlan(struct b53_device *dev, int port, bool enable,
 {
 	u8 mgmt, vc0, vc1, vc4 = 0, vc5;
 
+	if (!enable_filtering && dev->tag_protocol != DSA_TAG_PROTO_NONE)
+		enable = false;
+
 	b53_read8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, &mgmt);
 	b53_read8(dev, B53_VLAN_PAGE, B53_VLAN_CTRL0, &vc0);
 	b53_read8(dev, B53_VLAN_PAGE, B53_VLAN_CTRL1, &vc1);
@@ -916,7 +919,7 @@ int b53_configure_vlan(struct dsa_switch *ds)
 		b53_do_vlan_op(dev, VTA_CMD_CLEAR);
 	}
 
-	b53_enable_vlan(dev, -1, dev->vlan_enabled, dev->vlan_filtering);
+	b53_enable_vlan(dev, -1, true, dev->vlan_filtering);
 
 	/* Create an untagged VLAN entry for the default PVID in case
 	 * CONFIG_VLAN_8021Q is disabled and there are no calls to
maintainer
#1 lore raw

Hi,

Quoted 65 lines · Semih Baskan

b53 keeps the VLAN table enabled at all times, so a tagged frame whose VID is not in the table resolves to an empty member set and is dropped before it reaches the CPU. Documentation/networking/switchdev.rst requires a standalone port to keep every VLAN configured on top of it working, and a port that is not filtering to forward frames whose VID is absent from the table.

Turn the table off in that case. It also selects shared VLAN learning, which b53_arl_rw_op() already ties to the same flag. Switches with no tag protocol keep the CPU port tagged in every VLAN and identify the source port from that tag, so they stay VLAN aware.

b53_configure_vlan() has to stop passing dev->vlan_enabled back in as the requested state. b53_enable_vlan() stores its result there, so the disabled state latches and enabling VLAN filtering later would not re-enable the table.

An 8021q upper on a standalone port is the case that breaks, for example a PPPoE WAN on VLAN 35. The PADO comes back tagged and is dropped, so no session comes up and there is no default route.

Tested on an Asus RT-N18U (BCM53011 rev 5) against a PPPoE concentrator on a VLAN 35 subinterface. pppd timed out waiting for PADO before, and the session establishes after. With VLAN filtering enabled the table is still programmed and still enforces port membership.

Fixes 06cfb2df7eb0 ("net: dsa: don't advertise 'rx-vlan-filter' when not needed")

---
I tried setting ds->needs_standalone_vlan_filtering on hardware first. It is not sufficient on b53: f089652b6b16 ("net: dsa: b53: do not program vlans when vlan filtering is off") makes .port_vlan_add return before the hardware write while filtering is off, so the VID still never reaches the table. rx-vlan-filter flipped to on and tagged frames were still dropped, 0 of 5.

Relaxing that check instead also works, but only together with the opt-in above, since without the feature bit 8021q never calls .port_vlan_add at all. Making the port VLAN unaware follows the same switchdev rule that commit cites.

I also checked the case where another bridge on the same chip has vlan_filtering=1, since b53 sets vlan_filtering_is_global. DSA offloads the uppers on the standalone port too, so rx-vlan-filter goes on there and the tagged frames keep arriving. This patch cannot execute in that state.

 drivers/net/dsa/b53/b53_common.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/dsa/b53/b53_common.c +3 -0
index 3f5b9592794d..76378afe4993 100644
@@ -385,6 +385,9 @@ static void b53_enable_vlan(struct b53_device *dev, int port, bool enable,
 {
        u8 mgmt, vc0, vc1, vc4 = 0, vc5;
 
+       if (!enable_filtering && dev->tag_protocol != DSA_TAG_PROTO_NONE)
+               enable = false;
+
        b53_read8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, &mgmt);
        b53_read8(dev, B53_VLAN_PAGE, B53_VLAN_CTRL0, &vc0);
        b53_read8(dev, B53_VLAN_PAGE, B53_VLAN_CTRL1, &vc1);

drivers/net/dsa/b53/b53_common.c int b53_configure_vlan(struct dsa_switch *ds) · line 925

Quoted 8 lines
drivers/net/dsa/b53/b53_common.c +1 -1
@@ -916,7 +919,7 @@ int b53_configure_vlan(struct dsa_switch *ds)
                b53_do_vlan_op(dev, VTA_CMD_CLEAR);
        }
 
-       b53_enable_vlan(dev, -1, dev->vlan_enabled, dev->vlan_filtering);
+       b53_enable_vlan(dev, -1, true, dev->vlan_filtering);
 
        /* Create an untagged VLAN entry for the default PVID in case
         * CONFIG_VLAN_8021Q is disabled and there are no calls to

Unfortunately what this does is break modifying ARL entries with VID != 0, which is why I haven't added this.

While SVL is active, any ARL add/remove operations ignore the VID field/register and force it to 0, making existing static ARL entries with VID != 0 inaccessible, and any (static) ARL entries added will have their VID set to 0, regardless what the software entry said.

This causes the ARL hardware table to go out of sync with the bridge fdb/mdb software tables, and will lead to potentially hard to debug network issues.

The options to remedy this are:

1. keep track of all static fdb (and mdb) entries added to the hardware table, so we "sync" it on switching vlan filtering on/off (or find a way to do so without having a copy), or
2. while vlan filtering is off, have static vlan table entries for all possible VIDs, or
3. use direct memory access registers to directly modify the ARL table memory instead of going through default registers while SVL is enabled.

Neither one is a quick and easy fix.

1/2 make switching vlan filtering likely a costly operation (I test implemented 2, and it takes several seconds for SPI connected switches
- not sure if this is acceptable). 3 requires knowing the in-memory formats for each switch chip, which aren't publicly documented.

Best regards,
Jonas

#2 lore raw
Quoted 2 lines · Jonas Gorski

Unfortunately what this does is break modifying ARL entries with VID != 0, which is why I haven't added this.

Thank you for the review, your concerns are justified. I will drop this patch and let you know if I find a way to address them.

Best regards,
Semih

maintainer
#3 lore raw
In reply to Jonas Gorski
Quoted 4 lines · Jonas Gorski

While SVL is active, any ARL add/remove operations ignore the VID field/register and force it to 0, making existing static ARL entries with VID != 0 inaccessible, and any (static) ARL entries added will have their VID set to 0, regardless what the software entry said.

Is this a hardware limitation, or is it because global state (dev->vlan_enabled) blinds b53_arl_rw_op()'s attempts to look at FDB entries of the other type?

static int b53_arl_rw_op(struct b53_device *dev, unsigned int op)

{
	u8 reg;

	if (op > ARLTBL_RW)
		return -EINVAL;

	b53_read8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, &reg);
	reg |= ARLTBL_START_DONE;
	if (op)
		reg |= ARLTBL_RW;
	else
		reg &= ~ARLTBL_RW;
	if (dev->vlan_enabled)
		reg &= ~ARLTBL_IVL_SVL_SELECT;
	else
		reg |= ARLTBL_IVL_SVL_SELECT;
	b53_write8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, reg);

	return b53_arl_op_wait(dev);
}
#4 lore raw

Hi Vladimir,

Quoted 3 lines

Is this a hardware limitation, or is it because global state (dev->vlan_enabled) blinds b53_arl_rw_op()'s attempts to look at FDB entries of the other type?

I tested that separation on the RT-N18U (BCM53011 rev 5) during the measurements for the new series. Two builds kept ARLTBL_IVL_SVL_SELECT at IVL while disabling the table, so the ARL read/write ops were never switched to SVL:

- table off, select held at IVL, VC0 learning-mode bits cleared: a static entry with VID 100, present in the hardware dump before the toggle, is gone from the dump after it
- table off, select held at IVL, VC0 "individual VLAN learning mode" bits (bits 6:5) kept set as well: same result
- positive control, table on, identical toggle: the entry survives and the dump shows it

The observable is the fdb search path (bridge fdb show dev ... self), which does not go through b53_arl_rw_op() at all: b53_fdb_dump() only touches the ARL search registers. The same dump shows the entry before the toggle and shows survival in the table-on control, so it can tell the difference. With every mode input the driver has held at IVL, and with the reads happening through a path b53_arl_rw_op() cannot blind, the entry still does not survive disabling the table. On this chip that makes it a property of the table being off rather than an artifact of the select flip. Jonas also noted in the new thread that ARLTBL_IVL_SVL_SELECT is not implemented on 5301x at all, which fits:
the bit visibly did nothing in these tests.

I cannot rule out that the entries still sit in silicon and are merely unreachable in every mode the driver can program, but for the driver and for forwarding they behave as lost either way.

The measurements are summarized in the new series' cover letter:

https://lore.kernel.org/all/[email protected]/

Best regards,
Semih

maintainer
#5 lore raw
In reply to Vladimir Oltean
Quoted 10 lines · Vladimir Oltean
Quoted 4 lines · Jonas Gorski

While SVL is active, any ARL add/remove operations ignore the VID field/register and force it to 0, making existing static ARL entries with VID != 0 inaccessible, and any (static) ARL entries added will have their VID set to 0, regardless what the software entry said.

Is this a hardware limitation, or is it because global state (dev->vlan_enabled) blinds b53_arl_rw_op()'s attempts to look at FDB entries of the other type?

This is a hardware limitation.

The hardware hash function to determine the table index for a { VID,MAC } entry ignores the VID field (or rather treats it as 0) when filtering / 802.1q mode is disabled. This is done for all ARL accesses, both destination port lookup on forwarding as well as when doing ARL table operations.

You can manually write entries with VID != 0 as the VID used for table index calculation and the VID of the entry are two different register fields, but the entry will then be written to the wrong index, preventing them from being matched when looking them up on forwarding in filtering/802.1q mode.

I verified this by writing entries with VID != 0 with 802.1q mode disabled, and then looking up the MAC for VID 0 with 802.1q mode enabled, which then found the entry with the VID != 0, and looking up the MAC with the entry's VID did not find the correct entry.

Quoted 18 lines

static int b53_arl_rw_op(struct b53_device *dev, unsigned int op)

{
        u8 reg;

        if (op > ARLTBL_RW)
                return -EINVAL;

        b53_read8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, &reg);
        reg |= ARLTBL_START_DONE;
        if (op)
                reg |= ARLTBL_RW;
        else
                reg &= ~ARLTBL_RW;
        if (dev->vlan_enabled)
                reg &= ~ARLTBL_IVL_SVL_SELECT;
        else
                reg |= ARLTBL_IVL_SVL_SELECT;

This ARLTBL_IVL_SVL_SELECT bit is only implemented for a small subset of switches (bcm5302x / bcm58* and bcm53134). Additionally, according to the register description, this also requires enabling "per port IVL/SVL" mode, which is not enable by b53. The description of that also says that the VIDs used in SVL ports must not be used in IVL ports. No idea what the consequences are if they do.

Best regards,
Jonas

maintainer
#6 lore raw
Quoted 18 lines · Jonas Gorski
Quoted 17 lines

static int b53_arl_rw_op(struct b53_device *dev, unsigned int op)

{
        u8 reg;

        if (op > ARLTBL_RW)
                return -EINVAL;

        b53_read8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, &reg);
        reg |= ARLTBL_START_DONE;
        if (op)
                reg |= ARLTBL_RW;
        else
                reg &= ~ARLTBL_RW;
        if (dev->vlan_enabled)
                reg &= ~ARLTBL_IVL_SVL_SELECT;
        else
                reg |= ARLTBL_IVL_SVL_SELECT;
Quoted 6 lines

This ARLTBL_IVL_SVL_SELECT bit is only implemented for a small subset of switches (bcm5302x / bcm58* and bcm53134). Additionally, according to the register description, this also requires enabling "per port IVL/SVL" mode, which is not enable by b53. The description of that also says that the VIDs used in SVL ports must not be used in IVL ports. No idea what the consequences are if they do.

What does enabling per port IVL/SVL mode do?

maintainer
#7 lore raw
Quoted 20 lines · Vladimir Oltean
Quoted 18 lines · Jonas Gorski
Quoted 17 lines

static int b53_arl_rw_op(struct b53_device *dev, unsigned int op)

{
        u8 reg;

        if (op > ARLTBL_RW)
                return -EINVAL;

        b53_read8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, &reg);
        reg |= ARLTBL_START_DONE;
        if (op)
                reg |= ARLTBL_RW;
        else
                reg &= ~ARLTBL_RW;
        if (dev->vlan_enabled)
                reg &= ~ARLTBL_IVL_SVL_SELECT;
        else
                reg |= ARLTBL_IVL_SVL_SELECT;
Quoted 8 lines
Quoted 6 lines

This ARLTBL_IVL_SVL_SELECT bit is only implemented for a small subset of switches (bcm5302x / bcm58* and bcm53134). Additionally, according to the register description, this also requires enabling "per port IVL/SVL" mode, which is not enable by b53. The description of that also says that the VIDs used in SVL ports must not be used in IVL ports. No idea what the consequences are if they do.

What does enabling per port IVL/SVL mode do?

I don't have access to a datasheet or a device to test, Florian will need to answer that completely.

I can only guess from the register descriptions, which imply that enabling SVL for port makes any ARL lookups SVL based, i.e. VID is ignored/forced to 0. Presumably this essentially only applies to for frames received on that port for lookup / learning.

There is a global option for configuring IVL or SVL, but this option is only valid when VLAN-aware is enabled in the switch. Without it, learning is always SVL.

I do not know how the per port IVL/SVL actually register works, and whether you can enable IVL in VLAN-unaware mode (I would guess not).

Note that there is no differentiation in ARL entries themselves whether they are IVL or SVL entries, only the table index calculation changes.

Best regards,
Jonas

maintainer
#8 lore raw
Quoted 28 lines · Jonas Gorski
Quoted 27 lines · Vladimir Oltean
Quoted 24 lines · Jonas Gorski
Quoted 17 lines

static int b53_arl_rw_op(struct b53_device *dev, unsigned int op)

{
         u8 reg;

         if (op > ARLTBL_RW)
                 return -EINVAL;

         b53_read8(dev, B53_ARLIO_PAGE, B53_ARLTBL_RW_CTRL, &reg);
         reg |= ARLTBL_START_DONE;
         if (op)
                 reg |= ARLTBL_RW;
         else
                 reg &= ~ARLTBL_RW;
         if (dev->vlan_enabled)
                 reg &= ~ARLTBL_IVL_SVL_SELECT;
         else
                 reg |= ARLTBL_IVL_SVL_SELECT;

This ARLTBL_IVL_SVL_SELECT bit is only implemented for a small subset of switches (bcm5302x / bcm58* and bcm53134). Additionally, according to the register description, this also requires enabling "per port IVL/SVL" mode, which is not enable by b53. The description of that also says that the VIDs used in SVL ports must not be used in IVL ports. No idea what the consequences are if they do.

Quoted 4 lines
Quoted 1 line

What does enabling per port IVL/SVL mode do?

I don't have access to a datasheet or a device to test, Florian will need to answer that completely.

Not a whole lot of details in the datasheet, but essentially what is being provided is:

IVL: a given MAC address can only appear in one VLAN
SVL: a given MAC address can appear on different ports as long as the ports are not in the same VLAN.

Quoted 6 lines

I can only guess from the register descriptions, which imply that enabling SVL for port makes any ARL lookups SVL based, i.e. VID is ignored/forced to 0. Presumably this essentially only applies to for frames received on that port for lookup / learning.

Quoted 6 lines

There is a global option for configuring IVL or SVL, but this option is only valid when VLAN-aware is enabled in the switch. Without it, learning is always SVL.

I do not know how the per port IVL/SVL actually register works, and whether you can enable IVL in VLAN-unaware mode (I would guess not).

IVL/SVL applies to 802.1Q (VLAN enabled) mode, double tagging mode and individual double tagging mode, so yes this depends on VLAN being enabled globally.

Quoted 7 lines

Note that there is no differentiation in ARL entries themselves whether they are IVL or SVL entries, only the table index calculation changes.

Best regards,
Jonas

Signature · 1 line
Florian
maintainer
#9 lore raw
In reply to Jonas Gorski
Quoted 25 lines · Jonas Gorski

Unfortunately what this does is break modifying ARL entries with VID != 0, which is why I haven't added this.

While SVL is active, any ARL add/remove operations ignore the VID field/register and force it to 0, making existing static ARL entries with VID != 0 inaccessible, and any (static) ARL entries added will have their VID set to 0, regardless what the software entry said.

This causes the ARL hardware table to go out of sync with the bridge fdb/mdb software tables, and will lead to potentially hard to debug network issues.

The options to remedy this are:

1. keep track of all static fdb (and mdb) entries added to the hardware table, so we "sync" it on switching vlan filtering on/off (or find a way to do so without having a copy), or
2. while vlan filtering is off, have static vlan table entries for all possible VIDs, or
3. use direct memory access registers to directly modify the ARL table memory instead of going through default registers while SVL is enabled.

Neither one is a quick and easy fix.

Quoted 7 lines

1/2 make switching vlan filtering likely a costly operation (I test implemented 2, and it takes several seconds for SPI connected switches
- not sure if this is acceptable). 3 requires knowing the in-memory formats for each switch chip, which aren't publicly documented.

Best regards,
Jonas

I think the only reliable way to fix VLAN unaware mode in a way that's portable across all b53 variants is a variant of this patch: allow 802.1Q mode to be disabled. Then we need to deal with the fallout caused by it upon the ARL.

1. This looks implementable with some complexity isolated within the b53 driver:
- on any .port_fdb_add(), .port_fdb_del(), .port_mdb_add(), .port_mdb_del(), compare the VID of the entry with the dev->vlan_enabled state.
- if VID != 0 and dev->vlan_enabled, or if VID == 0 and !dev->vlan_enabled, commit the operation directly to the ARL, as is currently done
- if VID == 0 and !dev->vlan_enabled, or if VID != 0 and dev->vlan_enabled, operate on a software list, allocating, deleting or modifying a local representation of the ARL entry
- on vlan_filtering toggles from 0 to 1 or from 1 to 0, acquire dev->arl_mutex and flush out all static and dynamic ARL entries across the entire switch, commit the static ones from the software list and clear the software list
- dev->vlan_enabled will probably need to be merged with

     dev->vlan_filtering, since the vlan_enabled=1 vlan_filtering=0 case
     is broken

2. I see bcm_sf2 has support for B53_JOIN_ALL_VLAN_EN; IIUC this
proposal is a soft emulation of that. Would it work though? 2 concerns:
- in b53_switch_chips[] I see not all switches have a full 4K VLAN table
- unless b53 has a feature equivalent to MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_UNMODIFIED rather than the port-wide vl->untag, the emulation would either push a VLAN tag in originally untagged frames, or strip a VLAN tag from previously VLAN tagged frames. Neither option fits the bill for
what vlan_filtering=0 semantics expect (ignore the tag).

3. From a distance it doesn't sound bad, but I cannot comment on the feasibility of this and the scalability across the 4 b53_arl_ops; maybe Florian can.

The big advantage of option #1 is that it shouldn't depend on any HW functionality which is only present on some silicon variants. I don't see any downside except for the higher SW complexity in the control path.

We could also discuss falling back to software bridging for the vlan_filtering=0 case, but that penalizes the data path, so it would probably not be the option of choice.

maintainer
#10 lore raw
Quoted 14 lines · Vladimir Oltean

1. This looks implementable with some complexity isolated within the b53 driver:
- on any .port_fdb_add(), .port_fdb_del(), .port_mdb_add(), .port_mdb_del(), compare the VID of the entry with the dev->vlan_enabled state.
- if VID != 0 and dev->vlan_enabled, or if VID == 0 and !dev->vlan_enabled, commit the operation directly to the ARL, as is currently done
- if VID == 0 and !dev->vlan_enabled, or if VID != 0 and dev->vlan_enabled, operate on a software list, allocating, deleting or modifying a local representation of the ARL entry
- on vlan_filtering toggles from 0 to 1 or from 1 to 0, acquire dev->arl_mutex and flush out all static and dynamic ARL entries across the entire switch, commit the static ones from the software list and clear the software list
- dev->vlan_enabled will probably need to be merged with

Quoted 2 lines
     dev->vlan_filtering, since the vlan_enabled=1 vlan_filtering=0 case
     is broken

Actually this algorithm is too simplistic as I specified it. In this sequence:

ip link add br0 type bridge vlan_filtering 0 && ip link set swp0 master br0 bridge fdb add swp0 00:01:02:03:04:05 master static
ip link set br0 type bridge vlan_filtering 1
ip link set br0 type bridge vlan_filtering 0 # the 00:01:02:03:04:05 address would be lost here

Furthermore, the ARL is limited in size (1024, 2048 or 4096 entries), and we could mistakenly end up queuing more than we can commit.

I'm not yet sure what are reasonable amendments that would keep the complexity in check. Currently the most obvious would be to do a one-time allocation of dev->num_arl_bins * dev->num_arl_buckets, to always keep all entries in software, and only the active ones in hardware. But I need to think about this some more, as I'm aware OpenWrt typically doesn't operate with huge memory budgets.

I'm not sure that avoiding local copies is possible, because the ARL entries don't come from a single source - we have bridge FDB, bridge MDB, ndo_dflt_fdb_add(), dev->uc, dev->mc, dev->dev_addr, sometimes replicated per VLAN, etc. They all only converge at driver level.

maintainer
#11 lore raw
Quoted 35 lines · Vladimir Oltean
Quoted 8 lines · Vladimir Oltean

1. This looks implementable with some complexity isolated within the b53 driver:
- on any .port_fdb_add(), .port_fdb_del(), .port_mdb_add(), .port_mdb_del(), compare the VID of the entry with the dev->vlan_enabled state.
- if VID != 0 and dev->vlan_enabled, or if VID == 0 and !dev->vlan_enabled, commit the operation directly to the ARL, as is currently done
- if VID == 0 and !dev->vlan_enabled, or if VID != 0 and dev->vlan_enabled, operate on a software list, allocating, deleting or modifying a

Quoted 8 lines

local representation of the ARL entry
- on vlan_filtering toggles from 0 to 1 or from 1 to 0, acquire dev->arl_mutex and flush out all static and dynamic ARL entries across the entire switch, commit the static ones from the software list and clear the software list
- dev->vlan_enabled will probably need to be merged with dev->vlan_filtering, since the vlan_enabled=1 vlan_filtering=0 case is broken

Actually this algorithm is too simplistic as I specified it. In this sequence:

ip link add br0 type bridge vlan_filtering 0 && ip link set swp0 master br0 bridge fdb add swp0 00:01:02:03:04:05 master static
ip link set br0 type bridge vlan_filtering 1
ip link set br0 type bridge vlan_filtering 0 # the 00:01:02:03:04:05 address would be lost here

Furthermore, the ARL is limited in size (1024, 2048 or 4096 entries), and we could mistakenly end up queuing more than we can commit.

I'm not yet sure what are reasonable amendments that would keep the complexity in check. Currently the most obvious would be to do a one-time allocation of dev->num_arl_bins * dev->num_arl_buckets, to always keep all entries in software, and only the active ones in hardware. But I need to think about this some more, as I'm aware OpenWrt typically doesn't operate with huge memory budgets.

Quoted 5 lines

I'm not sure that avoiding local copies is possible, because the ARL entries don't come from a single source - we have bridge FDB, bridge MDB, ndo_dflt_fdb_add(), dev->uc, dev->mc, dev->dev_addr, sometimes replicated per VLAN, etc. They all only converge at driver level.

Would not we just be better off with enforcing vlan_filtering=1 for any bridge that is created? Unlike Marvell switches, the Roboswitch has no notion of egress as tag unmodified unfortunately.

Signature · 1 line
Florian
maintainer
#12 lore raw
In reply to Vladimir Oltean

Hi,

Quoted 54 lines · Vladimir Oltean
Quoted 25 lines · Jonas Gorski

Unfortunately what this does is break modifying ARL entries with VID != 0, which is why I haven't added this.

While SVL is active, any ARL add/remove operations ignore the VID field/register and force it to 0, making existing static ARL entries with VID != 0 inaccessible, and any (static) ARL entries added will have their VID set to 0, regardless what the software entry said.

This causes the ARL hardware table to go out of sync with the bridge fdb/mdb software tables, and will lead to potentially hard to debug network issues.

The options to remedy this are:

1. keep track of all static fdb (and mdb) entries added to the hardware table, so we "sync" it on switching vlan filtering on/off (or find a way to do so without having a copy), or
2. while vlan filtering is off, have static vlan table entries for all possible VIDs, or
3. use direct memory access registers to directly modify the ARL table memory instead of going through default registers while SVL is enabled.

Neither one is a quick and easy fix.

Quoted 7 lines

1/2 make switching vlan filtering likely a costly operation (I test implemented 2, and it takes several seconds for SPI connected switches
- not sure if this is acceptable). 3 requires knowing the in-memory formats for each switch chip, which aren't publicly documented.

Best regards,
Jonas

I think the only reliable way to fix VLAN unaware mode in a way that's portable across all b53 variants is a variant of this patch: allow 802.1Q mode to be disabled. Then we need to deal with the fallout caused by it upon the ARL.

1. This looks implementable with some complexity isolated within the b53 driver:
- on any .port_fdb_add(), .port_fdb_del(), .port_mdb_add(), .port_mdb_del(), compare the VID of the entry with the dev->vlan_enabled state.
- if VID != 0 and dev->vlan_enabled, or if VID == 0 and !dev->vlan_enabled, commit the operation directly to the ARL, as is currently done
- if VID == 0 and !dev->vlan_enabled, or if VID != 0 and dev->vlan_enabled, operate on a software list, allocating, deleting or modifying a local representation of the ARL entry
- on vlan_filtering toggles from 0 to 1 or from 1 to 0, acquire dev->arl_mutex and flush out all static and dynamic ARL entries across the entire switch, commit the static ones from the software list and clear the software list
- dev->vlan_enabled will probably need to be merged with

Quoted 2 lines
     dev->vlan_filtering, since the vlan_enabled=1 vlan_filtering=0 case
     is broken

You already found the issue with dropping the software entries; we need to keep them.

That said, maximum is 4k entries, and assuming an entry is 6 bytes mac
+ 2 bytes vid + 2 bytes port (mask), it would still be just 40kB, which is nothing compared to the buffers wifi drivers tend to consume (e.g. IIRC ath12k requires a device with 512? MiB RAM). The switch memory itself is 64kB, which I would assume is the maximum we may ever need.

The algorithm how the index is calculated is documented, so we could then just emulate the ARL table in software, and then even do collision checks, at least between static entries. Though we won't know if they collide with learned dynamic entries. Which makes me wonder, what's the expected priority here? Currently we reject adding a static entry if all bins are occupied by dynamic entries (unless a dynamic entry matches the static one).

One (non) issue here is that BCM5325 and BCM5365 do not support hardware flushing, so flushing can only by done by deleting all entries individually. That being said, they are also the ones where VLAN-unaware bridging works already, so they wouldn't need it.

Quoted 3 lines

2. I see bcm_sf2 has support for B53_JOIN_ALL_VLAN_EN; IIUC this proposal is a soft emulation of that. Would it work though? 2 concerns:
- in b53_switch_chips[] I see not all switches have a full 4K VLAN table

Funny enough, these are the same ones where DROP_VTABLE_MISS=0 floods instead of traps to IMP, so we don't need it for those.

Quoted 5 lines

- unless b53 has a feature equivalent to MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_UNMODIFIED rather than the port-wide vl->untag, the emulation would either push a VLAN tag in originally untagged frames, or strip a VLAN tag from previously VLAN tagged frames. Neither option fits the bill for
what vlan_filtering=0 semantics expect (ignore the tag).

Modern b53 switches supports full 4k VLAN table entries, including 0 and 4095. By using 0 as the VLAN for untagged frames and setting PVID to 0, we can leave all valid VIDs (1-4094) as tagged.

There is also a (global) bit in VLAN_CTRL5 to configure something similar to MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_UNMODIFIED:

* PRESV_NON1Q - en_preserv_non_1q_frame: (default 0)When
enabled, it preserves untagged frame as untagged frame at TX regardless of untag map in VLAN table, and preserves priority (802.1p) tagged frames as priority tagged frames at TX if untag map in VLAN Table is 0, but otherwise ntag priority tag.

This is obviously incompatible with VLAN-aware bridges, but should be fine for VLAN-unaware ones.

Quoted 4 lines

3. From a distance it doesn't sound bad, but I cannot comment on the feasibility of this and the scalability across the 4 b53_arl_ops; maybe Florian can.

The main issue here is finding out the various in-memory formats, as they are not documented in the datasheets or public code. And there will likely be more than 4, as devices may have the same ARL access register layouts, but their in-memory layouts can differ (AFAICT bcm53115 and bcm53118 despite same generation/family have different layouts).

Though it could be combined with 1. for those we don't have the layouts.

Quoted 8 lines

The big advantage of option #1 is that it shouldn't depend on any HW functionality which is only present on some silicon variants. I don't see any downside except for the higher SW complexity in the control path.

We could also discuss falling back to software bridging for the vlan_filtering=0 case, but that penalizes the data path, so it would probably not be the option of choice.

Right, that would be the quick workaround to allow functionality. We just need to not mark them as forwarded, as DROP_VTABLE_MISS=0 will have them sent to CPU already.

Best regards,
Jonas

#13 lore raw

Hi Jonas,

Quoted 3 lines · Jonas Gorski

Right, that would be the quick workaround to allow functionality. We just need to not mark them as forwarded, as DROP_VTABLE_MISS=0 will have them sent to CPU already.

On bcm5301x, miss frames reach the CPU only with port 8 as the sole management port.

All measured on the BCM53011, DROP_VTABLE_MISS=0 throughout:

- In-tree topology (port 5 as CPU): a tagged frame with a VID absent from the VLAN table reaches no CPU port, 0 of 7. Already reported in the offload series thread.
- All three CPU ports enabled per the in-tree bcm-ns.dtsi, with Rafal's V3 patches 1 and 2 applied so DSA prefers port 8, measured
yesterday: still 0 of 7. GMNGCFG reads 0xc2, dual IMP mode, and in that mode the miss path delivers to no port at all; bringing the port 5 conduit up or down changes nothing.
- Port 8 as the ONLY enabled CPU port (FRM_MNGP=0b10): miss frames are delivered to the CPU, indiscriminately.

So a software bridging fallback that relies on miss-to-CPU delivery works on chips like BCM63268/BCM53115, where you measured the CPU copy, but on bcm5301x it additionally requires either a single IMP DT (disabling ports 5 and 7, as the Netgear R8000 and Linksys Panamera do today) or driver code that forces FRM_MNGP to 0b10 when port 8 is the conduit. Rafal's pending "ARM: dts: BCM5301X: Enable extra CPU ports for Netgear R8000" re-enables those two ports on the R8000. On the port 5 DTs it would receive nothing.

Best regards,
Semih

#14 lore raw
In reply to Jonas Gorski

Hi Jonas,

Quoted 4 lines · Jonas Gorski

There is also a (global) bit in VLAN_CTRL5 to configure something similar to MV88E6XXX_G1_VTU_DATA_MEMBER_TAG_UNMODIFIED:

* PRESV_NON1Q - en_preserv_non_1q_frame: (default 0)

Measured on the BCM53011: the bit preserves both frame types.
Setup: vlan_filtering=1, VID 1 with the egress port dropped from the untag map, external endpoints on two front ports.

- Bit clear (the running default): an untagged frame egresses tagged VID 1, 12 of 12, and a priority tagged frame egresses tagged VID 1, 10 of 10.
- Bit set: the same untagged frames egress untagged, 10 of 10, and the same priority tagged frames egress with VID 0 preserved, 10 of 10.

The description's untag-map=1 case for priority tagged frames measures differently: they also egress with VID 0 kept, 10 of 10, where the text says they would be untagged. So with the bit set the preservation of non 1Q frames is unconditional.

Best regards,
Semih