Skip to contentSkip to conversation

net: l2switch: Provide support for L2 switch on i.MX28 SoC

RFC In review Lukasz Majewski 4 participants 24 messages Nov 25, 2020 – Jun 17, 2021

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

cover net: l2switch: Provide support for L2 switch on i.MX28 SoC

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

This code provides _very_ basic switch functionality (packets are passed between lan1 and lan2 ports and it is possible to send packets via eth0), at its main purpose is to establish the way of reusing the FEC driver. When this is done, one can add more advanced features to the switch (like vlan or port separation).

I also do have a request for testing on e.g. VF610 if this driver works on it too.
The L2 switch documentation is very scant on NXP's User Manual [0] and most understanding of how it really works comes from old (2.6.35) NXP driver [1]. The aforementioned old driver [1] was monolitic and now this patch set tries to mix FEC and DSA.

Open issues:
- I do have a hard time on understanding how to "disable" ENET-MAC{01} ports in DSA (via port_disable callback in dsa_switch_ops).
When I disable L2 switch port1,2 or the ENET-MAC{01} in control register, I cannot simply re-enable it with enabling this bit again. The old driver reset (and setup again) the whole switch.

- The L2 switch is part of the SoC silicon, so we cannot follow the "normal" DSA pattern with "attaching" it via mdio device. The switch reuses already well defined ENET-MAC{01}. For that reason the MoreThanIP switch driver is registered as platform device

- The question regarding power management - at least for my use case there is no need for runtime power management. The L2 switch shall work always at it connects other devices.

- The FEC clock is also used for L2 switch management and configuration (as the L2 switch is just in the same, large IP block). For now I just keep it enabled so DSA code can use it. It looks a bit problematic to export fec_enet_clk_enable() to be reused on DSA code.

Links:
[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013"
[1] - https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Dependencies:
This patch set depends on one, which adds DTS for XEA board. However, it shall be also possible to work on any board by adding L2 switch specific description.

https://marc.info/?l=devicetree&m=160632122703785&w=2
https://marc.info/?l=devicetree&m=160632122303783&w=2
https://marc.info/?l=devicetree&m=160632123203787&w=2

Those patches has been tested (applied) on 4.9.130-cip and v5.9 (vanila mainline kernel)

Lukasz Majewski (4):
net: fec: Move some defines to ./drivers/net/ethernet/freescale/fec.h
header
net: dsa: Provide DSA driver for NXP's More Than IP L2 switch
net: imx: l2switch: Adjust fec_main.c to provide support for L2 switch
ARM: dts: imx28: Add description for L2 switch on XEA board

 arch/arm/boot/dts/imx28-xea.dts           |  55 +++
 drivers/net/dsa/Kconfig                   |  11 +
 drivers/net/dsa/Makefile                  |   1 +
 drivers/net/dsa/mtip-l2switch.c           | 399 ++++++++++++++++++++++
 drivers/net/dsa/mtip-l2switch.h           | 239 +++++++++++++
 drivers/net/ethernet/freescale/fec.h      |  42 +++
 drivers/net/ethernet/freescale/fec_main.c | 148 ++++++--
 7 files changed, 874 insertions(+), 21 deletions(-)
 create mode 100644 drivers/net/dsa/mtip-l2switch.c
 create mode 100644 drivers/net/dsa/mtip-l2switch.h
Signature · 1 line
2.20.1
#1 lore raw

RFC 1/4 net: fec: Move some defines to ./drivers/net/ethernet/freescale/fec.h header

After this change ECR (control register) defines are moved to fec.h, so they can be reused by L2 switch code.

---
 drivers/net/ethernet/freescale/fec.h      | 3 +++
 drivers/net/ethernet/freescale/fec_main.c | 3 ---
 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/net/ethernet/freescale/fec.h +3 -0
index 832a2175636d..c555a421f647 100644
@@ -189,6 +189,9 @@
 #define FEC_RXIC2		0xfff
 #endif /* CONFIG_M5272 */
 
+/* FEC ECR bits definition */
+#define FEC_ECR_MAGICEN		(1 << 2)
+#define FEC_ECR_SLEEP		(1 << 3)
 
 /*
  *	Define the buffer descriptor structure.
drivers/net/ethernet/freescale/fec_main.c +0 -3
index fb37816a74db..bd29c84fd89a 100644
@@ -249,9 +249,6 @@ MODULE_PARM_DESC(macaddr, "FEC Ethernet MAC address");
 #define FEC_MMFR_RA(v)		((v & 0x1f) << 18)
 #define FEC_MMFR_TA		(2 << 16)
 #define FEC_MMFR_DATA(v)	(v & 0xffff)
-/* FEC ECR bits definition */
-#define FEC_ECR_MAGICEN		(1 << 2)
-#define FEC_ECR_SLEEP		(1 << 3)
 
 #define FEC_MII_TIMEOUT		30000 /* us */
 
Signature · 1 line
2.20.1
#2 lore raw

RFC 2/4 net: dsa: Provide DSA driver for NXP's More Than IP L2 switch

In reply to Lukasz Majewski

This change provides driver for DSA (Distributed Switch Architecture) subsystem for i.MX28 SoC (imx287 to be precise).

This code just is responsible for configuring this device as L2 bridge (no vlan, port separation supported).

This driver shall be regarded as a complementary one for NXP's FEC (fec_main.c).

---
 drivers/net/dsa/Kconfig         |  11 +
 drivers/net/dsa/Makefile        |   1 +
 drivers/net/dsa/mtip-l2switch.c | 399 ++++++++++++++++++++++++++++++++
 drivers/net/dsa/mtip-l2switch.h | 239 +++++++++++++++++++
 4 files changed, 650 insertions(+)
 create mode 100644 drivers/net/dsa/mtip-l2switch.c
 create mode 100644 drivers/net/dsa/mtip-l2switch.h
drivers/net/dsa/Kconfig +11 -0
index 468b3c4273c5..52013eb5afb3 100644
@@ -132,4 +132,15 @@ config NET_DSA_VITESSE_VSC73XX_PLATFORM
 	  This enables support for the Vitesse VSC7385, VSC7388, VSC7395
 	  and VSC7398 SparX integrated ethernet switches, connected over
 	  a CPU-attached address bus and work in memory-mapped I/O mode.
+
+config NET_DSA_MTIP_L2SW
+	tristate "MoreThanIP L2 switch support (IMX)"
+	depends on OF
+	depends on NET_DSA
+	depends on ARCH_MXS
+	select FIXED_PHY
+	help
+	  This enables support for the MoreThan IP on i.MX SoCs (e.g. iMX28)
+	  L2 switch.
+
 endmenu
drivers/net/dsa/Makefile +1 -0
index 4a943ccc2ca4..b947dc54db09 100644
@@ -17,6 +17,7 @@ obj-$(CONFIG_NET_DSA_SMSC_LAN9303_MDIO) += lan9303_mdio.o
 obj-$(CONFIG_NET_DSA_VITESSE_VSC73XX) += vitesse-vsc73xx-core.o
 obj-$(CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM) += vitesse-vsc73xx-platform.o
 obj-$(CONFIG_NET_DSA_VITESSE_VSC73XX_SPI) += vitesse-vsc73xx-spi.o
+obj-$(CONFIG_NET_DSA_MTIP_L2SW) += mtip-l2switch.o
 obj-y				+= b53/
 obj-y				+= microchip/
 obj-y				+= mv88e6xxx/
drivers/net/dsa/mtip-l2switch.c new file +399 -0

399 lines not on this page · raw message

drivers/net/dsa/mtip-l2switch.h new file +239 -0

239 lines not on this page · raw message

Signature · 1 line
2.20.1
#3 lore raw

RFC 3/4 net: imx: l2switch: Adjust fec_main.c to provide support for L2 switch

In reply to Lukasz Majewski

This patch provides the code for re-using generic FEC i.MX code for L2 switch.

The trick here is to set eth0 controller as a "fixed-link" and then use DSA subsystem to create lan{12} network devices for ports. The internal connection diagram can be found here [0].

This code has been developed on i.MX287 device, but L2 switch (from More ThanIP) can be also found on e.g. NXP's Vybrid VF610 SoC.

To reuse as much code as possible it was also necessary to introduce fec_hwp() wrapper, which either returns pointer to MAC or L2 switch interrupt and buffer descriptors.
Following registers shall be used with it:
FEC_{IEVENT|IMASK}, FEC_R_DES_ACTIVE_0, FEC_X_DES_ACTIVE_0, FEC_R_DES_START_0, FEC_X_DES_START_0

The biggest issue was to divide the legacy driver's code (from 2.6.35 [1]) between contemporary kernel FEC driver and DSA subsystem [2].

Only the code for setting up interrupts is protected by #ifdef CONFIG_NET_DSA_MTIP_L2SW

In other cases the fep->l2switch flag is checked (and the code is compiled into the driver) as there are only a few alterations (i.e. readl/writel) to the generic FEC code.

The most intrusive changes when L2 switch is used:
- FEC clock enabled
It is necessary to allow DSA normal operation (it shall be enabled as long as L2 switch is used)

- Disable RACC (Receive Data Accelerator)
When this feature is enabled the data is shifted by 16 bits and hence received packets are malformed when we try to pass data from one switch port to another.

- Enable PROMISC mode for ENET-MAC{01} ports

- L2 switch shall pass the whole pkt_len packet. Current FEC code subtract 4 to remove FCS

Links:
[0] - Chapter 29.3.1 in
"i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013"
[1]
From 2.6 i.MX kernel:
Repo: git://git.freescale.com/imx/linux-2.6-imx.git
Branch: imx_2.6.35_maintain
SHA1: b3912bb8a4caf3ec50909135e88af959982c43ca

[2] - https://github.com/lmajewski/linux-imx28-l2switch/commits/master

---
 drivers/net/ethernet/freescale/fec.h      |  39 ++++++
 drivers/net/ethernet/freescale/fec_main.c | 145 +++++++++++++++++++---
 2 files changed, 166 insertions(+), 18 deletions(-)
drivers/net/ethernet/freescale/fec.h +39 -0
index c555a421f647..acca7cbf9e6b 100644
@@ -29,8 +29,13 @@
  */
 #define FEC_IEVENT		0x004 /* Interrupt event reg */
 #define FEC_IMASK		0x008 /* Interrupt mask reg */
+#ifdef CONFIG_NET_DSA_MTIP_L2SW
+#define FEC_R_DES_ACTIVE_0	0x018 /* L2 switch Receive descriptor reg */
+#define FEC_X_DES_ACTIVE_0	0x01C /* L2 switch Transmit descriptor reg */
+#else
 #define FEC_R_DES_ACTIVE_0	0x010 /* Receive descriptor reg */
 #define FEC_X_DES_ACTIVE_0	0x014 /* Transmit descriptor reg */
+#endif
 #define FEC_ECNTRL		0x024 /* Ethernet control reg */
 #define FEC_MII_DATA		0x040 /* MII manage frame reg */
 #define FEC_MII_SPEED		0x044 /* MII speed control reg */
@@ -59,8 +64,13 @@
 #define FEC_R_DES_START_2	0x16c /* Receive descriptor ring 2 */
 #define FEC_X_DES_START_2	0x170 /* Transmit descriptor ring 2 */
 #define FEC_R_BUFF_SIZE_2	0x174 /* Maximum receive buff ring2 size */
+#ifdef CONFIG_NET_DSA_MTIP_L2SW
+#define FEC_R_DES_START_0	0x0C /* L2 switch Receive descriptor ring */
+#define FEC_X_DES_START_0	0x10 /* L2 switch Transmit descriptor ring */
+#else
 #define FEC_R_DES_START_0	0x180 /* Receive descriptor ring */
 #define FEC_X_DES_START_0	0x184 /* Transmit descriptor ring */
+#endif
 #define FEC_R_BUFF_SIZE_0	0x188 /* Maximum receive buff size */
 #define FEC_R_FIFO_RSFL		0x190 /* Receive FIFO section full threshold */
 #define FEC_R_FIFO_RSEM		0x194 /* Receive FIFO section empty threshold */
@@ -363,13 +373,25 @@ struct bufdesc_ex {
 #define FEC_ENET_BABR   ((uint)0x40000000)      /* Babbling receiver */
 #define FEC_ENET_BABT   ((uint)0x20000000)      /* Babbling transmitter */
 #define FEC_ENET_GRA    ((uint)0x10000000)      /* Graceful stop complete */
+#ifdef CONFIG_NET_DSA_MTIP_L2SW
+#define FEC_ENET_TXF_0	BIT(4)
+#define FEC_ENET_TXF_1	0
+#define FEC_ENET_TXF_2	0
+#else
 #define FEC_ENET_TXF_0	((uint)0x08000000)	/* Full frame transmitted */
 #define FEC_ENET_TXF_1	((uint)0x00000008)	/* Full frame transmitted */
 #define FEC_ENET_TXF_2	((uint)0x00000080)	/* Full frame transmitted */
+#endif
 #define FEC_ENET_TXB    ((uint)0x04000000)      /* A buffer was transmitted */
+#ifdef CONFIG_NET_DSA_MTIP_L2SW
+#define FEC_ENET_RXF_0	BIT(2)
+#define FEC_ENET_RXF_1	0
+#define FEC_ENET_RXF_2	0
+#else
 #define FEC_ENET_RXF_0	((uint)0x02000000)	/* Full frame received */
 #define FEC_ENET_RXF_1	((uint)0x00000002)	/* Full frame received */
 #define FEC_ENET_RXF_2	((uint)0x00000020)	/* Full frame received */
+#endif
 #define FEC_ENET_RXB    ((uint)0x01000000)      /* A buffer was received */
 #define FEC_ENET_MII    ((uint)0x00800000)      /* MII interrupt */
 #define FEC_ENET_EBERR  ((uint)0x00400000)      /* SDMA bus error */
@@ -380,6 +402,7 @@ struct bufdesc_ex {
 #define FEC_ENET_TS_TIMER       ((uint)0x00008000)
 
 #define FEC_DEFAULT_IMASK (FEC_ENET_TXF | FEC_ENET_RXF)
+#define FEC_NAPI_IMASK	FEC_ENET_MII
 #define FEC_RX_DISABLED_IMASK (FEC_DEFAULT_IMASK & (~FEC_ENET_RXF))
 
 /* ENET interrupt coalescing macro define */
@@ -587,9 +610,25 @@ struct fec_enet_private {
 	int pps_enable;
 	unsigned int next_counter;
 
+	/* More Than IP L2 switch */
+	void __iomem *hwpsw;
+	bool l2switch;
+
 	u64 ethtool_stats[];
 };
 
+/* L2 switch */
+#define L2SW_CTRL_REG_OFFSET (0x4)
+/* Get proper base address for either L2 switch or MAC ENET */
+static inline void __iomem *fec_hwp(struct fec_enet_private *fep)
+{
+#ifdef CONFIG_NET_DSA_MTIP_L2SW
+	if (fep->l2switch)
+		return fep->hwpsw;
+#endif
+	return fep->hwp;
+}
+
 void fec_ptp_init(struct platform_device *pdev, int irq_idx);
 void fec_ptp_stop(struct platform_device *pdev);
 void fec_ptp_start_cyclecounter(struct net_device *ndev);
drivers/net/ethernet/freescale/fec_main.c +127 -18

291 lines not on this page · raw message

Signature · 1 line
2.20.1
#4 lore raw

RFC 4/4 ARM: dts: imx28: Add description for L2 switch on XEA board

In reply to Lukasz Majewski

The 'eth_switch' node is now used to enable support for L2 switch. Moreover, a separate 'switch' node was introduced to keep the code more clean.

---
 arch/arm/boot/dts/imx28-xea.dts | 55 +++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm/boot/dts/imx28-xea.dts +55 -0
index 672080485b78..b8a896df02c0 100644
@@ -9,6 +9,30 @@
 
 / {
 	model = "XEA";
+
+	switch {
+		compatible = "imx,mtip-l2switch";
+		reg = <0x800f8000 0x400>, <0x800fC000 0x4000>;
+		ports {
+			port0@0 {
+				reg = <0>;
+				label = "lan1";
+				phy-handle = <&ethsw0>;
+			};
+
+			port1@1 {
+				reg = <1>;
+				label = "lan2";
+				phy-handle = <&ethsw1>;
+			};
+
+			port2@2 {
+				reg = <2>;
+				label = "cpu";
+				ethernet = <&eth_switch>;
+			};
+		};
+	};
 };
 
 &can0 {
@@ -23,6 +47,37 @@
 	status = "okay";
 };
 
+&eth_switch {
+	compatible = "fsl,imx28-l2switch";
+	reg = <0x800f0000 0x8000>, <0x800f8400 0x400>;
+	pinctrl-names = "default";
+	pinctrl-0 = <&mac0_pins_a>, <&mac1_pins_a>;
+	phy-mode = "rmii";
+	phy-supply = <&reg_fec_3v3>;
+	phy-reset-gpios = <&gpio2 13 0>;
+	phy-reset-duration = <100>;
+	interrupts = <100>, <101>, <102>;
+	clocks = <&clks 57>, <&clks 57>, <&clks 64>, <&clks 35>;
+	clock-names = "ipg", "ahb", "enet_out", "ptp";
+	local-mac-address = [ 00 11 22 AA BB CC ];
+	status = "okay";
+
+	fixed-link {
+		speed = <100>;
+		full-duplex;
+	};
+
+	mdio {
+		ethsw0: ethernet-phy@0 {
+			reg = <0>;
+		};
+
+		ethsw1: ethernet-phy@1 {
+			reg = <1>;
+		};
+	};
+};
+
 &pinctrl {
 	pinctrl-names = "default";
 	pinctrl-0 = <&hog_pins_a &hog_pins_tiva>;
Signature · 1 line
2.20.1
maintainer
#5 lore raw
In reply to Lukasz Majewski
Quoted 2 lines · Lukasz Majewski

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

Interesting. I need to take another look at the Vybrid manual. Last time i looked, i was more thinking of a pure switchdev driver, not a DSA driver. So i'm not sure this is the correct architecture. But has been a while since i looked at the datasheet.

Andrew

#6 lore raw
Quoted 8 lines · Andrew Lunn
Quoted 2 lines · Lukasz Majewski

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

Interesting. I need to take another look at the Vybrid manual. Last time i looked, i was more thinking of a pure switchdev driver, not a DSA driver. So i'm not sure this is the correct architecture. But has been a while since i looked at the datasheet.

Agreed the block diagram shows one DMA for each "switch port" which definitively fits more within the switchdev model than the DSA model that re-purposes an existing Ethernet MAC controller as-is and bolts on an integrated or external switch IC.

Signature · 1 line
Florian
maintainer
#7 lore raw
Quoted 12 lines · Florian Fainelli
Quoted 8 lines · Andrew Lunn
Quoted 2 lines · Lukasz Majewski

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

Interesting. I need to take another look at the Vybrid manual. Last time i looked, i was more thinking of a pure switchdev driver, not a DSA driver. So i'm not sure this is the correct architecture. But has been a while since i looked at the datasheet.

Quoted 4 lines

Agreed the block diagram shows one DMA for each "switch port" which definitively fits more within the switchdev model than the DSA model that re-purposes an existing Ethernet MAC controller as-is and bolts on an integrated or external switch IC.

Hi Florian

I'm not sure it is that simple. I'm looking at the Vybrid datasheet. There are two major configurations.

1) The switch is pass through, and does nothing. Then two DMA channels are used, one per external port. You basically just have two Ethernet interfaces

2) The switch is active. You then have a 3 port switch, 2 ports for the external interfaces, and one port connected to a single DMA channel.

So when in an active mode, it does look more like a DSA switch.

What is not yet clear to me is how you direct frames out specific interfaces. This is where i think we hit problems. I don't see a generic mechanism, which is probably why Lukasz put tagger as None. It does appear you can control the output of BPDUs, but it is not very friendly. You write a register with the port you would like the next BPDU to go out, queue the frame up on the DMA, and then poll a bit in the register which flips when the frame is actually processed in the switch. I don't see how you determine what port a BPDU came in on! Maybe you have to use the learning interface?

Ah, the ESW_FFEN register can be used to send a frame out a specific port. Write this register with the destination port, DMA a frame, and it goes out the specific port. You then need to write the register again for the next frame.

I get the feeling this is going to need a very close relationship between the 'tagger' and the DMA engine. I don't see how this can be done using the DSA architecture, the coupling is too loose.

It seems like the HW design assumes frames from the CPU will be switched using the switch internal FDB, making Linux integration "interesting"

It does not look like this is a classic DSA switch with a tagging protocol. It might be possible to do VLAN per port, in order to direct frames out a specific port?

Andrew

#8 lore raw

Hi Andrew, Florian,

Quoted 20 lines · Florian Fainelli
Quoted 14 lines · Andrew Lunn
Quoted 10 lines · Lukasz Majewski
Quoted 3 lines

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

Interesting. I need to take another look at the Vybrid manual. Last time i looked, i was more thinking of a pure switchdev driver, not a DSA driver. So i'm not sure this is the correct architecture. But has been a while since i looked at the datasheet.

Quoted 4 lines

Agreed the block diagram shows one DMA for each "switch port" which definitively fits more within the switchdev model than the DSA model that re-purposes an existing Ethernet MAC controller as-is and bolts on an integrated or external switch IC.

Quoted 7 lines

Hi Florian

I'm not sure it is that simple. I'm looking at the Vybrid datasheet. There are two major configurations.

1) The switch is pass through, and does nothing. Then two DMA channels are used, one per external port.

This is the "default" state (at least for imx28) - Chapter 29.3.2 on User Manual [0].

Then you use DMA0 and DMA1 to read/write data to ENET-MAC{01}.

Quoted 2 lines

You basically just have two Ethernet
interfaces

If I may add important note - on the imx28 the ENET-MAC1 is configured via ENET-MAC0 (it has FEC_QUIRK_SINGLE_MDIO set in fec_main.c). On this device it is clearly stated that ENET-MAC1 block functionality is reduced and its main purpose is to be used with L2 switch.

On the contrary, this flag is not set for vf610 in fec_main.c

Quoted 4 lines

2) The switch is active. You then have a 3 port switch, 2 ports for the external interfaces, and one port connected to a single DMA channel.

+1 (Only DMA0 is used)

Quoted 2 lines

So when in an active mode, it does look more like a DSA switch.

It also looked like this for me.

Quoted 4 lines

What is not yet clear to me is how you direct frames out specific interfaces. This is where i think we hit problems. I don't see a generic mechanism, which is probably why Lukasz put tagger as None.

I've put the "None" tag just to share the "testable" RFC code.

It is possible to "tag" frames - at least from the manual [0]:
Chapter: "29.4.9.2 Forced Forwarding".

With using register HW_ENET_SWI_FORCE_FWD_P0
29.9.34 ENET SWI Enable forced forwarding for a frame processed from port 0 (HW_ENET_SWI_FORCE_FWD_P0)

One can "tag" the packet going from port0 (internal one from SoC) to be forwarded to port1 (ENET-MAC0) or port2 (ENET-MAC1).

According to the legacy driver [1]:
"* It only replace the MAC lookup function,
* all other filtering(eg.VLAN verification) act as normal"

Quoted 7 lines

It
does appear you can control the output of BPDUs, but it is not very friendly. You write a register with the port you would like the next BPDU to go out, queue the frame up on the DMA, and then poll a bit in the register which flips when the frame is actually processed in the switch. I don't see how you determine what port a BPDU came in on! Maybe you have to use the learning interface?

The learning interface works with the legacy NXP driver (2.6.35) which copy can be found here[1].

Quoted 5 lines

Ah, the ESW_FFEN register can be used to send a frame out a specific port. Write this register with the destination port, DMA a frame, and it goes out the specific port. You then need to write the register again for the next frame.

It seems like the description of HW_ENET_SWI_FORCE_FWD_P0 described above for imx28.

Quoted 4 lines

I get the feeling this is going to need a very close relationship between the 'tagger' and the DMA engine. I don't see how this can be done using the DSA architecture, the coupling is too loose.

My first impression was that it could be possible to set this register in the DSA callback (which normally append the tag to the frame).

This would work if we can assure that after calling this callback this frame is transmitted (wait and poll?). Have I correctly understood your above concern?

I do know that L2 switch has some kind of buffer from DMA0 (port0 - its input port). However, I don't have access so such detailed manual.

Quoted 4 lines

It seems like the HW design assumes frames from the CPU will be switched using the switch internal FDB, making Linux integration "interesting"

The MoreThanIP L2 switch (on imx28) has 2K entries for setting FDB. It also uses some hash function to speed up access/presence assessment.

Quoted 3 lines

It does not look like this is a classic DSA switch with a tagging protocol.

This whole L2 switch implementation available on NXP's SoCs is a bit odd. It is very highly coupled with FEC, ENET and DMA.

The original driver (2.6.35) was just a copy of FEC driver with some switch adjustments.

Do you have any idea how to proceed?

The vf610 and imx28 are still produced and widely used, so this is still "actual" topic.

Quoted 2 lines

It might be possible to do VLAN per port, in order to direct frames out a specific port?

The old driver had code to support VLANs.

From the manual[0] (29.1):
" Programmable Ingress and Egress VLAN tag addition, removal and manipulation supporting single and double-tagged VLAN frames"

Also the "29.4.10 Frame Forwarding Tasks" from [0] sheds some more light on it.

From the manual (29.4.10.2) it looks like the VLAN tag can be appended. However, I don't know if it will work with VLAN built with L2 switch output ports.

Quoted 2 lines

Andrew

Links:

[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013"
[1] -
https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#9 lore raw
Quoted 5 lines
Quoted 3 lines

What is not yet clear to me is how you direct frames out specific interfaces. This is where i think we hit problems. I don't see a generic mechanism, which is probably why Lukasz put tagger as None.

I've put the "None" tag just to share the "testable" RFC code.

Tagging is a core feature of DSA. Without being able to direct a packet out a specific port, it is not really a DSA driver. It is also core requirement of integrating a switch into Linux. A DSA driver, or a pure switchdev driver expects to be able to forward frames out specific ports.

Quoted 7 lines

It is possible to "tag" frames - at least from the manual [0]:
Chapter: "29.4.9.2 Forced Forwarding".

With using register HW_ENET_SWI_FORCE_FWD_P0
29.9.34 ENET SWI Enable forced forwarding for a frame processed from port 0 (HW_ENET_SWI_FORCE_FWD_P0)

Quoted 6 lines

One can "tag" the packet going from port0 (internal one from SoC) to be forwarded to port1 (ENET-MAC0) or port2 (ENET-MAC1).

According to the legacy driver [1]:
"* It only replace the MAC lookup function,
* all other filtering(eg.VLAN verification) act as normal"

This might solve your outgoing frame problems. But you need to dive deep into how the FEC driver works, especially in a DSA like setup. The normal path would be, the slave interface passes a frame to the tagger driver, living in net/dsa/tag_*.c. Normally, it adds a header/trailer which the switch looks at. It then hands to packet over to the master Ethernet driver, which at some point will send the frame. Because the frame is self contained, we don't care what that ethernet driver actually does. It can add it to a queue and send it later. It can look at the QoS tags and send it with low priority after other frames, or could put it to the head of the queue and send it before other frames etc.

Since you don't have self contained frames, this is a problem. After writing to this register, you need to ensure what is transmitted next is the specific frame you intend. It cannot be added to an existing queue etc. You need to know when the frame has been sent, so you can re-write this register for the next frame.

This is why i said i don't know if the DSA architecture will work. You need a close coupling between the tagger setting the force bits, and the DMA engine sending the frame.

The other option is you totally ignore most of this and statically assign VLANs. Frames sent with VLAN 1 are forwarded out port 1. Frames sent with VLAN 2 are sent out port 2. You need the port to append/strip these VLAN tags for ingress/egress. tag_8021q.c gives you some code to help with this. But can you still use the hardware to switch frames between ports 1 and 2 without them going via the CPU?

Andrew.

#10 lore raw

Hi Andrew,

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

What is not yet clear to me is how you direct frames out specific interfaces. This is where i think we hit problems. I don't see a generic mechanism, which is probably why Lukasz put tagger as None.

Quoted 7 lines
Quoted 1 line

I've put the "None" tag just to share the "testable" RFC code.

Tagging is a core feature of DSA. Without being able to direct a packet out a specific port, it is not really a DSA driver. It is also core requirement of integrating a switch into Linux. A DSA driver, or a pure switchdev driver expects to be able to forward frames out specific ports.

Please find my answer, which I gave to Vladimir in the other mail (you were CC'ed).

As a backup plan - the vlan tagging may be worth to investigate.

Quoted 11 lines
Quoted 11 lines

It is possible to "tag" frames - at least from the manual [0]:
Chapter: "29.4.9.2 Forced Forwarding".

With using register HW_ENET_SWI_FORCE_FWD_P0
29.9.34 ENET SWI Enable forced forwarding for a frame processed from port 0 (HW_ENET_SWI_FORCE_FWD_P0)

One can "tag" the packet going from port0 (internal one from SoC) to be forwarded to port1 (ENET-MAC0) or port2 (ENET-MAC1).

Quoted 7 lines
Quoted 3 lines

According to the legacy driver [1]:
"* It only replace the MAC lookup function,
* all other filtering(eg.VLAN verification) act as normal"

This might solve your outgoing frame problems. But you need to dive deep into how the FEC driver works, especially in a DSA like setup.

Agree.

Quoted 10 lines

The normal path would be, the slave interface passes a frame to the tagger driver, living in net/dsa/tag_*.c. Normally, it adds a header/trailer which the switch looks at. It then hands to packet over to the master Ethernet driver, which at some point will send the frame. Because the frame is self contained, we don't care what that ethernet driver actually does. It can add it to a queue and send it later. It can look at the QoS tags and send it with low priority after other frames, or could put it to the head of the queue and send it before other frames etc.

Thanks for the explanation.

Quoted 5 lines

Since you don't have self contained frames, this is a problem. After writing to this register, you need to ensure what is transmitted next is the specific frame you intend. It cannot be added to an existing queue etc. You need to know when the frame has been sent, so you can re-write this register for the next frame.

This needs to be assessed as the documentation is very vague. I'm wondering how MTIP/NXP recommends usage of ESW_FFEN register.

Quoted 4 lines

This is why i said i don't know if the DSA architecture will work. You need a close coupling between the tagger setting the force bits, and the DMA engine sending the frame.

Maybe it would be just enough to program the ESW_FFEN register when ENET descriptor is programmed for DMA? Earlier we would append the superfluous tag in the tag_*.c ?

Quoted 7 lines

The other option is you totally ignore most of this and statically assign VLANs. Frames sent with VLAN 1 are forwarded out port 1. Frames sent with VLAN 2 are sent out port 2. You need the port to append/strip these VLAN tags for ingress/egress. tag_8021q.c gives you some code to help with this. But can you still use the hardware to switch frames between ports 1 and 2 without them going via the CPU?

Yes, it is possible to switch frames between ENET-MAC{01} ports without any interaction from CPU.

Quoted 2 lines

Andrew.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#11 lore raw
In reply to Lukasz Majewski

Hi Lukasz,

Quoted 16 lines · Lukasz Majewski

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

This code provides _very_ basic switch functionality (packets are passed between lan1 and lan2 ports and it is possible to send packets via eth0), at its main purpose is to establish the way of reusing the FEC driver. When this is done, one can add more advanced features to the switch (like vlan or port separation).

I also do have a request for testing on e.g. VF610 if this driver works on it too.
The L2 switch documentation is very scant on NXP's User Manual [0] and most understanding of how it really works comes from old (2.6.35) NXP driver [1]. The aforementioned old driver [1] was monolitic and now this patch set tries to mix FEC and DSA.

Quoted 6 lines

Open issues:
- I do have a hard time on understanding how to "disable" ENET-MAC{01} ports in DSA (via port_disable callback in dsa_switch_ops).
When I disable L2 switch port1,2 or the ENET-MAC{01} in control register, I cannot simply re-enable it with enabling this bit again. The old driver reset (and setup again) the whole switch.

You don't have to disable the ports if that does more harm than good, of course.

Quoted 4 lines

- The L2 switch is part of the SoC silicon, so we cannot follow the "normal" DSA pattern with "attaching" it via mdio device. The switch reuses already well defined ENET-MAC{01}. For that reason the MoreThanIP switch driver is registered as platform device

That is not a problem. Also, I would not say that the "normal" DSA pattern is to have an MDIO-attached switch. Maybe that was true 10 years ago. But now, we have DSA switches registered as platform devices, I2C devices, SPI devices, PCI devices. That is not an issue under any circumstance.

Quoted 4 lines

- The question regarding power management - at least for my use case there is no need for runtime power management. The L2 switch shall work always at it connects other devices.

Quoted 8 lines

- The FEC clock is also used for L2 switch management and configuration (as the L2 switch is just in the same, large IP block). For now I just keep it enabled so DSA code can use it. It looks a bit problematic to export fec_enet_clk_enable() to be reused on DSA code.

Links:
[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013"
[1] - https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Disclaimer: I don't know the details of imx28, it's just now that I downloaded the reference manual to see what it's about.

I would push back and say that the switch offers bridge acceleration for the FEC. The fact that the bridge acceleration is provided by a different vendor and requires access to an extra set of register blocks is immaterial. To qualify as a DSA switch, you need to have indirect networking I/O through a different network interface. You do not have that. What I would do is I would expand the fec driver into something that, on capable SoCs, detects bridging of the ENET_MAC0 and ENETC_MAC1 ports and configures the switch accordingly to offload that in a seamless manner for the user. This would also solve your power management issues, since the entire Ethernet block would be handled by a single driver. DSA is a complication you do not need. Convince me otherwise.

Also, side note.
Please, please, please, stop calling it "l2 switch" and use something more specific. If everybody writing a driver for the Linux kernel called their L2 switch "L2 switch", we would go crazy. The kernel is not a deep silo like the hardware team that integrated this MTIP switching IP into imx28, and for whom this L2 switch is the only switch that exists, and therefore for whom no further qualification was necessary. Andy, Peng or Fabio might be able to give you a reference to an internal code name that you can use, or something. Otherwise, I don't care if you need to invent a name yourself - be as creative as you feel like. mtip-fec-switch, charlie, matilda, brunhild, all fine by me.

#12 lore raw

Hi Vladimir,

Quoted 28 lines

Hi Lukasz,

Quoted 17 lines · Lukasz Majewski

This is the first attempt to add support for L2 switch available on some NXP devices - i.e. iMX287 or VF610. This patch set uses common FEC and DSA code.

This code provides _very_ basic switch functionality (packets are passed between lan1 and lan2 ports and it is possible to send packets via eth0), at its main purpose is to establish the way of reusing the FEC driver. When this is done, one can add more advanced features to the switch (like vlan or port separation).

I also do have a request for testing on e.g. VF610 if this driver works on it too.
The L2 switch documentation is very scant on NXP's User Manual [0] and most understanding of how it really works comes from old (2.6.35) NXP driver [1]. The aforementioned old driver [1] was monolitic and now this patch set tries to mix FEC and DSA.

Quoted 7 lines

Open issues:
- I do have a hard time on understanding how to "disable" ENET-MAC{01} ports in DSA (via port_disable callback in dsa_switch_ops). When I disable L2 switch port1,2 or the ENET-MAC{01} in control register, I cannot simply re-enable it with enabling this bit again. The old driver reset (and setup again) the whole switch.

Quoted 2 lines

You don't have to disable the ports if that does more harm than good, of course.

Ok.

Quoted 11 lines
Quoted 5 lines

- The L2 switch is part of the SoC silicon, so we cannot follow the "normal" DSA pattern with "attaching" it via mdio device. The switch reuses already well defined ENET-MAC{01}. For that reason the MoreThanIP switch driver is registered as platform device

That is not a problem. Also, I would not say that the "normal" DSA pattern is to have an MDIO-attached switch. Maybe that was true 10 years ago. But now, we have DSA switches registered as platform devices, I2C devices, SPI devices, PCI devices. That is not an issue under any circumstance.

Ok. The MTIP switch now registers as platform device.

Quoted 17 lines
Quoted 11 lines

- The question regarding power management - at least for my use case there is no need for runtime power management. The L2 switch shall work always at it connects other devices.

- The FEC clock is also used for L2 switch management and configuration (as the L2 switch is just in the same, large IP block). For now I just keep it enabled so DSA code can use it. It looks a bit problematic to export fec_enet_clk_enable() to be reused on DSA code.

Quoted 5 lines

Links:
[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013" [1] -
https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Quoted 5 lines

Disclaimer: I don't know the details of imx28, it's just now that I downloaded the reference manual to see what it's about.

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

And without the switch IP block enabled one shall be able to have software bridging in Linux in those two interfaces?

Quoted 3 lines

The fact that the bridge acceleration is provided by a different vendor and requires access to an extra set of register blocks is immaterial.

Am I correct that you mean not important above (i.e. immaterial == not important)?

Quoted 3 lines

To qualify as a DSA switch, you need to have
indirect networking I/O through a different network interface. You do not have that.

I do have eth0 (DMA0) -> MoreThanIP switch port 0 input

			 |
			 |----> switch port1 -> ENET-MAC0
			 |
			 |----> switch port2 -> ENET-MAC1
Quoted 4 lines

What I would do is I would expand the fec driver into something that, on capable SoCs, detects bridging of the ENET_MAC0 and ENETC_MAC1 ports and configures the switch accordingly to offload that in a seamless manner for the user.

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

And then configure the FEC driver to use this L2 switch driver?

Quoted 4 lines

This would also solve your
power management issues, since the entire Ethernet block would be handled by a single driver. DSA is a complication you do not need. Convince me otherwise.

From what I see the MoreThanIP IP block looks like a "typical" L2 switch (like lan9xxx), with VLAN tagging support, static and dynamic tables, forcing the packet to be passed to port [*], congestion management, switch input buffer, priority of packets/queues, broadcast, multicast, port snooping, and even IEEE1588 timestamps.

Seems like a lot of useful features.

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

2. The TAG is not appended to the frame outgoing from the "master" FEC port - it can be setup when DMA transfers packet to MTIP switch internal buffer.

Note:

[*] - The same situation is in the VF610 and IMX28:
The ESW_FFEN register - Bit 0 -> FEN

"When set, the next frame received from port 0 (the local DMA port) is forwarded to the ports defined in FD. The bit resets to zero automatically when one frame from port 0 has been processed by the switch (i.e. has been read from the port 0 input buffer; see Figure 32-1). Therefore, the bit must be set again as necessary. See also Section 32.5.8.2, "Forced Forwarding" for a description."

(Of course the "Section 32.5.8.2" is not available)

According to above the "tag" (engress port) is set when DMA transfers the packet to input MTIP buffer. This shall allow force forwarding as we can setup this bit when we normally append tag in the network stack.

I will investigate this issue - and check the port separation. If it works then DSA (or switchdev) shall be used?

(A side question - DSA uses switchdev, so when one shall use switchdev standalone?)

Quoted 9 lines

Also, side note.
Please, please, please, stop calling it "l2 switch" and use something more specific. If everybody writing a driver for the Linux kernel called their L2 switch "L2 switch", we would go crazy. The kernel is not a deep silo like the hardware team that integrated this MTIP switching IP into imx28, and for whom this L2 switch is the only switch that exists, and therefore for whom no further qualification was necessary.

The "l2 switch" name indeed was taken from the NXP's legacy driver :-)

Quoted 2 lines

Andy, Peng or Fabio might be able to give you a reference to an internal code name that you can use, or something.

I would prefer to obtain some kind of manual/doc/spec for MTIP IP block.

Quoted 3 lines

Otherwise, I don't care if you need to invent a name yourself - be as creative as you feel like. mtip-fec-switch, charlie, matilda, brunhild, all fine by me.

I think that "mtip-fec-switch" is the most reasonable name.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#13 lore raw
Quoted 2 lines

(A side question - DSA uses switchdev, so when one shall use switchdev standalone?)

DSA gives you a framework for an Ethernet switch connected to a host via Ethernet for the data plane. Generally, that Ethernet link to the switch is a MAC to MAC connection. It can be PHY to PHY. But those are just details. The important thing is you use an Ethernet driver on the host.

If you look at pure switchdev devices, they generally DMA frames directly into the switch. There is either one DMA queue per switch port, or there is a way to multiplex frames over one DMA queue, generally by additional fields in the buffer descriptor.

For this device, at the moment, it is hard to say which is the best fit. A lot will depend on how the FEC driver works, if you can reuse it, while still having the degree of control you need over the DMA channel. If you can reuse the FEC driver, then a DSA driver might work. If the coupling it too loose, and you have to take control of the DMA, then a pure switchdev driver seems more appropriate.

Andrew

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#14 lore raw

Hi Andrew,

Quoted 14 lines
Quoted 2 lines

(A side question - DSA uses switchdev, so when one shall use switchdev standalone?)

DSA gives you a framework for an Ethernet switch connected to a host via Ethernet for the data plane. Generally, that Ethernet link to the switch is a MAC to MAC connection. It can be PHY to PHY. But those are just details. The important thing is you use an Ethernet driver on the host.

If you look at pure switchdev devices, they generally DMA frames directly into the switch. There is either one DMA queue per switch port, or there is a way to multiplex frames over one DMA queue, generally by additional fields in the buffer descriptor.

Quoted 9 lines

For this device, at the moment, it is hard to say which is the best fit. A lot will depend on how the FEC driver works, if you can reuse it, while still having the degree of control you need over the DMA channel. If you can reuse the FEC driver, then a DSA driver might work. If the coupling it too loose, and you have to take control of the DMA, then a pure switchdev driver seems more appropriate.

Andrew

Thanks for the detailed explanation.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#15 lore raw
In reply to Lukasz Majewski
Quoted 5 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

You should think of the hardware as an accelerator, not a switch. The hardware is there to accelerate what linux can already do. You setup a software bridge in linux, and then offload L2 switching to the accelerator. You setup vlans in linux, and then offload the filtering of them to the accelerator. If there is something linux can do, but the hardware cannot accelerate, you leave linux to do it in software.

Quoted 8 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

And then configure the FEC driver to use this L2 switch driver?

That is what switchdev does. There are various hooks in the network stack which call into switchdev to ask it to offload operations to the accelerator.

Quoted 4 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

That does not matter. And there are memory mapped DSA switches. The DSA framework puts no restrictions on how the control plane works.

Quoted 1 line

(Of course the "Section 32.5.8.2" is not available)

It is in the Vybrid datasheet :-)

Andrew

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#16 lore raw

Hi Andrew,

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

Quoted 6 lines

You should think of the hardware as an accelerator, not a switch. The hardware is there to accelerate what linux can already do. You setup a software bridge in linux, and then offload L2 switching to the accelerator. You setup vlans in linux, and then offload the filtering of them to the accelerator. If there is something linux can do, but the hardware cannot accelerate, you leave linux to do it in software.

Ok.

Quoted 8 lines
Quoted 8 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

Quoted 5 lines
Quoted 1 line

And then configure the FEC driver to use this L2 switch driver?

That is what switchdev does. There are various hooks in the network stack which call into switchdev to ask it to offload operations to the accelerator.

Ok.

Quoted 6 lines
Quoted 5 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

Quoted 6 lines

That does not matter. And there are memory mapped DSA switches. The DSA framework puts no restrictions on how the control plane works.

Quoted 1 line

(Of course the "Section 32.5.8.2" is not available)

It is in the Vybrid datasheet :-)

Hmm...

I cannot find such chapter in the official documentation from NXP:
"VFxxx Controller Reference Manual, Rev. 0, 10/2016"

Maybe you have more verbose version? Could you share how the document is named?

Quoted 2 lines

Andrew

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#17 lore raw
Quoted 34 lines · Lukasz Majewski

Hi Andrew,

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

Quoted 6 lines

You should think of the hardware as an accelerator, not a switch. The hardware is there to accelerate what linux can already do. You setup a software bridge in linux, and then offload L2 switching to the accelerator. You setup vlans in linux, and then offload the filtering of them to the accelerator. If there is something linux can do, but the hardware cannot accelerate, you leave linux to do it in software.

Ok.

Quoted 8 lines
Quoted 8 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

Quoted 5 lines
Quoted 1 line

And then configure the FEC driver to use this L2 switch driver?

That is what switchdev does. There are various hooks in the network stack which call into switchdev to ask it to offload operations to the accelerator.

Ok.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

Hmm...

I cannot find such chapter in the official documentation from NXP:
"VFxxx Controller Reference Manual, Rev. 0, 10/2016"

I have

Vybrid Reference Manual
F-Series

Document Number: VYBRIDRM
Rev 7, 06/2014

Andrew

#18 lore raw
In reply to Andrew Lunn

Hi Andrew,

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

Quoted 13 lines

You should think of the hardware as an accelerator, not a switch. The hardware is there to accelerate what linux can already do. You setup a software bridge in linux, and then offload L2 switching to the accelerator. You setup vlans in linux, and then offload the filtering of them to the accelerator. If there is something linux can do, but the hardware cannot accelerate, you leave linux to do it in software.

Quoted 6 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

^^^^^^^^^^^^^ [*]

Quoted 6 lines
Quoted 2 lines

And then configure the FEC driver to use this L2 switch driver?

That is what switchdev does. There are various hooks in the network stack which call into switchdev to ask it to offload operations to the accelerator.

I'm a bit confused about the interfaces that pop up when I do enable bridging acceleration.

Without bridge I do have:
- eth0 (this is a 'primary' interface -> it also controls MII/PHY for eth1)
- eth1 (it uses the MII/PHY control from eth0)

Both interfaces works correctly.

And after starting the bridge (and switchdev) with commands from [*] I do have:

- br0 (created bridge - need to assign IP to it to communicate outside, even when routing is set via eth0, and eth0 has the same IP address)
- eth0 (just is used to control PHY - ifconfig up/down)
- eth1 (just is used to control PHY - ifconfig up/down)

And now the question, how internally shall I tackle the transmission (i.e. DMA setups)?

Now, I do use some hacks to re-use code for eth0 to perform the transmission from/to imx28 L2 switch. The eth1 is stopped (it only controls the PHY - responds to MII interrupts).

The above setup works, and the code adjustment for fec_main.c driver is really small.

However, I do wonder how conceptually it "mix" with br0 interface? I could leave br0 as is, but then why do I need to asign the IP address to it to communicate?
As I need to do it - then conceptually (re-)using eth0 internal structures (and the driver in fact) looks like some kind of abusement. However, adding the transmission handling to br0 net device would bloat and potentially duplicate the code.

I would prefer to re-use code from eth0 interface - it would be also easier to cleanu up after disabling the L2 switch.

Any feedback and help is more than welcome.

Quoted 6 lines
Quoted 5 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

Quoted 8 lines

That does not matter. And there are memory mapped DSA switches. The DSA framework puts no restrictions on how the control plane works.

Quoted 1 line

(Of course the "Section 32.5.8.2" is not available)

It is in the Vybrid datasheet :-)

Andrew

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

maintainer
#19 lore raw
Quoted 31 lines · Lukasz Majewski

Hi Andrew,

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

Quoted 13 lines

You should think of the hardware as an accelerator, not a switch. The hardware is there to accelerate what linux can already do. You setup a software bridge in linux, and then offload L2 switching to the accelerator. You setup vlans in linux, and then offload the filtering of them to the accelerator. If there is something linux can do, but the hardware cannot accelerate, you leave linux to do it in software.

Quoted 6 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

^^^^^^^^^^^^^ [*]

Quoted 6 lines
Quoted 2 lines

And then configure the FEC driver to use this L2 switch driver?

That is what switchdev does. There are various hooks in the network stack which call into switchdev to ask it to offload operations to the accelerator.

Quoted 7 lines

I'm a bit confused about the interfaces that pop up when I do enable bridging acceleration.

Without bridge I do have:
- eth0 (this is a 'primary' interface -> it also controls MII/PHY for eth1)
- eth1 (it uses the MII/PHY control from eth0)

You need to be careful with wording here. eth0 might provide an MDIO bus which eth1 PHY is connected to, but eth1 is controlling the PHY, not eth0.

Quoted 5 lines

Both interfaces works correctly.

And after starting the bridge (and switchdev) with commands from [*] I do have:

You don't start switchdev. switchdev is just an API between the network stack and the hardware accelerator.

Quoted 2 lines

- br0 (created bridge - need to assign IP to it to communicate outside, even when routing is set via eth0, and eth0 has the same IP address)

The IP address should only be on the bridge.

Quoted 2 lines

- eth0 (just is used to control PHY - ifconfig up/down)
- eth1 (just is used to control PHY - ifconfig up/down)

It is more than that. You can still send on these interfaces. e.g. if you are using ldap, or L2 PTP, the daemons will use these interfaces, not the bridge, since they want to send frames out a specific interface.

Quoted 5 lines

And now the question, how internally shall I tackle the transmission (i.e. DMA setups)?

Now, I do use some hacks to re-use code for eth0 to perform the transmission from/to imx28 L2 switch.

Avoid hacks. Always. Step back. Look at the hardware. How is the hardware meant to be used when using the switch?

You have two choices.

1) A DSA style driver. The SoC has an Ethernet interface connect to one port of the switch. In this case, there should be two additional ports of the switch connected to the outside world, i.e. the switch has 3 ports.

2) A pure switchdev driver. You DMA frames directly to the switch ingree/egress queues for each port connected to the outside world. In this case, you have a 2 port switch.

Once you get the architecture correct, then you can think about the driver. Maybe you can reuse parts of the FEC? Maybe not.

Andrew

maintainer
#20 lore raw
In reply to Lukasz Majewski
Quoted 19 lines · Lukasz Majewski
Quoted 19 lines
Quoted 10 lines

- The question regarding power management - at least for my use case there is no need for runtime power management. The L2 switch shall work always at it connects other devices.

- The FEC clock is also used for L2 switch management and configuration (as the L2 switch is just in the same, large IP block). For now I just keep it enabled so DSA code can use it. It looks a bit problematic to export fec_enet_clk_enable() to be reused on DSA code.

Quoted 5 lines

Links:
[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013" [1] -
https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Disclaimer: I don't know the details of imx28, it's just now that I downloaded the reference manual to see what it's about.

Quoted 8 lines
Quoted 2 lines

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

And without the switch IP block enabled one shall be able to have software bridging in Linux in those two interfaces?

So if the switch is bypassed through pin strapping of sx_ena, then the DMA0 would be connected to ENETC-MAC 0, DMA1 to ENET-MAC 1, and both these ports could be driven by the regular fec driver, am I right? This is how people use the imx28 with mainline linux right now, no?

When the sx_ena signal enables the switch, a hardware accelerator appears between the DMA engine and the same MACs. But that DMA engine is still compatible with the expectations of the fec driver. And the MACs still belong to the FEC. So, at the end of the day, there are still 2 FEC interfaces.

Where I was going is that from a user's perspective, it would be natural to have the exact same view of the system in both cases, aka still two network interfaces for MAC 0 and MAC 1, they still function in the same way (i.e. you can still ping through them) but with the additional ability to do hardware-accelerating bridging, if the MAC 0 and MAC 1 network interfaces are put under the same bridge.

Currently, you do not offer that, at all.
You split the MTIP switch configuration between DSA and the FEC driver. But you are still not exposing networking-capable net devices for MAC 0 and MAC 1. You still do I/O through the DMA engine.

So why use DSA at all? What benefit does it bring you? Why not do the entire switch configuration from within FEC, or a separate driver very closely related to it?

Quoted 6 lines
Quoted 3 lines

The fact that the bridge acceleration is provided by a different vendor and requires access to an extra set of register blocks is immaterial.

Am I correct that you mean not important above (i.e. immaterial == not important)?

Yes, sorry if that was not clear.

Quoted 9 lines
Quoted 3 lines

To qualify as a DSA switch, you need to have
indirect networking I/O through a different network interface. You do not have that.

I do have eth0 (DMA0) -> MoreThanIP switch port 0 input

			 |
			 |----> switch port1 -> ENET-MAC0
			 |
			 |----> switch port2 -> ENET-MAC1

The whole point of DSA is to intercept traffic from a different and completely unaware network interface, parse a tag, and masquerade the packet as though it came on a different, virtual, network interface corresponding to the switch. DSA offers this service so that:
- any switch works with any host Ethernet controller
- all vendors use the same mechanism and do not reinvent the same wheel The last part is especially relevant. Just grep net/core/ for "dsa" and be amazed at the number of hacks there. DSA can justify that only by scale. Aka "we have hardware that works this way, and doesn't work any other way. And lots of it".

That being said, in your case, the network interface is not unaware of the switch at all. Come on, you need to put the (allegedly) switch's MAC in promiscuous mode, from the "DSA master" driver! Hardware resource ownership is very DSA-unlike in the imx28/vybrid model, it seems. This is a very big deal.

And there is no tag to parse. You have a switch with a DMA engine which is a priori known to be register-compatible with the DMA engine used for plain FEC interfaces. It's not like you have a switch that can be connected via MII to anything and everything. Force forwarding is done via writing a register, again very much unlike DSA, and retrieving the source port on RX is up in the air.

Quoted 5 lines
Quoted 4 lines

What I would do is I would expand the fec driver into something that, on capable SoCs, detects bridging of the ENET_MAC0 and ENETC_MAC1 ports and configures the switch accordingly to offload that in a seamless manner for the user.

Quoted 8 lines

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

And then configure the FEC driver to use this L2 switch driver?

Yes, that can be summarized as:
One option to get this upstream would be to transform fec into a switchdev driver, or to create a mtip switchdev driver that shares a lot of code with the fec driver. The correct tool for code sharing is EXPORT_SYMBOL_GPL, not DSA.

Quoted 5 lines
Quoted 4 lines

This would also solve your
power management issues, since the entire Ethernet block would be handled by a single driver. DSA is a complication you do not need. Convince me otherwise.

Quoted 7 lines

From what I see the MoreThanIP IP block looks like a "typical" L2 switch (like lan9xxx), with VLAN tagging support, static and dynamic tables, forcing the packet to be passed to port [*], congestion management, switch input buffer, priority of packets/queues, broadcast, multicast, port snooping, and even IEEE1588 timestamps.

Seems like a lot of useful features.

I did not say that it is not a typical switch, or that it doesn't have useful features. I said that DSA does not help you. Adding a DSA_TAG_PROTO_NONE driver in 2020 is a no-go all around.

Quoted 4 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

Nope, that's not a difference.

Quoted 3 lines

2. The TAG is not appended to the frame outgoing from the "master" FEC port - it can be setup when DMA transfers packet to MTIP switch internal buffer.

That is a difference indeed. Since DSA switches are isolated from their host interface, and there has been a traditional separation at the MAC-to-MAC layer (or MAC-to-PHY-to-PHY-to-MAC in weird cases), the routing information is passed in-band via a header in the packet. The host interface has no idea that this header exists, it is just traffic as usual. The whole DSA infrastructure is built around intercepting and decoding that.

Quoted 15 lines

Note:

[*] - The same situation is in the VF610 and IMX28:
The ESW_FFEN register - Bit 0 -> FEN

"When set, the next frame received from port 0 (the local DMA port) is forwarded to the ports defined in FD. The bit resets to zero automatically when one frame from port 0 has been processed by the switch (i.e. has been read from the port 0 input buffer; see Figure 32-1). Therefore, the bit must be set again as necessary. See also Section 32.5.8.2, "Forced Forwarding" for a description."

(Of course the "Section 32.5.8.2" is not available)

Quoted 6 lines

According to above the "tag" (engress port) is set when DMA transfers the packet to input MTIP buffer. This shall allow force forwarding as we can setup this bit when we normally append tag in the network stack.

I will investigate this issue - and check the port separation. If it works then DSA (or switchdev) shall be used?

Source port identification and individual egress port addressing are things that should behave absolutely the same regardless of whether you have a DSA or a pure switchdev driver. I don't think that this is clear enough to you. DSA with DSA_TAG_PROTO_NONE is not the shortcut you're looking for. Please take a look at Documentation/networking/switchdev.rst, it explains pretty clearly that a switchdev port must still expose the same interface as any other net device.

Quoted 2 lines

(A side question - DSA uses switchdev, so when one shall use switchdev standalone?)

Short answer: if the system-side packet I/O interface is Ethernet and the hardware ownership is clearly delineated at the boundary of that Ethernet port, then it should be DSA, otherwise it shouldn't.

But nonetheless, this raises a fundamental question, and I'll indulge in attempting to answer it more comprehensively.

You seem to be tapping into an idea which has been circulating for a while, and which can be also found in Documentation/networking/dsa/dsa.rst:

-----------------------------[cut here]-----------------------------

TODO
====

Making SWITCHDEV and DSA converge towards an unified codebase
-------------------------------------------------------------

SWITCHDEV properly takes care of abstracting the networking stack with offload capable hardware, but does not enforce a strict switch device driver model. On the other DSA enforces a fairly strict device driver model, and deals with most of the switch specific. At some point we should envision a merger between these two subsystems and get the best of both worlds.

-----------------------------[cut here]-----------------------------

IMO this is never going to happen, nor should it.
To unify DSA and switchdev would mean to force plain switchdev drivers to have a stricter separation between the control path and the data path, a la DSA. So, just like DSA has separate drivers for the switch, the tagging protocol and for the DSA master, plain switchdev would need a separate driver too for the DMA/rings/queues, or whatever, of the switch (the piece of code that implements the NAPI instance). That driver would need to:
(a) expose a net device for the DMA interface
(b) fake a DSA tag that gets added by the DMA net device, just to be later removed by the switchdev net device.
This is just for compatibility with what DSA _needs_ to do to drive hardware that was _designed_ to work that way, and where the _hardware_ adds that tag. But non-DSA drivers don't need any of that, nor does it really help them in any way! So why should we model things this way? I don't think the lines between plain switchdev and DSA are blurry at all. And especially not in this case. You should not have a networking interface registered to the system for DMA-0 at all, just for MAC-0 and MAC-1.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#21 lore raw

Hi Vladimir,

Quoted 26 lines · Lukasz Majewski
Quoted 25 lines
Quoted 16 lines
Quoted 10 lines

- The question regarding power management - at least for my use case there is no need for runtime power management. The L2 switch shall work always at it connects other devices.

- The FEC clock is also used for L2 switch management and configuration (as the L2 switch is just in the same, large IP block). For now I just keep it enabled so DSA code can use it. It looks a bit problematic to export fec_enet_clk_enable() to be reused on DSA code.

Quoted 5 lines

Links:
[0] - "i.MX28 Applications Processor Reference Manual, Rev. 2, 08/2013" [1] -
https://github.com/lmajewski/linux-imx28-l2switch/commit/e3c7a6eab73401e021aef0070e1935a0dba84fb5

Quoted 5 lines

Disclaimer: I don't know the details of imx28, it's just now that I downloaded the reference manual to see what it's about.

I would push back and say that the switch offers bridge acceleration for the FEC.

Am I correct, that the "bridge acceleration" means in-hardware support for L2 packet bridging?

Quoted 7 lines
Quoted 2 lines

And without the switch IP block enabled one shall be able to have software bridging in Linux in those two interfaces?

So if the switch is bypassed through pin strapping of sx_ena, then the DMA0 would be connected to ENETC-MAC 0, DMA1 to ENET-MAC 1, and both these ports could be driven by the regular fec driver, am I right? This is how people use the imx28 with mainline linux right now, no?

Yes. This is the current situation.

Quoted 3 lines

When the sx_ena signal enables the switch, a hardware accelerator appears between the DMA engine and the same MACs.

Yes.

Quoted 2 lines

But that DMA engine
is still compatible with the expectations of the fec driver.

Yes. This is why I can reuse the fec_main.c driver.

Quoted 2 lines

And the
MACs still belong to the FEC.

Yes. I do use mdio communication between MAC and PHYs (by re-using FEC driver code).

Quoted 2 lines

So, at the end of the day, there are
still 2 FEC interfaces.

I'm a bit confused with the above sentence. What do you mean "2 FEC interfaces"?

There is the FEC driver (fec_main.c), which handles DMA0 management/setup and communication with PHYs via MDIO (the FEC driver setups MDIO, link type, speed, provides info if link is up or down).

Quoted 7 lines

Where I was going is that from a user's perspective, it would be natural to have the exact same view of the system in both cases, aka still two network interfaces for MAC 0 and MAC 1, they still function in the same way (i.e. you can still ping through them) but with the additional ability to do hardware-accelerating bridging, if the MAC 0 and MAC 1 network interfaces are put under the same bridge.

At least on imx287 ENET-MAC1 needs MAC0 to be configured:
https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/freescale/fec_main.c#L2065

Those two MACs are not fully independent - functionality of MAC1 is reduced.

Quoted 5 lines

Currently, you do not offer that, at all.
You split the MTIP switch configuration between DSA and the FEC driver. But you are still not exposing networking-capable net devices for MAC 0 and MAC 1.

As fair as I can tell (and tested it) - it is possible to bring up "lan{12}" interfaces with ip l s lan{12}

I would also like to set the static table with altering the FDB switch memory/configuration. Access to it seems natural via lan{12} interfaces (but of course could be possible from eth0).

Yes, I can send data via eth0 as I don't add tag to it. As pointed out by Andrew - I could use VLAN tags (or play with FEC_FFEN register).

Quoted 1 line

You still do I/O through the DMA engine.

Yes, DMA0 is connected to MTIP port0 (input port from the imx28 SoC). The MTIP port1 -> MAC0 and port2 -> MAC1

Quoted 4 lines

So why use DSA at all? What benefit does it bring you? Why not do the entire switch configuration from within FEC, or a separate driver very closely related to it?

Mine rationale to use DSA and FEC:
- Make as little changes to FEC as possible

- Provide separate driver to allow programming FDB, MDB, VLAN setup. This seems straightforward as MTIP has separate memory region (from FEC) for switch configuration, statistics, learning, static table programming. What is even more bizarre FEC and MTIP have the same 8 registers (with different base address and +4 offset :-) ) as interface to handle DMA0 transfers.

- According to MTIP description from NXP documentation, there is a separate register for frame forwarding, so it _shall_ also fit into DSA.

For me it would be enough to have:

- lan{12} - so I could enable/disable it on demand (control when switch ports are passing or not packets).

- Use standard net tools (like bridge) to setup FDB/MDB, vlan

- Read statistics from MTIP ports (all of them)

- I can use lan1 (bridged or not) to send data outside. It would be also correct to use eth0.

I'm for the most pragmatic (and simple) solution, which fulfill above requirements.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

The fact that the bridge acceleration is provided by a different vendor and requires access to an extra set of register blocks is immaterial.

Am I correct that you mean not important above (i.e. immaterial == not important)?

Yes, sorry if that was not clear.

Ok. I was just not sure.

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

To qualify as a DSA switch, you need to have indirect networking I/O through a different network interface. You do not have that.

I do have eth0 (DMA0) -> MoreThanIP switch port 0 input

			 |
			 |----> switch port1 -> ENET-MAC0
			 |
			 |----> switch port2 -> ENET-MAC1  
Quoted 10 lines

The whole point of DSA is to intercept traffic from a different and completely unaware network interface, parse a tag, and masquerade the packet as though it came on a different, virtual, network interface corresponding to the switch. DSA offers this service so that:
- any switch works with any host Ethernet controller
- all vendors use the same mechanism and do not reinvent the same wheel The last part is especially relevant. Just grep net/core/ for "dsa" and be amazed at the number of hacks there. DSA can justify that only by scale. Aka "we have hardware that works this way, and doesn't work any other way. And lots of it".

That is why I wanted to use FEC as "fixed-link" interface, so large portion of it would be disabled (e.g. the ENET-MAC management for eth0).

Quoted 3 lines

That being said, in your case, the network interface is not unaware of the switch at all.

Unfortunately, you are right here. FEC, MTIP, ENET-MAC are very closely coupled.

Quoted 4 lines

Come on, you need to put the (allegedly) switch's
MAC in promiscuous mode, from the "DSA master" driver! Hardware resource ownership is very DSA-unlike in the imx28/vybrid model, it seems. This is a very big deal.

Ok.

Quoted 7 lines

And there is no tag to parse. You have a switch with a DMA engine which is a priori known to be register-compatible with the DMA engine used for plain FEC interfaces. It's not like you have a switch that can be connected via MII to anything and everything. Force forwarding is done via writing a register, again very much unlike DSA, and retrieving the source port on RX is up in the air.

Yes. You are correct here.

Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

What I would do is I would expand the fec driver into something that, on capable SoCs, detects bridging of the ENET_MAC0 and ENETC_MAC1 ports and configures the switch accordingly to offload that in a seamless manner for the user.

Do you propose to catch some kind of notification when user calls:

ip link add name br0 type bridge; ip link set br0 up; ip link set lan1 up; ip link set lan2 up;
ip link set lan1 master br0; ip link set lan2 master br0; bridge link

Quoted 7 lines
Quoted 1 line

And then configure the FEC driver to use this L2 switch driver?

Yes, that can be summarized as:
One option to get this upstream would be to transform fec into a switchdev driver, or to create a mtip switchdev driver that shares a lot of code with the fec driver. The correct tool for code sharing is EXPORT_SYMBOL_GPL, not DSA.

One more note - at least for imx28 you can have L2 switch or FEC, not both.

Considering the above - you would recommend using the switchdev framework for MTIP?

Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

This would also solve your
power management issues, since the entire Ethernet block would be handled by a single driver. DSA is a complication you do not need. Convince me otherwise.

From what I see the MoreThanIP IP block looks like a "typical" L2 switch (like lan9xxx), with VLAN tagging support, static and dynamic tables, forcing the packet to be passed to port [*], congestion management, switch input buffer, priority of packets/queues, broadcast, multicast, port snooping, and even IEEE1588 timestamps.

Quoted 5 lines
Quoted 1 line

Seems like a lot of useful features.

I did not say that it is not a typical switch, or that it doesn't have useful features. I said that DSA does not help you. Adding a DSA_TAG_PROTO_NONE driver in 2020 is a no-go all around.

Ok. I see.

Quoted 7 lines
Quoted 5 lines

The differences from "normal" DSA switches:

1. It uses mapped memory (for its register space) for configuration/statistics gathering (instead of e.g. SPI, I2C)

Nope, that's not a difference.

Ok.

Quoted 5 lines
Quoted 4 lines

2. The TAG is not appended to the frame outgoing from the "master" FEC port - it can be setup when DMA transfers packet to MTIP switch internal buffer.

Quoted 8 lines

That is a difference indeed. Since DSA switches are isolated from their host interface, and there has been a traditional separation at the MAC-to-MAC layer (or MAC-to-PHY-to-PHY-to-MAC in weird cases), the routing information is passed in-band via a header in the packet. The host interface has no idea that this header exists, it is just traffic as usual. The whole DSA infrastructure is built around intercepting and decoding that.

Ok, so MTIP doesn't provide the core functionality for DSA out of the box.

Quoted 23 lines
Quoted 15 lines

Note:

[*] - The same situation is in the VF610 and IMX28:
The ESW_FFEN register - Bit 0 -> FEN

"When set, the next frame received from port 0 (the local DMA port) is forwarded to the ports defined in FD. The bit resets to zero automatically when one frame from port 0 has been processed by the switch (i.e. has been read from the port 0 input buffer; see Figure 32-1). Therefore, the bit must be set again as necessary. See also Section 32.5.8.2, "Forced Forwarding" for a description."

(Of course the "Section 32.5.8.2" is not available)

Quoted 7 lines

According to above the "tag" (engress port) is set when DMA transfers the packet to input MTIP buffer. This shall allow force forwarding as we can setup this bit when we normally append tag in the network stack.

I will investigate this issue - and check the port separation. If it works then DSA (or switchdev) shall be used?

Quoted 8 lines

Source port identification and individual egress port addressing are things that should behave absolutely the same regardless of whether you have a DSA or a pure switchdev driver. I don't think that this is clear enough to you. DSA with DSA_TAG_PROTO_NONE is not the shortcut you're looking for. Please take a look at Documentation/networking/switchdev.rst, it explains pretty clearly that a switchdev port must still expose the same interface as any other net device.

Ok.

Quoted 8 lines
Quoted 3 lines

(A side question - DSA uses switchdev, so when one shall use switchdev standalone?)

Short answer: if the system-side packet I/O interface is Ethernet and the hardware ownership is clearly delineated at the boundary of that Ethernet port, then it should be DSA, otherwise it shouldn't.

Ok.

Quoted 5 lines

But nonetheless, this raises a fundamental question, and I'll indulge in attempting to answer it more comprehensively.

You seem to be tapping into an idea which has been circulating for a while,

There was only NXP's legacy driver (2.6.35), which added support for MTIP switch. Up till then nobody tried to upstream it....

I do have this driver forward ported to v4.19.y:
https://github.com/lmajewski/linux-imx28-l2switch/commits/master

but its long term maintenance is at best awkward.

Quoted 20 lines

and which can be also found in
Documentation/networking/dsa/dsa.rst:

-----------------------------[cut here]-----------------------------

TODO
====

Making SWITCHDEV and DSA converge towards an unified codebase
-------------------------------------------------------------

SWITCHDEV properly takes care of abstracting the networking stack with offload capable hardware, but does not enforce a strict switch device driver model. On the other DSA enforces a fairly strict device driver model, and deals with most of the switch specific. At some point we should envision a merger between these two subsystems and get the best of both worlds.

-----------------------------[cut here]-----------------------------

Quoted 11 lines

IMO this is never going to happen, nor should it.
To unify DSA and switchdev would mean to force plain switchdev drivers to have a stricter separation between the control path and the data path, a la DSA. So, just like DSA has separate drivers for the switch, the tagging protocol and for the DSA master, plain switchdev would need a separate driver too for the DMA/rings/queues, or whatever, of the switch (the piece of code that implements the NAPI instance). That driver would need to:
(a) expose a net device for the DMA interface
(b) fake a DSA tag that gets added by the DMA net device, just to be later removed by the switchdev net device.

I also thought for a while to append "tag" byte ind tag_*.c file and then parse it (+ write FEC_FFEN) and remove when the DMA transfer is setup. But this seems to be not the "simple solution".

Quoted 8 lines

This is just for compatibility with what DSA _needs_ to do to drive hardware that was _designed_ to work that way, and where the _hardware_ adds that tag. But non-DSA drivers don't need any of that, nor does it really help them in any way! So why should we model things this way? I don't think the lines between plain switchdev and DSA are blurry at all. And especially not in this case. You should not have a networking interface registered to the system for DMA-0 at all, just for MAC-0 and MAC-1.

Indeed, there should be lan1, lan2 (and no eth0). Unfortunately, now FEC driver without switch enabled (on e.g. imx28) only exports eth0.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#22 lore raw
Quoted 6 lines · Lukasz Majewski
Quoted 3 lines

So why use DSA at all? What benefit does it bring you? Why not do the entire switch configuration from within FEC, or a separate driver very closely related to it?

Mine rationale to use DSA and FEC:
- Make as little changes to FEC as possible

Which is entirely possible if you stick to Vladimir suggestions of exporting services for the MTIP switch driver.

Quoted 7 lines

- Provide separate driver to allow programming FDB, MDB, VLAN setup. This seems straightforward as MTIP has separate memory region (from FEC) for switch configuration, statistics, learning, static table programming. What is even more bizarre FEC and MTIP have the same 8 registers (with different base address and +4 offset :-) ) as interface to handle DMA0 transfers.

OK, not sure how that is relevant here? The register organization should never ever dictate how to pick a particular subsystem.

Quoted 4 lines

- According to MTIP description from NXP documentation, there is a separate register for frame forwarding, so it _shall_ also fit into DSA.

And yet it does not, Vladimir went into great length into explaining what makes the MTIP + dual FEC different here and why it does not qualify for DSA. Basically any time you have DMA + integrated switch tightly coupled you have what we have coined a "pure switchdev" wrapper.

Quoted 7 lines

For me it would be enough to have:

- lan{12} - so I could enable/disable it on demand (control when switch ports are passing or not packets).

Quoted 6 lines

- Use standard net tools (like bridge) to setup FDB/MDB, vlan

- Read statistics from MTIP ports (all of them)

- I can use lan1 (bridged or not) to send data outside. It would be also correct to use eth0.

You know you can do that without having DSA, right? Look at mlxsw, look at rocker. You can call multiple times register_netdevice() with custom network devices that behave differently whether HW bridging offload is offered or not, whether the switch is declared in Device Tree or not.

Quoted 3 lines

I'm for the most pragmatic (and simple) solution, which fulfill above requirements.

The most pragmatic solution is to implement switchdev operations to offer HW bridging offload, VLAN programming, FDB/MDB programming.

It seems to me that you are trying to look for a framework to avoid doing a bit of middle layer work between switchdev and the FEC driver and that is not setting you for success.

Signature · 6 lines
Florian

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#23 lore raw

Hi Florian,

Quoted 10 lines · Lukasz Majewski
Quoted 6 lines
Quoted 3 lines

So why use DSA at all? What benefit does it bring you? Why not do the entire switch configuration from within FEC, or a separate driver very closely related to it?

Mine rationale to use DSA and FEC:
- Make as little changes to FEC as possible

Which is entirely possible if you stick to Vladimir suggestions of exporting services for the MTIP switch driver.

Ok.

Quoted 13 lines
Quoted 8 lines

- Provide separate driver to allow programming FDB, MDB, VLAN setup. This seems straightforward as MTIP has separate memory region
(from FEC) for switch configuration, statistics, learning, static table programming. What is even more bizarre FEC and MTIP have the same 8 registers (with different base address and +4 offset :-) ) as
interface to handle DMA0 transfers.

OK, not sure how that is relevant here? The register organization should never ever dictate how to pick a particular subsystem.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

- According to MTIP description from NXP documentation, there is a separate register for frame forwarding, so it _shall_ also fit
into DSA.

And yet it does not, Vladimir went into great length into explaining what makes the MTIP + dual FEC different here and why it does not qualify for DSA.

I'm very grateful for this insight and explanation from Vladimir.

Quoted 3 lines

Basically any time you have DMA + integrated switch tightly coupled you have what we have coined a "pure switchdev" wrapper.

Ok.

Quoted 12 lines
Quoted 12 lines

For me it would be enough to have:

- lan{12} - so I could enable/disable it on demand (control when switch ports are passing or not packets).

- Use standard net tools (like bridge) to setup FDB/MDB, vlan

- Read statistics from MTIP ports (all of them)

Quoted 8 lines
Quoted 2 lines

- I can use lan1 (bridged or not) to send data outside. It would be also correct to use eth0.

You know you can do that without having DSA, right? Look at mlxsw, look at rocker. You can call multiple times register_netdevice() with custom network devices that behave differently whether HW bridging offload is offered or not, whether the switch is declared in Device Tree or not.

I will look into those examples and try to follow them for MTIP.

Quoted 7 lines
Quoted 4 lines

I'm for the most pragmatic (and simple) solution, which fulfill above requirements.

The most pragmatic solution is to implement switchdev operations to offer HW bridging offload, VLAN programming, FDB/MDB programming.

Ok.

Quoted 4 lines

It seems to me that you are trying to look for a framework to avoid doing a bit of middle layer work between switchdev and the FEC driver and that is not setting you for success.

I'm not afraid to rework FEC. I just thought that DSA would be the best fit for the MTIP. However, after posting the RFC, the community gave me arguments that I was wrong.

I'm happy for having so detailed feedback :-).

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: [email protected]