Skip to contentSkip to conversation

RFC: dsa: add support for multiple CPU ports

DISCUSSION Mathieu Olivari 5 participants 16 messages Mar 10, 2015 – Mar 13, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

I’m writing a DSA driver for some QCA network switches already supported in OpenWrt using swconfig. They connect to the CPU using MDIO for configuration, and xMII ports for data. The main difference with what is supported comes from the fact that most of these switches actually have multiple xMII connections to the same CPU. Something like this:
(extending the picture from http://lwn.net/Articles/302333/)

	+-----------+       +-----------+
	|           | RGMII |           |
	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
	|        lan|       |           |
	+-----------+       +-----------+
	          |   MDIO     |
	          \------------/

In a typical configuration, we configure the switch to isolate WAN & LAN from each other.

The current DSA implementation is based on the assumption that there can be only one xMII connection between the switch and the CPU (this switch port is known as the CPU port), which obviously doesn't work anymore in that case. So I'm trying to figure out the best way to extend DSA to support these hardware.

Here is my proposal. I was hoping to get feedback before i get too much down the implementation path:
*add an optional phandle (cpu_dev) to the existing port device node, which would
reference the ethernet device node this port is connected to
*save the corresponding net_device in the "ports" field of "struct dsa_switch" *instrument the network layer to be able to look-up which switch port is
connected to a particular net_device.

The third point would allow to perform operations currently limited to switch ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB statistics in addition to the eth0 statistics.

Thoughts?

Thanks,
Mathieu

#1 lore raw
Quoted 21 lines · Mathieu Olivari

Hi all,

I???m writing a DSA driver for some QCA network switches already supported in OpenWrt using swconfig. They connect to the CPU using MDIO for configuration, and xMII ports for data. The main difference with what is supported comes from the fact that most of these switches actually have multiple xMII connections to the same CPU. Something like this:
(extending the picture from http://lwn.net/Articles/302333/)

	+-----------+       +-----------+
	|           | RGMII |           |
	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
	|        lan|       |           |
	+-----------+       +-----------+
	          |   MDIO     |
	          \------------/
Quoted 2 lines

In a typical configuration, we configure the switch to isolate WAN & LAN from each other.

Hi Mathieu

By default, all DSA ports are isolated from each other. If you want to join them together you need to setup a bridge and add the ports to the bridge. There are patches being worked on to push this bridge state down into the hardware, so the hardware will then bridge across these ports, rather than having to do it in software. So long as you don't add WAN to the bridge, it will be kept isolated.

I had a different solution in mind for multiple CPU ports. I've no idea if it actually works though, i've not had time to investigate. It would actually put the host CPU ports into a switch trunk, and use team/bond driver on the host. You then get one logical 2Gbp link to the switch and run DSA over that.

There have also been some patches to create trunks, but they were for normal ports, not CPU ports. They should however be a good starting point for what the switch driver needs to do to create a trunk towards the CPU.

I think this scheme might also work without having to change the DSA binding. There is nothing in the binding documentation that there can only be one CPU port. So if two or more are found, the DSA framework can do the trunking setup.

Andrew

#4 lore raw
In reply to Mathieu Olivari
Quoted 3 lines

The third point would allow to perform operations currently limited to switch ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB statistics in addition to the eth0 statistics.

Hi Mathieu

This already works with DSA. I have a WRT1900AC which has a 7 port switch. With DSA i have:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
    link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff

internet would be your WAN port in your diagram. I just named it the same as the label on the case.

I can get MIB statistics in the normal way for the DSA ports, e.g:

root@wrt1900ac:~# ethtool -S lan1
NIC statistics:
     tx_packets: 153009
     tx_bytes: 13307317
     rx_packets: 161136
     rx_bytes: 21840683
...

Andrew

#5 lore raw
Quoted 40 lines · Andrew Lunn
Quoted 3 lines

The third point would allow to perform operations currently limited to switch ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB statistics in addition to the eth0 statistics.

Hi Mathieu

This already works with DSA. I have a WRT1900AC which has a 7 port switch. With DSA i have:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
    link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff

internet would be your WAN port in your diagram. I just named it the same as the label on the case.

I can get MIB statistics in the normal way for the DSA ports, e.g:

root@wrt1900ac:~# ethtool -S lan1
NIC statistics:
     tx_packets: 153009
     tx_bytes: 13307317
     rx_packets: 161136
     rx_bytes: 21840683
...
Quoted 3 lines

Andrew

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

#6 lore raw
Quoted 17 lines

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/
Quoted 3 lines

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

Ah, sorry, i misunderstood you. As you say, the current DSA does not give you access to these counters. Apart from these counters, are there any other needs to have access to these ports? The core idea of hardware acceleration is that it should look like linux as normal. P0 and P6 are an internal detail, and you cannot actually use them. You cannot send packets, they are something special. So unless there is a good reason, we should not expose them.

Quoted 3 lines

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

But that is not what the user expects. They expect the statistic counters for ethN, not the switch port.

Andrew

#2 lore raw
In reply to Andrew Lunn
Quoted 34 lines · Andrew Lunn
Quoted 21 lines · Mathieu Olivari

Hi all,

I???m writing a DSA driver for some QCA network switches already supported in OpenWrt using swconfig. They connect to the CPU using MDIO for configuration, and xMII ports for data. The main difference with what is supported comes from the fact that most of these switches actually have multiple xMII connections to the same CPU. Something like this:
(extending the picture from http://lwn.net/Articles/302333/)

	+-----------+       +-----------+
	|           | RGMII |           |
	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
	|        lan|       |           |
	+-----------+       +-----------+
	          |   MDIO     |
	          \------------/
Quoted 2 lines

In a typical configuration, we configure the switch to isolate WAN & LAN from each other.

Hi Mathieu

By default, all DSA ports are isolated from each other. If you want to join them together you need to setup a bridge and add the ports to the bridge. There are patches being worked on to push this bridge state down into the hardware, so the hardware will then bridge across these ports, rather than having to do it in software. So long as you don't add WAN to the bridge, it will be kept isolated.

Quoted 6 lines

I had a different solution in mind for multiple CPU ports. I've no idea if it actually works though, i've not had time to investigate. It would actually put the host CPU ports into a switch trunk, and use team/bond driver on the host. You then get one logical 2Gbp link to the switch and run DSA over that.

I could see it working on the Tx path - as the destination port is specified in the header -, but on the Rx path, how would the switch figure out which CPU port it should send the packet to?

These switches doesn't have a concept of bonding, so this decision is generally based on the internal ARL table, and is automatically learnt by looking at the src MAC@ of the incoming packets.
When using bonding, the switch would see both eth0 & eth1 MAC@ on both of its CPU ports. The destination CPU port would be unexpected at best; I could see some switches being able to support this, but most of them would not.

Thoughts?

Quoted 10 lines

There have also been some patches to create trunks, but they were for normal ports, not CPU ports. They should however be a good starting point for what the switch driver needs to do to create a trunk towards the CPU.

I think this scheme might also work without having to change the DSA binding. There is nothing in the binding documentation that there can only be one CPU port. So if two or more are found, the DSA framework can do the trunking setup.

At the very least, we would need to treat "dsa,ethernet" as an array, and specify the list of ethernet device node that connects to the switch. I still think putting this information in the port section makes sense, as it represents the board layout more accurately than having a global node at a dsa level.

Quoted 1 line

Andrew

#7 lore raw
In reply to Andrew Lunn
Quoted 21 lines · Andrew Lunn
Quoted 17 lines

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/
Quoted 3 lines

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

Quoted 8 lines

Ah, sorry, i misunderstood you. As you say, the current DSA does not give you access to these counters. Apart from these counters, are there any other needs to have access to these ports? The core idea of hardware acceleration is that it should look like linux as normal. P0 and P6 are an internal detail, and you cannot actually use them. You cannot send packets, they are something special. So unless there is a good reason, we should not expose them.

That is true, but the switch does have information to share about this port (stats). Having a standard way to retrieve this information would be nice.

Quoted 6 lines
Quoted 3 lines

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

But that is not what the user expects. They expect the statistic counters for ethN, not the switch port.

The user would probably want to access both. It should not replace ethN counters obviously, but these are real switch port (just without any PHY), so the user would want to have the same level of information as it has on the other logical switch ports.

I was wondering whether it made sense to show counters for both ethN and P0 when running "ethtool -S ethN". Or maybe add a new argument to ethtool. Else, there is still debugfs but it means the same counters will be accessible through ethtool (for PHY ports) and debugfs (for MAC ports), which is not very consistent.

Quoted 2 lines

Andrew

#3 lore raw
In reply to Mathieu Olivari
Quoted 7 lines
Quoted 6 lines

I had a different solution in mind for multiple CPU ports. I've no idea if it actually works though, i've not had time to investigate. It would actually put the host CPU ports into a switch trunk, and use team/bond driver on the host. You then get one logical 2Gbp link to the switch and run DSA over that.

Quoted 7 lines

I could see it working on the Tx path - as the destination port is specified in the header -, but on the Rx path, how would the switch figure out which CPU port it should send the packet to?

These switches doesn't have a concept of bonding, so this decision is generally based on the internal ARL table, and is automatically learnt by looking at the src MAC@ of the incoming packets.

All Marvell switches, and according to Florian SF2 and B53 support trunk groups. I assume the switch does load balancing between ports in a trunk. On the CPU you can then use the header to determine which port the packet came from.

Quoted 3 lines

When using bonding, the switch would see both eth0 & eth1 MAC@ on both of its CPU ports. The destination CPU port would be unexpected at best; I could see some switches being able to support this, but most of them would not.

At the moment, this is purely an idea. I do have two boards where i can test it out with Marvell devices, once i get some spare time. Florian might be able to comment about SF2 and B53, if we thinks trunking and broadcom tags can be combined.

Quoted 2 lines

At the very least, we would need to treat "dsa,ethernet" as an array, and specify the list of ethernet device node that connects to the switch.

True.

Quoted 3 lines

I still think putting this information in the port section makes sense, as it represents the board layout more accurately than having a global node at a dsa level.

Yes, it does make sense to have it in port. But to keep backwards compatibility, we need at keep dsa,ethernet, with at least one interface.

Andrew

#10 lore raw
In reply to Mathieu Olivari

2015-03-10 13:42 GMT-07:00 Mathieu Olivari <[email protected]>:

Quoted 62 lines · Andrew Lunn
Quoted 40 lines
Quoted 3 lines

The third point would allow to perform operations currently limited to switch ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB statistics in addition to the eth0 statistics.

Hi Mathieu

This already works with DSA. I have a WRT1900AC which has a 7 port switch. With DSA i have:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
    link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff

internet would be your WAN port in your diagram. I just named it the same as the label on the case.

I can get MIB statistics in the normal way for the DSA ports, e.g:

root@wrt1900ac:~# ethtool -S lan1
NIC statistics:
     tx_packets: 153009
     tx_bytes: 13307317
     rx_packets: 161136
     rx_bytes: 21840683
...
Quoted 3 lines

Andrew

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/
Quoted 3 lines

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

That's right, we actually discussed that earlier during the swconfig discussions, and one of the reasons why this is not exposed is because it can be confusing for users, and the difference between your conduit
interfaces: eth0 or eth1, and their switch-facing interfaces: p0 or p6, is that p0 and p6 are only useful for counters, but not for much else, in particular not data passing.

Quoted 4 lines

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

Once we have that, interfaces p0 and p6 are not going to be sending/receiving traffic, their conduit interfaces: eth0 and eth1, will, and those could also maintain statistics in their own MIB (when using unmodified Ethernet MAC drivers). I agree this is useful in xMII cases to debug interface problems between a switch and a CPU Ethernet MAC for instance.

Signature · 1 line
Florian
#11 lore raw
Quoted 67 lines · Florian Fainelli

2015-03-10 13:42 GMT-07:00 Mathieu Olivari <[email protected]>:

Quoted 62 lines · Andrew Lunn
Quoted 40 lines
Quoted 3 lines

The third point would allow to perform operations currently limited to switch ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB statistics in addition to the eth0 statistics.

Hi Mathieu

This already works with DSA. I have a WRT1900AC which has a 7 port switch. With DSA i have:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
    link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff

internet would be your WAN port in your diagram. I just named it the same as the label on the case.

I can get MIB statistics in the normal way for the DSA ports, e.g:

root@wrt1900ac:~# ethtool -S lan1
NIC statistics:
     tx_packets: 153009
     tx_bytes: 13307317
     rx_packets: 161136
     rx_bytes: 21840683
...
Quoted 3 lines

Andrew

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/
Quoted 3 lines

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

Quoted 7 lines

That's right, we actually discussed that earlier during the swconfig discussions, and one of the reasons why this is not exposed is because it can be confusing for users, and the difference between your conduit
interfaces: eth0 or eth1, and their switch-facing interfaces: p0 or p6, is that p0 and p6 are only useful for counters, but not for much else, in particular not data passing.

When there is only one CPU port, the connection between CPU port and the other switch ports is implicit obviously. But if there is more, and if the switch doesn't support bonding, the user would probably expect to configure it & define how P0/P6 connects to the other ports.

We already have existing interface for eth0/eth1. Maybe we should consider allowing them to be bridge. User configuration would look like this:

brctl addbr br-lan
brctl addif br-lan eth1
brctl addif br-lan lan1
brctl addif br-lan lan2
brctl addif br-lan lan3
brctl addif br-lan lan4

Quoted 13 lines
Quoted 4 lines

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

Once we have that, interfaces p0 and p6 are not going to be sending/receiving traffic, their conduit interfaces: eth0 and eth1, will, and those could also maintain statistics in their own MIB (when using unmodified Ethernet MAC drivers). I agree this is useful in xMII cases to debug interface problems between a switch and a CPU Ethernet MAC for instance.

Signature · 1 line
Florian
maintainer
#8 lore raw
In reply to Mathieu Olivari

From: Mathieu Olivari <[email protected]>
Date: Tue, 10 Mar 2015 15:53:26 -0700

Quoted 5 lines

I was wondering whether it made sense to show counters for both ethN and P0 when running "ethtool -S ethN". Or maybe add a new argument to ethtool. Else, there is still debugfs but it means the same counters will be accessible through ethtool (for PHY ports) and debugfs (for MAC ports), which is not very consistent.

debugfs is only for things that are throw away and upon which userspace cannot depend.

#9 lore raw
In reply to Andrew Lunn
Quoted 21 lines · Tue, Mar 10, 2015 at 10:13:15PM CET, [email protected] wrote:
Quoted 17 lines

I was thinking about the MIB counters, maintained by the switch, on the CPU port(s).

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/
Quoted 3 lines

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU ports don't have a net_device, there isn't a way to access MIB stats of P0 & P6.

Quoted 7 lines

Ah, sorry, i misunderstood you. As you say, the current DSA does not give you access to these counters. Apart from these counters, are there any other needs to have access to these ports? The core idea of hardware acceleration is that it should look like linux as normal. P0 and P6 are an internal detail, and you cannot actually use them. You cannot send packets, they are something special. So unless there is a good reason, we should not expose them.

I completely agree. Let's keep this internal ports scop within the driver/dsa code.

Quoted 13 lines
Quoted 4 lines

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts, we could use it to lookup which switch port is connected to this net_dev and show the corresponding counters when running "ethtool -S ethN".

But that is not what the user expects. They expect the statistic counters for ethN, not the switch port.

Andrew
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
In reply to Mathieu Olivari
Quoted 10 lines

We already have existing interface for eth0/eth1. Maybe we should consider allowing them to be bridge. User configuration would look like this:

brctl addbr br-lan
brctl addif br-lan eth1
brctl addif br-lan lan1
brctl addif br-lan lan2
brctl addif br-lan lan3
brctl addif br-lan lan4

Think about this from the perspective of somebody how does not know there is a switch. When i look at this, to me it means packets from lan1, lan2, lan3, lan4 and eth1 are bridged together. So we are going to get packets sent out eth1 without a tag on it, and the switch is very likely to drop it, since the port is in a mode which expects a tag. You are not using the brctl command with its normal meaning, which is bad.

Configuring this at the bridge layer is also wrong. What conduit a DSA slave interfaces uses is a DSA layer issue.

Before we can come up with a nice solution, i think we first need to understand what the switches are capable off. If trunking does work, it is a relatively nice system for the user to configure, in that there is nothing for the user to configure! Assuming the switch can do reasonable load balancing, we also get near optimal usage of the two conduits.

I don't know when i will have time to play with this, and if somebody comes along with a better idea, i'm very open to adopting that idea over mine.

Andrew

#13 lore raw
Quoted 22 lines · Andrew Lunn
Quoted 10 lines

We already have existing interface for eth0/eth1. Maybe we should consider allowing them to be bridge. User configuration would look like this:

brctl addbr br-lan
brctl addif br-lan eth1
brctl addif br-lan lan1
brctl addif br-lan lan2
brctl addif br-lan lan3
brctl addif br-lan lan4

Think about this from the perspective of somebody how does not know there is a switch. When i look at this, to me it means packets from lan1, lan2, lan3, lan4 and eth1 are bridged together. So we are going to get packets sent out eth1 without a tag on it, and the switch is very likely to drop it, since the port is in a mode which expects a tag. You are not using the brctl command with its normal meaning, which is bad.

Configuring this at the bridge layer is also wrong. What conduit a DSA slave interfaces uses is a DSA layer issue.

Quoted 7 lines

Before we can come up with a nice solution, i think we first need to understand what the switches are capable off. If trunking does work, it is a relatively nice system for the user to configure, in that there is nothing for the user to configure! Assuming the switch can do reasonable load balancing, we also get near optimal usage of the two conduits.

Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support a mode which would allow them to use all the CPU ports as one logical bonding interface. I'm not sure how many of the switches in OpenWrt do.

I know some of these switches don't even support tagging. So I'd be surprised if they're all capable of bonding.

The base usage scenario for these switches is using port based vlan. The user knows very well there is a switch (OpenWrt even has a webpage for it), and he provides a static configuration by telling it how he want the ports to be connected to each other.
In the previous example, it would be something like creating 2 logical
switches: P0/P1 and P2/P3/P4/P5/P6. Or it could be something like (lan1 - P0/P1/P2) and (lan2 - P3/P4) and (lan3 - P5/P6). Or anything else. That's the mode that OpenWrt is using in pretty much all its target. It is also the way they get configured in most home routers in general.

I understand the value of using bonding. But we should also have a way to configure more simple hardware - which is typically using port based vlan.

Quoted 3 lines

I don't know when i will have time to play with this, and if somebody comes along with a better idea, i'm very open to adopting that idea over mine.

I think DSA should probably support both. I think it should allow bonding for switches who support it, but it should also allow for port based vlan mapped by the user.

Another way to support these could be to make the cpu port "optional". These switches would have no cpu port, all the ports would have a net_device (including the xMII ones). They would get bridged, and packets that are sent to interface connected to cpu would be dropped by the switch xmit function.
I didn't try that, but it should work.

Quoted 2 lines

Andrew

#14 lore raw
Quoted 3 lines

Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support a mode which would allow them to use all the CPU ports as one logical bonding interface.

I had a 60 second look at https://www.codeaurora.org/cgit/quic/qsdk/oss/lklm/qca-ssdk/

It looks like dess, isis, and isisc have some support for trunks, and the rest do not.

Quoted 2 lines

I know some of these switches don't even support tagging. So I'd be surprised if they're all capable of bonding.

So we are going to end up with multiple solutions depending on what the hardware is capable of.

Quoted 3 lines

I think DSA should probably support both. I think it should allow bonding for switches who support it, but it should also allow for port based vlan mapped by the user.

So the question is, how do you make the existing concepts in Linux support this, so configuration is how you expect it to be, without special cases, oddities, etc.

Andrew

#15 lore raw
Quoted 9 lines · Andrew Lunn
Quoted 3 lines

Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support a mode which would allow them to use all the CPU ports as one logical bonding interface.

I had a 60 second look at https://www.codeaurora.org/cgit/quic/qsdk/oss/lklm/qca-ssdk/

It looks like dess, isis, and isisc have some support for trunks, and the rest do not.

Yes, they do. But not on the CPU port, unfortunately.

Quoted 6 lines
Quoted 2 lines

I know some of these switches don't even support tagging. So I'd be surprised if they're all capable of bonding.

So we are going to end up with multiple solutions depending on what the hardware is capable of.

Quoted 8 lines
Quoted 3 lines

I think DSA should probably support both. I think it should allow bonding for switches who support it, but it should also allow for port based vlan mapped by the user.

So the question is, how do you make the existing concepts in Linux support this, so configuration is how you expect it to be, without special cases, oddities, etc.

I'll start by ignoring the second CPU port for now and see how far I go.

Quoted 1 line

Andrew