net: dsa: sja1105: Make HOSTPRIO a devlink param
Subsystems (6)
Vladimir Oltean
maintainerPATCH v2 1/1 net: dsa: sja1105: Make HOSTPRIO a devlink param
Unfortunately with this hardware, there is no way to transmit in-band QoS hints with management frames (i.e. VLAN PCP is ignored). The traffic class for these is fixed in the static config (which in turn requires a reset to change).
With the new ability to add time gates for individual traffic classes, there is a real danger that the user might unknowingly turn off the traffic class for PTP, BPDUs, LLDP etc.
So we need to manage this situation the best we can. There isn't any knob in Linux for this, so create a driver-specific devlink param which is a runtime u8. The default value is 7 (the highest priority traffic class).
Patch is largely inspired by the mv88e6xxx ATU_hash devlink param implementation.
---
Changes in v2:
Turned the NET_DSA_SJA1105_HOSTPRIO kernel config into a "hostprio" runtime devlink param.
.../networking/devlink-params-sja1105.txt | 9 ++ Documentation/networking/dsa/sja1105.rst | 19 +++- MAINTAINERS | 1 + drivers/net/dsa/sja1105/sja1105.h | 1 + drivers/net/dsa/sja1105/sja1105_main.c | 94 +++++++++++++++++++ 5 files changed, 122 insertions(+), 2 deletions(-) create mode 100644 Documentation/networking/devlink-params-sja1105.txt
Documentation/networking/devlink-params-sja1105.txt new file +9 -0
@@ -0,0 +1,9 @@ +hostprio [DEVICE, DRIVER-SPECIFIC] + Configure the traffic class which will be used for + management (link-local) traffic injected and trapped + to/from the CPU. This includes STP, PTP, LLDP etc, as + well as hardware-specific meta frames with RX + timestamps. Higher is better as long as you care about + your PTP frames. + Configuration mode: runtime + Type: u8. 0-7 valid.
Documentation/networking/dsa/sja1105.rst +17 -2
@@ -181,8 +181,23 @@ towards the switch, with the VLAN PCP bits set appropriately. Management traffic (having DMAC 01-80-C2-xx-xx-xx or 01-19-1B-xx-xx-xx) is the notable exception: the switch always treats it with a fixed priority and disregards any VLAN PCP bits even if present. The traffic class for management -traffic has a value of 7 (highest priority) at the moment, which is not -configurable in the driver. +traffic is configurable through a driver-specific devlink param called +``hostprio``, which by default has a value of 7 (highest priority):: + + devlink dev param show + spi/spi0.1: + name hostprio type driver-specific + values: + cmode runtime value 7 + + devlink dev param set spi/spi0.1 name hostprio value 5 cmode runtime + [ 389.903342] sja1105 spi0.1: Reset switch and programmed static config. Reason: Link-local traffic class + + devlink dev param show + spi/spi0.1: + name hostprio type driver-specific + values: + cmode runtime value 5 Below is an example of configuring a 500 us cyclic schedule on egress port ``swp5``. The traffic class gate for management traffic (7) is open for 100 us,
MAINTAINERS +1 -0
@@ -11688,6 +11688,7 @@ M: Vladimir Oltean <[email protected]> L: [email protected] S: Maintained F: drivers/net/dsa/sja1105 +F: Documentation/networking/devlink-params-sja1105.txt NXP TDA998X DRM DRIVER M: Russell King <[email protected]>
drivers/net/dsa/sja1105/sja1105.h +1 -0
@@ -122,6 +122,7 @@ enum sja1105_reset_reason {SJA1105_RX_HWTSTAMPING ,SJA1105_AGEING_TIME ,SJA1105_SCHEDULING , + SJA1105_HOSTPRIO, }; int sja1105_static_config_reload(struct sja1105_private *priv,
drivers/net/dsa/sja1105/sja1105_main.c +94 -0
124 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 37 lines · Vladimir Oltean
Unfortunately with this hardware, there is no way to transmit in-band QoS hints with management frames (i.e. VLAN PCP is ignored). The traffic class for these is fixed in the static config (which in turn requires a reset to change).
With the new ability to add time gates for individual traffic classes, there is a real danger that the user might unknowingly turn off the traffic class for PTP, BPDUs, LLDP etc.
So we need to manage this situation the best we can. There isn't any knob in Linux for this, so create a driver-specific devlink param which is a runtime u8. The default value is 7 (the highest priority traffic class).
Patch is largely inspired by the mv88e6xxx
ATU_hashdevlink param implementation.Signed-off-by Vladimir Oltean <[email protected]>---
Changes in v2:
Turned theNET_DSA_SJA1105_HOSTPRIOkernel config into a "hostprio" runtime devlink param..../networking/devlink-params-sja1105.txt | 9 ++ Documentation/networking/dsa/sja1105.rst | 19 +++- MAINTAINERS | 1 + drivers/net/dsa/sja1105/sja1105.h | 1 + drivers/net/dsa/sja1105/sja1105_main.c | 94 +++++++++++++++++++ 5 files changed, 122 insertions(+), 2 deletions(-) create mode 100644 Documentation/networking/devlink-params-sja1105.txtDocumentation/networking/devlink-params-sja1105.txt new file +0
-0
Documentation/networking/devlink-params-sja1105.txt line 7
Quoted 7 lines
Documentation/networking/devlink-params-sja1105.txt new file +7
-0@@ -0,0 +1,9 @@ +hostprio [DEVICE, DRIVER-SPECIFIC] + Configure the traffic class which will be used for + management (link-local) traffic injected and trapped + to/from the CPU. This includes STP, PTP, LLDP etc, as + well as hardware-specific meta frames with RX + timestamps. Higher is better as long as you care about + your PTP frames.
Vladimir,
I have some concerns about this. Firstly, I'm not sure why you need to expose this and who do you expect to be able to configure this? I'm asking because once you expose it to users there might not be a way back. mlxsw is upstream for over four years and the traffic classes for the different packet types towards the CPU are hard coded in the driver and based on "sane" defaults. It is therefore surprising to me that you already see the need to expose this.
Secondly, I find the name confusing. You call it "hostprio", but the description says "traffic class". These are two different things. Priority is a packet attribute based on which you can classify the packet to a transmission queue (traffic class). And if you have multiple transmission queues towards the CPU, how do you configure their scheduling and AQM? This relates to my next point.
Thirdly, the fact that "there isn't any knob in Linux for this" does not mean that we should not create a proper one. As I see it, the CPU port (switch side, not DSA master) is a port like any other and therefore should have a netdev. With a netdev you can properly configure its different QoS attributes using tc instead of introducing driver-specific devlink-params.
Yes, I understand that this is a large task compared to just adding this devlink-param, but adding a new user interface always scares me and I think we should do the right thing here.
Thanks
Vladimir Oltean
maintainerHi Ido,
Quoted 49 lines · Ido Schimmel
Quoted 37 lines · Vladimir Oltean
Unfortunately with this hardware, there is no way to transmit in-band QoS hints with management frames (i.e. VLAN PCP is ignored). The traffic class for these is fixed in the static config (which in turn requires a reset to change).
With the new ability to add time gates for individual traffic classes, there is a real danger that the user might unknowingly turn off the traffic class for PTP, BPDUs, LLDP etc.
So we need to manage this situation the best we can. There isn't any knob in Linux for this, so create a driver-specific devlink param which is a runtime u8. The default value is 7 (the highest priority traffic class).
Patch is largely inspired by the mv88e6xxx
ATU_hashdevlink param implementation.Signed-off-by Vladimir Oltean <[email protected]>---
Changes in v2:
Turned theNET_DSA_SJA1105_HOSTPRIOkernel config into a "hostprio" runtime devlink param..../networking/devlink-params-sja1105.txt | 9 ++ Documentation/networking/dsa/sja1105.rst | 19 +++- MAINTAINERS | 1 + drivers/net/dsa/sja1105/sja1105.h | 1 + drivers/net/dsa/sja1105/sja1105_main.c | 94 +++++++++++++++++++ 5 files changed, 122 insertions(+), 2 deletions(-) create mode 100644 Documentation/networking/devlink-params-sja1105.txtDocumentation/networking/devlink-params-sja1105.txt new file +0
-0Documentation/networking/devlink-params-sja1105.txt line 7
Quoted 7 lines
Documentation/networking/devlink-params-sja1105.txt new file +7
-0@@ -0,0 +1,9 @@ +hostprio [DEVICE, DRIVER-SPECIFIC] + Configure the traffic class which will be used for + management (link-local) traffic injected and trapped + to/from the CPU. This includes STP, PTP, LLDP etc, as + well as hardware-specific meta frames with RX + timestamps. Higher is better as long as you care about + your PTP frames.Vladimir,
Quoted 8 lines
I have some concerns about this. Firstly, I'm not sure why you need to expose this and who do you expect to be able to configure this? I'm asking because once you expose it to users there might not be a way back. mlxsw is upstream for over four years and the traffic classes for the different packet types towards the CPU are hard coded in the driver and based on "sane" defaults. It is therefore surprising to me that you already see the need to expose this.
WIth tc-taprio, it is up to the user / system administrator which of the 8 traffic classes has access to the Ethernet media and when (if ever).
For example, take this command:
sudo tc qdisc replace dev swp3 parent root handle 100 taprio \
num_tc 8 \
map 0 1 2 3 4 5 6 7 \
queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 \
base-time $base_time \
sched-entry S 03 300000 \
sched-entry S 02 300000 \
sched-entry S 06 400000 \
flags 2It will allow access to traffic classes 0 and 1 (S 03) for 300000 ns, traffic class 1 (S 02) for another 300000 ns, and to traffic classes 1 and 2 (S 06) for 400000 ns, then repeat the schedule. This is just an example.
But the base-time is a PTP time, so the switch needs a PTP daemon to keep track of time. But notice that the traffic class for PTP traffic (which is hardcoded as 7 in the driver) is not present in the tc-taprio schedule, so effectively PTP will be denied access to the media, so it won't work, so it will completely break the schedule. The user will be horribly confused by this. I also don't want to keep the PTP traffic class hardcoded at 7, since the system administrator might have other cyclic traffic schedules that are more time-sensitive than PTP itself.
Quoted 7 lines
Secondly, I find the name confusing. You call it "hostprio", but the description says "traffic class". These are two different things. Priority is a packet attribute based on which you can classify the packet to a transmission queue (traffic class). And if you have multiple transmission queues towards the CPU, how do you configure their scheduling and AQM? This relates to my next point.
HOSTPRIO is the way it is called in the hardware user guide:
https://www.nxp.com/docs/en/user-guide/UM10944.pdf
I don't see why I should call it something else, especially since it does not map over any other concept.
And since it is DSA, the Ethernet link between the CPU port and the DSA master is inherently single queue, even though the switch is multi-queue and the DSA master is multi-queue. Both the DSA master and the front-panel switch can have independent qdiscs that operate independently. It is a separate discussion.
Quoted 7 lines
Thirdly, the fact that "there isn't any knob in Linux for this" does not mean that we should not create a proper one. As I see it, the CPU port (switch side, not DSA master) is a port like any other and therefore should have a netdev. With a netdev you can properly configure its different QoS attributes using tc instead of introducing driver-specific devlink-params.
Right, but let me summarize the hardware operation to understand what this is really doing.
The switch has 2 MAC filters for link-local management traffic. I hardcoded them in the driver to 01-80-C2-xx-xx-xx and 01-1B-19-xx-xx-xx so that STP and PTP work by default. The switch checks the DMAC of frames against these masks very early in the packet processing pipeline, and if they match, they are trapped to the CPU. In fact, the match is so early that the analyzer module is bypassed and the frames do not get classified to a TC based on any QoS classification rules. The hardware designers recognized that this might be a problem, so they just invented a knob called HOSTPRIO, which all frames that are trapped to the CPU get assigned. On xmit, the MAC filters are active on the CPU port as well. So the switch wants to trap the link-local frames coming from the CPU and redirect them to the CPU, which it won't do because it's configured to avoid hairpinning. So it drops those frames when they come from the CPU port, due to lack of valid destinations. So the hardware designers invented another concept called "management routes" which are meant to bypass the MAC filters (which themselves bypass L2 forwarding). You pre-program a one-shot "management route" in the switch for a frame matching a certain DMAC, then you send it, then the switch figures out it matches this "management route" and properly sends it out the correct front-panel port. The point is that on xmit, the switch uses HOSTPRIO for the "management route" frames as well.
Whether the CPU port is a netdev or not does not change the fact that there is no way to "properly" model this using tc. In hardware, all frames that match the management routes or the MAC filtering rules get the fixed and global HOSTPRIO, end of story.
Quoted 4 lines
Yes, I understand that this is a large task compared to just adding this devlink-param, but adding a new user interface always scares me and I think we should do the right thing here.
I don't want to invent a wheel that has any pretense of being generic, when in reality it isn't generic at all. For traffic that the switch really classifies based on per-frame rules, I agree with you, but that's not the case here.
Quoted 1 line
Thanks
Thanks,
-Vladimir
Quoted 76 lines · Vladimir Oltean
Hi Ido,
Quoted 49 lines · Ido Schimmel
Quoted 37 lines · Vladimir Oltean
Unfortunately with this hardware, there is no way to transmit in-band QoS hints with management frames (i.e. VLAN PCP is ignored). The traffic class for these is fixed in the static config (which in turn requires a reset to change).
With the new ability to add time gates for individual traffic classes, there is a real danger that the user might unknowingly turn off the traffic class for PTP, BPDUs, LLDP etc.
So we need to manage this situation the best we can. There isn't any knob in Linux for this, so create a driver-specific devlink param which is a runtime u8. The default value is 7 (the highest priority traffic class).
Patch is largely inspired by the mv88e6xxx
ATU_hashdevlink param implementation.Signed-off-by Vladimir Oltean <[email protected]>---
Changes in v2:
Turned theNET_DSA_SJA1105_HOSTPRIOkernel config into a "hostprio" runtime devlink param..../networking/devlink-params-sja1105.txt | 9 ++ Documentation/networking/dsa/sja1105.rst | 19 +++- MAINTAINERS | 1 + drivers/net/dsa/sja1105/sja1105.h | 1 + drivers/net/dsa/sja1105/sja1105_main.c | 94 +++++++++++++++++++ 5 files changed, 122 insertions(+), 2 deletions(-) create mode 100644 Documentation/networking/devlink-params-sja1105.txtDocumentation/networking/devlink-params-sja1105.txt new file +0
-0Documentation/networking/devlink-params-sja1105.txt line 7
Quoted 7 lines
Documentation/networking/devlink-params-sja1105.txt new file +7
-0@@ -0,0 +1,9 @@ +hostprio [DEVICE, DRIVER-SPECIFIC] + Configure the traffic class which will be used for + management (link-local) traffic injected and trapped + to/from the CPU. This includes STP, PTP, LLDP etc, as + well as hardware-specific meta frames with RX + timestamps. Higher is better as long as you care about + your PTP frames.Vladimir,
Quoted 8 lines
I have some concerns about this. Firstly, I'm not sure why you need to expose this and who do you expect to be able to configure this? I'm asking because once you expose it to users there might not be a way back. mlxsw is upstream for over four years and the traffic classes for the different packet types towards the CPU are hard coded in the driver and based on "sane" defaults. It is therefore surprising to me that you already see the need to expose this.
WIth tc-taprio, it is up to the user / system administrator which of the 8 traffic classes has access to the Ethernet media and when (if ever).
For example, take this command:sudo tc qdisc replace dev swp3 parent root handle 100 taprio \
num_tc 8 \ map 0 1 2 3 4 5 6 7 \ queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 \ base-time $base_time \ sched-entry S 03 300000 \ sched-entry S 02 300000 \ sched-entry S 06 400000 \ flags 2
Quoted 10 lines
It will allow access to traffic classes 0 and 1 (S 03) for 300000 ns, traffic class 1 (S 02) for another 300000 ns, and to traffic classes 1 and 2 (S 06) for 400000 ns, then repeat the schedule. This is just an example.
But the base-time is a PTP time, so the switch needs a PTP daemon to keep track of time. But notice that the traffic class for PTP traffic (which is hardcoded as 7 in the driver) is not present in the tc-taprio schedule, so effectively PTP will be denied access to the media, so it won't work, so it will completely break the schedule. The user will be horribly confused by this.
OK, but I don't see how adding a new knob helps to avoid the confusion? Maybe add an extack here saying that HOSTPRIO traffic class has no access to the media. Alternatively, you can veto such configurations if it's acceptable (in which case you don't need the parameter).
Quoted 12 lines
I also don't want to keep the PTP traffic class hardcoded at 7, since the system administrator might have other cyclic traffic schedules that are more time-sensitive than PTP itself.
Quoted 7 lines
Secondly, I find the name confusing. You call it "hostprio", but the description says "traffic class". These are two different things. Priority is a packet attribute based on which you can classify the packet to a transmission queue (traffic class). And if you have multiple transmission queues towards the CPU, how do you configure their scheduling and AQM? This relates to my next point.
Quoted 4 lines
HOSTPRIO is the way it is called in the hardware user guide:
https://www.nxp.com/docs/en/user-guide/UM10944.pdf
I don't see why I should call it something else, especially since it does not map over any other concept.
It would be good to add the explanation you provided in this response to the documentation. It is not clear where the name comes from otherwise.
Quoted 30 lines
And since it is DSA, the Ethernet link between the CPU port and the DSA master is inherently single queue, even though the switch is multi-queue and the DSA master is multi-queue. Both the DSA master and the front-panel switch can have independent qdiscs that operate independently. It is a separate discussion.
Quoted 7 lines
Thirdly, the fact that "there isn't any knob in Linux for this" does not mean that we should not create a proper one. As I see it, the CPU port (switch side, not DSA master) is a port like any other and therefore should have a netdev. With a netdev you can properly configure its different QoS attributes using tc instead of introducing driver-specific devlink-params.
Right, but let me summarize the hardware operation to understand what this is really doing.
The switch has 2 MAC filters for link-local management traffic. I hardcoded them in the driver to 01-80-C2-xx-xx-xx and 01-1B-19-xx-xx-xx so that STP and PTP work by default. The switch checks the DMAC of frames against these masks very early in the packet processing pipeline, and if they match, they are trapped to the CPU. In fact, the match is so early that the analyzer module is bypassed and the frames do not get classified to a TC based on any QoS classification rules. The hardware designers recognized that this might be a problem, so they just invented a knob called HOSTPRIO, which all frames that are trapped to the CPU get assigned. On xmit, the MAC filters are active on the CPU port as well. So the switch wants to trap the link-local frames coming from the CPU and redirect them to the CPU, which it won't do because it's configured to avoid hairpinning. So it drops those frames when they come from the
Quoted 8 lines
CPU port, due to lack of valid destinations. So the hardware designers invented another concept called "management routes" which are meant to bypass the MAC filters (which themselves bypass L2 forwarding). You pre-program a one-shot "management route" in the switch for a frame matching a certain DMAC, then you send it, then the switch figures out it matches this "management route" and properly sends it out the correct front-panel port. The point is that on xmit, the switch uses HOSTPRIO for the "management route" frames as well.
Very weird model :( I understand your need for this knob now, but I suggest to add an extack in qdisc offload to make it clear to users that they are potentially shooting themselves in the foot.