tc-taprio offload for SJA1105 DSA
Subsystems (7)
Vladimir Oltean
maintainerv4 cover tc-taprio offload for SJA1105 DSA
This is the third attempt to submit the tc-taprio offload model for inclusion in the networking tree. The sja1105 switch driver will provide the first implementation of the offload. Only the bare minimum is added:
- The offload model and a DSA pass-through
- The hardware implementation
- The interaction with the netdev queues in the tagger code
- Documentation
What has been removed from previous attempts is support for PTP-as-clocksource in sja1105, as well as configuring the traffic class for management traffic. These will be added as soon as the offload model is settled.
Vinicius Costa Gomes (1):
taprio: Add support for hardware offloading
Vladimir Oltean (5):
net: dsa: Pass ndo_setup_tc slave callback to drivers
net: dsa: sja1105: Add static config tables for scheduling
net: dsa: sja1105: Advertise the 8 TX queues
net: dsa: sja1105: Configure the Time-Aware Scheduler via tc-taprio
offload
docs: net: dsa: sja1105: Add info about the Time-Aware Scheduler
Documentation/networking/dsa/sja1105.rst | 90 ++++ drivers/net/dsa/sja1105/Kconfig | 8 + drivers/net/dsa/sja1105/Makefile | 4 + drivers/net/dsa/sja1105/sja1105.h | 6 + .../net/dsa/sja1105/sja1105_dynamic_config.c | 8 + drivers/net/dsa/sja1105/sja1105_main.c | 26 +- .../net/dsa/sja1105/sja1105_static_config.c | 167 +++++++ .../net/dsa/sja1105/sja1105_static_config.h | 48 +- drivers/net/dsa/sja1105/sja1105_tas.c | 423 ++++++++++++++++++ drivers/net/dsa/sja1105/sja1105_tas.h | 41 ++ include/linux/netdevice.h | 1 + include/net/dsa.h | 2 + include/net/pkt_sched.h | 23 + include/uapi/linux/pkt_sched.h | 3 +- net/dsa/slave.c | 12 +- net/dsa/tag_sja1105.c | 3 +- net/sched/sch_taprio.c | 409 +++++++++++++++-- 17 files changed, 1222 insertions(+), 52 deletions(-) create mode 100644 drivers/net/dsa/sja1105/sja1105_tas.c create mode 100644 drivers/net/dsa/sja1105/sja1105_tas.h
Signature · 112 lines
For those who want to follow along with the hardware implementation, the
manual is here:
https://www.nxp.com/docs/en/user-guide/UM10944.pdf
Changes in v3 (actually v4 which is what v3 should have been):
- Removed "[PATCH v2 net-next 5/7] net: dsa: sja1105: Make HOSTPRIO a
kernel config", which was deemed non-critical at this point.
- A bit of cleanup in "[PATCH v3 net-next 5/6] net: dsa: sja1105:
Configure the Time-Aware Scheduler via tc-taprio offload".
Notable changes in v2:
- Made the series independent from PTP (which is temporarily removed)
- Changed the meaning of the gate_mask - it is now acting on traffic
classes even in the view exposed by taprio to drivers.
- Removed the next_sched hrtimer.
- Summarized one of the responses given to Vinicius into a new
documentation section.
The first version of the net-next patch series can be found here:
https://www.spinics.net/lists/netdev/msg597214.html
Changes in the first version of the net-next series compared to RFC v2:
- Made "flags 1" and "flags 2" mutually exclusive in the taprio qdisc
- Moved taprio_enable_offload and taprio_disable_offload out of atomic
context - spin_lock_bh(qdisc_lock(sch)). This allows drivers that
implement the ndo_setup_tc to sleep and for taprio memory to be
allocated with GFP_KERNEL. The only thing that was kept under the
spinlock is the assignment of the q->dequeue and q->peek pointers.
- Finally making proper use of own API - added a taprio_alloc helper to
avoid passing stack memory to drivers.
The second version of the RFC is at:
https://www.spinics.net/lists/netdev/msg596663.html
Changes in v2 of the RFC since v1:
- Adapted the taprio offload patch to work by specifying "flags 2" to
the iproute2-next tc. At the moment I don't clearly understand whether
the full offload and the txtime assist ("flags 1") are mutually
exclusive or not (i.e. whether a "flags 3" mode should be rejected,
which it currently isn't).
- Added reference counting to the taprio offload structure. Maybe the
function names and placement could have been better though. As for the
other complaint (cycle time calculation) it got fixed in the taprio
parser in the meantime.
- Converted sja1105 to use the hardware PTP registers, and save/restore
the PTP time across resets.
- Made the DSA callback for ndo_setup_tc a bit more generic, but I don't
know whether it fulfills expectations. Drivers still can't do blocking
operations in its execution context.
- Added a state machine for starting/stopping the scheduler based on the
last command run on the PTP clock.
The first RFC from July can be seen at:
https://lists.openwall.net/netdev/2019/07/07/81
Original cover letter:
Using Vinicius Costa Gomes' configuration interface for 802.1Qbv (later
resent by Voon Weifeng for the stmmac driver), I am submitting for
review a draft implementation of this offload for a DSA switch.
I don't want to insist too much on the hardware specifics of SJA1105
which isn't otherwise very compliant to the IEEE spec.
In order to be able to test with Vedang Patel's iproute2 patch for
taprio offload (https://www.spinics.net/lists/netdev/msg573072.html)
I had to actually revert the txtime-assist branch as it had changed the
iproute2 interface.
In terms of impact for DSA drivers, I would like to point out that:
- Maybe somebody should pre-populate qopt->cycle_time in case the user
does not provide one. Otherwise each driver needs to iterate over the
GCL once, just to set the cycle time (right now stmmac does as well).
- Configuring the switch over SPI cannot apparently be done from this
ndo_setup_tc callback because it runs in atomic context. I also have
some downstream patches to offload tc clsact matchall with mirred
action, but in that case it looks like the atomic context restriction
does not apply.
- I had to copy the struct tc_taprio_qopt_offload to driver private
memory because a static config needs to be constructed every time a
change takes place, and there are up to 4 switch ports that may take a
TAS configuration. I have created a private
tc_taprio_qopt_offload_copy() helper for this - I don't know whether
it's of any help in the general case.
There is more to be done however. The TAS needs to be integrated with
the PTP driver. This is because with a PTP clock source, the base time
is written dynamically to the PTPSCHTM (PTP schedule time) register and
must be a time in the future. Then the "real" base time of each port's
TAS config can be offset by at most ~50 ms (the DELTA field from the
Schedule Entry Points Table) relative to PTPSCHTM.
Because base times in the past are completely ignored by this hardware,
we need to decide if it's ok behaviorally for a driver to "roll" a past
base time into the immediate future by incrementally adding the cycle
time (so the phase doesn't change). If it is, then decide by how long in
the future it is ok to do so. Or alternatively, is it preferable if the
driver errors out if the user-supplied base time is in the past and the
hardware doesn't like it? But even then, there might be fringe cases
when the base time becomes a past PTP time right as the driver tries to
apply the config.
Also applying a tc-taprio offload to a second SJA1105 switch port will
inevitably need to roll the first port's (now past) base time into an
equivalent future time.
All of this is going to be complicated even further by the fact that
resetting the switch (to apply the tc-taprio offload) makes it reset its
PTP time.
2.17.1Vladimir Oltean
maintainerPATCH v4 2/6 net: dsa: Pass ndo_setup_tc slave callback to drivers
DSA currently handles shared block filters (for the classifier-action qdisc) in the core due to what I believe are simply pragmatic reasons - hiding the complexity from drivers and offerring a simple API for port mirroring.
Extend the dsa_slave_setup_tc function by passing all other qdisc offloads to the driver layer, where the driver may choose what it implements and how. DSA is simply a pass-through in this case.
---
Changes since v2:
- Added Florian Fainelli's Reviewed-by.
Changes since v1:
- Added Kurt Kanzenbach's Acked-by.
Changes since RFC:
- Removed the unused declaration of struct tc_taprio_qopt_offload.
include/net/dsa.h | 2 ++ net/dsa/slave.c | 12 ++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-)
include/net/dsa.h +2 -0
@@ -515,6 +515,8 @@ struct dsa_switch_ops { bool ingress);void (*port_mirror_del)(struct dsa_switch *ds,int port,struct dsa_mall_mirror_tc_entry *mirror); +int (*port_setup_tc)(struct dsa_switch *ds,int port, +enum tc_setup_type type,void *type_data);/* * Cross-chip operations
net/dsa/slave.c +8 -4
@@ -1035,12 +1035,16 @@ static int dsa_slave_setup_tc_block(struct net_device *dev,static int dsa_slave_setup_tc (struct net_device *dev,enum tc_setup_type type,void *type_data) { -switch (type) { -case TC_SETUP_BLOCK : +struct dsa_port *dp =dsa_slave_to_port (dev); +struct dsa_switch *ds = dp->ds; + +if (type ==TC_SETUP_BLOCK )return dsa_slave_setup_tc_block (dev, type_data); -default : + +if (!ds->ops->port_setup_tc)return -EOPNOTSUPP ; - } + +return ds->ops->port_setup_tc (ds, dp->index, type, type_data); } static void dsa_slave_get_stats64(struct net_device *dev,
Signature · 1 line
2.17.1
Vladimir Oltean
maintainerPATCH v4 1/6 taprio: Add support for hardware offloading
In reply to Vladimir OlteanFrom: Vinicius Costa Gomes <[email protected]>
This allows taprio to offload the schedule enforcement to capable network cards, resulting in more precise windows and less CPU usage.
The gate mask acts on traffic classes (groups of queues of same priority), as specified in IEEE 802.1Q-2018, and following the existing taprio and mqprio semantics.
It is up to the driver to perform conversion between tc and individual netdev queues if for some reason it needs to make that distinction.
Full offload is requested from the network interface by specifying "flags 2" in the tc qdisc creation command, which in turn corresponds to the TCA_TAPRIO_ATTR_FLAG_FULL_OFFLOAD bit.
The important detail here is the clockid which is implicitly /dev/ptpN for full offload, and hence not configurable.
A reference counting API is added to support the use case where Ethernet drivers need to keep the taprio offload structure locally (i.e. they are a multi-port switch driver, and configuring a port depends on the settings of other ports as well). The refcount_t variable is kept in a private structure (__tc_taprio_qopt_offload) and not exposed to drivers.
In the future, the private structure might also be expanded with a backpointer to taprio_sched *q, to implement the notification system described in the patch (of when admin became oper, or an error occurred, etc, so the offload can be monitored with 'tc qdisc show').
---
Changes since v2:
- None.
Changes since v1:
- Turned the next_sched hrtimer function into a simple taprio_offload_config_changed function called synchronously (for now) from taprio_enable_offload. But the idea is that the driver may have a lot more means to figure out when the admin schedule is no longer pending (perhaps even an interrupt), so leave an open window for implementing a notification system from the driver.
- Made it an error to specify 'clockid' with full offload.
- Created a wrapper __tc_taprio_qopt_offload structure which holds the refcount_t (for now) and maybe a backpointer to the qdisc_priv in the future.
- Renamed taprio_get and taprio_free to taprio_offload_get and taprio_offload_free. Renamed the "taprio" variable to "offload".
- Moved the reference counting helper implementations to sch_taprio.c.
- Removed the tc_mask_to_queue_mask manipulation done to the gate_mask before passing it on to drivers. Instead of netdev queue gates, they now see a mask of traffic class gates, which:
- They need to care about anyway, if they have a multi-queue device and they need to configure the queue-to-tc hardware mapping.
- Makes no difference to them if the hardware makes no distinction between queue and traffic class (there is only one egress queue per tc, having a fixed priority). The sja1105 hw is in this situation.
Changes since RFC:
- Made the combination of FULL_OFFLOAD and TXTIME_ASSIST invalid.
- Made ndo_setup_tc be called from sleepable context.
- Added a taprio_alloc helper to avoid passing stack memory to drivers.
- Made taprio_disable_offload take the extack as well.
- Conditioned the setup of the software (and txtime-assisted) implementation of taprio on there not being a full offload in place.
- Fixed a lockdep-related compilation bug.
include/linux/netdevice.h | 1 + include/net/pkt_sched.h | 23 ++ include/uapi/linux/pkt_sched.h | 3 +- net/sched/sch_taprio.c | 409 +++++++++++++++++++++++++++++---- 4 files changed, 392 insertions(+), 44 deletions(-)
include/linux/netdevice.h +1 -0
@@ -847,6 +847,7 @@ enum tc_setup_type {TC_SETUP_QDISC_ETF ,TC_SETUP_ROOT_QDISC ,TC_SETUP_QDISC_GRED , + TC_SETUP_QDISC_TAPRIO, };/* These structures hold the attributes of bpf state that are being passed
include/net/pkt_sched.h +23 -0
@@ -161,4 +161,27 @@ struct tc_etf_qopt_offload { s32 queue; }; +struct tc_taprio_sched_entry { +u8 command;/* TC_TAPRIO_CMD_* */ + +/* The gate_mask in the offloading side refers to traffic classes */ +u32 gate_mask; +u32 interval; +}; + +struct tc_taprio_qopt_offload { +u8 enable; +ktime_t base_time; +u64 cycle_time; +u64 cycle_time_extension; + +size_t num_entries; +struct tc_taprio_sched_entry entries[0 ]; +}; + +/* Reference counting */ +struct tc_taprio_qopt_offload *taprio_offload_get (struct tc_taprio_qopt_offload + *offload); +void taprio_offload_free (struct tc_taprio_qopt_offload *offload); +#endif
include/uapi/linux/pkt_sched.h +2 -1
@@ -1160,7 +1160,8 @@ enum {* [TCA_TAPRIO_ATTR_SCHED_ENTRY_INTERVAL] */ -#define TCA_TAPRIO_ATTR_FLAG_TXTIME_ASSIST 0x1 +#define TCA_TAPRIO_ATTR_FLAG_TXTIME_ASSIST BIT(0) +#define TCA_TAPRIO_ATTR_FLAG_FULL_OFFLOAD BIT(1)enum {TCA_TAPRIO_ATTR_UNSPEC ,
net/sched/sch_taprio.c +366 -43
521 lines not on this page · raw message
Signature · 1 line
2.17.1
Vladimir Oltean
maintainerPATCH v4 4/6 net: dsa: sja1105: Advertise the 8 TX queues
In reply to Vladimir OlteanThis is a preparation patch for the tc-taprio offload (and potentially for other future offloads such as tc-mqprio).
Instead of looking directly at skb->priority during xmit, let's get the netdev queue and the queue-to-traffic-class mapping, and put the resulting traffic class into the dsa_8021q PCP field. The switch is configured with a 1-to-1 PCP-to-ingress-queue-to-egress-queue mapping (see vlan_pmap in sja1105_main.c), so the effect is that we can inject into a front-panel's egress traffic class through VLAN tagging from Linux, completely transparently.
Unfortunately the switch doesn't look at the VLAN PCP in the case of management traffic to/from the CPU (link-local frames at 01-80-C2-xx-xx-xx or 01-1B-19-xx-xx-xx) so we can't alter the transmission queue of this type of traffic on a frame-by-frame basis. It is only selected through the "hostprio" setting which ATM is harcoded in the driver to 7.
---
Changes since v2:
- None.
Changes since v1:
- None, but the use of netdev_txq_to_tc is now finally correct after adjusting the gate_mask meaning in the taprio offload structure.
Changes since RFC:
- None.
drivers/net/dsa/sja1105/sja1105_main.c | 7 ++++++- net/dsa/tag_sja1105.c | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-)
drivers/net/dsa/sja1105/sja1105_main.c +6 -1
@@ -384,7 +384,9 @@ static int sja1105_init_general_params(struct sja1105_private *priv)/* Disallow dynamic changing of the mirror port */ .mirr_ptacu = 0, .switchid = priv->ds->index, -/* Priority queue for link-local frames trapped to CPU */ +/* Priority queue for link-local management frames +* (both ingress to and egress from CPU - PTP, STP etc) +*/ .hostprio = 7, .mac_fltres1 = SJA1105_LINKLOCAL_FILTER_A, .mac_flt1 = SJA1105_LINKLOCAL_FILTER_A_MASK,
@@ -1711,6 +1713,9 @@ static int sja1105_setup(struct dsa_switch *ds)*/ ds->vlan_filtering_is_global = true; +/* Advertise the 8 egress queues */ + ds->num_tx_queues =SJA1105_NUM_TC ; +/* The DSA/switchdev model brings up switch ports in standalone mode by * default, and that means vlan_filtering is 0 since they're not under * a bridge, so it's safe to set up switch tagging at this time.
net/dsa/tag_sja1105.c +2 -1
@@ -89,7 +89,8 @@ static struct sk_buff *sja1105_xmit(struct sk_buff *skb,struct dsa_port *dp =dsa_slave_to_port (netdev);struct dsa_switch *ds = dp->ds;u16 tx_vid =dsa_8021q_tx_vid (ds, dp->index); -u8 pcp = skb->priority; +u16 queue_mapping =skb_get_queue_mapping (skb); +u8 pcp =netdev_txq_to_tc (netdev, queue_mapping);/* Transmitting management traffic does not rely upon switch tagging, * but instead SPI-installed management routes. Part 2 of this
Signature · 1 line
2.17.1
Vladimir Oltean
maintainerPATCH v4 3/6 net: dsa: sja1105: Add static config tables for scheduling
In reply to Vladimir OlteanIn order to support tc-taprio offload, the TTEthernet egress scheduling core registers must be made visible through the static interface.
---
Changes since v2:
- None.
Changes since v1:
- None.
Changes since RFC:
- None.
.../net/dsa/sja1105/sja1105_dynamic_config.c | 8 + .../net/dsa/sja1105/sja1105_static_config.c | 167 ++++++++++++++++++ .../net/dsa/sja1105/sja1105_static_config.h | 48 ++++- 3 files changed, 222 insertions(+), 1 deletion(-)
drivers/net/dsa/sja1105/sja1105_dynamic_config.c +8 -0
@@ -488,6 +488,8 @@ sja1105et_general_params_entry_packing(void *buf, void *entry_ptr,/* SJA1105E/T: First generation */ struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN ] = { + [BLK_IDX_SCHEDULE ] = {0 }, + [BLK_IDX_SCHEDULE_ENTRY_POINTS ] = {0 }, [BLK_IDX_L2_LOOKUP ] = { .entry_packing = sja1105et_dyn_l2_lookup_entry_packing, .cmd_packing = sja1105et_l2_lookup_cmd_packing,
@@ -529,6 +531,8 @@ struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN] = { .packed_size = SJA1105ET_SIZE_MAC_CONFIG_DYN_CMD, .addr = 0x36, }, + [BLK_IDX_SCHEDULE_PARAMS] = {0}, + [BLK_IDX_SCHEDULE_ENTRY_POINTS_PARAMS] = {0}, [BLK_IDX_L2_LOOKUP_PARAMS] = { .entry_packing = sja1105et_l2_lookup_params_entry_packing, .cmd_packing = sja1105et_l2_lookup_params_cmd_packing,
@@ -552,6 +556,8 @@ struct sja1105_dynamic_table_ops sja1105et_dyn_ops[BLK_IDX_MAX_DYN] = {/* SJA1105P/Q/R/S: Second generation */ struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN ] = { + [BLK_IDX_SCHEDULE ] = {0 }, + [BLK_IDX_SCHEDULE_ENTRY_POINTS ] = {0 }, [BLK_IDX_L2_LOOKUP ] = { .entry_packing = sja1105pqrs_dyn_l2_lookup_entry_packing, .cmd_packing = sja1105pqrs_l2_lookup_cmd_packing,
@@ -593,6 +599,8 @@ struct sja1105_dynamic_table_ops sja1105pqrs_dyn_ops[BLK_IDX_MAX_DYN] = { .packed_size = SJA1105PQRS_SIZE_MAC_CONFIG_DYN_CMD, .addr = 0x4B, }, + [BLK_IDX_SCHEDULE_PARAMS] = {0}, + [BLK_IDX_SCHEDULE_ENTRY_POINTS_PARAMS] = {0}, [BLK_IDX_L2_LOOKUP_PARAMS] = { .entry_packing = sja1105et_l2_lookup_params_entry_packing, .cmd_packing = sja1105et_l2_lookup_params_cmd_packing,
drivers/net/dsa/sja1105/sja1105_static_config.c +167 -0
268 lines not on this page · raw message
drivers/net/dsa/sja1105/sja1105_static_config.h +47 -1
109 lines not on this page · raw message
Signature · 1 line
2.17.1
Vladimir Oltean
maintainerPATCH v4 6/6 docs: net: dsa: sja1105: Add info about the Time-Aware Scheduler
In reply to Vladimir OlteanWhile not an exhaustive usage tutorial, this describes the details needed to build more complex scenarios.
---
Changes since v2:
- Specified that the HOSTPRIO switch setting is not configurable at the moment, since I dropped the patch for configuring it.
Changes since v1:
- Patch is new.
Documentation/networking/dsa/sja1105.rst | 90 ++++++++++++++++++++++++ 1 file changed, 90 insertions(+)
Documentation/networking/dsa/sja1105.rst +90 -0
@@ -146,6 +146,96 @@ enslaves eth0 and eth1 (the DSA master of the switch ports). This is because in this mode, the switch ports beneath br0 are not capable of regular traffic, and are only used as a conduit for switchdev operations. +Offloads +======== + +Time-aware scheduling +--------------------- + +The switch supports a variation of the enhancements for scheduled traffic +specified in IEEE 802.1Q-2018 (formerly 802.1Qbv). This means it can be used to +ensure deterministic latency for priority traffic that is sent in-band with its +gate-open event in the network schedule. + +This capability can be managed through the tc-taprio offload ('flags 2'). The +difference compared to the software implementation of taprio is that the latter +would only be able to shape traffic originated from the CPU, but not +autonomously forwarded flows. + +The device has 8 traffic classes, and maps incoming frames to one of them based +on the VLAN PCP bits (if no VLAN is present, the port-based default is used). +As described in the previous sections, depending on the value of +``vlan_filtering``, the EtherType recognized by the switch as being VLAN can +either be the typical 0x8100 or a custom value used internally by the driver +for tagging. Therefore, the switch ignores the VLAN PCP if used in standalone +or bridge mode with ``vlan_filtering=0``, as it will not recognize the 0x8100 +EtherType. In these modes, injecting into a particular TX queue can only be +done by the DSA net devices, which populate the PCP field of the tagging header +on egress. Using ``vlan_filtering=1``, the behavior is the other way around: +offloaded flows can be steered to TX queues based on the VLAN PCP, but the DSA +net devices are no longer able to do that. To inject frames into a hardware TX +queue with VLAN awareness active, it is necessary to create a VLAN +sub-interface on the DSA master port, and send normal (0x8100) VLAN-tagged +towards the switch, with the VLAN PCP bits set appropriately. + +Management traffic (having DMAC 01-80-C2-xx-xx-xx or 01-19-1B-xx-xx-xx) is the +notable exception: the switch always treats it with a fixed priority and +disregards any VLAN PCP bits even if present. The traffic class for management +traffic has a value of 7 (highest priority) at the moment, which is not +configurable in the driver. + +Below is an example of configuring a 500 us cyclic schedule on egress port +``swp5``. The traffic class gate for management traffic (7) is open for 100 us, +and the gates for all other traffic classes are open for 400 us:: + + #!/bin/bash + + set -e -u -o pipefail + + NSEC_PER_SEC="1000000000" + + gatemask() { + local tc_list="$1" + local mask=0 + + for tc in ${tc_list}; do + mask=$((${mask} | (1 << ${tc}))) + done + + printf "%02x" ${mask} + } + + if ! systemctl is-active --quiet ptp4l; then + echo "Please start the ptp4l service" + exit + fi + + now=$(phc_ctl /dev/ptp1 get | gawk '/clock time is/ { print $5; }') + # Phase-align the base time to the start of the next second. + sec=$(echo "${now}" | gawk -F. '{ print $1; }') + base_time="$(((${sec} + 1) * ${NSEC_PER_SEC}))" + + tc qdisc add dev swp5 parent root handle 100 taprio \ + num_tc 8 \ + map 0 1 2 3 5 6 7 \ + queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 \ + base-time ${base_time} \ + sched-entry S $(gatemask 7) 100000 \ + sched-entry S $(gatemask "0 1 2 3 4 5 6") 400000 \ + flags 2 + +It is possible to apply the tc-taprio offload on multiple egress ports. There +are hardware restrictions related to the fact that no gate event may trigger +simultaneously on two ports. The driver checks the consistency of the schedules +against this restriction and errors out when appropriate. Schedule analysis is +needed to avoid this, which is outside the scope of the document. + +At the moment, the time-aware scheduler can only be triggered based on a +standalone clock and not based on PTP time. This means the base-time argument +from tc-taprio is ignored and the schedule starts right away. It also means it +is more difficult to phase-align the scheduler with the other devices in the +network. + Device Tree bindings and board design =====================================
Signature · 1 line
2.17.1
Vladimir Oltean
maintainerPATCH v4 5/6 net: dsa: sja1105: Configure the Time-Aware Scheduler via tc-taprio offload
In reply to Vladimir OlteanThis qdisc offload is the closest thing to what the SJA1105 supports in hardware for time-based egress shaping. The switch core really is built around SAE AS6802/TTEthernet (a TTTech standard) but can be made to operate similarly to IEEE 802.1Qbv with some constraints:
- The gate control list is a global list for all ports. There are 8 execution threads that iterate through this global list in parallel. I don't know why 8, there are only 4 front-panel ports.
- Care must be taken by the user to make sure that two execution threads never get to execute a GCL entry simultaneously. I created a O(n^4) checker for this hardware limitation, prior to accepting a taprio offload configuration as valid.
- The spec says that if a GCL entry's interval is shorter than the frame length, you shouldn't send it (and end up in head-of-line blocking). Well, this switch does anyway.
- The switch has no concept of ADMIN and OPER configurations. Because it's so simple, the TAS settings are loaded through the static config tables interface, so there isn't even place for any discussion about 'graceful switchover between ADMIN and OPER'. You just reset the switch and upload a new OPER config.
- The switch accepts multiple time sources for the gate events. Right now I am using the standalone clock source as opposed to PTP. So the base time parameter doesn't really do much. Support for the PTP clock source will be added in a future series.
---
Changes since v2:
- Made all functions that are entry points into sja1105_tas.c take a dsa_struct *ds argument instead of sja1105_private *priv. This also happens to avoid a build error reported by the Kbuild test robot.
- Moved the iteration over switch ports outside of sja1105_tas_check_conflicts, to address some checkpatch complaints.
- Renamed "new" -> "admin".
Changes since v1:
- Adapted to the naming convention changes in 01/07 (taprio_get -> taprio_offload_get, tas_config -> offload, etc).
Changes since RFC:
- Removed the sja1105_tas_config_work workqueue.
- Allocating memory with GFP_KERNEL.
- Made the ASCII art drawing fit in < 80 characters.
- Made most of the time-holding variables s64 instead of u64 (for fear of them not holding the result of signed arithmetics properly).
drivers/net/dsa/sja1105/Kconfig | 8 + drivers/net/dsa/sja1105/Makefile | 4 + drivers/net/dsa/sja1105/sja1105.h | 6 + drivers/net/dsa/sja1105/sja1105_main.c | 19 +- drivers/net/dsa/sja1105/sja1105_tas.c | 423 +++++++++++++++++++++++++ drivers/net/dsa/sja1105/sja1105_tas.h | 41 +++ 6 files changed, 500 insertions(+), 1 deletion(-) create mode 100644 drivers/net/dsa/sja1105/sja1105_tas.c create mode 100644 drivers/net/dsa/sja1105/sja1105_tas.h
drivers/net/dsa/sja1105/Kconfig +8 -0
@@ -23,3 +23,11 @@ config NET_DSA_SJA1105_PTP help This enables support for timestamping and PTP clock manipulations in the SJA1105 DSA driver. + +config NET_DSA_SJA1105_TAS + bool "Support for the Time-Aware Scheduler on NXP SJA1105" + depends on NET_DSA_SJA1105 + help + This enables support for the TTEthernet-based egress scheduling + engine in the SJA1105 DSA driver, which is controlled using a + hardware offload of the tc-tqprio qdisc.
drivers/net/dsa/sja1105/Makefile +4 -0
@@ -12,3 +12,7 @@ sja1105-objs := \ ifdef CONFIG_NET_DSA_SJA1105_PTP sja1105-objs += sja1105_ptp.o endif + +ifdef CONFIG_NET_DSA_SJA1105_TAS +sja1105-objs += sja1105_tas.o +endif
drivers/net/dsa/sja1105/sja1105.h +6 -0
@@ -20,6 +20,8 @@*/ #define SJA1105_AGEING_TIME_MS (ms) ((ms) / 10) +#include "sja1105_tas.h" +/* Keeps the different addresses between E/T and P/Q/R/S */ struct sja1105_regs {u64 device_id;
@@ -104,6 +106,7 @@ struct sja1105_private {*/ struct mutex mgmt_lock;struct sja1105_tagger_data tagger_data; +struct sja1105_tas_data tas_data; };#include "sja1105_dynamic_config.h"
@@ -120,6 +123,9 @@ typedef enum {SPI_WRITE =1 , } sja1105_spi_rw_mode_t; +/* From sja1105_main.c */ +int sja1105_static_config_reload (struct sja1105_private *priv); +/* From sja1105_spi.c */ int sja1105_spi_send_packed_buf(const struct sja1105_private *priv, sja1105_spi_rw_mode_t rw, u64 reg_addr,
drivers/net/dsa/sja1105/sja1105_main.c +18 -1
@@ -22,6 +22,7 @@#include <linux/if_ether.h> #include <linux/dsa/8021q.h> #include "sja1105.h" +#include "sja1105_tas.h" static void sja1105_hw_reset (struct gpio_desc *gpio,unsigned int pulse_len,unsigned int startup_delay)
@@ -1382,7 +1383,7 @@ static void sja1105_bridge_leave(struct dsa_switch *ds, int port,* modify at runtime (currently only MAC) and restore them after uploading, * such that this operation is relatively seamless. */ -static int sja1105_static_config_reload (struct sja1105_private *priv) +int sja1105_static_config_reload (struct sja1105_private *priv) {struct sja1105_mac_config_entry *mac;int speed_mbps[SJA1105_NUM_PORTS ];
@@ -1727,6 +1728,7 @@ static void sja1105_teardown(struct dsa_switch *ds) {struct sja1105_private *priv = ds->priv; +sja1105_tas_teardown (ds);cancel_work_sync (&priv->tagger_data.rxtstamp_work);skb_queue_purge (&priv->tagger_data.skb_rxtstamp_queue);sja1105_ptp_clock_unregister (priv);
@@ -2056,6 +2058,18 @@ static bool sja1105_port_txtstamp(struct dsa_switch *ds, int port,return true; } +static int sja1105_port_setup_tc (struct dsa_switch *ds,int port, +enum tc_setup_type type, +void *type_data) +{ +switch (type) { +case TC_SETUP_QDISC_TAPRIO : +return sja1105_setup_tc_taprio (ds, port, type_data); +default : +return -EOPNOTSUPP ; + } +} +static const struct dsa_switch_ops sja1105_switch_ops = { .get_tag_protocol = sja1105_get_tag_protocol, .setup = sja1105_setup,
@@ -2088,6 +2102,7 @@ static const struct dsa_switch_ops sja1105_switch_ops = { .port_hwtstamp_set = sja1105_hwtstamp_set, .port_rxtstamp = sja1105_port_rxtstamp, .port_txtstamp = sja1105_port_txtstamp, + .port_setup_tc = sja1105_port_setup_tc, };static int sja1105_check_device_id (struct sja1105_private *priv)
@@ -2197,6 +2212,8 @@ static int sja1105_probe(struct spi_device *spi) }mutex_init (&priv->mgmt_lock); +sja1105_tas_setup (ds); +return dsa_register_switch (priv->ds); }
drivers/net/dsa/sja1105/sja1105_tas.c new file +423 -0
423 lines not on this page · raw message
drivers/net/dsa/sja1105/sja1105_tas.h new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v4 net-next 4/6] net: dsa: sja1105: Advertise the 8 TX queues
In reply to Vladimir OlteanQuoted 11 lines · Vladimir Oltean
This is a preparation patch for the tc-taprio offload (and potentially for other future offloads such as tc-mqprio).
Instead of looking directly at
skb->priorityduring xmit, let's get the netdev queue and the queue-to-traffic-class mapping, and put the resulting traffic class into thedsa_8021qPCP field. The switch is configured with a 1-to-1 PCP-to-ingress-queue-to-egress-queue mapping (seevlan_pmapinsja1105_main.c), so the effect is that we can inject into a front-panel's egress traffic class through VLAN tagging from Linux, completely transparently.
Quoted 8 lines
Unfortunately the switch doesn't look at the VLAN PCP in the case of management traffic to/from the CPU (link-local frames at 01-80-C2-xx-xx-xx or 01-1B-19-xx-xx-xx) so we can't alter the transmission queue of this type of traffic on a frame-by-frame basis. It is only selected through the "hostprio" setting which ATM is harcoded in the driver to 7.
Signed-off-by Vladimir Oltean <[email protected]>
Signature · 1 line
Florian
RE: [PATCH v4 net-next 4/6] net: dsa: sja1105: Advertise the 8 TX queues
In reply to Vladimir OlteanFrom: Vladimir Oltean <[email protected]>
Date: Sep/15/2019, 03:00:01 (UTC+00:00)
Quoted 7 lines
Instead of looking directly at
skb->priorityduring xmit, let's get the netdev queue and the queue-to-traffic-class mapping, and put the resulting traffic class into thedsa_8021qPCP field. The switch is configured with a 1-to-1 PCP-to-ingress-queue-to-egress-queue mapping (seevlan_pmapinsja1105_main.c), so the effect is that we can inject into a front-panel's egress traffic class through VLAN tagging from Linux, completely transparently.
Wouldn't it be better to just rely on skb queue mapping as per userspace settings ? I mean this way you cant create some complex scenarios without having the VLAN ID need.
I generally use u32 filter and skbedit queue_mapping action to achieve this.
---
Thanks,
Jose Miguel Abreu
Vladimir Oltean
maintainerRe: [PATCH v4 net-next 4/6] net: dsa: sja1105: Advertise the 8 TX queues
Hi Jose,
Quoted 12 lines · Jose Abreu
From: Vladimir Oltean <[email protected]>
Date: Sep/15/2019, 03:00:01 (UTC+00:00)Quoted 7 lines
Instead of looking directly at
skb->priorityduring xmit, let's get the netdev queue and the queue-to-traffic-class mapping, and put the resulting traffic class into thedsa_8021qPCP field. The switch is configured with a 1-to-1 PCP-to-ingress-queue-to-egress-queue mapping (seevlan_pmapinsja1105_main.c), so the effect is that we can inject into a front-panel's egress traffic class through VLAN tagging from Linux, completely transparently.
Quoted 4 lines
Wouldn't it be better to just rely on skb queue mapping as per userspace settings ? I mean this way you cant create some complex scenarios without having the VLAN ID need.
I think I need to clarify the "without having the VLAN ID need" portion. This is a DSA switch, and the thing that all these switches have in common is that their data path is connected to the host system through an Ethernet port pair (as opposed to e.g. DMA and real queues). When you send a frame from the host through that Ethernet port, the switch's natural tendency is to forward/flood it according to FDB rules, which may not be what you want to achieve (i.e. you may want to xmit on a specific front-panel port only, to achieve a 'port multiplier' behavior out of it). Switch vendors achieve that by using a proprietary frame header which indicates stuff such as: egress port, QoS priority, etc. The host adds this tag to frames sent towards the switch, and the switch consumes it.
In the case of sja1105, the frame tagging format is actually 'open-spec', e.g. it is an actual VLAN header, just with a custom EtherType. Hence the QoS priority in this tagging format is the PCP field. Other DSA switch vendors still have a 'priority' field in their tagging format, even though it is not PCP.
So to answer your question, I do need VLAN tagging either way, for even more basic reasons. But it is VLAN only from the switch's hardware view. From the operating system's view it is no different than populating the 'priority' field of any other DSA frame tagger.
Quoted 3 lines
I generally use u32 filter and skbedit
queue_mappingaction to achieve this.
The trouble with relying on 'queue_mapping' only is that it's too
open-ended: what does a queue really mean for an Ethernet-managed switch?
In my interpretation of the mqprio/taprio offloads, the netdev queue is only an intermediary representation between the qdisc and the NIC's traffic classes. Now that makes sense, because even though the switch xmit procedure isn't multi-queue, there is still strict egress prioritization based on the traffic class mapping (inferred from VLAN PCP).
I have no idea how people are managing multi-queue net devices without an mqprio-type offload, if those netdev queues are not of equal priority. Technically, if you use the 'skbedit queue_mapping' action with this switch, it is your choice how you manage the net_device->tc_to_txq array, keeping in mind that tc 7 will have higher priority over tc 6. Alternatively, you can still use an mqprio-type offload, specify "num_tc 8 map 0 1 2 3 5 6 7 queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7", and then just use the 'skbedit priority' action. Then things would just work.
So I am just using the 'queues' term for this switch to express the strict priority scheduler. There isn't really any benefit to having more than one netdev queue exposed per traffic class.
Quoted 3 lines
---
Thanks,
Jose Miguel Abreu
Hope this clarifies,
-Vladimir
Re: [PATCH v4 net-next 2/6] net: dsa: Pass ndo_setup_tc slave callback to drivers
In reply to Vladimir OlteanQuoted 67 lines · Vladimir Oltean
DSA currently handles shared block filters (for the classifier-action qdisc) in the core due to what I believe are simply pragmatic reasons - hiding the complexity from drivers and offerring a simple API for port mirroring.
Extend the
dsa_slave_setup_tcfunction by passing all other qdisc offloads to the driver layer, where the driver may choose what it implements and how. DSA is simply a pass-through in this case.Signed-off-by Vladimir Oltean <[email protected]>Acked-by Kurt Kanzenbach <[email protected]>Reviewed-by Florian Fainelli <[email protected]>---
Changes since v2:
- Added Florian Fainelli's Reviewed-by.Changes since v1:
- Added Kurt Kanzenbach's Acked-by.Changes since RFC:
- Removed the unused declaration of structtc_taprio_qopt_offload.include/net/dsa.h | 2 ++ net/dsa/slave.c | 12 ++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-)include/net/dsa.h +2
-0@@ -515,6 +515,8 @@ struct dsa_switch_ops { bool ingress);void (*port_mirror_del)(struct dsa_switch *ds,int port,struct dsa_mall_mirror_tc_entry *mirror); +int (*port_setup_tc)(struct dsa_switch *ds,int port, +enum tc_setup_type type,void *type_data);/* * Cross-chip operations net/dsa/slave.c +8
-4@@ -1035,12 +1035,16 @@ static int dsa_slave_setup_tc_block(struct net_device *dev,static int dsa_slave_setup_tc (struct net_device *dev,enum tc_setup_type type,void *type_data) { -switch (type) { -case TC_SETUP_BLOCK : +struct dsa_port *dp =dsa_slave_to_port (dev); +struct dsa_switch *ds = dp->ds; + +if (type ==TC_SETUP_BLOCK )return dsa_slave_setup_tc_block (dev, type_data); -default : + +if (!ds->ops->port_setup_tc)return -EOPNOTSUPP ; - } + +return ds->ops->port_setup_tc (ds, dp->index, type, type_data); } static void dsa_slave_get_stats64(struct net_device *dev,Signature · 1 line
2.17.1
David Miller
maintainerFrom: Vladimir Oltean <[email protected]>
Date: Sun, 15 Sep 2019 04:59:57 +0300
Quoted 9 lines
This is the third attempt to submit the tc-taprio offload model for inclusion in the networking tree. The sja1105 switch driver will provide the first implementation of the offload. Only the bare minimum is added:
- The offload model and a DSA pass-through
- The hardware implementation
- The interaction with the netdev queues in the tagger code
- Documentation
Quoted 4 lines
What has been removed from previous attempts is support for PTP-as-clocksource in sja1105, as well as configuring the traffic class for management traffic. These will be added as soon as the offload model is settled.
Series applied, thanks.
Vladimir Oltean
maintainerQuoted 13 lines · David Miller
From: Vladimir Oltean <[email protected]>
Date: Sun, 15 Sep 2019 04:59:57 +0300Quoted 9 lines
This is the third attempt to submit the tc-taprio offload model for inclusion in the networking tree. The sja1105 switch driver will provide the first implementation of the offload. Only the bare minimum is added:
- The offload model and a DSA pass-through
- The hardware implementation
- The interaction with the netdev queues in the tagger code
- Documentation
Quoted 6 lines
Quoted 4 lines
What has been removed from previous attempts is support for PTP-as-clocksource in sja1105, as well as configuring the traffic class for management traffic. These will be added as soon as the offload model is settled.
Series applied, thanks.
Thanks a lot, that's what I call cutting it close!
-Vladimir