Skip to contentSkip to conversation

net/sched: Introduce the taprio scheduler

PATCH Applied Vinicius Costa Gomes 2 participants 4 messages Sep 29, 2018 – Oct 5, 2018

Subsystems (4)
lore
Share & tools
lore

Vinicius Costa Gomes

author maintainer
#0 lore raw

cover net/sched: Introduce the taprio scheduler

Hi,

Changes from the RFC:
- Moved some fields from the per-qdisc data structure to the per schedule entry one, mainly "expires" (now called "close_time", when an entry ends) and "budget" (how many bytes can be sent during an entry);

- Removed support for the schedule file, in favour of using iproute2 batch mode (only affects the iproute2 patches) (Jiri Pirko, Stephen Hemminger);

- Removed support for manually setting a cycle-time (it will be

    added in a later series);

Original cover letter
=====================
(lightly edited, updated references and usage)

This series provides a set of interfaces that can be used by applications that require (time-based) Scheduled Transmission of packets. It is comprised by 3 new components to the kernel:

- etf: the per-queue TxTime-Based scheduling qdisc;
- taprio: the per-port Time-Aware scheduler qdisc;
- SO_TXTIME: a socket option + cmsg APIs.

ETF and SO_TXTIME are already applied[1] into the net-next tree. This is the remaining piece.

Overview
========

The CBS qdisc proposal RFC [2] included some rough ideas about the design and API of a "taprio" (Time Aware Priority) qdisc. The idea of presenting the taprio ideas at that point (almost one year ago!) was to show our vision of how things would fit together going forward.

Quoted 1 line

From that concept stage to this (almost) realised stage the main

differences are:

- As of now, taprio is a software only implementation of a schedule

    executor;

- Instead of taprio centralising all the time based decisions, taprio can work together with ETF (the Earliest TxTime First), a qdisc meant to use the LaunchTime (or similar) feature of various network controllers;

In a nutshell, taprio is a root qdisc that can execute a pre-defined schedule, etf is a qdisc that provides time based admission control and "earliest deadline first" dequeue mode, and SO_TXTIME is a socket option that is used for enabling a socket to be used for time-based Tx and configuring its parameters.

taprio
======

This scheduler allows the network administrator to configure schedules for classes of traffic, the configuration interface is similar to what IEEE 802.1Q-2018 defines.

Example configuration:

$ tc qdisc add dev enp2s0 parent root handle 100 taprio \
	    num_tc 3 \
	    map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
	    queues 1@0 1@1 2@2 \
	    sched-entry S 01 300000 \
	    sched-entry S 02 300000 \
	    sched-entry S 04 300000 \
	    base-time 1528743495910289987 \
	    clockid CLOCK_TAI

This qdisc borrows a few concepts from mqprio and so, most the parameters are similar to mqprio. The main difference is the sequence of 'sched-entry' parameters, that constitute one schedule:

	      sched-entry S 01 300000
	      sched-entry S 02 300000
	      sched-entry S 04 300000

The format of each entry is:
sched-entry <CMD> <GATE MASK> <INTERVAL>

The only supported <CMD> is "S", which means "SetGateStates", following the IEEE 802.1Q-2018 definition (Table 8-7). <GATE MASK> is a bit-mask where each bit is a associated with a traffic class, so bit 0 (the least significant bit) being "on" means that traffic class 0 is "active" for that schedule entry. <INTERVAL> is a time duration in nanoseconds that specifies for how long that state defined by <CMD> and <GATE MASK> should be held before moving to the next entry.

This schedule is circular, that is, after the last entry is executed it starts from the first one, indefinitely.

The other parameters can be defined as follows:
- base-time: allows that multiple systems can have synchronised

    schedules, it specifies the instant when the schedule starts;

- clockid: specifies the reference clock to be used;

A more complete example can be found here, with instructions of how to test it:

https://gist.github.com/jeez/bd3afeff081ba64a695008dd8215866f [3]

The basic design of the scheduler is simple, after we calculate the first expiration of the hrtimer, we set the next expiration to be the previous plus the current entry's interval. At each time the function runs, we set the current_entry, which has a gate_mask (that controls which traffic classes are allowed to "go out" during each interval), and we reuse this callback to "kick" the qdisc (this is the reason that the usual qdisc watchdog isn't used).

Future work
===========

- Add support for multiple schedules, so something like the Admin and Oper schedules from IEEE 802.1Q-2018 can be implemented, probably "cycle-time" will be re-implemented at this time;

- Add support for HW offloading;

- Add support for Frame Preemption related commands (formerly

    802.1Qbu, now part of 802.1Q);

Known Issues
============

- As taprio is a software only implementation, and there's another layer of queuing in the network controller, packets can still leave the controller outside their "correct" windows. This happens mostly for low-priority classes, and only if they are 'starved' by the higher priority ones;

This series is also hosted on github and can be found at [4]. The companion iproute2 patches can be found at [5].

Cheers,

author maintainer
#1 lore raw

PATCH 1/1 tc: Add support for configuring the taprio scheduler

This traffic scheduler allows traffic classes states (transmission allowed/not allowed, in the simplest case) to be scheduled, according to a pre-generated time sequence. This is the basis of the IEEE 802.1Qbv specification.

Example configuration:

tc qdisc replace dev enp3s0 parent root handle 100 taprio \

          num_tc 3 \
	  map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
	  queues 1@0 1@1 2@2 \
	  base-time 1528743495910289987 \
	  sched-entry S 01 300000 \
	  sched-entry S 02 300000 \
	  sched-entry S 04 300000 \
	  clockid CLOCK_TAI

The configuration format is similar to mqprio. The main difference is the presence of a schedule, built by multiple "sched-entry" definitions, each entry has the following format:

sched-entry <CMD> <GATE MASK> <INTERVAL>

The only supported <CMD> is "S", which means "SetGateStates", following the IEEE 802.1Qbv-2015 definition (Table 8-6). <GATE MASK> is a bitmask where each bit is a associated with a traffic class, so bit 0 (the least significant bit) being "on" means that traffic class 0 is "active" for that schedule entry. <INTERVAL> is a time duration in nanoseconds that specifies for how long that state defined by <CMD> and <GATE MASK> should be held before moving to the next entry.

This schedule is circular, that is, after the last entry is executed it starts from the first one, indefinitely.

The other parameters can be defined as follows:

- base-time: specifies the instant when the schedule starts, if 'base-time' is a time in the past, the schedule will start at

base-time + (N * cycle-time)

where N is the smallest integer so the resulting time is greater than "now", and "cycle-time" is the sum of all the intervals of the entries in the schedule;

- clockid: specifies the reference clock to be used;

The parameters should be similar to what the IEEE 802.1Q family of specification defines.

---
 include/uapi/linux/pkt_sched.h |  46 ++
 net/sched/Kconfig              |  11 +
 net/sched/Makefile             |   1 +
 net/sched/sch_taprio.c         | 962 +++++++++++++++++++++++++++++++++
 4 files changed, 1020 insertions(+)
 create mode 100644 net/sched/sch_taprio.c
include/uapi/linux/pkt_sched.h +46 -0
index e9b7244ac381..89ee47c2f17d 100644
@@ -1084,4 +1084,50 @@ enum {
 	CAKE_ATM_MAX
 };
 
+
+/* TAPRIO */
+enum {
+	TC_TAPRIO_CMD_SET_GATES = 0x00,
+	TC_TAPRIO_CMD_SET_AND_HOLD = 0x01,
+	TC_TAPRIO_CMD_SET_AND_RELEASE = 0x02,
+};
+
+enum {
+	TCA_TAPRIO_SCHED_ENTRY_UNSPEC,
+	TCA_TAPRIO_SCHED_ENTRY_INDEX, /* u32 */
+	TCA_TAPRIO_SCHED_ENTRY_CMD, /* u8 */
+	TCA_TAPRIO_SCHED_ENTRY_GATE_MASK, /* u32 */
+	TCA_TAPRIO_SCHED_ENTRY_INTERVAL, /* u32 */
+	__TCA_TAPRIO_SCHED_ENTRY_MAX,
+};
+#define TCA_TAPRIO_SCHED_ENTRY_MAX (__TCA_TAPRIO_SCHED_ENTRY_MAX - 1)
+
+/* The format for schedule entry list is:
+ * [TCA_TAPRIO_SCHED_ENTRY_LIST]
+ *   [TCA_TAPRIO_SCHED_ENTRY]
+ *     [TCA_TAPRIO_SCHED_ENTRY_CMD]
+ *     [TCA_TAPRIO_SCHED_ENTRY_GATES]
+ *     [TCA_TAPRIO_SCHED_ENTRY_INTERVAL]
+ */
+enum {
+	TCA_TAPRIO_SCHED_UNSPEC,
+	TCA_TAPRIO_SCHED_ENTRY,
+	__TCA_TAPRIO_SCHED_MAX,
+};
+
+#define TCA_TAPRIO_SCHED_MAX (__TCA_TAPRIO_SCHED_MAX - 1)
+
+enum {
+	TCA_TAPRIO_ATTR_UNSPEC,
+	TCA_TAPRIO_ATTR_PRIOMAP, /* struct tc_mqprio_qopt */
+	TCA_TAPRIO_ATTR_SCHED_ENTRY_LIST, /* nested of entry */
+	TCA_TAPRIO_ATTR_SCHED_BASE_TIME, /* s64 */
+	TCA_TAPRIO_ATTR_SCHED_SINGLE_ENTRY, /* single entry */
+	TCA_TAPRIO_ATTR_SCHED_CLOCKID, /* s32 */
+	TCA_TAPRIO_PAD,
+	__TCA_TAPRIO_ATTR_MAX,
+};
+
+#define TCA_TAPRIO_ATTR_MAX (__TCA_TAPRIO_ATTR_MAX - 1)
+
 #endif
net/sched/Kconfig +11 -0
index e95741388311..1b9afdee5ba9 100644
@@ -194,6 +194,17 @@ config NET_SCH_ETF
 	  To compile this code as a module, choose M here: the
 	  module will be called sch_etf.
 
+config NET_SCH_TAPRIO
+	tristate "Time Aware Priority (taprio) Scheduler"
+	help
+	  Say Y here if you want to use the Time Aware Priority (taprio) packet
+	  scheduling algorithm.
+
+	  See the top of <file:net/sched/sch_taprio.c> for more details.
+
+	  To compile this code as a module, choose M here: the
+	  module will be called sch_taprio.
+
 config NET_SCH_GRED
 	tristate "Generic Random Early Detection (GRED)"
 	---help---
net/sched/Makefile +1 -0
index f0403f49edcb..8a40431d7b5c 100644
@@ -57,6 +57,7 @@ obj-$(CONFIG_NET_SCH_HHF)	+= sch_hhf.o
 obj-$(CONFIG_NET_SCH_PIE)	+= sch_pie.o
 obj-$(CONFIG_NET_SCH_CBS)	+= sch_cbs.o
 obj-$(CONFIG_NET_SCH_ETF)	+= sch_etf.o
+obj-$(CONFIG_NET_SCH_TAPRIO)	+= sch_taprio.o
 
 obj-$(CONFIG_NET_CLS_U32)	+= cls_u32.o
 obj-$(CONFIG_NET_CLS_ROUTE4)	+= cls_route.o
net/sched/sch_taprio.c new file +962 -0

962 lines not on this page · raw message

Signature · 1 line
2.19.0
maintainer Applied
#2 lore raw

Re: [PATCH net-next v1 1/1] tc: Add support for configuring the taprio scheduler

From: Vinicius Costa Gomes <[email protected]>
Date: Fri, 28 Sep 2018 17:59:43 -0700

Quoted 45 lines

This traffic scheduler allows traffic classes states (transmission allowed/not allowed, in the simplest case) to be scheduled, according to a pre-generated time sequence. This is the basis of the IEEE 802.1Qbv specification.

Example configuration:

tc qdisc replace dev enp3s0 parent root handle 100 taprio \

          num_tc 3 \
	  map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 \
	  queues 1@0 1@1 2@2 \
	  base-time 1528743495910289987 \
	  sched-entry S 01 300000 \
	  sched-entry S 02 300000 \
	  sched-entry S 04 300000 \
	  clockid CLOCK_TAI

The configuration format is similar to mqprio. The main difference is the presence of a schedule, built by multiple "sched-entry" definitions, each entry has the following format:

sched-entry <CMD> <GATE MASK> <INTERVAL>

The only supported <CMD> is "S", which means "SetGateStates", following the IEEE 802.1Qbv-2015 definition (Table 8-6). <GATE MASK> is a bitmask where each bit is a associated with a traffic class, so bit 0 (the least significant bit) being "on" means that traffic class 0 is "active" for that schedule entry. <INTERVAL> is a time duration in nanoseconds that specifies for how long that state defined by <CMD> and <GATE MASK> should be held before moving to the next entry.

This schedule is circular, that is, after the last entry is executed it starts from the first one, indefinitely.

The other parameters can be defined as follows:

- base-time: specifies the instant when the schedule starts, if 'base-time' is a time in the past, the schedule will start at

base-time + (N * cycle-time)

where N is the smallest integer so the resulting time is greater than "now", and "cycle-time" is the sum of all the intervals of the entries in the schedule;

Quoted 6 lines

- clockid: specifies the reference clock to be used;

The parameters should be similar to what the IEEE 802.1Q family of specification defines.

Applied.

author maintainer
#3 lore raw
In reply to Vinicius Costa Gomes

Hi,

Just a small correction, one link on the cover letter is wrong.

Vinicius Costa Gomes <[email protected]> writes:

[...]

Quoted 10 lines

[1] https://patchwork.ozlabs.org/cover/938991/

[2] https://patchwork.ozlabs.org/cover/808504/

[3] github doesn't make it clear, but the gist can be cloned like this:

$ git clone https://gist.github.com/jeez/bd3afeff081ba64a695008dd8215866f taprio-test

[4] https://github.com/vcgomes/linux/tree/taprio-v1

The correct link is:

[4] https://github.com/vcgomes/net-next

Quoted 7 lines

[5] https://github.com/vcgomes/iproute2/tree/taprio-v1

Vinicius Costa Gomes (1):
tc: Add support for configuring the taprio scheduler

Quoted 9 lines
 include/uapi/linux/pkt_sched.h |  46 ++
 net/sched/Kconfig              |  11 +
 net/sched/Makefile             |   1 +
 net/sched/sch_taprio.c         | 962 +++++++++++++++++++++++++++++++++
 4 files changed, 1020 insertions(+)
 create mode 100644 net/sched/sch_taprio.c
Signature · 1 line
2.19.0

Cheers,