Skip to contentSkip to conversation

TC: Introduce qevents

RFC PATCH In review Petr Machata 5 participants 26 messages May 26, 2020 – Jun 3, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover TC: Introduce qevents

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

For example:

# tc qdisc add dev eth0 root handle 1: \
	red limit 500K avpkt 1K qevent early block 10
# tc filter add block 10 \
	matchall action mirred egress mirror dev eth1

Patch #1 of this set introduces several helpers to allow easy and uniform addition of qevents to qdiscs. The following two patches, #2 and #3, then add two qevents to the RED qdisc: "early" qevent fires when a packet is early-dropped; "mark" qevent, when it is ECN-marked.

This patch set does not deal with offloading. The idea there is that a driver will be able to figure out that a given block is used in qevent context by looking at binder type. A future patch-set will add a qdisc pointer to struct flow_block_offload, which a driver will be able to consult to glean the TC or other relevant attributes.

Petr Machata (3):
net: sched: Introduce helpers for qevent blocks
net: sched: sch_red: Split init and change callbacks
net: sched: sch_red: Add qevents "early" and "mark"

 include/net/flow_offload.h     |   2 +
 include/net/pkt_cls.h          |  48 +++++++++++++++
 include/uapi/linux/pkt_sched.h |   2 +
 net/sched/cls_api.c            | 107 +++++++++++++++++++++++++++++++++
 net/sched/sch_red.c            | 100 ++++++++++++++++++++++++++----
 5 files changed, 247 insertions(+), 12 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

RFC PATCH 1/3 net: sched: Introduce helpers for qevent blocks

Qevents are attach points for TC blocks, where filters can be put that are executed when "interesting events" take place in a qdisc. The data to keep and the functions to invoke to maintain a qevent will be largely the same between qevents. Therefore introduce sched-wide helpers for qevent management.

Currently, similarly to ingress and egress blocks of clsact pseudo-qdisc, blocks attachment cannot be changed after the qdisc is created. To that end, add a helper tcf_qevent_validate_change(), which verifies whether block index attribute is not attached, or if it is, whether its value matches the current one (i.e. there is no material change).

The function tcf_qevent_handle() is supposed to be invoked when qdisc hits the "interesting event" corresponding to this block.

---
 include/net/pkt_cls.h |  48 +++++++++++++++++++
 net/sched/cls_api.c   | 107 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 155 insertions(+)
include/net/pkt_cls.h +48 -0
index ed65619cbc47..efb20e3c2c98 100644
@@ -32,6 +32,13 @@ struct tcf_block_ext_info {
 	u32 block_index;
 };
 
+struct tcf_qevent {
+	int			attr_name;
+	struct tcf_block	*block;
+	struct tcf_block_ext_info info;
+	struct tcf_proto __rcu *filter_chain;
+};
+
 struct tcf_block_cb;
 bool tcf_queue_work(struct rcu_work *rwork, work_func_t func);
 
@@ -552,6 +559,47 @@ int tc_setup_cb_reoffload(struct tcf_block *block, struct tcf_proto *tp,
 			  void *cb_priv, u32 *flags, unsigned int *in_hw_count);
 unsigned int tcf_exts_num_actions(struct tcf_exts *exts);
 
+#ifdef CONFIG_NET_CLS_ACT
+int tcf_qevent_init(struct tcf_qevent *qe, struct Qdisc *sch,
+		    enum flow_block_binder_type binder_type,
+		    struct nlattr *block_index_attr,
+		    struct netlink_ext_ack *extack);
+void tcf_qevent_destroy(struct tcf_qevent *qe, struct Qdisc *sch);
+int tcf_qevent_validate_change(struct tcf_qevent *qe,
+			       struct nlattr *block_index_attr,
+			       struct netlink_ext_ack *extack);
+struct sk_buff *tcf_qevent_handle(struct tcf_qevent *qe, struct Qdisc *sch,
+				  struct sk_buff *skb, struct sk_buff **to_free,
+				  int *ret);
+#else
+static inline int tcf_qevent_init(struct tcf_qevent *qe, struct Qdisc *sch,
+				  enum flow_block_binder_type binder_type,
+				  struct nlattr *block_index_attr,
+				  struct netlink_ext_ack *extack)
+{
+	return 0;
+}
+
+static inline void tcf_qevent_destroy(struct tcf_qevent *qe, struct Qdisc *sch)
+{
+}
+
+static inline int tcf_qevent_validate_change(struct tcf_qevent *qe,
+					     struct nlattr *block_index_attr,
+					     struct netlink_ext_ack *extack)
+{
+	return 0;
+}
+
+static inline struct sk_buff *
+tcf_qevent_handle(struct tcf_qevent *qe, struct Qdisc *sch,
+		  struct sk_buff *skb, struct sk_buff **to_free,
+		  int *ret)
+{
+	return skb;
+}
+#endif
+
 struct tc_cls_u32_knode {
 	struct tcf_exts *exts;
 	struct tcf_result *res;
net/sched/cls_api.c +107 -0

113 lines not on this page · raw message

Signature · 1 line
2.20.1
#2 lore raw

RFC PATCH 2/3 net: sched: sch_red: Split init and change callbacks

In reply to Petr Machata

In the following patches, RED will get two qevents. The implementation will be clearer if the callback for change is not a pure subset of the callback for init. Split the two and promote attribute parsing to the callbacks themselves from the common code, because it will be handy there.

---
 net/sched/sch_red.c | 43 +++++++++++++++++++++++++++++++------------
 1 file changed, 31 insertions(+), 12 deletions(-)
net/sched/sch_red.c +31 -12
index 555a1b9e467f..c52a40ad5e59 100644
@@ -215,12 +215,11 @@ static const struct nla_policy red_policy[TCA_RED_MAX + 1] = {
 	[TCA_RED_FLAGS] = NLA_POLICY_BITFIELD32(TC_RED_SUPPORTED_FLAGS),
 };
 
-static int red_change(struct Qdisc *sch, struct nlattr *opt,
-		      struct netlink_ext_ack *extack)
+static int __red_change(struct Qdisc *sch, struct nlattr **tb,
+			struct netlink_ext_ack *extack)
 {
 	struct Qdisc *old_child = NULL, *child = NULL;
 	struct red_sched_data *q = qdisc_priv(sch);
-	struct nlattr *tb[TCA_RED_MAX + 1];
 	struct nla_bitfield32 flags_bf;
 	struct tc_red_qopt *ctl;
 	unsigned char userbits;
@@ -228,14 +227,6 @@ static int red_change(struct Qdisc *sch, struct nlattr *opt,
 	int err;
 	u32 max_P;
 
-	if (opt == NULL)
-		return -EINVAL;
-
-	err = nla_parse_nested_deprecated(tb, TCA_RED_MAX, opt, red_policy,
-					  NULL);
-	if (err < 0)
-		return err;
-
 	if (tb[TCA_RED_PARMS] == NULL ||
 	    tb[TCA_RED_STAB] == NULL)
 		return -EINVAL;
@@ -323,11 +314,39 @@ static int red_init(struct Qdisc *sch, struct nlattr *opt,
 		    struct netlink_ext_ack *extack)
 {
 	struct red_sched_data *q = qdisc_priv(sch);
+	struct nlattr *tb[TCA_RED_MAX + 1];
+	int err;
+
+	if (!opt)
+		return -EINVAL;
+
+	err = nla_parse_nested_deprecated(tb, TCA_RED_MAX, opt, red_policy,
+					  extack);
+	if (err < 0)
+		return err;
 
 	q->qdisc = &noop_qdisc;
 	q->sch = sch;
 	timer_setup(&q->adapt_timer, red_adaptative_timer, 0);
-	return red_change(sch, opt, extack);
+	return __red_change(sch, tb, extack);
+}
+
+static int red_change(struct Qdisc *sch, struct nlattr *opt,
+		      struct netlink_ext_ack *extack)
+{
+	struct red_sched_data *q = qdisc_priv(sch);
+	struct nlattr *tb[TCA_RED_MAX + 1];
+	int err;
+
+	if (!opt)
+		return -EINVAL;
+
+	err = nla_parse_nested_deprecated(tb, TCA_RED_MAX, opt, red_policy,
+					  extack);
+	if (err < 0)
+		return err;
+
+	return __red_change(sch, tb, extack);
 }
 
 static int red_dump_offload_stats(struct Qdisc *sch)
Signature · 1 line
2.20.1
maintainer
#3 lore raw

Re: [RFC PATCH net-next 2/3] net: sched: sch_red: Split init and change callbacks

net/sched/sch_red.c static int red_init(struct Qdisc *sch, struct nlattr *opt, · line 337

Quoted 4 lines · On Tue, 26 May 2020 20:10:06 +0300 Petr Machata wrote:
net/sched/sch_red.c +4 -0
+static int red_change(struct Qdisc *sch, struct nlattr *opt,
+		      struct netlink_ext_ack *extack)
+{
+	struct red_sched_data *q = qdisc_priv(sch);

net/sched/sch_red.c: In function red_change:
net/sched/sch_red.c:337:25: warning: unused variable q [-Wunused-variable]
337 | struct red_sched_data *q = qdisc_priv(sch);
| ^

Needs to go to the next patch.

Quoted 13 lines
net/sched/sch_red.c +12 -0
+	struct nlattr *tb[TCA_RED_MAX + 1];
+	int err;
+
+	if (!opt)
+		return -EINVAL;
+
+	err = nla_parse_nested_deprecated(tb, TCA_RED_MAX, opt, red_policy,
+					  extack);
+	if (err < 0)
+		return err;
+
+	return __red_change(sch, tb, extack);
 }
#4 lore raw

Re: [RFC PATCH net-next 2/3] net: sched: sch_red: Split init and change callbacks

Quoted 11 lines · Jakub Kicinski

net/sched/sch_red.c static int red_init(struct Qdisc *sch, struct nlattr *opt, · line 337

Quoted 4 lines · On Tue, 26 May 2020 20:10:06 +0300 Petr Machata wrote:
net/sched/sch_red.c +4 -0
+static int red_change(struct Qdisc *sch, struct nlattr *opt,
+		      struct netlink_ext_ack *extack)
+{
+	struct red_sched_data *q = qdisc_priv(sch);

net/sched/sch_red.c: In function red_change:
net/sched/sch_red.c:337:25: warning: unused variable q [-Wunused-variable]
337 | struct red_sched_data *q = qdisc_priv(sch);
| ^

Quoted 1 line

Needs to go to the next patch.

Indeed. I neglected to do my usual "git rebase -x" dance :-/

#5 lore raw

RFC PATCH 3/3 net: sched: sch_red: Add qevents "early" and "mark"

In reply to Petr Machata

In order to allow acting on dropped and/or ECN-marked packets, add two new qevents to the RED qdisc: "early" and "mark". Filters attached at "early" block are executed as packets are early-dropped, those attached at the "mark" block are executed as packets are ECN-marked.

Two new attributes are introduced: TCA_RED_EARLY_BLOCK with the block index for the "early" qevent, and TCA_RED_MARK_BLOCK for the "mark" qevent. Absence of these attributes signifies "don't care": no block is allocated in that case, or the existing blocks are left intact in case of the change callback.

For purposes of offloading, blocks attached to these qevents appear with newly-introduced binder types, FLOW_BLOCK_BINDER_TYPE_RED_EARLY and FLOW_BLOCK_BINDER_TYPE_RED_MARK.

---
 include/net/flow_offload.h     |  2 ++
 include/uapi/linux/pkt_sched.h |  2 ++
 net/sched/sch_red.c            | 59 +++++++++++++++++++++++++++++++++-
 3 files changed, 62 insertions(+), 1 deletion(-)
include/net/flow_offload.h +2 -0
index 4001ffb04f0d..635d2bb57550 100644
@@ -416,6 +416,8 @@ enum flow_block_binder_type {
 	FLOW_BLOCK_BINDER_TYPE_UNSPEC,
 	FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS,
 	FLOW_BLOCK_BINDER_TYPE_CLSACT_EGRESS,
+	FLOW_BLOCK_BINDER_TYPE_RED_EARLY,
+	FLOW_BLOCK_BINDER_TYPE_RED_MARK,
 };
 
 struct flow_block {
include/uapi/linux/pkt_sched.h +2 -0
index a95f3ae7ab37..ff3f4830e049 100644
@@ -257,6 +257,8 @@ enum {
 	TCA_RED_STAB,
 	TCA_RED_MAX_P,
 	TCA_RED_FLAGS,		/* bitfield32 */
+	TCA_RED_EARLY_BLOCK,	/* u32 */
+	TCA_RED_MARK_BLOCK,	/* u32 */
 	__TCA_RED_MAX,
 };
 
net/sched/sch_red.c +58 -1

107 lines not on this page · raw message

Signature · 1 line
2.20.1
#6 lore raw

RFC PATCH 1/4 uapi: pkt_sched: Add two new RED attributes

In reply to Petr Machata
---
 include/uapi/linux/pkt_sched.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/pkt_sched.h +2 -0
index 0c02737c..203272df 100644
@@ -257,6 +257,8 @@ enum {
 	TCA_RED_STAB,
 	TCA_RED_MAX_P,
 	TCA_RED_FLAGS,		/* bitfield32 */
+	TCA_RED_EARLY_BLOCK,	/* u32 */
+	TCA_RED_MARK_BLOCK,	/* u32 */
 	__TCA_RED_MAX,
 };
 
Signature · 1 line
2.20.1
#7 lore raw

RFC PATCH 2/4 tc: Add helpers to support qevent parsing

Introduce a set of helpers to make it easy to add support for qevents into qdisc.

The idea behind this is that qevent types will be generally reused between qdiscs, rather than each having a completely idiosyncratic set of qevents. The qevent module holds functions for parsing of these common qevent types, and for dispatching to one of the parsers based on the qevent name.

---
 tc/Makefile    |  1 +
 tc/tc_qevent.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tc/tc_qevent.h | 34 +++++++++++++++++++++
 3 files changed, 118 insertions(+)
 create mode 100644 tc/tc_qevent.c
 create mode 100644 tc/tc_qevent.h
tc/Makefile +1 -0
index e31cbc12..10567f33 100644
@@ -121,6 +121,7 @@ TCLIB += tc_red.o
 TCLIB += tc_cbq.o
 TCLIB += tc_estimator.o
 TCLIB += tc_stab.o
+TCLIB += tc_qevent.o
 
 CFLAGS += -DCONFIG_GACT -DCONFIG_GACT_PROB
 ifneq ($(IPT_LIB_DIR),)
tc/tc_qevent.c new file +83 -0

83 lines not on this page · raw message

tc/tc_qevent.h new file +34 -0
new file mode 100644
index 00000000..548f3fc6
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _TC_QEVENT_H_
+#define _TC_QEVENT_H_
+
+#include <linux/types.h>
+
+struct qevent_base {
+	__u32 block_idx;
+};
+
+struct qevent_util {
+	const char *id;
+	int (*parse_qevent)(int *argc, char ***argv, void *data);
+	void *data;
+};
+
+#define QEVENT(_name, _parser, _data)					\
+	{								\
+		.id = _name,						\
+		.parse_qevent = qevent_parse_##_parser,			\
+		.data = ({						\
+				struct qevent_##_parser *__data = _data; \
+				__data;					\
+		}),							\
+	}
+
+int qevent_parse(struct qevent_util *qevents, int *p_argc, char ***p_argv);
+
+struct qevent_plain {
+	struct qevent_base base;
+};
+int qevent_parse_plain(int *p_argc, char ***p_argv, void *p_data);
+
+#endif
Signature · 1 line
2.20.1
#8 lore raw

RFC PATCH 3/4 man: tc: Describe qevents

In reply to Petr Machata

Add some general remarks about qevents.

---
 man/man8/tc.8 | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
man/man8/tc.8 +19 -0
index e8e0cd0f..970440f6 100644
@@ -254,6 +254,25 @@ Traffic control filter that matches every packet. See
 .BR tc-matchall (8)
 for details.
 
+.SH QEVENTS
+Qdiscs may invoke user-configured actions when certain interesting events
+take place in the qdisc. Each qevent can either be unused, or can have a
+block attached to it. To this block are then attached filters using the "tc
+block BLOCK_IDX" syntax. The block is executed when the qevent associated
+with the attachment point takes place. For example, packet could be
+dropped, or delayed, etc., depending on the qdisc and the qevent in
+question.
+
+For example:
+.PP
+.RS
+tc qdisc add dev eth0 root handle 1: red limit 500K avpkt 1K \\
+   qevent early block 10
+.RE
+.RS
+tc filter add block 10 matchall action mirred egress mirror dev eth1
+.RE
+
 .SH CLASSLESS QDISCS
 The classless qdiscs are:
 .TP
Signature · 1 line
2.20.1
#9 lore raw

RFC PATCH 4/4 tc: q_red: Add support for qevents "mark" and "early"

In reply to Petr Machata

The "early" qevent matches packets that have been early-dropped. The "mark" qevent matches packets that have been ECN-marked.

---
 man/man8/tc-red.8 | 18 +++++++++++++++++-
 tc/q_red.c        | 23 ++++++++++++++++++++++-
 2 files changed, 39 insertions(+), 2 deletions(-)
man/man8/tc-red.8 +17 -1
index b5aaa986..e74dd330 100644
@@ -17,7 +17,11 @@ packets
 rate
 .B ] [ probability
 chance
-.B ] [ adaptive ]
+.B ] [ adaptive ] [ qevent early block
+index
+.B ] [ qevent mark block
+index
+.B ]
 
 .SH DESCRIPTION
 Random Early Detection is a classless qdisc which manages its queue size
@@ -134,6 +138,18 @@ Goal of Adaptive RED is to make 'probability' dynamic value between 1% and 50% t
 .B (max - min) / 2
 .fi
 
+.SH QEVENTS
+See tc (8) for some general notes about qevents. The RED qdisc supports the
+following qevents:
+
+.TP
+early
+The associated block is executed when packets are early-dropped. This includes
+non-ECT packets in ECN mode.
+.TP
+mark
+The associated block is executed when packets are marked in ECN mode.
+
 .SH EXAMPLE
 
 .P
tc/q_red.c +22 -1
index 53181c82..7e7dfa05 100644
@@ -22,6 +22,7 @@
 
 #include "utils.h"
 #include "tc_util.h"
+#include "tc_qevent.h"
 
 #include "tc_red.h"
 
@@ -30,7 +31,8 @@ static void explain(void)
 	fprintf(stderr,
 		"Usage: ... red	limit BYTES [min BYTES] [max BYTES] avpkt BYTES [burst PACKETS]\n"
 		"		[adaptive] [probability PROBABILITY] [bandwidth KBPS]\n"
-		"		[ecn] [harddrop] [nodrop]\n");
+		"		[ecn] [harddrop] [nodrop]\n"
+		"		[qevent early block IDX] [qevent mark block IDX]\n");
 }
 
 #define RED_SUPPORTED_FLAGS (TC_RED_HISTORIC_FLAGS | TC_RED_NODROP)
@@ -38,6 +40,14 @@ static void explain(void)
 static int red_parse_opt(struct qdisc_util *qu, int argc, char **argv,
 			 struct nlmsghdr *n, const char *dev)
 {
+	struct qevent_plain qe_early = {};
+	struct qevent_plain qe_mark = {};
+	struct qevent_util qevents[] = {
+		QEVENT("early", plain, &qe_early),
+		QEVENT("mark", plain, &qe_mark),
+		{},
+	};
+
 	struct nla_bitfield32 flags_bf = {
 		.selector = RED_SUPPORTED_FLAGS,
 	};
@@ -109,6 +119,11 @@ static int red_parse_opt(struct qdisc_util *qu, int argc, char **argv,
 			flags_bf.value |= TC_RED_ADAPTATIVE;
 		} else if (strcmp(*argv, "adaptive") == 0) {
 			flags_bf.value |= TC_RED_ADAPTATIVE;
+		} else if (matches(*argv, "qevent") == 0) {
+			NEXT_ARG();
+			if (qevent_parse(qevents, &argc, &argv))
+				return -1;
+			continue;
 		} else if (strcmp(*argv, "help") == 0) {
 			explain();
 			return -1;
@@ -162,6 +177,12 @@ static int red_parse_opt(struct qdisc_util *qu, int argc, char **argv,
 	max_P = probability * pow(2, 32);
 	addattr_l(n, 1024, TCA_RED_MAX_P, &max_P, sizeof(max_P));
 	addattr_l(n, 1024, TCA_RED_FLAGS, &flags_bf, sizeof(flags_bf));
+	if (qe_early.base.block_idx)
+		addattr32(n, 1024, TCA_RED_EARLY_BLOCK,
+			  qe_early.base.block_idx);
+	if (qe_mark.base.block_idx)
+		addattr32(n, 1024, TCA_RED_MARK_BLOCK,
+			  qe_mark.base.block_idx);
 	addattr_nest_end(n, tail);
 	return 0;
 }
Signature · 1 line
2.20.1
#10 lore raw
In reply to Petr Machata
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

So, perhaps adding a "position" in tc filter is better? Something like:

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

And obviously default position must be "enqueue". Makes sense?

(The word "position" may be not accurate, but hope you get my point here.)

Thanks.

#11 lore raw
Quoted 27 lines · Cong Wang
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

Quoted 4 lines

So, perhaps adding a "position" in tc filter is better? Something like:

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

Position would just be a chain index.

Quoted 1 line

And obviously default position must be "enqueue". Makes sense?

Chain 0.

So if I understand the proposal correctly: a qdisc has a classification block (cl_ops->tcf_block). Qevents then reference a chain on that classification block.

If the above is correct, I disagree that this is a better model. RED does not need to classify anything, modelling this as a classification block is wrong. It should be another block. (Also shareable, because as an operator you likely want to treat all, say, early drops the same, and therefore to add just one rule for all 128 or so of your qdiscs.)

#12 lore raw
Quoted 31 lines · Petr Machata
Quoted 27 lines · Cong Wang
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

Quoted 6 lines
Quoted 4 lines

So, perhaps adding a "position" in tc filter is better? Something like:

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

Position would just be a chain index.

Why? By position, I mean a place where we _execute_ tc filters on a qdisc, currently there is only "enqueue". I don't see how this is close to a chain which is basically a group of filters.

Quoted 8 lines
Quoted 2 lines

And obviously default position must be "enqueue". Makes sense?

Chain 0.

So if I understand the proposal correctly: a qdisc has a classification block (cl_ops->tcf_block). Qevents then reference a chain on that classification block.

No, I am suggesting to replace your qevents with position, because as I said it does not fit well there.

Quoted 6 lines

If the above is correct, I disagree that this is a better model. RED does not need to classify anything, modelling this as a classification block is wrong. It should be another block. (Also shareable, because as an operator you likely want to treat all, say, early drops the same, and therefore to add just one rule for all 128 or so of your qdiscs.)

You can still choose not to classify anything by using matchall. No one is saying you have to do classification.

If you want to jump to a block, you can just use a goto action like normal. So your above example can be replaced with:

# tc qdisc add dev eth0 root handle 1: \
        red limit 500K avpkt 1K
# tc filter add dev eth0 parent 1:0 position early_drop matchall \
   action goto chain 10
# tc chain add dev eth0 index 10 ...

See the difference?

Thanks.

#13 lore raw
Quoted 34 lines · Cong Wang
Quoted 33 lines · Petr Machata
Quoted 29 lines · Cong Wang
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

So, perhaps adding a "position" in tc filter is better? Something like:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

Position would just be a chain index.

Why? By position, I mean a place where we _execute_ tc filters on a qdisc, currently there is only "enqueue". I don't see how this is close to a chain which is basically a group of filters.

OK, I misunderstood what you mean.

So you propose to have further division within the block? To have sort of namespaces within blocks or chains, where depending on the context, only filters in the corresponding namespace are executed?

Quoted 13 lines
Quoted 8 lines
Quoted 2 lines

And obviously default position must be "enqueue". Makes sense?

Chain 0.

So if I understand the proposal correctly: a qdisc has a classification block (cl_ops->tcf_block). Qevents then reference a chain on that classification block.

No, I am suggesting to replace your qevents with position, because as I said it does not fit well there.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines

If the above is correct, I disagree that this is a better model. RED does not need to classify anything, modelling this as a classification block is wrong. It should be another block. (Also shareable, because as an operator you likely want to treat all, say, early drops the same, and therefore to add just one rule for all 128 or so of your qdiscs.)

You can still choose not to classify anything by using matchall. No one is saying you have to do classification.

The point here is filter reuse, not classification.

Quoted 1 line

If you want to jump to a block, you can just use a goto action like

I don't think you can jump to a block. You can jump to a chain within the same block.

Quoted 11 lines

normal. So your above example can be replaced with:

# tc qdisc add dev eth0 root handle 1: \
        red limit 500K avpkt 1K
# tc filter add dev eth0 parent 1:0 position early_drop matchall \
   action goto chain 10
# tc chain add dev eth0 index 10 ...

See the difference?

#14 lore raw
Quoted 3 lines · Petr Machata

So you propose to have further division within the block? To have sort of namespaces within blocks or chains, where depending on the context, only filters in the corresponding namespace are executed?

What I suggest is to let filters (or chain or block) decide where they belong to, because I think that fit naturally.

What you suggest is to let qdisc's decide where they put filters, which looks odd to me.

Hope it is all clear now.

Thanks.

#15 lore raw
Quoted 7 lines · Cong Wang
Quoted 3 lines · Petr Machata

So you propose to have further division within the block? To have sort of namespaces within blocks or chains, where depending on the context, only filters in the corresponding namespace are executed?

What I suggest is to let filters (or chain or block) decide where they belong to, because I think that fit naturally.

So filters would have this attribute that marks them for execution in the qevent context?

Does "goto chain" keep this position? I.e. are the executed filters from the "position" context or not? If they keep the position, then this new system can be equivalently modeled as simply a block binding point.

If "goto chain" loses the position, that is just a block binding point and a chain to start on, instead of the default zero.

So introducing the position does not seem to allow us to model anything that could not be modeled before. But it is a more complicated system.

Quoted 2 lines

What you suggest is to let qdisc's decide where they put filters, which looks odd to me.

Ultimately the qdisc decides what qevents to expose. Qevents are closely tied to the inner workings of a qdisc algorithm, they can't be probed as modules the way qdiscs, filters and actions can. If a user wishes to make use of them, they will have to let qdiscs "dictate" where to put the filters one way or another.

#16 lore raw
Quoted 9 lines · Petr Machata
Quoted 5 lines · Cong Wang
Quoted 3 lines · Petr Machata

So you propose to have further division within the block? To have sort of namespaces within blocks or chains, where depending on the context, only filters in the corresponding namespace are executed?

Quoted 5 lines
Quoted 2 lines

What I suggest is to let filters (or chain or block) decide where they belong to, because I think that fit naturally.

So filters would have this attribute that marks them for execution in the qevent context?

If you view it as position rather than qevent, sure, we already need to specify the "context" for tc filter creations anyway, "dev... parent ..." is is context to locate the filter placeholders. This is why it makes sense to add one more piece, say "position", that is "dev... parent... position...".

It is you who calls it qevent which of course looks like only belong to qdisc's.

Quoted 6 lines

Ultimately the qdisc decides what qevents to expose. Qevents are closely tied to the inner workings of a qdisc algorithm, they can't be probed as modules the way qdiscs, filters and actions can. If a user wishes to make use of them, they will have to let qdiscs "dictate" where to put the filters one way or another.

For a specific event like early drop, sure. But if we think it generally, "enqueue" has the same problem, there are a few qdisc's don't even support filtering (noop). We can just return ENOSUPP anyway.

maintainer
#17 lore raw
In reply to Petr Machata
Quoted 46 lines · Thu, May 28, 2020 at 11:48:50AM CEST, [email protected] wrote:
Quoted 34 lines · Cong Wang
Quoted 33 lines · Petr Machata
Quoted 29 lines · Cong Wang
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

So, perhaps adding a "position" in tc filter is better? Something like:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

Position would just be a chain index.

Why? By position, I mean a place where we _execute_ tc filters on a qdisc, currently there is only "enqueue". I don't see how this is close to a chain which is basically a group of filters.

Quoted 5 lines

OK, I misunderstood what you mean.

So you propose to have further division within the block? To have sort of namespaces within blocks or chains, where depending on the context, only filters in the corresponding namespace are executed?

Please take the block as a whole entity. It has one entry ->chain0 processing. The gotos to other chains should be contained within the block.

Please don't divide the block. If you want to process the block from a different entry point, please process it as a whole.

Quoted 23 lines
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines

And obviously default position must be "enqueue". Makes sense?

Chain 0.

So if I understand the proposal correctly: a qdisc has a classification block (cl_ops->tcf_block). Qevents then reference a chain on that classification block.

No, I am suggesting to replace your qevents with position, because as I said it does not fit well there.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines

If the above is correct, I disagree that this is a better model. RED does not need to classify anything, modelling this as a classification block is wrong. It should be another block. (Also shareable, because as an operator you likely want to treat all, say, early drops the same, and therefore to add just one rule for all 128 or so of your qdiscs.)

You can still choose not to classify anything by using matchall. No one is saying you have to do classification.

Quoted 6 lines

The point here is filter reuse, not classification.

Quoted 1 line

If you want to jump to a block, you can just use a goto action like

I don't think you can jump to a block. You can jump to a chain within the same block.

Correct.

entrypoint->
block X

     chain 0
     chain A
     chain B
     chain ...

There's no goto/jump out of the block.

Quoted 12 lines
Quoted 12 lines

normal. So your above example can be replaced with:

# tc qdisc add dev eth0 root handle 1: \
        red limit 500K avpkt 1K
# tc filter add dev eth0 parent 1:0 position early_drop matchall \
   action goto chain 10
# tc chain add dev eth0 index 10 ...

See the difference?

maintainer
#18 lore raw
In reply to Cong Wang
Quoted 27 lines · Wed, May 27, 2020 at 06:09:03AM CEST, [email protected] wrote:
Quoted 13 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

Quoted 7 lines

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

This concept does not fit well into qdisc, essentially you still want to install filters (and actions) somewhere on qdisc, but currently all filters are executed at enqueue, basically you want to execute them at other pre-defined locations too, for example early drop.

Quoted 6 lines

So, perhaps adding a "position" in tc filter is better? Something like:

tc qdisc add dev x root handle 1: ... # same as before
tc filter add dev x parent 1:0 position early_drop matchall action....

And obviously default position must be "enqueue". Makes sense?

Well, if you look at the examples in the cover letter, I think that they are showing something very similar you are talking about:

# tc qdisc add dev eth0 root handle 1: \
        red limit 500K avpkt 1K qevent early block 10
# tc filter add block 10 \
        matchall action mirred egress mirror dev eth1

The first command just says "early drop position should be processed by block 10"

The second command just adds a filter to the block 10.

We have this concept of blocks, we use them in "tc filter" as a handle.

The block as a unit could be attached to be processed not only to "enqueue" but to anything else, like some qdisc stage.

Looks quite neat to me.

Quoted 5 lines

(The word "position" may be not accurate, but hope you get my point here.)

Thanks.

#19 lore raw
Quoted 4 lines · Jiri Pirko

The first command just says "early drop position should be processed by block 10"

The second command just adds a filter to the block 10.

This is exactly why it looks odd to me, because it _reads_ like 'tc qdisc' creates the block to hold tc filters... I think tc filters should create whatever placeholder for themselves.

I know in memory block (or chain or filters) are stored in qdisc, but it is still different to me who initiates the creation.

Thanks.

#20 lore raw
Quoted 9 lines · Cong Wang
Quoted 4 lines · Jiri Pirko

The first command just says "early drop position should be processed by block 10"

The second command just adds a filter to the block 10.

This is exactly why it looks odd to me, because it _reads_ like 'tc qdisc' creates the block to hold tc filters... I think tc filters should create whatever placeholder for themselves.

Look at clsact. It creates blocks in exactly the same way.

Quoted 2 lines

I know in memory block (or chain or filters) are stored in qdisc, but it is still different to me who initiates the creation.

The block binding mechanics are not new. The patch just reuses them. If you are unhappy about how this is currently done, I too would see merit in creating a block explicitly, like with chains. But it has nothing to do with this patchset, which would just naturally pick up whatever this new mechanic is.

maintainer
#21 lore raw
In reply to Cong Wang
Quoted 9 lines · Mon, Jun 01, 2020 at 09:50:23PM CEST, [email protected] wrote:
Quoted 4 lines · Jiri Pirko

The first command just says "early drop position should be processed by block 10"

The second command just adds a filter to the block 10.

This is exactly why it looks odd to me, because it _reads_ like 'tc qdisc' creates the block to hold tc filters... I think tc filters should create whatever placeholder for themselves.

That is how it is done already today. We have the block object. The instances are created separatelly (clsact for example), user may specify the block id to identify the block. Then the user may use this block id to add/remove filters directly to/from the block.

What you propose with "position" on the other hand look unnatural for me. It would slice the exising blocks in some way. Currently, the block has 1 clearly defined entrypoint. With "positions", all of the sudden there would be many of them? I can't really imagine how that is supposed to work :/

Quoted 3 lines

I know in memory block (or chain or filters) are stored in qdisc, but it is still different to me who initiates the creation.

Qdiscs create blocks.

Quoted 2 lines

Thanks.

#22 lore raw
Quoted 17 lines · Jiri Pirko
Quoted 9 lines · Mon, Jun 01, 2020 at 09:50:23PM CEST, [email protected] wrote:
Quoted 4 lines · Jiri Pirko

The first command just says "early drop position should be processed by block 10"

The second command just adds a filter to the block 10.

This is exactly why it looks odd to me, because it _reads_ like 'tc qdisc' creates the block to hold tc filters... I think tc filters should create whatever placeholder for themselves.

That is how it is done already today. We have the block object. The instances are created separatelly (clsact for example), user may specify the block id to identify the block. Then the user may use this block id to add/remove filters directly to/from the block.

Quoted 5 lines

What you propose with "position" on the other hand look unnatural for me. It would slice the exising blocks in some way. Currently, the block has 1 clearly defined entrypoint. With "positions", all of the sudden there would be many of them? I can't really imagine how that is supposed to work :/

I imagine we could introduce multiple blocks for a qdisc.

Currently we have:

struct some_qdisc_data {
struct tcf_block *block;
};

Maybe we can extend it to:

struct some_qdisc_data {
struct tcf_block *blocks[3];
};

#define ENQUEUE 0
#define DEQUEUE 1
#define DROP 2

static struct tcf_block *foo_tcf_block(struct Qdisc *sch, unsigned long cl,
struct netlink_ext_ack
*extack, int position)

{
        struct some_qdisc_data *q = qdisc_priv(sch);

        if (cl)
                return NULL;
        return q->block[position];
}

Just some scratches on my mind.

Thanks.

#23 lore raw
Quoted 1 line · Cong Wang

I imagine we could introduce multiple blocks for a qdisc.

Yes, and that's what the patchset does. If you look at struct tcf_qevent, it is just some block bookkeeping and an attribute name.

Quoted 10 lines

Currently we have:

struct some_qdisc_data {
struct tcf_block *block;
};

Maybe we can extend it to:

struct some_qdisc_data {
struct tcf_block *blocks[3];

Yeah, except not all qdiscs will implement all qevents, so let's instead make it a handful of fields, like in the patchset.

Quoted 9 lines

};

#define ENQUEUE 0
#define DEQUEUE 1
#define DROP 2

static struct tcf_block *foo_tcf_block(struct Qdisc *sch, unsigned long cl,
struct netlink_ext_ack
*extack, int position)

Quoted 7 lines
{
        struct some_qdisc_data *q = qdisc_priv(sch);

        if (cl)
                return NULL;
        return q->block[position];
}

Interestingly, this is close to my original approach, pre-RFC. But there needs to be this global list of all existing qevents. On its own, that's a negative--at least it's an extra uAPI to maintain. What does it bring?

It theoretically allows one to refer to blocks symbolically, through binding point coordinates (dev D parent P qevent Q) not by indices (block B). But then one block could be referenced by several different coordinates, which is confusing. That is the reason TC disallows editing filters on shared blocks. Qevents should be the same.

What else is there?

#24 lore raw
In reply to Petr Machata

Hi Petr,

Quoted 51 lines · Petr Machata

The Spectrum hardware allows execution of one of several actions as a result of queue management events: tail-dropping, early-dropping, marking a packet, or passing a configured latency threshold or buffer size. Such packets can be mirrored, trapped, or sampled.

Modeling the action to be taken as simply a TC action is very attractive, but it is not obvious where to put these actions. At least with ECN marking one could imagine a tree of qdiscs and classifiers that effectively accomplishes this task, albeit in an impractically complex manner. But there is just no way to match on dropped-ness of a packet, let alone dropped-ness due to a particular reason.

To allow configuring user-defined actions as a result of inner workings of a qdisc, this patch set introduces a concept of qevents. Those are attach points for TC blocks, where filters can be put that are executed as the packet hits well-defined points in the qdisc algorithms. The attached blocks can be shared, in a manner similar to clsact ingress and egress blocks, arbitrary classifiers with arbitrary actions can be put on them, etc.

For example:

# tc qdisc add dev eth0 root handle 1: \
        red limit 500K avpkt 1K qevent early block 10
# tc filter add block 10 \
        matchall action mirred egress mirror dev eth1

Patch #1 of this set introduces several helpers to allow easy and uniform addition of qevents to qdiscs. The following two patches, #2 and #3, then add two qevents to the RED qdisc: "early" qevent fires when a packet is early-dropped; "mark" qevent, when it is ECN-marked.

This patch set does not deal with offloading. The idea there is that a driver will be able to figure out that a given block is used in qevent context by looking at binder type. A future patch-set will add a qdisc pointer to struct flow_block_offload, which a driver will be able to consult to glean the TC or other relevant attributes.

Petr Machata (3):
net: sched: Introduce helpers for qevent blocks
net: sched: sch_red: Split init and change callbacks
net: sched: sch_red: Add qevents "early" and "mark"

 include/net/flow_offload.h     |   2 +
 include/net/pkt_cls.h          |  48 +++++++++++++++
 include/uapi/linux/pkt_sched.h |   2 +
 net/sched/cls_api.c            | 107 +++++++++++++++++++++++++++++++++
 net/sched/sch_red.c            | 100 ++++++++++++++++++++++++++----
 5 files changed, 247 insertions(+), 12 deletions(-)
Quoted 3 lines

--
2.20.1

I only took a cursory glance at your patches. Can these "qevents" be added to code outside of the packet scheduler, like to the bridge, for example? Or can the bridge mark the packets somehow, and then any generic qdisc be able to recognize this mark without specific code? A very common use case which is currently not possible to implement is to rate-limit flooded (broadcast, unknown unicast, unknown multicast) traffic. Can your "qevents" be used to describe this, or must it be described separately?

Thanks,
-Vladimir

#25 lore raw
Quoted 8 lines · Vladimir Oltean

I only took a cursory glance at your patches. Can these "qevents" be added to code outside of the packet scheduler, like to the bridge, for example? Or can the bridge mark the packets somehow, and then any generic qdisc be able to recognize this mark without specific code? A very common use case which is currently not possible to implement is to rate-limit flooded (broadcast, unknown unicast, unknown multicast) traffic. Can your "qevents" be used to describe this, or must it be described separately?

You mean something like a "flood" qevent? In principle nothing prevents this, but it does not strike me as a very good fit. These events are meant to be used on qdiscs, hence the "q" in the name. I am not sure it makes sense to reuse them for bridge traffic policing.

Peeking in 802.1Q, I see "Managed objects for per-stream filtering and policing". If that's related, it seems like it would be conservative to model the policing directly in the bridge, instead of this round-about through TC.