Skip to contentSkip to conversation

sch_htb: Avoid grafting on htb_destroy_class_offload when destroying htb

PATCH Superseded Rahul Rameshbabu 2 participants 3 messages Jan 11, 2023 – Jan 12, 2023

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sch_htb: Avoid grafting on htb_destroy_class_offload when destroying htb

When destroying the htb, the caller may already have grafted a new qdisc that is not part of the htb structure being destroyed. htb_destroy_class_offload should not peek at the qdisc of the netdev queue. Peek at old qdisc and graft only when deleting a leaf class in the htb, rather than when deleting the htb itself.

This fix resolves two use cases.

1. Using tc to destroy the htb.
2. Using tc to replace the htb with another qdisc (which also leads to the htb being destroyed).

Fixes d03b195b5aa0 ("sch_htb: Hierarchical QoS hardware offload")
Cc Eric Dumazet <[email protected]>
Cc Maxim Mikityanskiy <[email protected]>
---
 net/sched/sch_htb.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)
net/sched/sch_htb.c +13 -10
index 2238edece1a4..360ce8616fd2 100644
@@ -1557,14 +1557,13 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 
 	WARN_ON(!q);
 	dev_queue = htb_offload_get_queue(cl);
-	old = htb_graft_helper(dev_queue, NULL);
-	if (destroying)
-		/* Before HTB is destroyed, the kernel grafts noop_qdisc to
-		 * all queues.
+	if (!destroying) {
+		old = htb_graft_helper(dev_queue, NULL);
+		/* Last qdisc grafted should be the same as cl->leaf.q when
+		 * calling htb_destroy
 		 */
-		WARN_ON(!(old->flags & TCQ_F_BUILTIN));
-	else
 		WARN_ON(old != q);
+	}
 
 	if (cl->parent) {
 		_bstats_update(&cl->parent->bstats_bias,
@@ -1581,10 +1580,14 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 	};
 	err = htb_offload(qdisc_dev(sch), &offload_opt);
 
-	if (!err || destroying)
-		qdisc_put(old);
-	else
-		htb_graft_helper(dev_queue, old);
+	/* htb_offload related errors when destroying cannot be handled */
+	WARN_ON(err && destroying);
+	if (!destroying) {
+		if (!err)
+			qdisc_put(old);
+		else
+			htb_graft_helper(dev_queue, old);
+	}
 
 	if (last_child)
 		return err;
Signature · 7 lines
2.36.2

Previous related discussions

[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/all/CANn89iJSsFPBp5dYm3y6Jbbpuwbb9P+X3gmqk6zow0VWgx1Q-A@mail.gmail.com/
#1 lore raw
Quoted 11 lines · Rahul Rameshbabu

When destroying the htb, the caller may already have grafted a new qdisc that is not part of the htb structure being destroyed. htb_destroy_class_offload should not peek at the qdisc of the netdev queue. Peek at old qdisc and graft only when deleting a leaf class in the htb, rather than when deleting the htb itself.

This fix resolves two use cases.

1. Using tc to destroy the htb.
2. Using tc to replace the htb with another qdisc (which also leads to the htb being destroyed).

Please elaborate in the commit message what exactly was broken in these cases, i.e. premature dev_activate in both cases, and also accidental overwriting of the qdisc in case 2.

Quoted 15 lines
Fixes d03b195b5aa0 ("sch_htb: Hierarchical QoS hardware offload")
Cc Eric Dumazet <[email protected]>
Cc Maxim Mikityanskiy <[email protected]>
---
 net/sched/sch_htb.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)
net/sched/sch_htb.c +0 -0
index 2238edece1a4..360ce8616fd2 100644
@@ -1557,14 +1557,13 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1563

Quoted 10 lines
net/sched/sch_htb.c +4 -4
 	WARN_ON(!q);
 	dev_queue = htb_offload_get_queue(cl);
-	old = htb_graft_helper(dev_queue, NULL);
-	if (destroying)
-		/* Before HTB is destroyed, the kernel grafts noop_qdisc to
-		 * all queues.
+	if (!destroying) {
+		old = htb_graft_helper(dev_queue, NULL);
+		/* Last qdisc grafted should be the same as cl->leaf.q when
+		 * calling htb_destroy

Did you mean "when calling htb_delete"?

Worth also commenting that on destroying, graft is done by qdisc_graft, and the latter also qdisc_puts the old one. Just to explain why we skip steps on destroying.

Quoted 9 lines
net/sched/sch_htb.c +1 -2
 		 */
-		WARN_ON(!(old->flags & TCQ_F_BUILTIN));
-	else
 		WARN_ON(old != q);
+	}
 
 	if (cl->parent) {
 		_bstats_update(&cl->parent->bstats_bias,

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1584

Quoted 9 lines
net/sched/sch_htb.c +2 -4
@@ -1581,10 +1580,14 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 	};
 	err = htb_offload(qdisc_dev(sch), &offload_opt);
 
-	if (!err || destroying)
-		qdisc_put(old);
-	else
-		htb_graft_helper(dev_queue, old);
+	/* htb_offload related errors when destroying cannot be handled */
+	WARN_ON(err && destroying);

Not sure whether we want to WARN on this error...

On destroying, we call htb_offload with TC_HTB_LEAF_DEL_LAST_FORCE, which makes the mlx5e driver proceed with deleting the node even if it failed to create a replacement node. Normally it cancels the deletion to keep the integrity of hardware structures, but on htb_destroy it doesn't matter, because everything is going to be torn down anyway. An error is still returned by the driver, but it's safe to ignore it, not worth a WARN at all.

Another error flow, when the firmware command to delete a node fails for some reason, doesn't even lead to returning an error, because the worst that happens is a leak of hardware resources, and we can't do anything meaningful about it at that stage.

So, I don't think this WARN_ON is helpful, unless you also want to change the way mlx5e returns errors.

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1590

Quoted 6 lines
net/sched/sch_htb.c +6 -0
+	if (!destroying) {
+		if (!err)
+			qdisc_put(old);
+		else
+			htb_graft_helper(dev_queue, old);
+	}

Looks good. I also suggest removing NULL-initialization of old to make sure one will get a compiler warning about an uninitialized variable if one changes the code in the future and accidentally uses old in the destroying flow.

Quoted 11 lines
net/sched/sch_htb.c +0 -0
 	if (last_child)
 		return err;
Signature · 7 lines
2.36.2

Previous related discussions

[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/all/CANn89iJSsFPBp5dYm3y6Jbbpuwbb9P+X3gmqk6zow0VWgx1Q-A@mail.gmail.com/
#2 lore raw
Quoted 9 lines · Maxim Mikityanskiy
Quoted 8 lines · Rahul Rameshbabu

When destroying the htb, the caller may already have grafted a new qdisc that is not part of the htb structure being destroyed. htb_destroy_class_offload should not peek at the qdisc of the netdev queue. Peek at old qdisc and graft only when deleting a leaf class in the htb, rather than when deleting the htb itself.

This fix resolves two use cases.

Quoted 7 lines
Quoted 3 lines

1. Using tc to destroy the htb.
2. Using tc to replace the htb with another qdisc (which also leads to the htb being destroyed).

Please elaborate in the commit message what exactly was broken in these cases, i.e. premature dev_activate in both cases, and also accidental overwriting of the qdisc in case 2.

Ack.

Quoted 29 lines
Quoted 16 lines
Fixes d03b195b5aa0 ("sch_htb: Hierarchical QoS hardware offload")
Cc Eric Dumazet <[email protected]>
Cc Maxim Mikityanskiy <[email protected]>
---
 net/sched/sch_htb.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)
net/sched/sch_htb.c +0 -0
index 2238edece1a4..360ce8616fd2 100644
@@ -1557,14 +1557,13 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1563

Quoted 10 lines
net/sched/sch_htb.c +4 -4
 	WARN_ON(!q);
 	dev_queue = htb_offload_get_queue(cl);
-	old = htb_graft_helper(dev_queue, NULL);
-	if (destroying)
-		/* Before HTB is destroyed, the kernel grafts noop_qdisc to
-		 * all queues.
+	if (!destroying) {
+		old = htb_graft_helper(dev_queue, NULL);
+		/* Last qdisc grafted should be the same as cl->leaf.q when
+		 * calling htb_destroy

Did you mean "when calling htb_delete"?

Quoted 4 lines

Worth also commenting that on destroying, graft is done by qdisc_graft, and the latter also qdisc_puts the old one. Just to explain why we skip steps on destroying.

Yes, I did mean htb_delete. Ack.

Quoted 21 lines
Quoted 9 lines
net/sched/sch_htb.c +1 -2
 		 */
-		WARN_ON(!(old->flags & TCQ_F_BUILTIN));
-	else
 		WARN_ON(old != q);
+	}
 
 	if (cl->parent) {
 		_bstats_update(&cl->parent->bstats_bias,

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1584

Quoted 9 lines
net/sched/sch_htb.c +2 -4
@@ -1581,10 +1580,14 @@ static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl,
 	};
 	err = htb_offload(qdisc_dev(sch), &offload_opt);
 
-	if (!err || destroying)
-		qdisc_put(old);
-	else
-		htb_graft_helper(dev_queue, old);
+	/* htb_offload related errors when destroying cannot be handled */
+	WARN_ON(err && destroying);

Not sure whether we want to WARN on this error...

Quoted 7 lines

On destroying, we call htb_offload with TC_HTB_LEAF_DEL_LAST_FORCE, which makes the mlx5e driver proceed with deleting the node even if it failed to create a replacement node. Normally it cancels the deletion to keep the integrity of hardware structures, but on htb_destroy it doesn't matter, because everything is going to be torn down anyway. An error is still returned by the driver, but it's safe to ignore it, not worth a WARN at all.

I see. This makes sense to me.

Quoted 5 lines

Another error flow, when the firmware command to delete a node fails for some reason, doesn't even lead to returning an error, because the worst that happens is a leak of hardware resources, and we can't do anything meaningful about it at that stage.

This was what I was trying to catch with the WARN_ON, with the hope that at worst it wouldn't have any false positives. However, if there are errors due to certain operation modes like TC_HTB_LEAF_DEL_LAST_FORCE where the htb is still destroyed, this WARN_ON seems to be problematic more than helpful. Will remove in my next revision.

Quoted 4 lines

So, I don't think this WARN_ON is helpful, unless you also want to change the way mlx5e returns errors.

Quoted 11 lines

net/sched/sch_htb.c static int htb_destroy_class_offload(struct Qdisc *sch, struct htb_class *cl, · line 1590

Quoted 6 lines
net/sched/sch_htb.c +6 -0
+	if (!destroying) {
+		if (!err)
+			qdisc_put(old);
+		else
+			htb_graft_helper(dev_queue, old);
+	}

Looks good. I also suggest removing NULL-initialization of old to make sure one will get a compiler warning about an uninitialized variable if one changes the code in the future and accidentally uses old in the destroying flow.

Ack.

Quoted 12 lines
Quoted 12 lines
net/sched/sch_htb.c +0 -0
 	if (last_child)
 		return err;
Signature · 7 lines
2.36.2

Previous related discussions

[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/all/CANn89iJSsFPBp5dYm3y6Jbbpuwbb9P+X3gmqk6zow0VWgx1Q-A@mail.gmail.com/