Skip to contentSkip to conversation

net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

PATCH Applied Jamal Hadi Salim 3 participants 4 messages Aug 25 – Aug 27

Subsystems (4)
lore
Share & tools
lore

Jamal Hadi Salim

author maintainer
#0 lore raw

PATCH v2 1/1 net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall as illustrated by [email protected].

Cap the final qdisc_pkt_len() to QDISC_PKT_LEN_MAX so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Introduce cap QDISC_PKT_LEN_MAX (1 << 20) = 1 MiB which is well above any legitimate single-skb wire length: the largest current skb->len is GSO_MAX_SIZE (524280), and an ATM-style size table (53/48 cell tax) amplifies that to ~578 KB, both comfortably below 1 MiB. At the same time, 1 MiB bounds the deficit refill loop to ~1M iterations per packet with quantum=1, which completes in a few milliseconds well under the demonstrated softlockup threshold (~10^9 iterations).

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2:
- Use a dedicated QDISC_PKT_LEN_MAX define in include/net/pkt_sched.h instead of GSO_MAX_SIZE (Jakub Kicinski, Eric Dumazet). __qdisc_calculate_pkt_len() deals with single-skb wire-length accounting, not GSO segments; GSO_MAX_SIZE is semantically wrong. QDISC_PKT_LEN_MAX = (1 << 20) = 1 MiB avoids truncating legitimate stab accounting (BIG TCP + ATM-style 53/48 table ~ 578 KB) while still bounding the deficit loop to ~1M iterations at quantum=1 (completes in a few ms, well under the softlockup threshold of ~10^9 iterations).

---
 include/net/pkt_sched.h | 1 +
 net/sched/sch_api.c     | 7 +++++--
 2 files changed, 6 insertions(+), 2 deletions(-)
include/net/pkt_sched.h +1 -0
index 18a419cd9d94..90d3e7943b19 100644
@@ -12,6 +12,7 @@
 
 #define DEFAULT_TX_QUEUE_LEN	1000
 #define STAB_SIZE_LOG_MAX	30
+#define QDISC_PKT_LEN_MAX	(1 << 20)	/* 1 MiB */
 
 struct qdisc_walker {
 	int	stop;
net/sched/sch_api.c +5 -2
index 65b35528d125..90503e59e6e3 100644
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
 	pkt_len <<= stab->szopts.size_log;
 out:
-	if (unlikely(pkt_len < 1))
-		pkt_len = 1;
+	/* A size table can inflate qdisc_pkt_len() beyond any real packet
+	 * (via overhead, the data table, or size_log); cap it so deficit
+	 * schedulers such as DRR/ETS terminate their refill loops.
+	 */
+	pkt_len = clamp_t(int, pkt_len, 1, QDISC_PKT_LEN_MAX);
 	qdisc_skb_cb(skb)->pkt_len = pkt_len;
 }
 
Signature · 1 line
2.43.0
maintainer
#1 lore raw

This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible. Full review at: https://netdev-ai.bots.linux.dev/sashiko/
---
net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

This adds QDISC_PKT_LEN_MAX (1 MiB) and clamps the value computed by __qdisc_calculate_pkt_len() into the range [1, QDISC_PKT_LEN_MAX], with the stated goal of preventing a crafted TCA_STAB size table from driving the DRR and ETS deficit refill loops into a multi-billion iteration spin under the qdisc lock.

A couple of questions about the fix and about the arithmetic it now consumes.

Quoted 17 lines
include/net/pkt_sched.h +1 -0
index 18a419cd9d949..90d3e7943b19a 100644
@@ -12,6 +12,7 @@
 
 #define DEFAULT_TX_QUEUE_LEN	1000
 #define STAB_SIZE_LOG_MAX	30
+#define QDISC_PKT_LEN_MAX	(1 << 20)	/* 1 MiB */
 
 struct qdisc_walker {
 	int	stop;
net/sched/sch_api.c +0 -0
index 65b35528d125f..90503e59e6e3b 100644

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 611

Quoted 2 lines
net/sched/sch_api.c +0 -0
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
 	pkt_len <<= stab->szopts.size_log;

[Severity: Medium]
This isn't a bug introduced by this patch, but the new clamp consumes the result of arithmetic that is done entirely in a signed int on unvalidated user input, so it may not bound what it is intended to bound.

In __qdisc_calculate_pkt_len():

	pkt_len = skb->len + stab->szopts.overhead;
	...
	pkt_len = stab->data[stab->szopts.tsize - 1] *
			(slot / stab->szopts.tsize) +
			stab->data[slot % stab->szopts.tsize];

	pkt_len <<= stab->szopts.size_log;

qdisc_get_stab() only validates tsize consistency and size_log/cell_log against STAB_SIZE_LOG_MAX:

	if (s->size_log > STAB_SIZE_LOG_MAX ||
	    s->cell_log > STAB_SIZE_LOG_MAX) {
		NL_SET_ERR_MSG(extack, "Invalid logarithmic size of size table");
		return ERR_PTR(-EINVAL);
	}

The overhead field and the u16 data table are not range checked (the changelog itself notes overhead 999999999 is accepted), and size_log may be up to 30, so the shift overflows for any table entry >= 4.

Is the signed left shift here reportable under CONFIG_UBSAN_SHIFT, and can that be reached by a namespace-local CAP_NET_ADMIN user?

With the usual wrapping behaviour, is the new clamp monotone in the attacker-controlled inputs? For example data[slot] = 65535 with size_log = 16 wraps to -65536, which clamp_t(int, ...) turns into 1, so shaping qdiscs such as TBF, HTB and fq would account one byte for a packet the table describes as huge.

Would doing this arithmetic in u64 (or unsigned with saturation) and narrowing before the clamp make the new bound actually hold?

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 619

Quoted 10 lines
net/sched/sch_api.c +5 -2
 out:
-	if (unlikely(pkt_len < 1))
-		pkt_len = 1;
+	/* A size table can inflate qdisc_pkt_len() beyond any real packet
+	 * (via overhead, the data table, or size_log); cap it so deficit
+	 * schedulers such as DRR/ETS terminate their refill loops.
+	 */
+	pkt_len = clamp_t(int, pkt_len, 1, QDISC_PKT_LEN_MAX);
 	qdisc_skb_cb(skb)->pkt_len = pkt_len;
 }

[Severity: High]
Does this clamp actually bound the DRR/ETS refill loops, or only one of their inputs?

drr_dequeue() round-robins the active class list, and each unsuccessful visit advances only the head class by one quantum:

net/sched/sch_drr.c:drr_dequeue() {
	while (1) {
		cl = list_first_entry(&q->active, struct drr_class, alist);
		skb = cl->qdisc->ops->peek(cl->qdisc);
		...
		len = qdisc_pkt_len(skb);
		if (len <= cl->deficit) {
			...
			return skb;
		}

		WRITE_ONCE(cl->deficit, cl->deficit + READ_ONCE(cl->quantum));
		list_move_tail(&cl->alist, &q->active);
	}
}

Since every active class advances at the same rate, isn't the iteration count before the first packet becomes eligible roughly nr_active_classes * (qdisc_pkt_len / quantum) rather than qdisc_pkt_len / quantum?

The minimum accepted quantum is 1, since drr_change_class() only rejects zero:

net/sched/sch_drr.c:drr_change_class() {
	quantum = nla_get_u32(tb[TCA_DRR_QUANTUM]);
	if (quantum == 0) {
		NL_SET_ERR_MSG(extack, "Specified DRR quantum cannot be zero");
		return -EINVAL;
	}
}

and drr_enqueue() re-arms the worst case on every reactivation:

net/sched/sch_drr.c:drr_enqueue() {
	if (!cl_is_active(cl)) {
		list_add_tail(&cl->alist, &q->active);
		WRITE_ONCE(cl->deficit, READ_ONCE(cl->quantum));
	}
}

So with quantum 1, the new 2^20 cap and roughly 1024 active classes (class creation is not bounded, and drr_classify() maps skb->priority to a classid so an unprivileged socket can steer packets per class), does a single drr_dequeue() still run on the order of 10^9 iterations with the root qdisc spinlock held and softirqs disabled?

The same shape appears in ets_qdisc_dequeue(), which is the other consumer named in the new comment:

net/sched/sch_ets.c:ets_qdisc_dequeue() {
		cl->deficit += READ_ONCE(cl->quantum);
		list_move_tail(&cl->alist, &q->active);
}

with an added per-iteration scan of the q->nstrict strict bands.

Also, is this reachable with no TCA_STAB at all, for instance ~16k classes with quantum 1 and ordinary 64 KiB packets? If so, the deficit-loop stall does not appear to be gated on size-table amplification in the first place.

And can __qdisc_run() interrupt it?

net/sched/sch_generic.c:__qdisc_run() {
	while (qdisc_restart(q, &packets, quota)) {
		quota -= packets;
		if (quota <= 0) {
			...
			break;
		}
	}
}

The quota is only evaluated after qdisc_restart() returns, so it cannot preempt the internal refill loop.

Would bounding the loop itself, either with an iteration limit or by requiring a quantum that is sane relative to the maximum pkt_len, be needed instead of clamping one of its inputs?

Signature · 1 line
This is an AI-generated review.
author maintainer
#2 lore raw
Quoted 4 lines · Paolo Abeni

This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible. Full review at: https://netdev-ai.bots.linux.dev/sashiko/

I have two followups both targetted for net-next. Both are valid but pre-existing different-root-cause bugs, not things this patch introduces or should fold in tbh.

cheers,
jamal

Quoted 172 lines

---
net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

This adds QDISC_PKT_LEN_MAX (1 MiB) and clamps the value computed by __qdisc_calculate_pkt_len() into the range [1, QDISC_PKT_LEN_MAX], with the stated goal of preventing a crafted TCA_STAB size table from driving the DRR and ETS deficit refill loops into a multi-billion iteration spin under the qdisc lock.

A couple of questions about the fix and about the arithmetic it now consumes.

Quoted 17 lines
include/net/pkt_sched.h +1 -0
index 18a419cd9d949..90d3e7943b19a 100644
@@ -12,6 +12,7 @@
 
 #define DEFAULT_TX_QUEUE_LEN 1000
 #define STAB_SIZE_LOG_MAX    30
+#define QDISC_PKT_LEN_MAX    (1 << 20)       /* 1 MiB */
 
 struct qdisc_walker {
      int     stop;
net/sched/sch_api.c +0 -0
index 65b35528d125f..90503e59e6e3b 100644

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 611

Quoted 2 lines
net/sched/sch_api.c +0 -0
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
      pkt_len <<= stab->szopts.size_log;

[Severity: Medium]
This isn't a bug introduced by this patch, but the new clamp consumes the result of arithmetic that is done entirely in a signed int on unvalidated user input, so it may not bound what it is intended to bound.

In __qdisc_calculate_pkt_len():

        pkt_len = skb->len + stab->szopts.overhead;
        ...
        pkt_len = stab->data[stab->szopts.tsize - 1] *
                        (slot / stab->szopts.tsize) +
                        stab->data[slot % stab->szopts.tsize];

        pkt_len <<= stab->szopts.size_log;

qdisc_get_stab() only validates tsize consistency and size_log/cell_log against STAB_SIZE_LOG_MAX:

        if (s->size_log > STAB_SIZE_LOG_MAX ||
            s->cell_log > STAB_SIZE_LOG_MAX) {
                NL_SET_ERR_MSG(extack, "Invalid logarithmic size of size table");
                return ERR_PTR(-EINVAL);
        }

The overhead field and the u16 data table are not range checked (the changelog itself notes overhead 999999999 is accepted), and size_log may be up to 30, so the shift overflows for any table entry >= 4.

Is the signed left shift here reportable under CONFIG_UBSAN_SHIFT, and can that be reached by a namespace-local CAP_NET_ADMIN user?

With the usual wrapping behaviour, is the new clamp monotone in the attacker-controlled inputs? For example data[slot] = 65535 with size_log = 16 wraps to -65536, which clamp_t(int, ...) turns into 1, so shaping qdiscs such as TBF, HTB and fq would account one byte for a packet the table describes as huge.

Would doing this arithmetic in u64 (or unsigned with saturation) and narrowing before the clamp make the new bound actually hold?

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 619

Quoted 10 lines
net/sched/sch_api.c +5 -2
 out:
-     if (unlikely(pkt_len < 1))
-             pkt_len = 1;
+     /* A size table can inflate qdisc_pkt_len() beyond any real packet
+      * (via overhead, the data table, or size_log); cap it so deficit
+      * schedulers such as DRR/ETS terminate their refill loops.
+      */
+     pkt_len = clamp_t(int, pkt_len, 1, QDISC_PKT_LEN_MAX);
      qdisc_skb_cb(skb)->pkt_len = pkt_len;
 }

[Severity: High]
Does this clamp actually bound the DRR/ETS refill loops, or only one of their inputs?

drr_dequeue() round-robins the active class list, and each unsuccessful visit advances only the head class by one quantum:

net/sched/sch_drr.c:drr_dequeue() {
        while (1) {
                cl = list_first_entry(&q->active, struct drr_class, alist);
                skb = cl->qdisc->ops->peek(cl->qdisc);
                ...
                len = qdisc_pkt_len(skb);
                if (len <= cl->deficit) {
                        ...
                        return skb;
                }

                WRITE_ONCE(cl->deficit, cl->deficit + READ_ONCE(cl->quantum));
                list_move_tail(&cl->alist, &q->active);
        }
}

Since every active class advances at the same rate, isn't the iteration count before the first packet becomes eligible roughly nr_active_classes * (qdisc_pkt_len / quantum) rather than qdisc_pkt_len / quantum?

The minimum accepted quantum is 1, since drr_change_class() only rejects zero:

net/sched/sch_drr.c:drr_change_class() {
        quantum = nla_get_u32(tb[TCA_DRR_QUANTUM]);
        if (quantum == 0) {
                NL_SET_ERR_MSG(extack, "Specified DRR quantum cannot be zero");
                return -EINVAL;
        }
}

and drr_enqueue() re-arms the worst case on every reactivation:

net/sched/sch_drr.c:drr_enqueue() {
        if (!cl_is_active(cl)) {
                list_add_tail(&cl->alist, &q->active);
                WRITE_ONCE(cl->deficit, READ_ONCE(cl->quantum));
        }
}

So with quantum 1, the new 2^20 cap and roughly 1024 active classes (class creation is not bounded, and drr_classify() maps skb->priority to a classid so an unprivileged socket can steer packets per class), does a single drr_dequeue() still run on the order of 10^9 iterations with the root qdisc spinlock held and softirqs disabled?

The same shape appears in ets_qdisc_dequeue(), which is the other consumer named in the new comment:

net/sched/sch_ets.c:ets_qdisc_dequeue() {
                cl->deficit += READ_ONCE(cl->quantum);
                list_move_tail(&cl->alist, &q->active);
}

with an added per-iteration scan of the q->nstrict strict bands.

Also, is this reachable with no TCA_STAB at all, for instance ~16k classes with quantum 1 and ordinary 64 KiB packets? If so, the deficit-loop stall does not appear to be gated on size-table amplification in the first place.

And can __qdisc_run() interrupt it?

net/sched/sch_generic.c:__qdisc_run() {
        while (qdisc_restart(q, &packets, quota)) {
                quota -= packets;
                if (quota <= 0) {
                        ...
                        break;
                }
        }
}

The quota is only evaluated after qdisc_restart() returns, so it cannot preempt the internal refill loop.

Would bounding the loop itself, either with an iteration limit or by requiring a quantum that is sane relative to the maximum pkt_len, be needed instead of clamping one of its inputs?
--
This is an AI-generated review.

#3 lore raw
In reply to Jamal Hadi Salim
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 10 lines · On Tue, 25 Aug 2026 04:14:03 -0400 you wrote:

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall as illustrated by [email protected].

[...]

Here is the summary with links:
- [net,v2] net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

    https://git.kernel.org/netdev/net/c/8f735d64382d

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html