net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
Subsystems (4)
Jamal Hadi Salim
maintainerPATCH v2 1/1 net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall as illustrated by [email protected].
Cap the final qdisc_pkt_len() to QDISC_PKT_LEN_MAX so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.
Introduce cap QDISC_PKT_LEN_MAX (1 << 20) = 1 MiB which is well above any legitimate single-skb wire length: the largest current skb->len is GSO_MAX_SIZE (524280), and an ATM-style size table (53/48 cell tax) amplifies that to ~578 KB, both comfortably below 1 MiB. At the same time, 1 MiB bounds the deficit refill loop to ~1M iterations per packet with quantum=1, which completes in a few milliseconds well under the demonstrated softlockup threshold (~10^9 iterations).
Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.
---
v1 -> v2:
- Use a dedicated QDISC_PKT_LEN_MAX define in include/net/pkt_sched.h instead of GSO_MAX_SIZE (Jakub Kicinski, Eric Dumazet). __qdisc_calculate_pkt_len() deals with single-skb wire-length accounting, not GSO segments; GSO_MAX_SIZE is semantically wrong. QDISC_PKT_LEN_MAX = (1 << 20) = 1 MiB avoids truncating legitimate stab accounting (BIG TCP + ATM-style 53/48 table ~ 578 KB) while still bounding the deficit loop to ~1M iterations at quantum=1 (completes in a few ms, well under the softlockup threshold of ~10^9 iterations).
--- include/net/pkt_sched.h | 1 + net/sched/sch_api.c | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-)
include/net/pkt_sched.h +1 -0
@@ -12,6 +12,7 @@#define DEFAULT_TX_QUEUE_LEN 1000#define STAB_SIZE_LOG_MAX 30 +#define QDISC_PKT_LEN_MAX (1 << 20)/* 1 MiB */ struct qdisc_walker {int stop;
net/sched/sch_api.c +5 -2
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,QDISC_PKT_LEN_MAX );qdisc_skb_cb (skb)->pkt_len = pkt_len; }
Signature · 1 line
2.43.0
Paolo Abeni
maintainerThis is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible. Full review at: https://netdev-ai.bots.linux.dev/sashiko/
---
net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
This adds QDISC_PKT_LEN_MAX (1 MiB) and clamps the value computed by __qdisc_calculate_pkt_len() into the range [1, QDISC_PKT_LEN_MAX], with the stated goal of preventing a crafted TCA_STAB size table from driving the DRR and ETS deficit refill loops into a multi-billion iteration spin under the qdisc lock.
A couple of questions about the fix and about the arithmetic it now consumes.
Quoted 17 lines
include/net/pkt_sched.h +1
-0@@ -12,6 +12,7 @@#define DEFAULT_TX_QUEUE_LEN 1000#define STAB_SIZE_LOG_MAX 30 +#define QDISC_PKT_LEN_MAX (1 << 20)/* 1 MiB */ struct qdisc_walker {int stop;net/sched/sch_api.c +0
-0
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 611
Quoted 2 lines
net/sched/sch_api.c +0
-0@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log;
[Severity: Medium]
This isn't a bug introduced by this patch, but the new clamp consumes the result of arithmetic that is done entirely in a signed int on unvalidated user input, so it may not bound what it is intended to bound.
In __qdisc_calculate_pkt_len():
pkt_len = skb->len + stab->szopts.overhead; ... pkt_len = stab->data[stab->szopts.tsize - 1] * (slot / stab->szopts.tsize) + stab->data[slot % stab->szopts.tsize]; pkt_len <<= stab->szopts.size_log;
qdisc_get_stab() only validates tsize consistency and size_log/cell_log against STAB_SIZE_LOG_MAX:
if (s->size_log > STAB_SIZE_LOG_MAX ||
s->cell_log > STAB_SIZE_LOG_MAX) {
NL_SET_ERR_MSG(extack, "Invalid logarithmic size of size table");
return ERR_PTR(-EINVAL);
}The overhead field and the u16 data table are not range checked (the changelog itself notes overhead 999999999 is accepted), and size_log may be up to 30, so the shift overflows for any table entry >= 4.
Is the signed left shift here reportable under CONFIG_UBSAN_SHIFT, and can that be reached by a namespace-local CAP_NET_ADMIN user?
With the usual wrapping behaviour, is the new clamp monotone in the attacker-controlled inputs? For example data[slot] = 65535 with size_log = 16 wraps to -65536, which clamp_t(int, ...) turns into 1, so shaping qdiscs such as TBF, HTB and fq would account one byte for a packet the table describes as huge.
Would doing this arithmetic in u64 (or unsigned with saturation) and narrowing before the clamp make the new bound actually hold?
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 619
Quoted 10 lines
net/sched/sch_api.c +5
-2out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,QDISC_PKT_LEN_MAX );qdisc_skb_cb (skb)->pkt_len = pkt_len; }
[Severity: High]
Does this clamp actually bound the DRR/ETS refill loops, or only one of their inputs?
drr_dequeue() round-robins the active class list, and each unsuccessful visit advances only the head class by one quantum:
net/sched/sch_drr.c:drr_dequeue() {
while (1) {
cl = list_first_entry(&q->active, struct drr_class, alist);
skb = cl->qdisc->ops->peek(cl->qdisc);
...
len = qdisc_pkt_len(skb);
if (len <= cl->deficit) {
...
return skb;
}
WRITE_ONCE(cl->deficit, cl->deficit + READ_ONCE(cl->quantum));
list_move_tail(&cl->alist, &q->active);
}
}Since every active class advances at the same rate, isn't the iteration count before the first packet becomes eligible roughly nr_active_classes * (qdisc_pkt_len / quantum) rather than qdisc_pkt_len / quantum?
The minimum accepted quantum is 1, since drr_change_class() only rejects zero:
net/sched/sch_drr.c:drr_change_class() {
quantum = nla_get_u32(tb[TCA_DRR_QUANTUM]);
if (quantum == 0) {
NL_SET_ERR_MSG(extack, "Specified DRR quantum cannot be zero");
return -EINVAL;
}
}and drr_enqueue() re-arms the worst case on every reactivation:
net/sched/sch_drr.c:drr_enqueue() {
if (!cl_is_active(cl)) {
list_add_tail(&cl->alist, &q->active);
WRITE_ONCE(cl->deficit, READ_ONCE(cl->quantum));
}
}So with quantum 1, the new 2^20 cap and roughly 1024 active classes (class creation is not bounded, and drr_classify() maps skb->priority to a classid so an unprivileged socket can steer packets per class), does a single drr_dequeue() still run on the order of 10^9 iterations with the root qdisc spinlock held and softirqs disabled?
The same shape appears in ets_qdisc_dequeue(), which is the other consumer named in the new comment:
net/sched/sch_ets.c:ets_qdisc_dequeue() {
cl->deficit += READ_ONCE(cl->quantum);
list_move_tail(&cl->alist, &q->active);
}with an added per-iteration scan of the q->nstrict strict bands.
Also, is this reachable with no TCA_STAB at all, for instance ~16k classes with quantum 1 and ordinary 64 KiB packets? If so, the deficit-loop stall does not appear to be gated on size-table amplification in the first place.
And can __qdisc_run() interrupt it?
net/sched/sch_generic.c:__qdisc_run() {
while (qdisc_restart(q, &packets, quota)) {
quota -= packets;
if (quota <= 0) {
...
break;
}
}
}The quota is only evaluated after qdisc_restart() returns, so it cannot preempt the internal refill loop.
Would bounding the loop itself, either with an iteration limit or by requiring a quantum that is sane relative to the maximum pkt_len, be needed instead of clamping one of its inputs?
Signature · 1 line
This is an AI-generated review.
Jamal Hadi Salim
maintainerQuoted 4 lines · Paolo Abeni
This is an AI-generated review of your patch. The human sending this email has considered the AI review valid, or at least plausible. Full review at: https://netdev-ai.bots.linux.dev/sashiko/
I have two followups both targetted for net-next. Both are valid but pre-existing different-root-cause bugs, not things this patch introduces or should fold in tbh.
cheers,
jamal
Quoted 172 lines
---
net/sched: boundqdisc_pkt_lento prevent qdisc soft lockupThis adds
QDISC_PKT_LEN_MAX(1 MiB) and clamps the value computed by__qdisc_calculate_pkt_len()into the range [1,QDISC_PKT_LEN_MAX], with the stated goal of preventing a craftedTCA_STABsize table from driving the DRR and ETS deficit refill loops into a multi-billion iteration spin under the qdisc lock.A couple of questions about the fix and about the arithmetic it now consumes.
Quoted 17 lines
include/net/pkt_sched.h +1
-0@@ -12,6 +12,7 @@#define DEFAULT_TX_QUEUE_LEN 1000#define STAB_SIZE_LOG_MAX 30 +#define QDISC_PKT_LEN_MAX (1 << 20)/* 1 MiB */ struct qdisc_walker {int stop;net/sched/sch_api.c +0
-0net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 611
Quoted 2 lines
net/sched/sch_api.c +0
-0@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log;[Severity: Medium]
This isn't a bug introduced by this patch, but the new clamp consumes the result of arithmetic that is done entirely in a signed int on unvalidated user input, so it may not bound what it is intended to bound.In
__qdisc_calculate_pkt_len():pkt_len = skb->len + stab->szopts.overhead; ... pkt_len = stab->data[stab->szopts.tsize - 1] * (slot / stab->szopts.tsize) + stab->data[slot % stab->szopts.tsize]; pkt_len <<= stab->szopts.size_log;
qdisc_get_stab()only validates tsize consistency and size_log/cell_log againstSTAB_SIZE_LOG_MAX:if (s->size_log > STAB_SIZE_LOG_MAX || s->cell_log > STAB_SIZE_LOG_MAX) { NL_SET_ERR_MSG(extack, "Invalid logarithmic size of size table"); return ERR_PTR(-EINVAL); }The overhead field and the u16 data table are not range checked (the changelog itself notes overhead 999999999 is accepted), and
size_logmay be up to 30, so the shift overflows for any table entry >= 4.Is the signed left shift here reportable under
CONFIG_UBSAN_SHIFT, and can that be reached by a namespace-localCAP_NET_ADMINuser?With the usual wrapping behaviour, is the new clamp monotone in the attacker-controlled inputs? For example data[slot] = 65535 with
size_log= 16 wraps to -65536, whichclamp_t(int, ...)turns into 1, so shaping qdiscs such as TBF, HTB and fq would account one byte for a packet the table describes as huge.Would doing this arithmetic in u64 (or unsigned with saturation) and narrowing before the clamp make the new bound actually hold?
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 619
Quoted 10 lines
net/sched/sch_api.c +5
-2out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,QDISC_PKT_LEN_MAX );qdisc_skb_cb (skb)->pkt_len = pkt_len; }[Severity: High]
Does this clamp actually bound the DRR/ETS refill loops, or only one of their inputs?
drr_dequeue()round-robins the active class list, and each unsuccessful visit advances only the head class by one quantum:net/sched/sch_drr.c:drr_dequeue() { while (1) { cl = list_first_entry(&q->active, struct drr_class, alist); skb = cl->qdisc->ops->peek(cl->qdisc); ... len = qdisc_pkt_len(skb); if (len <= cl->deficit) { ... return skb; } WRITE_ONCE(cl->deficit, cl->deficit + READ_ONCE(cl->quantum)); list_move_tail(&cl->alist, &q->active); } }Since every active class advances at the same rate, isn't the iteration count before the first packet becomes eligible roughly
nr_active_classes* (qdisc_pkt_len/ quantum) rather thanqdisc_pkt_len/ quantum?The minimum accepted quantum is 1, since
drr_change_class()only rejects zero:net/sched/sch_drr.c:drr_change_class() { quantum = nla_get_u32(tb[TCA_DRR_QUANTUM]); if (quantum == 0) { NL_SET_ERR_MSG(extack, "Specified DRR quantum cannot be zero"); return -EINVAL; } }and
drr_enqueue()re-arms the worst case on every reactivation:net/sched/sch_drr.c:drr_enqueue() { if (!cl_is_active(cl)) { list_add_tail(&cl->alist, &q->active); WRITE_ONCE(cl->deficit, READ_ONCE(cl->quantum)); } }So with quantum 1, the new 2^20 cap and roughly 1024 active classes (class creation is not bounded, and
drr_classify()mapsskb->priorityto a classid so an unprivileged socket can steer packets per class), does a singledrr_dequeue()still run on the order of 10^9 iterations with the root qdisc spinlock held and softirqs disabled?The same shape appears in
ets_qdisc_dequeue(), which is the other consumer named in the new comment:net/sched/sch_ets.c:ets_qdisc_dequeue() { cl->deficit += READ_ONCE(cl->quantum); list_move_tail(&cl->alist, &q->active); }with an added per-iteration scan of the
q->nstrictstrict bands.Also, is this reachable with no
TCA_STABat all, for instance ~16k classes with quantum 1 and ordinary 64 KiB packets? If so, the deficit-loop stall does not appear to be gated on size-table amplification in the first place.And can
__qdisc_run()interrupt it?net/sched/sch_generic.c:__qdisc_run() { while (qdisc_restart(q, &packets, quota)) { quota -= packets; if (quota <= 0) { ... break; } } }The quota is only evaluated after
qdisc_restart()returns, so it cannot preempt the internal refill loop.Would bounding the loop itself, either with an iteration limit or by requiring a quantum that is sane relative to the maximum
pkt_len, be needed instead of clamping one of its inputs?
--
This is an AI-generated review.
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 10 lines · On Tue, 25 Aug 2026 04:14:03 -0400 you wrote:
qdisc_get_stab()accepts a user-supplied size table, and__qdisc_calculate_pkt_len()amplifiesqdisc_pkt_len()through the overhead, the size-table data (u16), andsize_log(up toSTAB_SIZE_LOG_MAX). A crafted stab can therefore setqdisc_pkt_len()to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall as illustrated by [email protected].[...]
Here is the summary with links:
- [net,v2] net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
https://git.kernel.org/netdev/net/c/8f735d64382d
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html