net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
Subsystems (4)
Jamal Hadi Salim
maintainerPATCH v2 1/1 net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup
qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.
Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.
Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.
---
v1 -> v2
1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.
2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions
3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.
[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]
--- net/sched/sch_api.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +5 -2
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,GSO_MAX_SIZE );qdisc_skb_cb (skb)->pkt_len = pkt_len; }
Signature · 1 line
2.43.0
Quoted 54 lines · Jamal Hadi Salim
qdisc_get_stab()accepts a user-supplied size table, and__qdisc_calculate_pkt_len()amplifiesqdisc_pkt_len()through the overhead, the size-table data (u16), andsize_log(up toSTAB_SIZE_LOG_MAX). A crafted stab can therefore setqdisc_pkt_len()to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.Cap the final
qdisc_pkt_len()toGSO_MAX_SIZEso the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.Conditions to recreate the bug:
-CONFIG_NET_SCHED=y,CONFIG_NET_SCH_DRR=y(orCONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a craftedTCA_STABthat amplifiesqdisc_pkt_lento ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-localCAP_NET_ADMIN.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by [email protected]Tested-by Victor Nogueira <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>---
v1 -> v21. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole
tdc.shinstead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.2. Better Fix: cap the final
qdisc_pkt_len()toGSO_MAX_SIZE(524280)in__qdisc_calculate_pkt_len()per sashikos[1][2] suggestions3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.
[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]--- net/sched/sch_api.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)net/sched/sch_api.c +0
-0
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617
Quoted 10 lines
net/sched/sch_api.c +5
-2@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,GSO_MAX_SIZE );
Yeah, although it is a bit strange to mention GSO_MAX_SIZE in a function which does not take care of GSO packets.
Jamal Hadi Salim
maintainerQuoted 57 lines · Eric Dumazet
Quoted 55 lines · Jamal Hadi Salim
qdisc_get_stab()accepts a user-supplied size table, and__qdisc_calculate_pkt_len()amplifiesqdisc_pkt_len()through the overhead, the size-table data (u16), andsize_log(up toSTAB_SIZE_LOG_MAX). A crafted stab can therefore setqdisc_pkt_len()to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.Cap the final
qdisc_pkt_len()toGSO_MAX_SIZEso the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.Conditions to recreate the bug:
-CONFIG_NET_SCHED=y,CONFIG_NET_SCH_DRR=y(orCONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a craftedTCA_STABthat amplifiesqdisc_pkt_lento ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-localCAP_NET_ADMIN.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by [email protected]Tested-by Victor Nogueira <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>---
v1 -> v21. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole
tdc.shinstead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.2. Better Fix: cap the final
qdisc_pkt_len()toGSO_MAX_SIZE(524280)in__qdisc_calculate_pkt_len()per sashikos[1][2] suggestions3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.
[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]--- net/sched/sch_api.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)net/sched/sch_api.c +0
-0@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
Quoted 12 lines
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617
Quoted 9 lines
net/sched/sch_api.c +5
-2pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,GSO_MAX_SIZE );Yeah, although it is a bit strange to mention
GSO_MAX_SIZEin a function which does not take care of GSO packets.
well... there are many many references to gso under net/sched. What dont you like about this specific things?
cheers,
jamal
Quoted 72 lines · Jamal Hadi Salim
Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jamal Hadi Salim
qdisc_get_stab()accepts a user-supplied size table, and__qdisc_calculate_pkt_len()amplifiesqdisc_pkt_len()through the overhead, the size-table data (u16), andsize_log(up toSTAB_SIZE_LOG_MAX). A crafted stab can therefore setqdisc_pkt_len()to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.Cap the final
qdisc_pkt_len()toGSO_MAX_SIZEso the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.Conditions to recreate the bug:
-CONFIG_NET_SCHED=y,CONFIG_NET_SCH_DRR=y(orCONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a craftedTCA_STABthat amplifiesqdisc_pkt_lento ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-localCAP_NET_ADMIN.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by [email protected]Tested-by Victor Nogueira <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>---
v1 -> v21. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole
tdc.shinstead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.2. Better Fix: cap the final
qdisc_pkt_len()toGSO_MAX_SIZE(524280)in__qdisc_calculate_pkt_len()per sashikos[1][2] suggestions3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.
[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]--- net/sched/sch_api.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)net/sched/sch_api.c +0
-0@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,Quoted 12 lines
net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617
Quoted 9 lines
net/sched/sch_api.c +5
-2pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,GSO_MAX_SIZE );Yeah, although it is a bit strange to mention
GSO_MAX_SIZEin a function which does not take care of GSO packets.
Quoted 2 lines
well... there are many many references to gso under net/sched. What dont you like about this specific things?
I just mentioned that __qdisc_calculate_pkt_len() never was updated to deal with GSO packets.
It assumes packets with a single set of headers.
This is fine, I am sure nobody uses this stuff anymore.
Jamal Hadi Salim
maintainerQuoted 74 lines · Eric Dumazet
Quoted 72 lines · Jamal Hadi Salim
Quoted 69 lines · Eric Dumazet
Quoted 54 lines · Jamal Hadi Salim
qdisc_get_stab()accepts a user-supplied size table, and__qdisc_calculate_pkt_len()amplifiesqdisc_pkt_len()through the overhead, the size-table data (u16), andsize_log(up toSTAB_SIZE_LOG_MAX). A crafted stab can therefore setqdisc_pkt_len()to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.Cap the final
qdisc_pkt_len()toGSO_MAX_SIZEso the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.Conditions to recreate the bug:
-CONFIG_NET_SCHED=y,CONFIG_NET_SCH_DRR=y(orCONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a craftedTCA_STABthat amplifiesqdisc_pkt_lento ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-localCAP_NET_ADMIN.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by [email protected]Tested-by Victor Nogueira <[email protected]>Signed-off-by Jamal Hadi Salim <[email protected]>---
v1 -> v21. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole
tdc.shinstead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.2. Better Fix: cap the final
qdisc_pkt_len()toGSO_MAX_SIZE(524280)in__qdisc_calculate_pkt_len()per sashikos[1][2] suggestions3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.
[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]--- net/sched/sch_api.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)net/sched/sch_api.c +0
-0net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617
Quoted 10 lines
net/sched/sch_api.c +5
-2@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log; out: -if (unlikely (pkt_len <1 )) - pkt_len =1 ; +/* A size table can inflate qdisc_pkt_len() beyond any real packet +* (via overhead, the data table, or size_log); cap it so deficit +* schedulers such as DRR/ETS terminate their refill loops. +*/ + pkt_len =clamp_t (int, pkt_len,1 ,GSO_MAX_SIZE );Yeah, although it is a bit strange to mention
GSO_MAX_SIZEin a function which does not take care of GSO packets.
Quoted 8 lines
Quoted 2 lines
well... there are many many references to gso under net/sched. What dont you like about this specific things?
I just mentioned that
__qdisc_calculate_pkt_len()never was updated to deal with GSO packets.It assumes packets with a single set of headers.
I see. I was looking for something reasonable. The problem is maximum real skb->len is UINT_MAX and with stab amplification it can get a small packet into a GB.
Would you prefer a new #define in pkt_sched.h to put an upper bound (QDISC_PKT_LEN_MAX)?
Something else?
cheers,
jamal
Quoted 1 line
This is fine, I am sure nobody uses this stuff anymore.
Jakub Kicinski
maintainerQuoted 6 lines · On Wed, 19 Aug 2026 11:30:44 -0400 Jamal Hadi Salim wrote:
I see. I was looking for something reasonable. The problem is maximum real
skb->lenisUINT_MAXand with stab amplification it can get a small packet into a GB.
Would you prefer a new #define inpkt_sched.hto put an upper bound (QDISC_PKT_LEN_MAX)?
Something else?
Let's go with the define you mentioned to make progress.
Signature · 1 line
pw-bot: cr