Skip to contentSkip to conversation

net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

PATCH Superseded Jamal Hadi Salim 3 participants 6 messages Aug 19 – Aug 24

Subsystems (4)
lore
Share & tools
lore

Jamal Hadi Salim

author maintainer
#0 lore raw

PATCH v2 1/1 net/sched: bound qdisc_pkt_len to prevent qdisc soft lockup

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.

Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2

1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.

2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions

3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.

[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]

---
 net/sched/sch_api.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +5 -2
index 65b35528d125..ad4f117ff55a 100644
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
 	pkt_len <<= stab->szopts.size_log;
 out:
-	if (unlikely(pkt_len < 1))
-		pkt_len = 1;
+	/* A size table can inflate qdisc_pkt_len() beyond any real packet
+	 * (via overhead, the data table, or size_log); cap it so deficit
+	 * schedulers such as DRR/ETS terminate their refill loops.
+	 */
+	pkt_len = clamp_t(int, pkt_len, 1, GSO_MAX_SIZE);
 	qdisc_skb_cb(skb)->pkt_len = pkt_len;
 }
 
Signature · 1 line
2.43.0
#1 lore raw
Quoted 54 lines · Jamal Hadi Salim

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.

Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2

1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.

2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions

3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.

[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]

---
 net/sched/sch_api.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +0 -0
index 65b35528d125..ad4f117ff55a 100644

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617

Quoted 10 lines
net/sched/sch_api.c +5 -2
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
        pkt_len <<= stab->szopts.size_log;
 out:
-       if (unlikely(pkt_len < 1))
-               pkt_len = 1;
+       /* A size table can inflate qdisc_pkt_len() beyond any real packet
+        * (via overhead, the data table, or size_log); cap it so deficit
+        * schedulers such as DRR/ETS terminate their refill loops.
+        */
+       pkt_len = clamp_t(int, pkt_len, 1, GSO_MAX_SIZE);

Yeah, although it is a bit strange to mention GSO_MAX_SIZE in a function which does not take care of GSO packets.

author maintainer
#2 lore raw
Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jamal Hadi Salim

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.

Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2

1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.

2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions

3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.

[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]

---
 net/sched/sch_api.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +0 -0
index 65b35528d125..ad4f117ff55a 100644
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
Quoted 12 lines

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617

Quoted 9 lines
net/sched/sch_api.c +5 -2
        pkt_len <<= stab->szopts.size_log;
 out:
-       if (unlikely(pkt_len < 1))
-               pkt_len = 1;
+       /* A size table can inflate qdisc_pkt_len() beyond any real packet
+        * (via overhead, the data table, or size_log); cap it so deficit
+        * schedulers such as DRR/ETS terminate their refill loops.
+        */
+       pkt_len = clamp_t(int, pkt_len, 1, GSO_MAX_SIZE);

Yeah, although it is a bit strange to mention GSO_MAX_SIZE in a function which does not take care of GSO packets.

well... there are many many references to gso under net/sched. What dont you like about this specific things?

cheers,
jamal

#3 lore raw
Quoted 72 lines · Jamal Hadi Salim
Quoted 57 lines · Eric Dumazet
Quoted 55 lines · Jamal Hadi Salim

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.

Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2

1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.

2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions

3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.

[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]

---
 net/sched/sch_api.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +0 -0
index 65b35528d125..ad4f117ff55a 100644
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
Quoted 12 lines

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617

Quoted 9 lines
net/sched/sch_api.c +5 -2
        pkt_len <<= stab->szopts.size_log;
 out:
-       if (unlikely(pkt_len < 1))
-               pkt_len = 1;
+       /* A size table can inflate qdisc_pkt_len() beyond any real packet
+        * (via overhead, the data table, or size_log); cap it so deficit
+        * schedulers such as DRR/ETS terminate their refill loops.
+        */
+       pkt_len = clamp_t(int, pkt_len, 1, GSO_MAX_SIZE);

Yeah, although it is a bit strange to mention GSO_MAX_SIZE in a function which does not take care of GSO packets.

Quoted 2 lines

well... there are many many references to gso under net/sched. What dont you like about this specific things?

I just mentioned that __qdisc_calculate_pkt_len() never was updated to deal with GSO packets.

It assumes packets with a single set of headers.

This is fine, I am sure nobody uses this stuff anymore.

author maintainer
#4 lore raw
Quoted 74 lines · Eric Dumazet
Quoted 72 lines · Jamal Hadi Salim
Quoted 69 lines · Eric Dumazet
Quoted 54 lines · Jamal Hadi Salim

qdisc_get_stab() accepts a user-supplied size table, and __qdisc_calculate_pkt_len() amplifies qdisc_pkt_len() through the overhead, the size-table data (u16), and size_log (up to STAB_SIZE_LOG_MAX). A crafted stab can therefore set qdisc_pkt_len() to ~1 GiB for an ordinary skb. Per-flow deficit schedulers such as DRR and ETS replenish one quantum per loop iteration; with a tiny quantum (1) they spin billions of times under the qdisc lock, producing a soft lockup / RCU stall.

Cap the final qdisc_pkt_len() to GSO_MAX_SIZE so the size-table amplification cannot drive deficit schedulers into an unbounded loop. A legitimate size table (e.g. qfq's overhead 999999999, which is handled by dropping) is still accepted.

Conditions to recreate the bug:
- CONFIG_NET_SCHED=y, CONFIG_NET_SCH_DRR=y (or CONFIG_NET_SCH_ETS=y).
- Attach a DRR (or ETS) root qdisc with a crafted TCA_STAB that amplifies qdisc_pkt_len to ~1 GiB (e.g. size_log=15, data=[32768]).
- Add a class with a tiny quantum of 1 and send one small packet; the deficit loop spins billions of times under the qdisc lock and trips the softlockup detector (panic with kernel.softlockup_panic=1).
- Reachable as root or from an unprivileged user in a fresh user+net namespace (unshare -Urn) with namespace-local CAP_NET_ADMIN.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by [email protected]

---
v1 -> v2

1. Sigh. The v1 overhead cap broke the existing qfq tdc test 5993, caught by running the whole tdc.sh instead of affected qdiscs reported reported by poc. That test legitimately uses stab overhead 999999999 qfq and expects the qdisc to be accepted (exit 0) with packets dropped.

2. Better Fix: cap the final qdisc_pkt_len() to GSO_MAX_SIZE(524280) in __qdisc_calculate_pkt_len() per sashikos[1][2] suggestions

3. Given existence of tdc 5993 we dont need the tdc test created earlier since the essence of that test is covered in tdc 5993.

[1] https://sashiko.dev/#/patchset/[email protected]
[2] https://netdev-ai.bots.linux.dev/sashiko/#/patchset/[email protected]

---
 net/sched/sch_api.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
net/sched/sch_api.c +0 -0
index 65b35528d125..ad4f117ff55a 100644

net/sched/sch_api.c void __qdisc_calculate_pkt_len(struct sk_buff *skb, · line 617

Quoted 10 lines
net/sched/sch_api.c +5 -2
@@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb,
 
        pkt_len <<= stab->szopts.size_log;
 out:
-       if (unlikely(pkt_len < 1))
-               pkt_len = 1;
+       /* A size table can inflate qdisc_pkt_len() beyond any real packet
+        * (via overhead, the data table, or size_log); cap it so deficit
+        * schedulers such as DRR/ETS terminate their refill loops.
+        */
+       pkt_len = clamp_t(int, pkt_len, 1, GSO_MAX_SIZE);

Yeah, although it is a bit strange to mention GSO_MAX_SIZE in a function which does not take care of GSO packets.

Quoted 8 lines
Quoted 2 lines

well... there are many many references to gso under net/sched. What dont you like about this specific things?

I just mentioned that __qdisc_calculate_pkt_len() never was updated to deal with GSO packets.

It assumes packets with a single set of headers.

I see. I was looking for something reasonable. The problem is maximum real skb->len is UINT_MAX and with stab amplification it can get a small packet into a GB.
Would you prefer a new #define in pkt_sched.h to put an upper bound (QDISC_PKT_LEN_MAX)?
Something else?

cheers,
jamal

Quoted 1 line

This is fine, I am sure nobody uses this stuff anymore.

maintainer
#5 lore raw
Quoted 6 lines · On Wed, 19 Aug 2026 11:30:44 -0400 Jamal Hadi Salim wrote:

I see. I was looking for something reasonable. The problem is maximum real skb->len is UINT_MAX and with stab amplification it can get a small packet into a GB.
Would you prefer a new #define in pkt_sched.h to put an upper bound (QDISC_PKT_LEN_MAX)?
Something else?

Let's go with the define you mentioned to make progress.

Signature · 1 line
pw-bot: cr