New: netem is broken in 6.18
Subsystems (2)
Stephen Hemminger
Acked-by Jamal Hadi SalimRegression caused by:
commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
Author: William Liu <[email protected]> Date: Tue Jul 8 16:43:26 2025 +0000
net/sched: Restrict conditions for adding duplicating netems to qdisc tree
netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.
Previous approaches suggested in discussions in chronological order:
1) Track duplication status or ttl in the sk_buff struct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].
2) Restrict netem_enqueue recursion depth like in act_mirred with a per cpu variable. However, netem_dequeue can call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.
3) Use the same approach as in 2, but add metadata in netem_skb_cb to handle the netem_dequeue case and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.
4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.
[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/
Begin forwarded message:
Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18
https://bugzilla.kernel.org/show_bug.cgi?id=220774
Bug ID: 220774
Summary: netem is broken in 6.18
Product: Networking
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No[jschung@localhost ~]$ cat test.sh #!/bin/bash
DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"
tc qdisc del dev $DEV root > /dev/null 2>&1 tc qdisc add dev $DEV root handle 1: mq
for i in $(seq 1 $NUM_QUEUES); do
HANDLE_ID=$((i * 10))
PARENT_ID="1:$i"
tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netemduplicate $DUPLICATE_PERCENT
done
[jschung@localhost ~]$ sudo ./test.sh
[ 2976.073299] netem: change failed
Error: netem: cannot mix duplicating netems with other netems in tree.
[jschung@localhost ~]$ uname -r 6.18.0-rc4
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
Hi Will, Jamal and Jakub,
I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.
Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.
Regards,
Cong
Quoted 104 lines · Stephen Hemminger
Regression caused by:
commit
ec8e0e3d7adef940cdf9475e2352c0680189d14eAuthor: William Liu <[email protected]> Date: Tue Jul 8 16:43:26 2025 +0000net/sched: Restrict conditions for adding duplicating netems to qdisc tree
netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in
netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.Previous approaches suggested in discussions in chronological order:
1) Track duplication status or ttl in the
sk_buffstruct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].2) Restrict
netem_enqueuerecursion depth like inact_mirredwith a per cpu variable. However,netem_dequeuecan call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.3) Use the same approach as in 2, but add metadata in
netem_skb_cbto handle thenetem_dequeuecase and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.
[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/Fixes 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")Reported-by William Liu <[email protected]>Reported-by Savino Dicanosa <[email protected]>Signed-off-by William Liu <[email protected]>Signed-off-by Savino Dicanosa <[email protected]>Acked-by Jamal Hadi Salim <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>Begin forwarded message:
Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18https://bugzilla.kernel.org/show_bug.cgi?id=220774
Bug ID: 220774 Summary: netem is broken in 6.18 Product: Networking Version: 2.5 Hardware: All OS: Linux Status: NEW Severity: high Priority: P3 Component: Other Assignee: [email protected] Reporter: [email protected] Regression: No[jschung@localhost ~]$ cat
test.sh#!/bin/bashDEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"tc qdisc del dev $DEV root >
/dev/null2>&1 tc qdisc add dev $DEV root handle 1: mqfor i in $(seq 1 $NUM_QUEUES); do
HANDLE_ID=$((i * 10)) PARENT_ID="1:$i" tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netemduplicate $DUPLICATE_PERCENT
done[jschung@localhost ~]$ sudo ./test.sh
[ 2976.073299] netem: change failedError: netem: cannot mix duplicating netems with other netems in tree.
[jschung@localhost ~]$ uname -r 6.18.0-rc4
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are the assignee for the bug.
Quoted 6 lines · Cong Wang
Hi Will, Jamal and Jakub,
I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.
Quoted 100 lines
Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.
Regards,
CongQuoted 92 lines · Stephen Hemminger
Regression caused by:
commit
ec8e0e3d7adef940cdf9475e2352c0680189d14eAuthor: William Liu <[email protected]> Date: Tue Jul 8 16:43:26 2025 +0000net/sched: Restrict conditions for adding duplicating netems to qdisc tree
netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in
netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.Previous approaches suggested in discussions in chronological order:
1) Track duplication status or ttl in the
sk_buffstruct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].2) Restrict
netem_enqueuerecursion depth like inact_mirredwith a per cpu variable. However,netem_dequeuecan call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.3) Use the same approach as in 2, but add metadata in
netem_skb_cbto handle thenetem_dequeuecase and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.
[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/Fixes 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")Reported-by William Liu <[email protected]>Reported-by Savino Dicanosa <[email protected]>Signed-off-by William Liu <[email protected]>Signed-off-by Savino Dicanosa <[email protected]>Acked-by Jamal Hadi Salim <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>Begin forwarded message:
Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18https://bugzilla.kernel.org/show_bug.cgi?id=220774
Bug ID: 220774 Summary: netem is broken in 6.18 Product: Networking Version: 2.5 Hardware: All OS: Linux Status: NEW Severity: high Priority: P3 Component: Other Assignee: [email protected] Reporter: [email protected] Regression: No[jschung@localhost ~]$ cat
test.sh#!/bin/bashDEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"tc qdisc del dev $DEV root >
/dev/null2>&1 tc qdisc add dev $DEV root handle 1: mqfor i in $(seq 1 $NUM_QUEUES); do
HANDLE_ID=$((i * 10)) PARENT_ID="1:$i" tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netemduplicate $DUPLICATE_PERCENT
done
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
cheers,
jamal
Quoted 12 lines
Quoted 12 lines
[jschung@localhost ~]$ sudo ./test.sh
[ 2976.073299] netem: change failedError: netem: cannot mix duplicating netems with other netems in tree.
[jschung@localhost ~]$ uname -r 6.18.0-rc4
--
You may reply to this email to add a comment.You are receiving this mail because:
You are the assignee for the bug.
Quoted 4 lines · Jamal Hadi Salim
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.
BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).
Regards,
Cong
Quoted 12 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.
Quoted 6 lines
BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).
Regards,
Cong
The bug still needs to be fixed.
eBPF would still have the same kind of issues.
Jakub Kicinski
maintainerQuoted 15 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 12 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.
Quoted 6 lines
Quoted 3 lines
BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).
The bug still needs to be fixed.
eBPF would still have the same kind of issues.
I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow the mq+netem hierarchy?
#!/bin/bash
set -euo pipefail
DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15
usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in
-d) DEV="$2"; shift 2;;
-q) QUEUE="$2"; shift 2;;
-P) RTP_DST_PORT="$2"; shift 2;;
*) usage;; endac
done || true
[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }
if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fi
echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
echo "some flows will still land on :$QUEUE"
}
echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"
tc qdisc show dev "$DEV"
echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true
if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fi
echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'
Sent with Proton Mail secure email.
Quoted 125 lines · Jamal Hadi Salim
Quoted 5 lines · On Thu, Nov 20, 2025 at 11:29 PM Cong Wang [email protected] wrote:
Hi Will, Jamal and Jakub,
I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.
Quoted 97 lines
Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.
Regards,
CongQuoted 89 lines · Stephen Hemminger
Regression caused by:
commit
ec8e0e3d7adef940cdf9475e2352c0680189d14e
Author: William Liu [email protected]
Date: Tue Jul 8 16:43:26 2025 +0000net/sched: Restrict conditions for adding duplicating netems to qdisc tree
netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in
netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.Previous approaches suggested in discussions in chronological order:
1) Track duplication status or ttl in the
sk_buffstruct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].2) Restrict
netem_enqueuerecursion depth like inact_mirredwith a per cpu variable. However,netem_dequeuecan call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.3) Use the same approach as in 2, but add metadata in
netem_skb_cbto handle thenetem_dequeuecase and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.
[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/Fixes 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")Reported-by William Liu [email protected]Reported-by Savino Dicanosa [email protected]Signed-off-by William Liu [email protected]Signed-off-by Savino Dicanosa [email protected]Acked-by Jamal Hadi Salim [email protected]Signed-off-by Jakub Kicinski [email protected]Begin forwarded message:
Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18https://bugzilla.kernel.org/show_bug.cgi?id=220774
Bug ID: 220774
Summary: netem is broken in 6.18
Product: Networking
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No[jschung@localhost ~]$ cat
test.sh
#!/bin/bashDEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"tc qdisc del dev $DEV root >
/dev/null2>&1
tc qdisc add dev $DEV root handle 1: mqfor i in $(seq 1 $NUM_QUEUES); do
HANDLE_ID=$((i * 10))
PARENT_ID="1:$i"
tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem duplicate $DUPLICATE_PERCENT
done[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
cheers,
jamalQuoted 12 lines
Quoted 12 lines
[jschung@localhost ~]$ sudo ./test.sh
[ 2976.073299] netem: change failedError: netem: cannot mix duplicating netems with other netems in tree.
[jschung@localhost ~]$ uname -r
6.18.0-rc4--
You may reply to this email to add a comment.You are receiving this mail because:
You are the assignee for the bug.
Quoted 21 lines · Jakub Kicinski
Quoted 19 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 15 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).
Quoted 6 lines
Quoted 2 lines
The bug still needs to be fixed.
eBPF would still have the same kind of issues.I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow the mq+netem hierarchy?
Yes, something like that - but imo it would be fugly to do that check in current code just for mq.
I proposed to add another qdisc ops which checks for cross-qdisc semantics. This ops will be invoked by a qdisc's change/init - sort of what pci quirks ops does. For example in this case it will disallow setting up netem when it would cause loops but have the quirck checker allow multiple children such as with mq with as long as none is looping. I believe some sort of "feature bits" checking should work here.
The majority of the bugs we are dealing with follow the formula of:
a) create a nonsense hierarchy of qdiscs which has no pratical value,
b) start sending packets
c) netlink cmds to change hierarchy some more; It's more fun if you can get packets stuck - the formula in this case includes non-working conserving qdsiscs somewhere in the hierarchy
d) send more packets
e) profit
Current init/change are not really designed to catch things that deal with children or parents, so catching #a or #c is a challenge. We do "make it work" eventually but it keeps adding these "checks" for nonsense setups adds more unnecessary code.
Perhaps its time to start this effort and use this specific scenario as a use case.
cheers,
jamal
Hi 정지수,
Quoted 51 lines · 정지수
#!/bin/bash
set -euo pipefail
DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P
UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in-d) DEV="$2"; shift 2;; -q) QUEUE="$2"; shift 2;; -P) RTP_DST_PORT="$2"; shift 2;; *) usage;;endac
done || true[[ -d
/sys/class/net/$DEV]] || { echo "No such dev $DEV"; exit 1; }if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fiecho "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
echo "some flows will still land on :$QUEUE"
}echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"tc qdisc show dev "$DEV"
echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true
Quoted 9 lines
if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fiecho
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'
Thanks for the config.
If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?
cheers,
jamal
Quoted 20 lines · Jakub Kicinski
Quoted 19 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 15 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim
[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?
An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).
Quoted 5 lines
Quoted 2 lines
The bug still needs to be fixed.
eBPF would still have the same kind of issues.I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow
This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.
Quoted 1 line
the mq+netem hierarchy?
This would make the code even uglier, it is already ugly enough to hard-code and single out this case in the code.
Not to mention there could be other combinations we don't know yet.
We need to revert it and fix the original issue with changing the problematic duplication behavior.
Regards,
Cong
Quoted 8 lines · Jamal Hadi Salim
If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?
I am not speaking for Ji-Soo, but which tc action are you talking about here?
Personally, I am not aware of any tc action could achieve the same netem duplication. gact offers some randomness, mirred offers duplication, but it is not trivial to just combine them to be same as netem duplication.
Regards,
Cong Wang
Jakub Kicinski
maintainerQuoted 5 lines · On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
Quoted 2 lines
I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow
This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.
I'm curious why we did.. Link?
Part of the original issue was that alternative proposed fixes did not work and still allowed for local DOSes [1] [2].
An easy fix that was suggested was to not enqueue from the root [3][4]. However, this changes behavior that has existed since the beginning, immediately caused a new bug, and Hemminger mentioned that the original behavior was necessary for "trying to keep proper semantics and accounting especially when doing rate limits." An argument can be made that Cong's change doesn't violate manpage semantics, but that may still break user setups, just like the current scenario.
Jamal had a good idea on using tc_skb_ext extensions, but I pointed out a few implementation issues we would have to deal with [5]. From what I understood though, that solution would preserve all the existing behaviors and avoid the DOS bug. It's been a while so let me know if that is not the case. Otherwise, I can try to implement it - I am quite busy though so it might take a while.
Quoted 4 lines · Jakub Kicinski
On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow
This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.
I'm curious why we did.. Link?
The issue was due to filters redirecting packets iirc [6]. We decided to move with it and come back to it if someone did rely on this rare setup [7].
If anyone wants a quick refresher on the bug, please take a read at [6] as it contains all the details and issues.
[1] https://lore.kernel.org/netdev/[email protected]/T/#u
[2] https://lore.kernel.org/netdev/[email protected]/T/#u
[3] https://lore.kernel.org/netdev/[email protected]/T/#u
[4] https://lore.kernel.org/netdev/CAM0EoMmTZon=nFmLsDPKhDEzHruw701iV9=mq92At9oKo0LGpA@mail.gmail.com/T/#u
[5] https://lore.kernel.org/netdev/pGE9OHWRSf4oJwC4gS0oPonBy8_0WsDthxgLzBYGBtMVeT_EDc-HAz8NbhJxcWe0NEUrf_a7Fyq2op5FVFujfc2KyO-I38Yx_HlQhFwB0Cs=@willsroot.io/
[6] https://lore.kernel.org/netdev/CAM0EoMnd0nZxJW3zpEuBGWTwB3AnJSnj242f9hMpcLdBWdcbfQ@mail.gmail.com/
[7] https://lore.kernel.org/netdev/[email protected]/
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
Quoted 2 lines
I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow
This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.
I'm curious why we did.. Link?
https://lore.kernel.org/all/[email protected]/#t
Jamal just denied the use case and let users complain.
This strategy does not work, since majority users would need to wait until LTS gets hit by this regression. (IMHO, it is also unethical to knowingly break valid use cases, regardless of purpose.)
Regards,
Cong Wang
Mr Salim,
I do not understand what your saying. You provide a bash script, and I can test
- Ji-Soo
에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <[email protected]> 님이 작성함:
Quoted 74 lines
Hi 정지수,
Quoted 45 lines · On Sat, Nov 22, 2025 at 1:56 AM 정지수 [email protected] wrote:
#!/bin/bash
set -euo pipefail
DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P
UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in(quoted diff) +0
-3-d) DEV="$2"; shift 2;; -q) QUEUE="$2"; shift 2;; -P) RTP_DST_PORT="$2"; shift 2;;*) usage;;
endac
done || true[[ -d
/sys/class/net/$DEV]] || { echo "No such dev $DEV"; exit 1; }if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fiecho "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || { echo "some flows will still land on :$QUEUE"
}echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"tc qdisc show dev "$DEV"
echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || trueQuoted 8 lines
if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true fiecho
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'Thanks for the config.
If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?cheers,
jamal
Hi,
Quoted 5 lines · 정지수
Mr Salim,
I do not understand what your saying. You provide a bash script, and I can test
Will you be willing to go on a zoom/gmeet call? I think it will be easier to communicate.
cheers,
jamal
Quoted 78 lines
- Ji-Soo
에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <[email protected]> 님이 작성함:
Quoted 74 lines
Hi 정지수,
Quoted 45 lines · On Sat, Nov 22, 2025 at 1:56 AM 정지수 [email protected] wrote:
#!/bin/bash
set -euo pipefail
DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P
UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in(quoted diff) +0
-3-d) DEV="$2"; shift 2;; -q) QUEUE="$2"; shift 2;; -P) RTP_DST_PORT="$2"; shift 2;;*) usage;;
endac
done || true[[ -d
/sys/class/net/$DEV]] || { echo "No such dev $DEV"; exit 1; }if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fiecho "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || { echo "some flows will still land on :$QUEUE"
}echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"tc qdisc show dev "$DEV"
echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || trueQuoted 8 lines
if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true fiecho
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'Thanks for the config.
If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?cheers,
jamal