Skip to contentSkip to conversation

New: netem is broken in 6.18

Bug 220774 Stephen Hemminger 6 participants 16 messages Nov 10, 2025 – Nov 27, 2025

Subsystems (2)
lore
Share & tools
lore

Stephen Hemminger

author Acked-by Jamal Hadi Salim
#0 lore raw

Regression caused by:

commit ec8e0e3d7adef940cdf9475e2352c0680189d14e

Author: William Liu <[email protected]>
Date:   Tue Jul 8 16:43:26 2025 +0000

net/sched: Restrict conditions for adding duplicating netems to qdisc tree

netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.

Previous approaches suggested in discussions in chronological order:

1) Track duplication status or ttl in the sk_buff struct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].

2) Restrict netem_enqueue recursion depth like in act_mirred with a per cpu variable. However, netem_dequeue can call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.

3) Use the same approach as in 2, but add metadata in netem_skb_cb to handle the netem_dequeue case and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.

4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.

[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/

Begin forwarded message:

Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18

https://bugzilla.kernel.org/show_bug.cgi?id=220774

            Bug ID: 220774
           Summary: netem is broken in 6.18
           Product: Networking
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

[jschung@localhost ~]$ cat test.sh #!/bin/bash

DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"

tc qdisc del dev $DEV root > /dev/null 2>&1 tc qdisc add dev $DEV root handle 1: mq

for i in $(seq 1 $NUM_QUEUES); do

    HANDLE_ID=$((i * 10))
    PARENT_ID="1:$i"
    tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem

duplicate $DUPLICATE_PERCENT
done

[jschung@localhost ~]$ sudo ./test.sh

[  2976.073299] netem: change failed

Error: netem: cannot mix duplicating netems with other netems in tree.

[jschung@localhost ~]$ uname -r 6.18.0-rc4

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.
#1 lore raw

Hi Will, Jamal and Jakub,

I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.

Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.

Regards,
Cong

Quoted 104 lines · Stephen Hemminger

Regression caused by:

commit ec8e0e3d7adef940cdf9475e2352c0680189d14e

Author: William Liu <[email protected]>
Date:   Tue Jul 8 16:43:26 2025 +0000

net/sched: Restrict conditions for adding duplicating netems to qdisc tree

netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.

Previous approaches suggested in discussions in chronological order:

1) Track duplication status or ttl in the sk_buff struct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].

2) Restrict netem_enqueue recursion depth like in act_mirred with a per cpu variable. However, netem_dequeue can call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.

3) Use the same approach as in 2, but add metadata in netem_skb_cb to handle the netem_dequeue case and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.

4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.

[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/

Begin forwarded message:

Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18

https://bugzilla.kernel.org/show_bug.cgi?id=220774

            Bug ID: 220774
           Summary: netem is broken in 6.18
           Product: Networking
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

[jschung@localhost ~]$ cat test.sh #!/bin/bash

DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"

tc qdisc del dev $DEV root > /dev/null 2>&1 tc qdisc add dev $DEV root handle 1: mq

for i in $(seq 1 $NUM_QUEUES); do

    HANDLE_ID=$((i * 10))
    PARENT_ID="1:$i"
    tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem

duplicate $DUPLICATE_PERCENT
done

[jschung@localhost ~]$ sudo ./test.sh

[  2976.073299] netem: change failed

Error: netem: cannot mix duplicating netems with other netems in tree.

[jschung@localhost ~]$ uname -r 6.18.0-rc4

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.
#2 lore raw
Quoted 6 lines · Cong Wang

Hi Will, Jamal and Jakub,

I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.

Quoted 100 lines

Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.

Regards,
Cong

Quoted 92 lines · Stephen Hemminger

Regression caused by:

commit ec8e0e3d7adef940cdf9475e2352c0680189d14e

Author: William Liu <[email protected]>
Date:   Tue Jul 8 16:43:26 2025 +0000

net/sched: Restrict conditions for adding duplicating netems to qdisc tree

netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.

Previous approaches suggested in discussions in chronological order:

1) Track duplication status or ttl in the sk_buff struct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].

2) Restrict netem_enqueue recursion depth like in act_mirred with a per cpu variable. However, netem_dequeue can call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.

3) Use the same approach as in 2, but add metadata in netem_skb_cb to handle the netem_dequeue case and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.

4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.

[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/

Begin forwarded message:

Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18

https://bugzilla.kernel.org/show_bug.cgi?id=220774

            Bug ID: 220774
           Summary: netem is broken in 6.18
           Product: Networking
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

[jschung@localhost ~]$ cat test.sh #!/bin/bash

DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"

tc qdisc del dev $DEV root > /dev/null 2>&1 tc qdisc add dev $DEV root handle 1: mq

for i in $(seq 1 $NUM_QUEUES); do

    HANDLE_ID=$((i * 10))
    PARENT_ID="1:$i"
    tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem

duplicate $DUPLICATE_PERCENT
done

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

cheers,
jamal

Quoted 12 lines
Quoted 12 lines

[jschung@localhost ~]$ sudo ./test.sh

[  2976.073299] netem: change failed

Error: netem: cannot mix duplicating netems with other netems in tree.

[jschung@localhost ~]$ uname -r 6.18.0-rc4

--
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

#3 lore raw
Quoted 4 lines · Jamal Hadi Salim

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.

BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).

Regards,
Cong

#4 lore raw
Quoted 12 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.

Quoted 6 lines

BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).

Regards,
Cong

The bug still needs to be fixed.
eBPF would still have the same kind of issues.

maintainer
#5 lore raw
Quoted 15 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 12 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.

Quoted 6 lines
Quoted 3 lines

BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).

The bug still needs to be fixed.
eBPF would still have the same kind of issues.

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow the mq+netem hierarchy?

#11 lore raw
In reply to Jamal Hadi Salim

#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in

    -d) DEV="$2"; shift 2;;
    -q) QUEUE="$2"; shift 2;;
    -P) RTP_DST_PORT="$2"; shift 2;;
    *) usage;;

endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }

if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fi

echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
echo "some flows will still land on :$QUEUE"
}

echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true

if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'

Sent with Proton Mail secure email.

Quoted 125 lines · Jamal Hadi Salim
Quoted 5 lines · On Thu, Nov 20, 2025 at 11:29 PM Cong Wang [email protected] wrote:

Hi Will, Jamal and Jakub,

I already warned you many times before you applied it. Now we have users complaining, please let me know if you still respect users.

Quoted 97 lines

Also, Jamal, if I remember correctly, you said you will work on a long term solution, now after 4 months, please let us know what your plan is.

Regards,
Cong

Quoted 89 lines · Stephen Hemminger

Regression caused by:

commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
Author: William Liu [email protected]
Date: Tue Jul 8 16:43:26 2025 +0000

net/sched: Restrict conditions for adding duplicating netems to qdisc tree

netem_enqueue's duplication prevention logic breaks when a netem resides in a qdisc tree with other netems - this can lead to a soft lockup and OOM loop in netem_dequeue, as seen in [1]. Ensure that a duplicating netem cannot exist in a tree with other netems.

Previous approaches suggested in discussions in chronological order:

1) Track duplication status or ttl in the sk_buff struct. Considered too specific a use case to extend such a struct, though this would be a resilient fix and address other previous and potential future DOS bugs like the one described in loopy fun [2].

2) Restrict netem_enqueue recursion depth like in act_mirred with a per cpu variable. However, netem_dequeue can call enqueue on its child, and the depth restriction could be bypassed if the child is a netem.

3) Use the same approach as in 2, but add metadata in netem_skb_cb to handle the netem_dequeue case and track a packet's involvement in duplication. This is an overly complex approach, and Jamal notes that the skb cb can be overwritten to circumvent this safeguard.

4) Prevent the addition of a netem to a qdisc tree if its ancestral path contains a netem. However, filters and actions can cause a packet to change paths when re-enqueued to the root from netem duplication, leading us to the current solution: prevent a duplicating netem from inhabiting the same tree as other netems.

[1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
[2] https://lwn.net/Articles/719297/

Begin forwarded message:

Date: Mon, 10 Nov 2025 19:13:57 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 220774] New: netem is broken in 6.18

https://bugzilla.kernel.org/show_bug.cgi?id=220774

Bug ID: 220774
Summary: netem is broken in 6.18
Product: Networking
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: high
Priority: P3
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No

[jschung@localhost ~]$ cat test.sh
#!/bin/bash

DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"

tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc add dev $DEV root handle 1: mq

for i in $(seq 1 $NUM_QUEUES); do
HANDLE_ID=$((i * 10))
PARENT_ID="1:$i"
tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem duplicate $DUPLICATE_PERCENT
done

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

cheers,
jamal

Quoted 12 lines
Quoted 12 lines

[jschung@localhost ~]$ sudo ./test.sh

[ 2976.073299] netem: change failed

Error: netem: cannot mix duplicating netems with other netems in tree.

[jschung@localhost ~]$ uname -r
6.18.0-rc4

--
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

#6 lore raw
In reply to Jakub Kicinski
Quoted 21 lines · Jakub Kicinski
Quoted 19 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 15 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.

BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).

Quoted 6 lines
Quoted 2 lines

The bug still needs to be fixed.
eBPF would still have the same kind of issues.

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow the mq+netem hierarchy?

Yes, something like that - but imo it would be fugly to do that check in current code just for mq.
I proposed to add another qdisc ops which checks for cross-qdisc semantics. This ops will be invoked by a qdisc's change/init - sort of what pci quirks ops does. For example in this case it will disallow setting up netem when it would cause loops but have the quirck checker allow multiple children such as with mq with as long as none is looping. I believe some sort of "feature bits" checking should work here.

The majority of the bugs we are dealing with follow the formula of:
a) create a nonsense hierarchy of qdiscs which has no pratical value,
b) start sending packets
c) netlink cmds to change hierarchy some more; It's more fun if you can get packets stuck - the formula in this case includes non-working conserving qdsiscs somewhere in the hierarchy
d) send more packets
e) profit
Current init/change are not really designed to catch things that deal with children or parents, so catching #a or #c is a challenge. We do "make it work" eventually but it keeps adding these "checks" for nonsense setups adds more unnecessary code.

Perhaps its time to start this effort and use this specific scenario as a use case.

cheers,
jamal

#12 lore raw
In reply to 정지수

Hi 정지수,

Quoted 51 lines · 정지수

#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in

    -d) DEV="$2"; shift 2;;
    -q) QUEUE="$2"; shift 2;;
    -P) RTP_DST_PORT="$2"; shift 2;;
    *) usage;;

endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }

if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fi

echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
echo "some flows will still land on :$QUEUE"
}

echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true

Quoted 9 lines

if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'

Thanks for the config.

If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?

cheers,
jamal

#7 lore raw
In reply to Jakub Kicinski
Quoted 20 lines · Jakub Kicinski
Quoted 19 lines · On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
Quoted 15 lines · On Fri, 21 Nov 2025 13:45:06 -0800 Cong Wang <[email protected]> wrote:
Quoted 4 lines · Jamal Hadi Salim

[email protected]: Can you please provide more details about what you are trying to do so we can see if a different approach can be prescribed?

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a working cmdline.

BTW, Jamal, if your plan is to solve them one by one, even if it could work, it wouldn't scale. There are still many users don't get hit by this regression yet (not until hitting LTS or major distro).

Quoted 5 lines
Quoted 2 lines

The bug still needs to be fixed.
eBPF would still have the same kind of issues.

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow

This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.

Quoted 1 line

the mq+netem hierarchy?

This would make the code even uglier, it is already ugly enough to hard-code and single out this case in the code.

Not to mention there could be other combinations we don't know yet.

We need to revert it and fix the original issue with changing the problematic duplication behavior.

Regards,
Cong

#13 lore raw
In reply to Jamal Hadi Salim
Quoted 8 lines · Jamal Hadi Salim

If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?

I am not speaking for Ji-Soo, but which tc action are you talking about here?

Personally, I am not aware of any tc action could achieve the same netem duplication. gact offers some randomness, mirred offers duplication, but it is not trivial to just combine them to be same as netem duplication.

Regards,
Cong Wang

maintainer
#8 lore raw
In reply to Cong Wang
Quoted 5 lines · On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
Quoted 2 lines

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow

This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.

I'm curious why we did.. Link?

#9 lore raw

Part of the original issue was that alternative proposed fixes did not work and still allowed for local DOSes [1] [2].

An easy fix that was suggested was to not enqueue from the root [3][4]. However, this changes behavior that has existed since the beginning, immediately caused a new bug, and Hemminger mentioned that the original behavior was necessary for "trying to keep proper semantics and accounting especially when doing rate limits." An argument can be made that Cong's change doesn't violate manpage semantics, but that may still break user setups, just like the current scenario.

Jamal had a good idea on using tc_skb_ext extensions, but I pointed out a few implementation issues we would have to deal with [5]. From what I understood though, that solution would preserve all the existing behaviors and avoid the DOS bug. It's been a while so let me know if that is not the case. Otherwise, I can try to implement it - I am quite busy though so it might take a while.

Quoted 4 lines · Jakub Kicinski

On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow

This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.

I'm curious why we did.. Link?

The issue was due to filters redirecting packets iirc [6]. We decided to move with it and come back to it if someone did rely on this rare setup [7].

If anyone wants a quick refresher on the bug, please take a read at [6] as it contains all the details and issues.

[1] https://lore.kernel.org/netdev/[email protected]/T/#u
[2] https://lore.kernel.org/netdev/[email protected]/T/#u
[3] https://lore.kernel.org/netdev/[email protected]/T/#u
[4] https://lore.kernel.org/netdev/CAM0EoMmTZon=nFmLsDPKhDEzHruw701iV9=mq92At9oKo0LGpA@mail.gmail.com/T/#u
[5] https://lore.kernel.org/netdev/pGE9OHWRSf4oJwC4gS0oPonBy8_0WsDthxgLzBYGBtMVeT_EDc-HAz8NbhJxcWe0NEUrf_a7Fyq2op5FVFujfc2KyO-I38Yx_HlQhFwB0Cs=@willsroot.io/
[6] https://lore.kernel.org/netdev/CAM0EoMnd0nZxJW3zpEuBGWTwB3AnJSnj242f9hMpcLdBWdcbfQ@mail.gmail.com/
[7] https://lore.kernel.org/netdev/[email protected]/

#10 lore raw
In reply to Jakub Kicinski
Quoted 8 lines · Jakub Kicinski
Quoted 5 lines · On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
Quoted 2 lines

I guess we forgot about mq.. IIRC mq doesn't come into play in duplication, we should be able to just adjust the check to allow

This is not true, I warned you and Jamal with precisely the mq+netem combination before applying the patch, both of you chose to ignore.

I'm curious why we did.. Link?

https://lore.kernel.org/all/[email protected]/#t

Jamal just denied the use case and let users complain.

This strategy does not work, since majority users would need to wait until LTS gets hit by this regression. (IMHO, it is also unethical to knowingly break valid use cases, regardless of purpose.)

Regards,
Cong Wang

#14 lore raw
In reply to Jamal Hadi Salim

Mr Salim,

I do not understand what your saying. You provide a bash script, and I can test

- Ji-Soo

에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <[email protected]> 님이 작성함:

Quoted 74 lines

Hi 정지수,

Quoted 45 lines · On Sat, Nov 22, 2025 at 1:56 AM 정지수 [email protected] wrote:

#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in

(quoted diff) +0 -3
-d) DEV="$2"; shift 2;;
-q) QUEUE="$2"; shift 2;;
-P) RTP_DST_PORT="$2"; shift 2;;

*) usage;;
endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }

if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fi

echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || { echo "some flows will still land on :$QUEUE"
}

echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true

Quoted 8 lines

if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'

Thanks for the config.

If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?

cheers,
jamal

#15 lore raw

Hi,

Quoted 5 lines · 정지수

Mr Salim,

I do not understand what your saying. You provide a bash script, and I can test

Will you be willing to go on a zoom/gmeet call? I think it will be easier to communicate.

cheers,
jamal

Quoted 78 lines

- Ji-Soo

에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <[email protected]> 님이 작성함:

Quoted 74 lines

Hi 정지수,

Quoted 45 lines · On Sat, Nov 22, 2025 at 1:56 AM 정지수 [email protected] wrote:

#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; } while [[ $# -gt 0 ]]; do
case "$1" in

(quoted diff) +0 -3
-d) DEV="$2"; shift 2;;
-q) QUEUE="$2"; shift 2;;
-P) RTP_DST_PORT="$2"; shift 2;;

*) usage;;
endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }

if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then echo "Setting root qdisc to mq.."
tc qdisc replace dev "$DEV" root handle 1: mq
fi

echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..." ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || { echo "some flows will still land on :$QUEUE"
}

echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.." tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true

Quoted 8 lines

if command -v tshark >/dev/null 2>&1; then
echo "tshark live RTP view :"
timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):" tc -s qdisc show dev "$DEV" | sed -n '1,200p'

Thanks for the config.

If you can talk about it: I was more interested in what your end goal is. From the dev name it seems $DEV is a wireless device? Are you replicating these RTP packets across different ssids mapped to different hw queues? Are you forwarding these packets? The ethtool config indicates the RX direction but the netem replication is on the tx.
And in the short term if a tc action could achieve what you are trying to achieve - would that work for you?

cheers,
jamal