net/sched: use the backlog for nested mirred ingress
Subsystems (5)
cover net/sched: use the backlog for nested mirred ingress
TC mirred has a protection against excessive stack growth, but that protection doesn't really guarantee the absence of recursion, nor it guards against loops. Patch 1/2 rewords "recursion" to "nesting" to make this more clear.
We can leverage on this existing mechanism to prevent TCP / SCTP from doing soft lock-up in some specific scenarios that uses mirred egress->ingress:
patch 2 changes mirred so that the networking backlog is used for nested mirred ingress actions.
Davide Caratti (2):
net/sched: act_mirred: better wording on protection against excessive
stack growth
act_mirred: use the backlog for nested calls to mirred ingress
net/sched/act_mirred.c | 23 ++++++--- .../selftests/net/forwarding/tc_actions.sh | 49 ++++++++++++++++++- 2 files changed, 63 insertions(+), 9 deletions(-)
Signature · 1 line
2.38.1
PATCH 1/2 net/sched: act_mirred: better wording on protection against excessive stack growth
with commit e2ca070f89ec ("net: sched: protect against stack overflow in TC act_mirred"), act_mirred protected itself against excessive stack growth using per_cpu counter of nested calls to tcf_mirred_act(), and capping it to MIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.
--- net/sched/act_mirred.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
net/sched/act_mirred.c +8 -8
@@ -29,8 +29,8 @@static LIST_HEAD (mirred_list);static DEFINE_SPINLOCK (mirred_list_lock); -#define MIRRED_RECURSION_LIMIT 4 -static DEFINE_PER_CPU(unsigned int, mirred_rec_level); +#define MIRRED_NEST_LIMIT 4 +static DEFINE_PER_CPU(unsigned int, mirred_nest_level);static bool tcf_mirred_is_act_redirect (int action) {
@@ -226,7 +226,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,struct sk_buff *skb2 = skb;bool m_mac_header_xmit;struct net_device *dev; -unsigned int rec_level; +unsigned int nest_level;int retval, err =0 ;bool use_reinsert;bool want_ingress;
@@ -237,11 +237,11 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,int mac_len;bool at_nh; - rec_level =__this_cpu_inc_return (mirred_rec_level); -if (unlikely (rec_level >MIRRED_RECURSION_LIMIT )) { + nest_level =__this_cpu_inc_return (mirred_nest_level); +if (unlikely (nest_level >MIRRED_NEST_LIMIT )) {net_warn_ratelimited ("Packet exceeded mirred recursion limit on dev %s\n" ,netdev_name (skb->dev)); -__this_cpu_dec (mirred_rec_level); +__this_cpu_dec (mirred_nest_level);return TC_ACT_SHOT ; }
@@ -310,7 +310,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb, err =tcf_mirred_forward (want_ingress, skb);if (err)tcf_action_inc_overlimit_qstats (&m->common); -__this_cpu_dec (mirred_rec_level); +__this_cpu_dec (mirred_nest_level);return TC_ACT_CONSUMED ; } }
@@ -322,7 +322,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,if (tcf_mirred_is_act_redirect (m_eaction)) retval =TC_ACT_SHOT ; } -__this_cpu_dec (mirred_rec_level); +__this_cpu_dec (mirred_nest_level);return retval; }
Signature · 1 line
2.38.1
PATCH 2/2 act_mirred: use the backlog for nested calls to mirred ingress
In reply to Davide CarattiWilliam reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).
Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when tcf_mirred_act() has a nest level greater than one:
change tcf_mirred_forward() accordingly.
Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred egress->ingress (when the nest level is 1).
[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and return NET_RX_SUCCESS inside
tcf_mirred_forward().
--- net/sched/act_mirred.c | 7 +++ .../selftests/net/forwarding/tc_actions.sh | 49 ++++++++++++++++++- 2 files changed, 55 insertions(+), 1 deletion(-)
net/sched/act_mirred.c +7 -0
@@ -206,12 +206,19 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,return err; } +static bool is_mirred_nested (void ) +{ +return unlikely (__this_cpu_read (mirred_nest_level) >1 ); +} +static int tcf_mirred_forward (bool want_ingress,struct sk_buff *skb) {int err;if (!want_ingress) err =tcf_dev_queue_xmit (skb, dev_queue_xmit); +else if (is_mirred_nested ()) + err =netif_rx (skb);else err =netif_receive_skb (skb);
tools/testing/selftests/net/forwarding/tc_actions.sh +48 -1
61 lines not on this page · raw message
Signature · 1 line
2.38.1
Re: [PATCH net-next 1/2] net/sched: act_mirred: better wording on protection against excessive stack growth
In reply to Davide CarattiQuoted 10 lines · Davide Caratti
with commit
e2ca070f89ec("net: sched: protect against stack overflow in TCact_mirred"),act_mirredprotected itself against excessive stack growth usingper_cpucounter of nested calls totcf_mirred_act(), and capping it toMIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.CC Jamal Hadi Salim <[email protected]>Signed-off-by Davide Caratti <[email protected]>
Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
In reply to Davide CarattiQuoted 24 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when
tcf_mirred_act()has a nest level greater than one:
changetcf_mirred_forward()accordingly.Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred
egress->ingress(when the nest level is 1).[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and returnNET_RX_SUCCESSinside
tcf_mirred_forward().
Quoted 3 lines
Reported-by William Zhao <[email protected]>CC Xin Long <[email protected]>Signed-off-by Davide Caratti <[email protected]>
Re: [PATCH net-next 1/2] net/sched: act_mirred: better wording on protection against excessive stack growth
In reply to Marcelo Ricardo LeitnerQuoted 10 lines · Marcelo Ricardo Leitner
Quoted 8 lines · Davide Caratti
with commit
e2ca070f89ec("net: sched: protect against stack overflow in TCact_mirred"),act_mirredprotected itself against excessive stack growth usingper_cpucounter of nested calls totcf_mirred_act(), and capping it toMIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.
Quoted 4 lines
Quoted 2 lines
CC Jamal Hadi Salim <[email protected]>Signed-off-by Davide Caratti <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>
cheers,
jamal
Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
In reply to Marcelo Ricardo LeitnerQuoted 26 lines · Marcelo Ricardo Leitner
Quoted 24 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when
tcf_mirred_act()has a nest level greater than one:
changetcf_mirred_forward()accordingly.Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred
egress->ingress(when the nest level is 1).[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and returnNET_RX_SUCCESSinside
tcf_mirred_forward().
Quoted 5 lines
Quoted 3 lines
Reported-by William Zhao <[email protected]>CC Xin Long <[email protected]>Signed-off-by Davide Caratti <[email protected]>Reviewed-by Marcelo Ricardo Leitner <[email protected]>
cheers,
jamal
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:
Quoted 10 lines · On Fri, 20 Jan 2023 18:01:38 +0100 you wrote:
TC mirred has a protection against excessive stack growth, but that protection doesn't really guarantee the absence of recursion, nor it guards against loops. Patch 1/2 rewords "recursion" to "nesting" to make this more clear.
We can leverage on this existing mechanism to prevent TCP / SCTP from doing soft lock-up in some specific scenarios that uses mirredegress->ingress:
patch 2 changes mirred so that the networking backlog is used for nested mirred ingress actions.[...]
Here is the summary with links:
- [net-next,1/2] net/sched: act_mirred: better wording on protection against excessive stack growth
https://git.kernel.org/netdev/net-next/c/78dcdffe0418 - [net-next,2/2] act_mirred: use the backlog for nested calls to mirred ingress https://git.kernel.org/netdev/net-next/c/ca22da2fbd69
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Jamal Hadi Salim
maintainerMirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
In reply to Davide CarattiQuoted 14 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when
tcf_mirred_act()has a nest level greater than one:
changetcf_mirred_forward()accordingly.
Quoted 4 lines
Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred
egress->ingress(when the nest level is 1).
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
----
sudo ip netns add p4node
sudo ip link add p4port0 address 10:00:00:01:AA:BB type veth peer port0 address 10:00:00:02:AA:BB
sudo ip link set dev port0 netns p4node
sudo ip a add 10.0.0.1/24 dev p4port0
sudo ip neigh add 10.0.0.2 dev p4port0 lladdr 10:00:00:02:aa:bb sudo ip netns exec p4node ip a add 10.0.0.2/24 dev port0 sudo ip netns exec p4node ip l set dev port0 up
sudo ip l set dev p4port0 up
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
ping -I p4port0 10.0.0.2 -c 1
-----
reverting the patch fixes things and it gets caught by the nested recursion check.
Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.
cheers,
jamal
Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
hello Jamal, thanks for looking at this!
Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).
[...]
Quoted 3 lines
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
[...]
Quoted 3 lines
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop. I don't see it much different than what we can obtain with bridges:
# ip link add name one type veth peer name two # ip link add name three type veth peer name four # for n in even odd; do ip link add name $n type bridge; done # for n in one two three four even odd; do ip link set dev $n up; done # for n in one three; do ip link set dev $n master odd; done # for n in two four; do ip link set dev $n master even; done
there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of bpf_redirect() helper)
Quoted 2 lines
reverting the patch fixes things and it gets caught by the nested recursion check.
the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress. And even if it would fix mirred egress->ingress loops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]
Quoted 4 lines
Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.
TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?
thanks!
Signature · 7 lines
davide [1] by the way: the POC patch at https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075 silcences lockdep false warnings, and it preserves the splat when the real deadlock happens with TC marred egress. If you agree I will send it soon to this ML for review.
Jamal Hadi Salim
maintainerRe: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
Quoted 19 lines · Davide Caratti
hello Jamal, thanks for looking at this!
Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).[...]
Quoted 3 lines
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
Quoted 8 lines
[...]
Quoted 3 lines
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.
Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...
Quoted 9 lines
I don't see it much different than what we can obtain with bridges:
# ip link add name one type veth peer name two # ip link add name three type veth peer name four # for n in even odd; do ip link add name $n type bridge; done # for n in one two three four even odd; do ip link set dev $n up; done # for n in one three; do ip link set dev $n master odd; done # for n in two four; do ip link set dev $n master even; done
Sure that is another way to reproduce.
Quoted 5 lines
there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of
bpf_redirect()helper)
I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)
Quoted 11 lines
Quoted 2 lines
reverting the patch fixes things and it gets caught by the nested recursion check.
the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress.
And even if it would fix mirred
egress->ingressloops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]
Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on egress->egress (the example only showed ingress-ingress).
We will try restoring the ttl and see if it continues to work with your patch intact... unless there are other ideas.
Quoted 8 lines
Quoted 4 lines
Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?
I dont think running STP will save us from this, unless i am mistaken. This happens within the kernel before the packet hits the "wire". Besides this happens without using any bridging.
Quoted 5 lines
[1] by the way: the POC patch at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075
silcences lockdep false warnings, and it preserves the splat when the real deadlock happens with TC marred egress. If you agree I will send it soon to this ML for review.
please do.
cheers,
jamal
Jamal Hadi Salim
maintainerRe: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
Hi Davide,
Quoted 45 lines · Jamal Hadi Salim
Quoted 19 lines · Davide Caratti
hello Jamal, thanks for looking at this!
Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti
William reports kernel soft-lockups on some OVS topologies when TC mirred
egress->ingressaction is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).[...]
Quoted 3 lines
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
Quoted 8 lines
[...]
Quoted 3 lines
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.
Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...
Quoted 9 lines
I don't see it much different than what we can obtain with bridges:
# ip link add name one type veth peer name two # ip link add name three type veth peer name four # for n in even odd; do ip link add name $n type bridge; done # for n in one two three four even odd; do ip link set dev $n up; done # for n in one three; do ip link set dev $n master odd; done # for n in two four; do ip link set dev $n master even; done
Quoted 1 line
Sure that is another way to reproduce.
Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?
cheers,
jamal
[1]https://lore.kernel.org/all/[email protected]/
Quoted 53 lines
Quoted 5 lines
there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of
bpf_redirect()helper)I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)
Quoted 11 lines
Quoted 2 lines
reverting the patch fixes things and it gets caught by the nested recursion check.
the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress.
And even if it would fix mirred
egress->ingressloops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on
egress->egress(the example only showed ingress-ingress).We will try restoring the ttl and see if it continues to work with your patch intact... unless there are other ideas.
Quoted 8 lines
Quoted 4 lines
Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?I dont think running STP will save us from this, unless i am mistaken. This happens within the kernel before the packet hits the "wire". Besides this happens without using any bridging.
Quoted 5 lines
[1] by the way: the POC patch at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075
silcences lockdep false warnings, and it preserves the splat when the real deadlock happens with TC marred egress. If you agree I will send it soon to this ML for review.please do.
cheers,
jamal
Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
hello Jamal,
Quoted 2 lines · Jamal Hadi Salim
Hi Davide,
[...]
Quoted 28 lines
Quoted 28 lines
Quoted 4 lines
Quoted 3 lines
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
Quoted 8 lines
[...]
Quoted 3 lines
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.
Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...
Quoted 9 lines
I don't see it much different than what we can obtain with bridges:
# ip link add name one type veth peer name two # ip link add name three type veth peer name four # for n in even odd; do ip link add name $n type bridge; done # for n in one two three four even odd; do ip link set dev $n up; done # for n in one three; do ip link set dev $n master odd; done # for n in two four; do ip link set dev $n master even; done
Quoted 7 lines
Quoted 1 line
Sure that is another way to reproduce.
Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?
I'm also more optimistic on the use of qdisc cb for that purpose :) Just share the code, i will be happy to review/test.
With regular TC mirred, the deadlock happened with TCP and SCTP socket locks, as they were sending an ACK back for a packet that was sent by the peer using egress->ingress.
AFAIR there is a small reproducer in tc_actions.sh kselftest, namely
mirred_egress_to_ingress_tcp_test()
maybe it's useful for pre-verification also.
[...]
my 2 cents below:
Quoted 4 lines
Quoted 4 lines
I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)
[...]
Quoted 3 lines
Quoted 3 lines
Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on
egress->egress(the example only showed ingress-ingress).
if you try to make a loop using mirred egress/redirect, the first packet will trigger a deadlock on the root qdisc lock - see [1]. It's worse than a loop, because user can't fix it by just removing the "offending" mirred action. Would the ttl be helpful here?
(in the meanwhile, I ill try to figure out if it's possible at least to silence false lockdep warnings without using dynamic keys, as per Eric reply).
TIA!
Signature · 3 lines
davide [1] https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200
Jamal Hadi Salim
maintainerRe: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress
Hi Davide,
Quoted 51 lines · Davide Caratti
hello Jamal,
Quoted 2 lines · Jamal Hadi Salim
Hi Davide,
[...]
Quoted 28 lines
Quoted 28 lines
Quoted 4 lines
Quoted 3 lines
I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):
Quoted 8 lines
[...]
Quoted 3 lines
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0
the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.
Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...
Quoted 9 lines
I don't see it much different than what we can obtain with bridges:
# ip link add name one type veth peer name two # ip link add name three type veth peer name four # for n in even odd; do ip link add name $n type bridge; done # for n in one two three four even odd; do ip link set dev $n up; done # for n in one three; do ip link set dev $n master odd; done # for n in two four; do ip link set dev $n master even; doneQuoted 7 lines
Quoted 1 line
Sure that is another way to reproduce.
Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?
I'm also more optimistic on the use of qdisc cb for that purpose :) Just share the code, i will be happy to review/test.
With regular TC mirred, the deadlock happened with TCP and SCTP socket locks, as they were sending an ACK back for a packet that was sent by the peer usingegress->ingress.
Quoted 6 lines
AFAIR there is a small reproducer in
tc_actions.shkselftest, namely
mirred_egress_to_ingress_tcp_test()maybe it's useful for pre-verification also.
Ah, good - didnt realize there was a reproducer for your use case. We'll try it out.
Quoted 11 lines
[...]
my 2 cents below:
Quoted 4 lines
Quoted 4 lines
I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)
[...]
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines
Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on
egress->egress(the example only showed ingress-ingress).if you try to make a loop using mirred egress/redirect, the first packet will trigger a deadlock on the root qdisc lock - see [1]. It's worse than a loop, because user can't fix it by just removing the "offending" mirred action. Would the ttl be helpful here?
Possible. So the test is just to create a loop?
Lets test with the reproducer you pointed out then see where we go from there.
Quoted 4 lines
(in the meanwhile, I ill try to figure out if it's possible at least to silence false lockdep warnings without using dynamic keys, as per Eric reply).
Sorry, wasnt helpful - i have been in travel mode for the last week.
cheers,
jamal
Quoted 8 lines
TIA!
--
davide[1] https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200
Questions regarding "act_mirred: use the backlog for nested calls to mirred ingress"
In reply to Davide CarattiHello Davide,
In the following commit:
commit ca22da2fbd693b54dc8e3b7b54ccc9f7e9ba3640
Author: Davide Caratti <[email protected]>
Date: Fri Jan 20 18:01:40 2023 +0100
act_mirred: use the backlog for nested calls to mirred ingress
you added the mirred_egress_to_ingress_tcp_test kselftest that, I noticed, hangs with "Ncat: TIMEOUT." if openvswitch module is loaded.
Is this the right behaviour or was such configuration not intended?
Regards,
Lenar Khannanov