Skip to contentSkip to conversation

net/sched: use the backlog for nested mirred ingress

PATCH Applied Davide Caratti 5 participants 15 messages Jan 20, 2023 – Aug 2, 2024

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover net/sched: use the backlog for nested mirred ingress

TC mirred has a protection against excessive stack growth, but that protection doesn't really guarantee the absence of recursion, nor it guards against loops. Patch 1/2 rewords "recursion" to "nesting" to make this more clear.
We can leverage on this existing mechanism to prevent TCP / SCTP from doing soft lock-up in some specific scenarios that uses mirred egress->ingress:
patch 2 changes mirred so that the networking backlog is used for nested mirred ingress actions.

Davide Caratti (2):
net/sched: act_mirred: better wording on protection against excessive
stack growth
act_mirred: use the backlog for nested calls to mirred ingress

 net/sched/act_mirred.c                        | 23 ++++++---
 .../selftests/net/forwarding/tc_actions.sh    | 49 ++++++++++++++++++-
 2 files changed, 63 insertions(+), 9 deletions(-)
Signature · 1 line
2.38.1
#1 lore raw

PATCH 1/2 net/sched: act_mirred: better wording on protection against excessive stack growth

with commit e2ca070f89ec ("net: sched: protect against stack overflow in TC act_mirred"), act_mirred protected itself against excessive stack growth using per_cpu counter of nested calls to tcf_mirred_act(), and capping it to MIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.

CC Jamal Hadi Salim <[email protected]>
---
 net/sched/act_mirred.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
net/sched/act_mirred.c +8 -8
index 7284bcea7b0b..c8abb5136491 100644
@@ -29,8 +29,8 @@
 static LIST_HEAD(mirred_list);
 static DEFINE_SPINLOCK(mirred_list_lock);
 
-#define MIRRED_RECURSION_LIMIT    4
-static DEFINE_PER_CPU(unsigned int, mirred_rec_level);
+#define MIRRED_NEST_LIMIT    4
+static DEFINE_PER_CPU(unsigned int, mirred_nest_level);
 
 static bool tcf_mirred_is_act_redirect(int action)
 {
@@ -226,7 +226,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,
 	struct sk_buff *skb2 = skb;
 	bool m_mac_header_xmit;
 	struct net_device *dev;
-	unsigned int rec_level;
+	unsigned int nest_level;
 	int retval, err = 0;
 	bool use_reinsert;
 	bool want_ingress;
@@ -237,11 +237,11 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,
 	int mac_len;
 	bool at_nh;
 
-	rec_level = __this_cpu_inc_return(mirred_rec_level);
-	if (unlikely(rec_level > MIRRED_RECURSION_LIMIT)) {
+	nest_level = __this_cpu_inc_return(mirred_nest_level);
+	if (unlikely(nest_level > MIRRED_NEST_LIMIT)) {
 		net_warn_ratelimited("Packet exceeded mirred recursion limit on dev %s\n",
 				     netdev_name(skb->dev));
-		__this_cpu_dec(mirred_rec_level);
+		__this_cpu_dec(mirred_nest_level);
 		return TC_ACT_SHOT;
 	}
 
@@ -310,7 +310,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,
 			err = tcf_mirred_forward(want_ingress, skb);
 			if (err)
 				tcf_action_inc_overlimit_qstats(&m->common);
-			__this_cpu_dec(mirred_rec_level);
+			__this_cpu_dec(mirred_nest_level);
 			return TC_ACT_CONSUMED;
 		}
 	}
@@ -322,7 +322,7 @@ TC_INDIRECT_SCOPE int tcf_mirred_act(struct sk_buff *skb,
 		if (tcf_mirred_is_act_redirect(m_eaction))
 			retval = TC_ACT_SHOT;
 	}
-	__this_cpu_dec(mirred_rec_level);
+	__this_cpu_dec(mirred_nest_level);
 
 	return retval;
 }
Signature · 1 line
2.38.1
#4 lore raw

PATCH 2/2 act_mirred: use the backlog for nested calls to mirred ingress

In reply to Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when tcf_mirred_act() has a nest level greater than one:
change tcf_mirred_forward() accordingly.

Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred egress->ingress (when the nest level is 1).

[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and return NET_RX_SUCCESS inside
tcf_mirred_forward().

---
 net/sched/act_mirred.c                        |  7 +++
 .../selftests/net/forwarding/tc_actions.sh    | 49 ++++++++++++++++++-
 2 files changed, 55 insertions(+), 1 deletion(-)
net/sched/act_mirred.c +7 -0
index c8abb5136491..8037ec9b1d31 100644
@@ -206,12 +206,19 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
 	return err;
 }
 
+static bool is_mirred_nested(void)
+{
+	return unlikely(__this_cpu_read(mirred_nest_level) > 1);
+}
+
 static int tcf_mirred_forward(bool want_ingress, struct sk_buff *skb)
 {
 	int err;
 
 	if (!want_ingress)
 		err = tcf_dev_queue_xmit(skb, dev_queue_xmit);
+	else if (is_mirred_nested())
+		err = netif_rx(skb);
 	else
 		err = netif_receive_skb(skb);
 
tools/testing/selftests/net/forwarding/tc_actions.sh +48 -1

61 lines not on this page · raw message

Signature · 1 line
2.38.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH net-next 1/2] net/sched: act_mirred: better wording on protection against excessive stack growth

In reply to Davide Caratti
Quoted 10 lines · Davide Caratti

with commit e2ca070f89ec ("net: sched: protect against stack overflow in TC act_mirred"), act_mirred protected itself against excessive stack growth using per_cpu counter of nested calls to tcf_mirred_act(), and capping it to MIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.

CC Jamal Hadi Salim <[email protected]>
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

In reply to Davide Caratti
Quoted 24 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when tcf_mirred_act() has a nest level greater than one:
change tcf_mirred_forward() accordingly.

Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred egress->ingress (when the nest level is 1).

[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and return NET_RX_SUCCESS inside
tcf_mirred_forward().

Quoted 3 lines
maintainer Acked-by
#3 lore raw

Re: [PATCH net-next 1/2] net/sched: act_mirred: better wording on protection against excessive stack growth

In reply to Marcelo Ricardo Leitner
Quoted 10 lines · Marcelo Ricardo Leitner
Quoted 8 lines · Davide Caratti

with commit e2ca070f89ec ("net: sched: protect against stack overflow in TC act_mirred"), act_mirred protected itself against excessive stack growth using per_cpu counter of nested calls to tcf_mirred_act(), and capping it to MIRRED_RECURSION_LIMIT. However, such protection does not detect recursion/loops in case the packet is enqueued to the backlog (for example, when the mirred target device has RPS or skb timestamping enabled). Change the wording from "recursion" to "nesting" to make it more clear to readers.

Quoted 4 lines
Quoted 2 lines
CC Jamal Hadi Salim <[email protected]>

cheers,
jamal

maintainer Acked-by
#6 lore raw

Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

In reply to Marcelo Ricardo Leitner
Quoted 26 lines · Marcelo Ricardo Leitner
Quoted 24 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when tcf_mirred_act() has a nest level greater than one:
change tcf_mirred_forward() accordingly.

Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred egress->ingress (when the nest level is 1).

[1] https://lore.kernel.org/netdev/33dc43f587ec1388ba456b4915c75f02a8aae226.1663945716.git.dcaratti@redhat.com/
[2] https://lore.kernel.org/netdev/Y0w%[email protected]/
[3] such behavior is not guaranteed: for example, if RPS or skb RX
timestamping is enabled on the mirred target device, the kernel
can defer receiving the skb and return NET_RX_SUCCESS inside
tcf_mirred_forward().

Quoted 5 lines
Quoted 3 lines

cheers,
jamal

#14 lore raw
In reply to Davide Caratti
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:

Quoted 10 lines · On Fri, 20 Jan 2023 18:01:38 +0100 you wrote:

TC mirred has a protection against excessive stack growth, but that protection doesn't really guarantee the absence of recursion, nor it guards against loops. Patch 1/2 rewords "recursion" to "nesting" to make this more clear.
We can leverage on this existing mechanism to prevent TCP / SCTP from doing soft lock-up in some specific scenarios that uses mirred egress->ingress:
patch 2 changes mirred so that the networking backlog is used for nested mirred ingress actions.

[...]

Here is the summary with links:
- [net-next,1/2] net/sched: act_mirred: better wording on protection against excessive stack growth

    https://git.kernel.org/netdev/net-next/c/78dcdffe0418
  - [net-next,2/2] act_mirred: use the backlog for nested calls to mirred ingress
    https://git.kernel.org/netdev/net-next/c/ca22da2fbd69

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
maintainer
#7 lore raw

Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

In reply to Davide Caratti
Quoted 14 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

Enqueueing to backlog proved to fix this soft lockup; however, as Cong noticed [2], we should preserve - when possible - the current mirred behavior that counts as "overlimits" any eventual packet drop subsequent to the mirred forwarding action [3]. A compromise solution might use the backlog only when tcf_mirred_act() has a nest level greater than one:
change tcf_mirred_forward() accordingly.

Quoted 4 lines

Also, add a kselftest that can reproduce the lockup and verifies TC mirred ability to account for further packet drops after TC mirred egress->ingress (when the nest level is 1).

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

----
sudo ip netns add p4node
sudo ip link add p4port0 address 10:00:00:01:AA:BB type veth peer port0 address 10:00:00:02:AA:BB
sudo ip link set dev port0 netns p4node
sudo ip a add 10.0.0.1/24 dev p4port0
sudo ip neigh add 10.0.0.2 dev p4port0 lladdr 10:00:00:02:aa:bb sudo ip netns exec p4node ip a add 10.0.0.2/24 dev port0 sudo ip netns exec p4node ip l set dev port0 up
sudo ip l set dev p4port0 up
sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

ping -I p4port0 10.0.0.2 -c 1
-----

reverting the patch fixes things and it gets caught by the nested recursion check.

Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.

cheers,
jamal

#8 lore raw

Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

hello Jamal, thanks for looking at this!

Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

[...]

Quoted 3 lines

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

[...]

Quoted 3 lines

sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop. I don't see it much different than what we can obtain with bridges:

# ip link add name one type veth peer name two
# ip link add name three type veth peer name four
# for n in even odd; do ip link add name $n type bridge; done
# for n in one two three four even odd; do ip link set dev $n up; done
# for n in one three; do ip link set dev $n master odd; done
# for n in two four; do ip link set dev $n master even; done

there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of bpf_redirect() helper)

Quoted 2 lines

reverting the patch fixes things and it gets caught by the nested recursion check.

the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress. And even if it would fix mirred egress->ingress loops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]

Quoted 4 lines

Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.

TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?

thanks!

Signature · 7 lines
davide

[1] by the way: the POC patch at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075
silcences lockdep false warnings, and it preserves the splat when the
real deadlock happens with TC marred egress. If you agree I will send
it soon to this ML for review.
maintainer
#9 lore raw

Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

Quoted 19 lines · Davide Caratti

hello Jamal, thanks for looking at this!

Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

[...]

Quoted 3 lines

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

Quoted 8 lines

[...]

Quoted 3 lines

sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.

Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...

Quoted 9 lines

I don't see it much different than what we can obtain with bridges:

# ip link add name one type veth peer name two
# ip link add name three type veth peer name four
# for n in even odd; do ip link add name $n type bridge; done
# for n in one two three four even odd; do ip link set dev $n up; done
# for n in one three; do ip link set dev $n master odd; done
# for n in two four; do ip link set dev $n master even; done

Sure that is another way to reproduce.

Quoted 5 lines

there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of bpf_redirect() helper)

I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)

Quoted 11 lines
Quoted 2 lines

reverting the patch fixes things and it gets caught by the nested recursion check.

the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress.

And even if it would fix mirred egress->ingress loops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]

Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on egress->egress (the example only showed ingress-ingress).

We will try restoring the ttl and see if it continues to work with your patch intact... unless there are other ideas.

Quoted 8 lines
Quoted 4 lines

Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.

TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?

I dont think running STP will save us from this, unless i am mistaken. This happens within the kernel before the packet hits the "wire". Besides this happens without using any bridging.

Quoted 5 lines

[1] by the way: the POC patch at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075
silcences lockdep false warnings, and it preserves the splat when the real deadlock happens with TC marred egress. If you agree I will send it soon to this ML for review.

please do.

cheers,
jamal

maintainer
#10 lore raw

Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

Hi Davide,

Quoted 45 lines · Jamal Hadi Salim
Quoted 19 lines · Davide Caratti

hello Jamal, thanks for looking at this!

Quoted 8 lines · Jamal Hadi Salim
Quoted 6 lines · Davide Caratti

William reports kernel soft-lockups on some OVS topologies when TC mirred egress->ingress action is hit by local TCP traffic [1].
The same can also be reproduced with SCTP (thanks Xin for verifying), when client and server reach themselves through mirred egress to ingress, and one of the two peers sends a "heartbeat" packet (from within a timer).

[...]

Quoted 3 lines

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

Quoted 8 lines

[...]

Quoted 3 lines

sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.

Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...

Quoted 9 lines

I don't see it much different than what we can obtain with bridges:

# ip link add name one type veth peer name two
# ip link add name three type veth peer name four
# for n in even odd; do ip link add name $n type bridge; done
# for n in one two three four even odd; do ip link set dev $n up; done
# for n in one three; do ip link set dev $n master odd; done
# for n in two four; do ip link set dev $n master even; done
Quoted 1 line

Sure that is another way to reproduce.

Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?

cheers,
jamal

[1]https://lore.kernel.org/all/[email protected]/

Quoted 53 lines
Quoted 5 lines

there is a practical difference: with bridges we have protocols (like STP) that can detect and act-upon loops - while TC mirred needs some facility on top (not 100% sure, but the same might also apply to similar tools, such as users of bpf_redirect() helper)

I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)

Quoted 11 lines
Quoted 2 lines

reverting the patch fixes things and it gets caught by the nested recursion check.

the price of that revert is: we'll see those soft-lockups again with L4 protocols when peers communicate through mirred egress -> ingress.

And even if it would fix mirred egress->ingress loops, we would still suffer from soft-lockups (on the qdisc root lock) when the same rule is done with mirred egress (see an example at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200)
[1]

Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on egress->egress (the example only showed ingress-ingress).

We will try restoring the ttl and see if it continues to work with your patch intact... unless there are other ideas.

Quoted 8 lines
Quoted 4 lines

Frankly, I believe we should restore a proper ttl from what was removed here:
https://lore.kernel.org/all/[email protected]/
The headaches(and time consumed) trying to save the 3-4 bits removing the ttl field is not worth it imo.

TTL would protect us against loops when they are on the same node:
what do you think about inserting a rule that detects BPDU before the mirred ingress rule?

I dont think running STP will save us from this, unless i am mistaken. This happens within the kernel before the packet hits the "wire". Besides this happens without using any bridging.

Quoted 5 lines

[1] by the way: the POC patch at
https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782654075
silcences lockdep false warnings, and it preserves the splat when the real deadlock happens with TC marred egress. If you agree I will send it soon to this ML for review.

please do.

cheers,
jamal

#11 lore raw

Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

hello Jamal,

Quoted 2 lines · Jamal Hadi Salim

Hi Davide,

[...]

Quoted 28 lines
Quoted 28 lines
Quoted 4 lines
Quoted 3 lines

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

Quoted 8 lines

[...]

Quoted 3 lines

sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.

Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...

Quoted 9 lines

I don't see it much different than what we can obtain with bridges:

# ip link add name one type veth peer name two
# ip link add name three type veth peer name four
# for n in even odd; do ip link add name $n type bridge; done
# for n in one two three four even odd; do ip link set dev $n up; done
# for n in one three; do ip link set dev $n master odd; done
# for n in two four; do ip link set dev $n master even; done
Quoted 7 lines
Quoted 1 line

Sure that is another way to reproduce.

Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?

I'm also more optimistic on the use of qdisc cb for that purpose :) Just share the code, i will be happy to review/test.
With regular TC mirred, the deadlock happened with TCP and SCTP socket locks, as they were sending an ACK back for a packet that was sent by the peer using egress->ingress.

AFAIR there is a small reproducer in tc_actions.sh kselftest, namely

mirred_egress_to_ingress_tcp_test()

maybe it's useful for pre-verification also.

[...]

my 2 cents below:

Quoted 4 lines
Quoted 4 lines

I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)

[...]

Quoted 3 lines
Quoted 3 lines

Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on egress->egress (the example only showed ingress-ingress).

if you try to make a loop using mirred egress/redirect, the first packet will trigger a deadlock on the root qdisc lock - see [1]. It's worse than a loop, because user can't fix it by just removing the "offending" mirred action. Would the ttl be helpful here?

(in the meanwhile, I ill try to figure out if it's possible at least to silence false lockdep warnings without using dynamic keys, as per Eric reply).

TIA!

Signature · 3 lines
davide

[1] https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200
maintainer
#12 lore raw

Re: Mirred broken WAS(Re: [PATCH net-next 2/2] act_mirred: use the backlog for nested calls to mirred ingress

Hi Davide,

Quoted 51 lines · Davide Caratti

hello Jamal,

Quoted 2 lines · Jamal Hadi Salim

Hi Davide,

[...]

Quoted 28 lines
Quoted 28 lines
Quoted 4 lines
Quoted 3 lines

I am afraid this broke things. Here's a simple use case which causes an infinite loop (that we found while testing blockcasting but simplified to demonstrate the issue):

Quoted 8 lines

[...]

Quoted 3 lines

sudo ip netns exec p4node tc qdisc add dev port0 clsact sudo ip netns exec p4node tc filter add dev port0 ingress protocol ip prio 10 matchall action mirred ingress redirect dev port0

the above rule is taking packets from port0 ingress and putting it again in the mirred ingress of the same device, hence the loop.

Right - that was intentional to show the loop. We are worrying about extending mirred now to also broadcast (see the blockcast discussion) to more ports making the loop even worse. The loop should terminate at some point - in this case it does not...

Quoted 9 lines

I don't see it much different than what we can obtain with bridges:

# ip link add name one type veth peer name two
# ip link add name three type veth peer name four
# for n in even odd; do ip link add name $n type bridge; done
# for n in one two three four even odd; do ip link set dev $n up; done
# for n in one three; do ip link set dev $n master odd; done
# for n in two four; do ip link set dev $n master even; done
Quoted 7 lines
Quoted 1 line

Sure that is another way to reproduce.

Ok, so i can verify that re-introduction of the ttl field in the skb[1] fixes the issue. But restoring that patch may cause too much bikeshedding. Victor will work on a better approach using the cb struct instead - there may. Are you able to test with/out your patch and see if this same patch fixes it?

I'm also more optimistic on the use of qdisc cb for that purpose :) Just share the code, i will be happy to review/test.
With regular TC mirred, the deadlock happened with TCP and SCTP socket locks, as they were sending an ACK back for a packet that was sent by the peer using egress->ingress.

Quoted 6 lines

AFAIR there is a small reproducer in tc_actions.sh kselftest, namely

mirred_egress_to_ingress_tcp_test()

maybe it's useful for pre-verification also.

Ah, good - didnt realize there was a reproducer for your use case. We'll try it out.

Quoted 11 lines

[...]

my 2 cents below:

Quoted 4 lines
Quoted 4 lines

I dont think we can run something equivalent inside the kernel. The ttl worked fine. BTW, the example shown breaks even when you have everything running on a single cpu (and packets being queued on the backlog)

[...]

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

Yes, we need to make sure those are fixed with whatever replacement.. The loops will happen even on egress->egress (the example only showed ingress-ingress).

if you try to make a loop using mirred egress/redirect, the first packet will trigger a deadlock on the root qdisc lock - see [1]. It's worse than a loop, because user can't fix it by just removing the "offending" mirred action. Would the ttl be helpful here?

Possible. So the test is just to create a loop?
Lets test with the reproducer you pointed out then see where we go from there.

Quoted 4 lines

(in the meanwhile, I ill try to figure out if it's possible at least to silence false lockdep warnings without using dynamic keys, as per Eric reply).

Sorry, wasnt helpful - i have been in travel mode for the last week.

cheers,
jamal

Quoted 8 lines

TIA!

--
davide

[1] https://github.com/multipath-tcp/mptcp_net-next/issues/451#issuecomment-1782690200

#13 lore raw

Questions regarding "act_mirred: use the backlog for nested calls to mirred ingress"

In reply to Davide Caratti

Hello Davide,

In the following commit:

commit ca22da2fbd693b54dc8e3b7b54ccc9f7e9ba3640
Author: Davide Caratti <[email protected]>
Date:   Fri Jan 20 18:01:40 2023 +0100

    act_mirred: use the backlog for nested calls to mirred ingress

you added the mirred_egress_to_ingress_tcp_test kselftest that, I noticed, hangs with "Ncat: TIMEOUT." if openvswitch module is loaded.

Is this the right behaviour or was such configuration not intended?

Regards,
Lenar Khannanov