Skip to contentSkip to conversation

nf_flowtable: ensure dst.dev is not blackhole

PATCH In review Ritaro Takenaka 2 participants 7 messages Apr 25, 2022 – May 9, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nf_flowtable: ensure dst.dev is not blackhole

Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev even if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.

---
 net/netfilter/nf_flow_table_ip.c | 5 +++++
 1 file changed, 5 insertions(+)
net/netfilter/nf_flow_table_ip.c +5 -0
index 32c0eb1b4..12f81661d 100644
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
 	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
 		return NF_ACCEPT;
 
+	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
+		flow_offload_teardown(flow);
+		return NF_ACCEPT;
+	}
+
 	if (skb_try_make_writable(skb, thoff + hdrsize))
 		return NF_DROP;
 
Signature · 1 line
2.25.1
#1 lore raw

Hi,

Quoted 7 lines · Ritaro Takenaka

Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev even if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.

dst_check() should deal with this.

In 5.4, this check is only enabled for xfrm.

Quoted 24 lines
---
 net/netfilter/nf_flow_table_ip.c | 5 +++++
 1 file changed, 5 insertions(+)
net/netfilter/nf_flow_table_ip.c +5 -0
index 32c0eb1b4..12f81661d 100644
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
 	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
 		return NF_ACCEPT;
 
+	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
+		flow_offload_teardown(flow);
+		return NF_ACCEPT;
+	}
+
 	if (skb_try_make_writable(skb, thoff + hdrsize))
 		return NF_DROP;
 
Signature · 1 line
2.25.1
#2 lore raw

Thanks for your reply.

Quoted 1 line

In 5.4, this check is only enabled for xfrm.

Packet loss occurs with xmit_neigh (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check() periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.

Quoted 1 line

dst_check() should deal with this.

When dst_check() is used, the performance degradation is not negligible. From 940 Mbps to 700 Mbps with QCA9563 simple firewall.

Quoted 40 lines · Pablo Neira Ayuso

Hi,

Quoted 7 lines · Ritaro Takenaka

Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev even if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.

dst_check() should deal with this.

In 5.4, this check is only enabled for xfrm.

Quoted 24 lines
---
 net/netfilter/nf_flow_table_ip.c | 5 +++++
 1 file changed, 5 insertions(+)
net/netfilter/nf_flow_table_ip.c +5 -1
index 32c0eb1b4..12f81661d 100644
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
      if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
              return NF_ACCEPT;
 
+     if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
+             flow_offload_teardown(flow);
+             return NF_ACCEPT;
+     }
+
      if (skb_try_make_writable(skb, thoff + hdrsize))
              return NF_DROP;
 
--

2.25.1

#3 lore raw
In reply to Pablo Neira Ayuso

Thanks for your reply.

Quoted 1 line

In 5.4, this check is only enabled for xfrm.

Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.

Quoted 1 line

dst_check() should deal with this.

When dst_check is used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

I am sorry if you have received this mail twice.

Quoted 39 lines · Pablo Neira Ayuso

Hi,

Quoted 7 lines · Ritaro Takenaka

Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev even if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.

dst_check() should deal with this.

In 5.4, this check is only enabled for xfrm.

Quoted 24 lines
---
 net/netfilter/nf_flow_table_ip.c | 5 +++++
 1 file changed, 5 insertions(+)
net/netfilter/nf_flow_table_ip.c +5 -0
index 32c0eb1b4..12f81661d 100644
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
 	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
 		return NF_ACCEPT;
 
+	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
+		flow_offload_teardown(flow);
+		return NF_ACCEPT;
+	}
+
 	if (skb_try_make_writable(skb, thoff + hdrsize))
 		return NF_DROP;
 
Signature · 1 line
2.25.1
#4 lore raw
Quoted 11 lines · Ritaro Takenaka

Thanks for your reply.

Quoted 1 line

In 5.4, this check is only enabled for xfrm.

Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.

Quoted 1 line

dst_check() should deal with this.

When dst_check is used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

You mention 5.10 above.

Starting 5.12, dst_check() uses INDIRECT_CALL_INET.

Is dst_check() still slow with >= 5.12?

Asking this because my understanding (at this stage) is that this check for blackhole_netdev is a faster way to check for stale cached routes.

#5 lore raw
Quoted 14 lines · Ritaro Takenaka
Quoted 11 lines

Thanks for your reply.

Quoted 1 line

In 5.4, this check is only enabled for xfrm.

Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.

Quoted 1 line

dst_check() should deal with this.

When dst_check is used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

You mention 5.10 above.

Quoted 7 lines

Starting 5.12, dst_check() uses INDIRECT_CALL_INET.

Is dst_check() still slow with >= 5.12?

Asking this because my understanding (at this stage) is that this check for blackhole_netdev is a faster way to check for stale cached routes.

I did the performance tests with 5.15, confirmed dst_check() is not slower than checking for blackhole_netdev.

Good, dst_check() can be used.

Then, stale routes check should be moved from nf_flow_offload_gc_step() to nf_flow_offload(_ipv6)_hook(). Is it correct?

#6 lore raw
Quoted 23 lines · Ritaro Takenaka
Quoted 14 lines · Ritaro Takenaka
Quoted 11 lines

Thanks for your reply.

Quoted 1 line

In 5.4, this check is only enabled for xfrm.

Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.

Quoted 1 line

dst_check() should deal with this.

When dst_check is used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

You mention 5.10 above.

Quoted 7 lines

Starting 5.12, dst_check() uses INDIRECT_CALL_INET.

Is dst_check() still slow with >= 5.12?

Asking this because my understanding (at this stage) is that this check for blackhole_netdev is a faster way to check for stale cached routes.

Quoted 7 lines

I did the performance tests with 5.15, confirmed dst_check() is not slower than checking for blackhole_netdev.

Good, dst_check() can be used.

Then, stale routes check should be moved from nf_flow_offload_gc_step() to nf_flow_offload(_ipv6)_hook(). Is it correct?

Then, the check from packet path needs to be restored.