nf_flowtable: ensure dst.dev is not blackhole
Subsystems (2)
PATCH 1/1 nf_flowtable: ensure dst.dev is not blackhole
Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev even if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.
--- net/netfilter/nf_flow_table_ip.c | 5 +++++ 1 file changed, 5 insertions(+)
net/netfilter/nf_flow_table_ip.c +5 -0
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,if (nf_flow_state_check (flow, ip6h->nexthdr, skb, thoff))return NF_ACCEPT ; +if (unlikely (tuplehash->tuple.dst_cache->dev == blackhole_netdev)) { +flow_offload_teardown (flow); +return NF_ACCEPT ; + } +if (skb_try_make_writable (skb, thoff + hdrsize))return NF_DROP ;
Signature · 1 line
2.25.1
Pablo Neira Ayuso
maintainerHi,
Quoted 7 lines · Ritaro Takenaka
Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls
dst_dev_put()which makes dst.devblackhole_netdeveven if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit227e1e4d0d6c(netfilter:nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.
dst_check() should deal with this.
In 5.4, this check is only enabled for xfrm.
Quoted 24 lines
Signed-off-by Ritaro Takenaka <[email protected]>--- net/netfilter/nf_flow_table_ip.c | 5 +++++ 1 file changed, 5 insertions(+)net/netfilter/nf_flow_table_ip.c +5
-0@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,if (nf_flow_state_check (flow, ip6h->nexthdr, skb, thoff))return NF_ACCEPT ; +if (unlikely (tuplehash->tuple.dst_cache->dev == blackhole_netdev)) { +flow_offload_teardown (flow); +return NF_ACCEPT ; + } +if (skb_try_make_writable (skb, thoff + hdrsize))return NF_DROP ;Signature · 1 line
2.25.1
Thanks for your reply.
Quoted 1 line
In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit_neigh (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check() periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.
Quoted 1 line
dst_check()should deal with this.
When dst_check() is used, the performance degradation is not negligible. From 940 Mbps to 700 Mbps with QCA9563 simple firewall.
Quoted 40 lines · Pablo Neira Ayuso
Hi,
Quoted 7 lines · Ritaro Takenaka
Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls
dst_dev_put()which makes dst.devblackhole_netdeveven if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit227e1e4d0d6c(netfilter:nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.
dst_check()should deal with this.In 5.4, this check is only enabled for xfrm.
Quoted 24 lines
Signed-off-by Ritaro Takenaka <[email protected]>--- net/netfilter/nf_flow_table_ip.c | 5 +++++ 1 file changed, 5 insertions(+)net/netfilter/nf_flow_table_ip.c +5
-1@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,if (nf_flow_state_check (flow, ip6h->nexthdr, skb, thoff))return NF_ACCEPT ; +if (unlikely (tuplehash->tuple.dst_cache->dev == blackhole_netdev)) { +flow_offload_teardown (flow); +return NF_ACCEPT ; + } +if (skb_try_make_writable (skb, thoff + hdrsize))return NF_DROP ; --2.25.1
Thanks for your reply.
Quoted 1 line
In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.
Quoted 1 line
dst_check()should deal with this.
When dst_check is used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.
I am sorry if you have received this mail twice.
Quoted 39 lines · Pablo Neira Ayuso
Hi,
Quoted 7 lines · Ritaro Takenaka
Fixes sporadic IPv6 packet loss when flow offloading is enabled. IPv6 route GC calls
dst_dev_put()which makes dst.devblackhole_netdeveven if dst is cached in flow offload. If a packet passes through this invalid flow, packet loss will occur.
This is from Commit227e1e4d0d6c(netfilter:nf_flowtable: skip device lookup from interface index), as outdev was cached independently before. Packet loss is reported on OpenWrt with Linux 5.4 and later.
dst_check()should deal with this.In 5.4, this check is only enabled for xfrm.
Quoted 24 lines
Signed-off-by Ritaro Takenaka <[email protected]>--- net/netfilter/nf_flow_table_ip.c | 5 +++++ 1 file changed, 5 insertions(+)net/netfilter/nf_flow_table_ip.c +5
-0@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,if (nf_flow_state_check (flow, ip6h->nexthdr, skb, thoff))return NF_ACCEPT ; +if (unlikely (tuplehash->tuple.dst_cache->dev == blackhole_netdev)) { +flow_offload_teardown (flow); +return NF_ACCEPT ; + } +if (skb_try_make_writable (skb, thoff + hdrsize))return NF_DROP ;Signature · 1 line
2.25.1
Pablo Neira Ayuso
maintainerQuoted 11 lines · Ritaro Takenaka
Thanks for your reply.
Quoted 1 line
In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runsdst_checkperiodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.Quoted 1 line
dst_check()should deal with this.When
dst_checkis used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.
You mention 5.10 above.
Starting 5.12, dst_check() uses INDIRECT_CALL_INET.
Is dst_check() still slow with >= 5.12?
Asking this because my understanding (at this stage) is that this check for blackhole_netdev is a faster way to check for stale cached routes.
Quoted 14 lines · Ritaro Takenaka
Quoted 11 lines
Thanks for your reply.
Quoted 1 line
In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runsdst_checkperiodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.Quoted 1 line
dst_check()should deal with this.When
dst_checkis used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.You mention 5.10 above.
Quoted 7 lines
Starting 5.12,
dst_check()usesINDIRECT_CALL_INET.Is
dst_check()still slow with >= 5.12?Asking this because my understanding (at this stage) is that this check for
blackhole_netdevis a faster way to check for stale cached routes.
I did the performance tests with 5.15, confirmed dst_check() is not slower than checking for blackhole_netdev.
Good, dst_check() can be used.
Then, stale routes check should be moved from nf_flow_offload_gc_step() to nf_flow_offload(_ipv6)_hook(). Is it correct?
Pablo Neira Ayuso
maintainerQuoted 23 lines · Ritaro Takenaka
Quoted 14 lines · Ritaro Takenaka
Quoted 11 lines
Thanks for your reply.
Quoted 1 line
In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runsdst_checkperiodically. Route GC and flowtable GC are not synchronized, so it is necessary to check each packet.Quoted 1 line
dst_check()should deal with this.When
dst_checkis used, the performance degradation is not negligible. From 900 Mbps to 700 Mbps with QCA9563 simple firewall.You mention 5.10 above.
Quoted 7 lines
Starting 5.12,
dst_check()usesINDIRECT_CALL_INET.Is
dst_check()still slow with >= 5.12?Asking this because my understanding (at this stage) is that this check for
blackhole_netdevis a faster way to check for stale cached routes.
Quoted 7 lines
I did the performance tests with 5.15, confirmed
dst_check()is not slower than checking forblackhole_netdev.Good,
dst_check()can be used.Then, stale routes check should be moved from
nf_flow_offload_gc_step()to nf_flow_offload(_ipv6)_hook(). Is it correct?
Then, the check from packet path needs to be restored.