netfilter: nf_dup_netdev: scrub duplicates to preserve the direct path
Subsystems (3)
PATCH 1/1 netfilter: nf_dup_netdev: scrub duplicates to preserve the direct path
The nftables 'dup' action clones the skb with its full glory of metadata, including references to its destination and conntrack information. As a consequence, a link failure on the duplicate's egress path ends up doing the same as it would for the direct path, for example invalidating the original packet's destination, which typically breaks all TCP connections to that address.
In other words, the "dup" path has the potential to wreak havoc in the direct path as a consequence of secondary link failures. This is very bad behavior for a monitoring tool, which is the most obvious application of 'dup'.
This patch fixes all similar scenarii by calling skb_scrub_pkt() on the clone, severing its link to precious direct-path state.
Note: the second argument of skb_scrub_pkt(), the boolean "packet is crossing netns", is intentionally set to 'false', as a 'true' involves exaggerate scrubbing, e.g. of the timestamp, which a monitoring 'dup' typically wants to preserve.
--- net/netfilter/nf_dup_netdev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
net/netfilter/nf_dup_netdev.c +3 -1
@@ -63,8 +63,10 @@ void nf_dup_netdev_egress(const struct nft_pktinfo *pkt, int oif)return ; skb =skb_clone (pkt->skb,GFP_ATOMIC ); -if (skb) +if (skb) { +skb_scrub_packet (skb, false);nf_do_netdev_egress (skb, dev,nft_hook (pkt)); + } }EXPORT_SYMBOL_GPL (nf_dup_netdev_egress);
Signature · 1 line
2.47.3
Pablo Neira Ayuso
maintainerHi,
Quoted 11 lines · Alexandre Ferrieux
The nftables 'dup' action clones the skb with its full glory of metadata, including references to its destination and conntrack information. As a consequence, a link failure on the duplicate's egress path ends up doing the same as it would for the direct path, for example invalidating the original packet's destination, which typically breaks all TCP connections to that address.
In other words, the "dup" path has the potential to wreak havoc in the direct path as a consequence of secondary link failures. This is very bad behavior for a monitoring tool, which is the most obvious application of 'dup'.
Can you describe your use-case a bit and how it breaks?
Quoted 2 lines
This patch fixes all similar scenarii by calling
skb_scrub_pkt()on the clone, severing its link to precious direct-path state.
This patch is targetted at the net tree, but nf.git is preferred.
Quoted 4 lines
Note: the second argument of
skb_scrub_pkt(), the boolean "packet is crossing netns", is intentionally set to 'false', as a 'true' involves exaggerate scrubbing, e.g. of the timestamp, which a monitoring 'dup' typically wants to preserve.
Yes, the skb->mark should really remain in place for the duplication. As for the conntrack and dst, you have to explain what it breaks on your end.
Quoted 25 lines
Signed-off-by Alexandre Ferrieux <[email protected]>--- net/netfilter/nf_dup_netdev.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/netfilter/nf_dup_netdev.c +3
-1@@ -63,8 +63,10 @@ void nf_dup_netdev_egress(const struct nft_pktinfo *pkt, int oif)return ; skb =skb_clone (pkt->skb,GFP_ATOMIC ); -if (skb) +if (skb) { +skb_scrub_packet (skb, false);nf_do_netdev_egress (skb, dev,nft_hook (pkt)); + } }EXPORT_SYMBOL_GPL (nf_dup_netdev_egress);Signature · 1 line
2.47.3
Quoted 10 lines · Pablo Neira Ayuso
Hi,
Quoted 7 lines · Alexandre Ferrieux
The nftables 'dup' action clones the skb with its full glory of metadata, including references to its destination and conntrack information. As a consequence, a link failure on the duplicate's egress path ends up doing the same as it would for the direct path, for example invalidating the original packet's destination, which typically breaks all TCP connections to that address.
Quoted 6 lines
Quoted 4 lines
In other words, the "dup" path has the potential to wreak havoc in the direct path as a consequence of secondary link failures. This is very bad behavior for a monitoring tool, which is the most obvious application of 'dup'.
Can you describe your use-case a bit and how it breaks?
Sure:
- assume hosts A.eth0 and B.eth0 have active production traffic (say TCP)
- assume we have a monitoring tool on A that dups eth0's egress to some other
interface $MON
nft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch dup to $MON
- assume something goes wrong on $MON generating link failures. In my case it
was a GRETAP with L3 destination suddenly unreachable.
- The next A->B packet goes through normally, but its duplicate hitsipv4_link_failure(), hence the dst (which is B) is expired.
- As a result, (say) TCP disruptions occur. The thermometer killed the patient :)
Note: as a straightforward repro, you can simply witness "noise" in simple ping sessions, with ghost unreach reports muxed with normal measurement:
ip link add gre1 type gretap remote 192.168.1.99 ;# on the LAN, nonexistent
IP => will generate link failures
ip link set dev gre1 up
nft add table netdev ta
nft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch counter dup to gre1
ping -n 8.8.8.8
=>
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=19.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=36.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=30.3 ms
From 192.168.1.13 icmp_seq=5 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=5 ttl=115 time=31.0 ms
From 192.168.1.13 icmp_seq=6 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=6 ttl=115 time=38.0 ms
From 192.168.1.13 icmp_seq=7 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=7 ttl=115 time=36.8 ms
From 192.168.1.13 icmp_seq=8 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=8 ttl=115 time=23.5 ms
^CQuoted 5 lines
Quoted 3 lines
This patch fixes all similar scenarii by calling
skb_scrub_pkt()on the clone, severing its link to precious direct-path state.This patch is targetted at the net tree, but nf.git is preferred.
Okay, will retarget :)
Quoted 2 lines
As for the conntrack and dst, you have to explain what it breaks on your end.
Dst as shown above. Conntrack is more speculation, but my take is that in any case the dup path should *never* have any kind of retroaction on the observed path, so any "complex state" attached to the direct path should be absolutely isolated from "whatever happens on the dup path". Am I mistaken ?
-Alex
Hi,
Quoted 59 lines
Quoted 8 lines · Alexandre Ferrieux
Quoted 5 lines
In other words, the "dup" path has the potential to wreak havoc in the direct path as a consequence of secondary link failures. This is very bad behavior for a monitoring tool, which is the most obvious application of 'dup'.
Can you describe your use-case a bit and how it breaks?
Sure:
- assume hosts A.eth0 and B.eth0 have active production traffic (say TCP)
- assume we have a monitoring tool on A that dups eth0's egress to some other
interface $MONnft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch dup to $MON- assume something goes wrong on $MON generating link failures. In my case it
was a GRETAP with L3 destination suddenly unreachable.- The next
A->Bpacket goes through normally, but its duplicate hitsipv4_link_failure(), hence the dst (which is B) is expired.- As a result, (say) TCP disruptions occur. The thermometer killed the patient :)
Note: as a straightforward repro, you can simply witness "noise" in simple ping sessions, with ghost unreach reports muxed with normal measurement:
ip link add gre1 type gretap remote 192.168.1.99 ;# on the LAN, nonexistent
IP => will generate link failures
ip link set dev gre1 up
nft add table netdev ta
nft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch counter dup to gre1
ping -n 8.8.8.8
=>PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=19.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=12.8 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=36.0 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=30.3 ms From 192.168.1.13 icmp_seq=5 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=5 ttl=115 time=31.0 ms From 192.168.1.13 icmp_seq=6 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=6 ttl=115 time=38.0 ms From 192.168.1.13 icmp_seq=7 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=7 ttl=115 time=36.8 ms From 192.168.1.13 icmp_seq=8 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=8 ttl=115 time=23.5 ms ^CQuoted 5 lines
Quoted 3 lines
This patch fixes all similar scenarii by calling
skb_scrub_pkt()on the clone, severing its link to precious direct-path state.This patch is targetted at the net tree, but nf.git is preferred.
Quoted 8 lines
Okay, will retarget :)
Quoted 2 lines
As for the conntrack and dst, you have to explain what it breaks on your end.
Dst as shown above. Conntrack is more speculation, but my take is that in any case the dup path should *never* have any kind of retroaction on the observed path, so any "complex state" attached to the direct path should be absolutely isolated from "whatever happens on the dup path". Am I mistaken ?
What do you think ? Is the description of the problem satisfactory ? Is the proposed solution acceptable ?
-Alex
Pablo Neira Ayuso
maintainerHi,
Quoted 65 lines · Alexandre Ferrieux
Hi,
Quoted 63 lines
Quoted 8 lines · Alexandre Ferrieux
Quoted 5 lines
In other words, the "dup" path has the potential to wreak havoc in the direct path as a consequence of secondary link failures. This is very bad behavior for a monitoring tool, which is the most obvious application of 'dup'.
Can you describe your use-case a bit and how it breaks?
Sure:
- assume hosts A.eth0 and B.eth0 have active production traffic (say TCP)
- assume we have a monitoring tool on A that dups eth0's egress to some other
interface $MONnft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch dup to $MON- assume something goes wrong on $MON generating link failures. In my case it
was a GRETAP with L3 destination suddenly unreachable.- The next
A->Bpacket goes through normally, but its duplicate hitsipv4_link_failure(), hence the dst (which is B) is expired.- As a result, (say) TCP disruptions occur. The thermometer killed the patient :)
Note: as a straightforward repro, you can simply witness "noise" in simple ping sessions, with ghost unreach reports muxed with normal measurement:
ip link add gre1 type gretap remote 192.168.1.99 ;# on the LAN, nonexistent
IP => will generate link failures
ip link set dev gre1 up
nft add table netdev ta
nft add chain netdev ta ch '{type filter hook egress device "eth0" priority
filter ; policy accept ; }'
nft add rule netdev ta ch counter dup to gre1
ping -n 8.8.8.8
=>PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=19.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=12.8 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=36.0 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=30.3 ms From 192.168.1.13 icmp_seq=5 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=5 ttl=115 time=31.0 ms From 192.168.1.13 icmp_seq=6 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=6 ttl=115 time=38.0 ms From 192.168.1.13 icmp_seq=7 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=7 ttl=115 time=36.8 ms From 192.168.1.13 icmp_seq=8 Destination Host Unreachable 64 bytes from 8.8.8.8: icmp_seq=8 ttl=115 time=23.5 ms ^CQuoted 5 lines
Quoted 3 lines
This patch fixes all similar scenarii by calling
skb_scrub_pkt()on the clone, severing its link to precious direct-path state.This patch is targetted at the net tree, but nf.git is preferred.
Okay, will retarget :)
Quoted 2 lines
As for the conntrack and dst, you have to explain what it breaks on your end.
Quoted 7 lines
Quoted 4 lines
Dst as shown above. Conntrack is more speculation, but my take is that in any case the dup path should *never* have any kind of retroaction on the observed path, so any "complex state" attached to the direct path should be absolutely isolated from "whatever happens on the dup path". Am I mistaken ?
What do you think ? Is the description of the problem satisfactory ? Is the proposed solution acceptable ?
I think conntrack should be set to untracked for the dup.
And dst should be resetted if skb_valid_dst() returns true, so DST_METADATA can still be processed for duplicated packets.