document danger of '-j REJECT'ing of '-m state INVALID' packets
Subsystems (3)
PATCH 1/1 document danger of '-j REJECT'ing of '-m state INVALID' packets
From: Maciej Żenczykowski <[email protected]>
This appears to be a common, but hard to debug, misconfiguration.
--- extensions/libip6t_REJECT.man | 15 +++++++++++++++ extensions/libipt_REJECT.man | 15 +++++++++++++++ 2 files changed, 30 insertions(+)
extensions/libip6t_REJECT.man +15 -0
@@ -30,3 +30,18 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP if you are using connection tracking and \fBACCEPT\fP'ing +\fBESTABLISHED\fP (and possibly \fBRELATED\fP) state packets, do not +indiscriminately \fBREJECT\fP (especially with \fITCP RST\fP) \fBINVALID\fP +state packets. Sometimes naturally occuring packet reordering will result +in packets being considered \fBINVALID\fP and the generated \fITCP RST\fP +will abort an otherwise healthy connection. +.P +Suggested use: +.br + -A INPUT -m state ESTABLISHED,RELATED -j ACCEPT +.br + -A INPUT -m state INVALID -j DROP +.br +(and -j REJECT rules go here at the end)
extensions/libipt_REJECT.man +15 -0
@@ -30,3 +30,18 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP if you are using connection tracking and \fBACCEPT\fP'ing +\fBESTABLISHED\fP (and possibly \fBRELATED\fP) state packets, do not +indiscriminately \fBREJECT\fP (especially with \fITCP RST\fP) \fBINVALID\fP +state packets. Sometimes naturally occuring packet reordering will result +in packets being considered \fBINVALID\fP and the generated \fITCP RST\fP +will abort an otherwise healthy connection. +.P +Suggested use: +.br + -A INPUT -m state ESTABLISHED,RELATED -j ACCEPT +.br + -A INPUT -m state INVALID -j DROP +.br +(and -j REJECT rules go here at the end)
Signature · 1 line
2.26.2.645.ge9eca65c58-goog
Quoted 11 lines · Maciej Żenczykowski
extensions/libip6t_REJECT.man +0
-0@@ -30,3 +30,18 @@ TCP RST packet to be sent back. This is mainly useful for blockinghosts (which won't accept your mail otherwise). \fBtcp\-reset\fP
can only be used with kernel versions 2.6.14 or later.
+.PP
+\fIWarning:\fP if you are using connection tracking and \fBACCEPT\fP'ing +\fBESTABLISHED\fP (and possibly \fBRELATED\fP) state packets, do not
extensions/libip6t_REJECT.man TCP RST packet to be sent back. This is mainly useful for blocking · line 39
Quoted 4 lines
extensions/libip6t_REJECT.man +4
-0+indiscriminately \fBREJECT\fP (especially with \fITCP RST\fP) \fBINVALID\fP +state packets. Sometimes naturally occuring packet reordering will result +in packets being considered \fBINVALID\fP and the generated \fITCP RST\fP +will abort an otherwise healthy connection.
I fail to understand the problem here.
1. Because ESTABLISHED and INVALID are mutually exclusive, there is no ordering dependency between two rules of the kind {EST=>ACCEPT, INV=>REJ}, and thus their order plays no role.
2. Given packets D,R (data, rst) leads to state(ct(D))=EST, state(ct(R))=EST in the normal case. When this gets reordered to R,D, then we end up with state(ct(R))=EST, state(ct(D))=INV. Though the outcome of nfct changes, I do not think that will be of consequence, because in the absence of filtering, the tcp layer should be discarding/rejecting D.
3. Natural reordering of D1,D2 to D2,D1 should not cause nfct to drop the ct at reception of D1 and turn the state to INV. Reordering can happen at any time, and we'd be having more reports of problems if it did, wouldn't we...
So I've never tried to figure out how things break, just observed that they do - first many many years ago (close to 15ish) - between my wifi connected laptop at home and my university server in the same city. I've kept an INVALID->DROP rule in all my firewalls since then and not had problems. I vaguely recall seeing delayed packets when I debugged it back then.
See for example: https://github.com/moby/libnetwork/issues/1090 for others running into this.
Now we've hit an issue at work where a network misconfiguration has asymmetric one way pathing with a result that some packets were getting *massively* delayed, and it's been causing user firewalls to generate tcp resets for 'too old' 'already ack'ed' packets (ie. dups).
While this is of course a misconfig, and it shouldn't happen, in practice it sometimes simply does.
All it takes is for a packet to get into a long queue, and the network path to shift (immediately after it) to a less congested path. Due to bufferbloat those long queues can take seconds to drain and exceed path rtt by orders of magnitude.
I *think* what happens is:
A non-final tcp packet gets massively delayed, the packet past that makes it through to the receive, and triggers an ACK with SACK, which makes it back to the sender and triggers a retransmit and the connections keeps on making forward progress, then eventually the delayed packet arrives and it's no longer considered valid and triggers a tcp reset. Massively of course depends on the rtt and retransmit aggressiveness.
Here's my attempt to demonstrate what I believe the problem to be:
(on a freshly booted clean/empty/idle fedora 31 vm)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -m state --state INVALID -j DROP
modprobe ifb
ip link set dev ifb0 up
tc qdisc add dev ifb0 root netem reorder 99% 0% delay 10s
tc qdisc add dev eth0 clsact
tc filter add dev eth0 ingress u32 match u32 0 0 action mirred egress redirect dev ifb0
wget -O /dev/null https://git.kernel.org/torvalds/t/linux-5.7-rc4.tar.gz
iptables-save -c
.../dev/null [ <=>
] 169.58M 2.93MB/s in 45s 2020-05-09 10:35:44 (3.81 MB/s) - ‘/dev/null’ saved [177819073]
...
[31750:181080717] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT [244:1403178] -A INPUT -m state --state INVALID -j DROP
Now if I reboot, and run the same script, except instead of the INVALID/DROP rule I do
iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
then the download never finishes (it hangs after 15MB @ 2MB/s and eventually times out).
[4170:16758894] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT [37:147454] -A INPUT -p tcp -j REJECT --reject-with tcp-reset
(arguably since this is a VM, and thus NAT'ed by my host, and then again by the real ipv4 NAT, the setup isn't entirely clear, but I hope it makes my point: INVALID state needs to be dropped, not rejected)
Side note, it doesn't have to be nearly as aggressive as the above.
With just:
tc qdisc replace dev ifb0 root netem reorder 99.9% 0% delay 1s
I still see 169.58M @ 7.02MB/s in 26s:
[24263:180667450] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT [27:174654] -A INPUT -m state --state INVALID -j DROP
[0:0] -A INPUT -p tcp -j REJECT --reject-with tcp-reset
And the connection still freezes without the INVALID/DROP rule (after 43MiB this time)
PATCH 1/1 doc: document danger of applying REJECT to INVALID CTs
In reply to Maciej Żenczykowski---
Maciej's explanation on how INVALID+REJECT can lead to problems looks convincing. I hereby present new manpage wording in the form of "if A, then B" to better build the argument of avoiding REJECT. So the issue is not caused by an _incoming_ TCP RST as the initial mail might have suggested, but by RST generated by REJECT (--reject-with tcp-reset).
It is conceivable to me that a connection termination may occur with not only TCP+RST, but also with TCP+ICMP and UDP+ICMP, so I trimmed any protocol-specific wording too. Also trimmed is any mention of -j ACCEPT, because rule order is not the point of the argument.
extensions/libip6t_REJECT.man | 21 +++++++++++++++++++++ extensions/libipt_REJECT.man | 21 +++++++++++++++++++++ 2 files changed, 42 insertions(+)
extensions/libip6t_REJECT.man +21 -0
@@ -30,3 +30,24 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscrimnately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these: +.PP +Consider a source host retransmitting an original packet P as P_2 for any +reason, and P_2 getting routed via a different path (load balancing/policy +routing, or anything of the kind). Additionally, let P_2 experience so much +delay that the source host issues \fIanother\fP retransmission, P_3, with P_3 +being succesful in reaching its destination and advancing the connection state +normally. The delayed P_2, when it eventually is processed, may be considered +to be not associated with any connection tracking entry. Generating a reject +packet for such a belated packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT -m conntrack --ctstate INVALID -j REJECT +.PP +do consider using: +.PP +-A INPUT -m conntrack --ctstate INVALID -j DROP
extensions/libipt_REJECT.man +21 -0
@@ -30,3 +30,24 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP You should not indiscrimnately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these: +.PP +Consider a source host retransmitting an original packet P as P_2 for any +reason, and P_2 getting routed via a different path (load balancing/policy +routing, or anything of the kind). Additionally, let P_2 experience so much +delay that the source host issues \fIanother\fP retransmission, P_3, with P_3 +being succesful in reaching its destination and advancing the connection state +normally. The delayed P_2, when it eventually is processed, may be considered +to be not associated with any connection tracking entry. Generating a reject +packet for such a belated packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT -m conntrack --ctstate INVALID -j REJECT +.PP +do consider using: +.PP +-A INPUT -m conntrack --ctstate INVALID -j DROP
Signature · 1 line
2.26.2
Re: [PATCH] doc: document danger of applying REJECT to INVALID CTs
I *think* that your talk of 3 packets is not needed, ie. the initial delayed packet doesn't have to be a retransmission.
It can be the first copy of that segment that gets massively delayed and arrives late and causes problems,
by virtue of arriving after the retransmission already caused the connection to move on.
Other than that this does seem perhaps a bit cleared than what I wrote.
Re: [PATCH] doc: document danger of applying REJECT to INVALID CTs
Also maybe the example should be:
instead of just:
-A INPUT ... -j REJECT
do:
-A INPUT ... -m conntrack --ctstate INVALID -j DROP -A INPUT ... -j REJECT
PATCH v2 1/1 doc: document danger of applying REJECT to INVALID CTs
In reply to Maciej Żenczykowski---
Simplify the trigger case by dropping mentions of P_3. New -A commands as proposed.
extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++ extensions/libipt_REJECT.man | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)
extensions/libip6t_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscrimnately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
extensions/libipt_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP You should not indiscrimnately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
Signature · 1 line
2.26.2
Re: [PATCH v2] doc: document danger of applying REJECT to INVALID CTs
Re: [PATCH v2] doc: document danger of applying REJECT to INVALID CTs
In reply to Jan EngelhardtQuoted 15 lines · Jan Engelhardt
Signed-off-by Jan Engelhardt <[email protected]>---
Simplify the trigger case by dropping mentions of
P_3. New -A commands as proposed.extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++ extensions/libipt_REJECT.man | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)extensions/libip6t_REJECT.man +0
-0
extensions/libip6t_REJECT.man TCP RST packet to be sent back. This is mainly useful for blocking · line 34
Quoted 5 lines
extensions/libip6t_REJECT.man +2
-0@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscrimnately apply the REJECT target to
^ typo
extensions/libip6t_REJECT.man TCP RST packet to be sent back. This is mainly useful for blocking · line 40
Quoted 6 lines
extensions/libip6t_REJECT.man +6
-0+packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection
^ typo
PATCH v3 1/1 doc: document danger of applying REJECT to INVALID CTs
---
Spello fix near "indiscriminately".
extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++ extensions/libipt_REJECT.man | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)
extensions/libip6t_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
extensions/libipt_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
Signature · 1 line
2.26.2
Re: [PATCH v3] doc: document danger of applying REJECT to INVALID CTs
you still missed: succesful -> successful
Quoted 67 lines · Jan Engelhardt
Signed-off-by Jan Engelhardt <[email protected]>---
Spello fix near "indiscriminately".
extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++ extensions/libipt_REJECT.man | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)extensions/libip6t_REJECT.man +20
-0@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECTextensions/libipt_REJECT.man +20
-1@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT --
Quoted 2 lines
2.26.2
Maciej Żenczykowski, Kernel Networking Developer @ Google
PATCH v4 1/1 doc: document danger of applying REJECT to INVALID CTs
---
4th time is the charm?!
extensions/libip6t_REJECT.man | 20 ++++++++++++++++++++ extensions/libipt_REJECT.man | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+)
extensions/libip6t_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). \fBtcp\-reset\fP can only be used with kernel versions 2.6.14 or later. +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
extensions/libipt_REJECT.man +20 -0
@@ -30,3 +30,23 @@ TCP RST packet to be sent back. This is mainly useful for blocking hosts (which won't accept your mail otherwise). .IP (*) Using icmp\-admin\-prohibited with kernels that do not support it will result in a plain DROP instead of REJECT +.PP +\fIWarning:\fP You should not indiscriminately apply the REJECT target to +packets whose connection state is classified as INVALID; instead, you should +only DROP these. +.PP +Consider a source host transmitting a packet P, with P experiencing so much +delay along its path that the source host issues a retransmission, P_2, with +P_2 being succesful in reaching its destination and advancing the connection +state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject +packet for this packet would then terminate the healthy connection. +.PP +So, instead of: +.PP +-A INPUT ... -j REJECT +.PP +do consider using: +.PP +-A INPUT ... -m conntrack --ctstate INVALID -j DROP +-A INPUT ... -j REJECT
Signature · 1 line
2.26.2
Re: [PATCH v4] doc: document danger of applying REJECT to INVALID CTs
Apparently no, did you send the wrong patch?
But since you'll have to resend again, 2 more minor stylistic comments.
Quoted 1 line
+P_2 being succesful in reaching its destination and advancing the connection
successful
Review
Quoted 2 lines
(quoted diff) +2
-0+state normally. It is conceivable that the late-arriving P may be considered to +be not associated with any connection tracking entry. Generating a reject
s/be not/not be/ is probably better
Quoted 1 line
+only DROP these.
would 'those' be better?
Quoted 1 line
+P_2 being succesful in reaching its destination and advancing the connection
ditto