2.6.12 netfilter: local packets marked as invalid
Subsystems (4)
Hi,
Some Gentoo users have reported very long application startup times in 2.6.12. This seems to be because the applications are attempting to connect to local ports such as sunrpc/portmap (where these services are not running), but some packets are being dropped, so the application load just pauses until the connection times out.
There was a similar problem reported recently ("2.6.12: connection tracking broken?"), regarding bridge devices. No bridge devices are involved here.
This is easy to reproduce, and the problem exists on both Linux 2.6.12 and 2.6.13-rc2. This was not a problem on Linux 2.6.11.
Taking a simple configuration:
# iptables -P INPUT ACCEPT # iptables -P OUTPUT ACCEPT # iptables -P FORWARD ACCEPT
I don't have a webserver running. If I try and telnet to port 80, I immediately get connection refused, as expected:
# telnet 127.0.0.1 80 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused
I now add another rule, to drop invalid packets, and retry the telnet connection.
# iptables -A INPUT -m state --state INVALID -j DROP # telnet 127.0.0.1 80 <very long pause> telnet: connect to address 127.0.0.1: Connection timed out
During the pause, netstat reports the connection state as SYN_SENT:
Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 1 localhost:39066 localhost:http SYN_SENT
I now flush the filter table, and make the kernel log invalid packets:
# iptables -t filter -F #iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "inv "
When retrying the telnet test, this appears in the logs:
Jul 8 14:53:04 dsd inv IN=lo OUT=
MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=15 DF PROTO=TCP SPT=80 DPT=58950 WINDOW=0 RES=0x00 ACK RST URGP=0
Does this mean that the kernel thinks its own ACK RST packet is invalid?
There is a Gentoo bug on this here:
http://bugs.gentoo.org/96948
...but I think I got the imporant info into this message.
Let me know if I can provide any more info. For those interested in a temporary workaround, you can explicitly allow all local traffic, i.e.
# iptables -A INPUT -i lo -j ACCEPT # iptables -A INPUT -m state --state INVALID -j DROP
Thanks.
Daniel
Quoted 8 lines · Daniel Drake
When retrying the telnet test, this appears in the logs:
Jul 8 14:53:04 dsd inv IN=lo OUT=MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=15 DF PROTO=TCP SPT=80 DPT=58950 WINDOW=0 RES=0x00 ACK RST URGP=0
Does this mean that the kernel thinks its own ACK RST packet is invalid?
I think I know what happens. In 2.6.12 we started dropping the conntrack reference when a packet leaves IP, so packets on loopback are tracked twice (LOCAL_OUT/PRE_ROUTING). TCP connection tracking destroys a conntrack entry when the only reply is an RST. So when the packet is tracked for the second time in PRE_ROUTING, the conntrack entry can't be found anymore and the packet is considered invalid.
You could confirm this theory by logging invalid packets in LOCAL_OUT and in PRE_ROUTING - only PRE_ROUTING should trigger. I'm going to think about a solution meanwhile.
Regards
Patrick
Quoted 3 lines · Patrick McHardy
You could confirm this theory by logging invalid packets in
LOCAL_OUTand inPRE_ROUTING- onlyPRE_ROUTINGshould trigger. I'm going to think about a solution meanwhile.
You'll have to forgive my lack of netfilter knowledge, I set up my firewall ages ago and haven't really touched it since :)
How can I do this with iptables?
Thanks,
Daniel
Quoted 2 lines · Daniel Drake
You'll have to forgive my lack of netfilter knowledge, I set up my firewall ages ago and haven't really touched it since :)
We decided to revert the responsible change because it caused problems in other areas as well. This patch should fix your problem.
[NETFILTER]: Revert nf_reset change
Revert the nf_reset change that caused so much trouble, drop conntrack references manually before packets are queued to packet sockets.
---
commit c64e9fafffd203886f6c57149803f1824febd9bb
tree 40a90ab1427c25f7a36109eabd1a53b0a8f144be
parent 9fd73bed06eb5cf309e0cb5cdf3249a54fd3ff22
author Phil Oester <[email protected]> Mon, 11 Jul 2005 02:04:08 committer Patrick McHardy <[email protected]> Mon, 11 Jul 2005 02:04:08
net/ipv4/ip_output.c | 9 --------- net/ipv4/netfilter/ip_conntrack_standalone.c | 7 +++++++ net/packet/af_packet.c | 6 ++++++ 3 files changed, 13 insertions(+), 9 deletions(-)
net/ipv4/ip_output.c +0 -9
@@ -107,7 +107,6 @@ static int ip_dev_loopback_xmit(struct s newskb->pkt_type =PACKET_LOOPBACK ; newskb->ip_summed =CHECKSUM_UNNECESSARY ;BUG_TRAP (newskb->dst); -nf_reset (newskb);netif_rx (newskb);return 0 ; }
@@ -188,14 +187,6 @@ static inline int ip_finish_output2(stru skb = skb2; } -#ifdef CONFIG_BRIDGE_NETFILTER -/* bridge-netfilter defers calling some IP hooks to the bridge layer -* and still needs the conntrack reference. -*/ - if (skb->nf_bridge == NULL) -#endif -nf_reset (skb); -if (hh) {int hh_alen;
net/ipv4/netfilter/ip_conntrack_standalone.c +7 -0
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag( const struct net_device *out, int (*okfn)(struct sk_buff *)) { +#if !defined(CONFIG_IP_NF_NAT ) && !defined(CONFIG_IP_NF_NAT_MODULE ) +/* Previously seen (loopback)? Ignore. Do this before +fragment check. */ +if ((*pskb)->nfct) +return NF_ACCEPT ; +#endif +/* Gather fragments. */ if ((*pskb)->nh.iph->frag_off &htons (IP_MF |IP_OFFSET )) { *pskb = ip_ct_gather_frags(*pskb,
net/packet/af_packet.c +6 -0
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_bufdst_release (skb->dst); skb->dst =NULL ; +/* drop conntrack reference */ +nf_reset (skb); + spkt = (struct sockaddr_pkt *)skb->cb;skb_push (skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *skdst_release (skb->dst); skb->dst =NULL ; +/* drop conntrack reference */ +nf_reset (skb); +spin_lock (&sk->sk_receive_queue.lock); po->stats.tp_packets++;__skb_queue_tail (&sk->sk_receive_queue, skb);
Quoted 2 lines · Patrick McHardy
We decided to revert the responsible change because it caused problems in other areas as well. This patch should fix your problem.
Thanks, it works. If you decide to revisit this in the future, feel free to send me a patch and I will help test it.
Daniel
From: Patrick McHardy <[email protected]>
Date: Tue, 12 Jul 2005 13:58:27 +0200
Quoted 6 lines
Quoted 2 lines · Daniel Drake
You'll have to forgive my lack of netfilter knowledge, I set up my firewall ages ago and haven't really touched it since :)
We decided to revert the responsible change because it caused problems in other areas as well. This patch should fix your problem.
Applied.
Now the question is what to do about the 2.6.12.x stable tree. I think we put the offending change there, now we need to revert it there too. Patrick, could you push this patch to [email protected] so we can resolve that too?
Thanks a lot.
Quoted 10 lines · * David S. Miller ([email protected]) wrote:
From: Patrick McHardy <[email protected]>
Date: Tue, 12 Jul 2005 13:58:27 +0200Quoted 6 lines
Quoted 2 lines · Daniel Drake
You'll have to forgive my lack of netfilter knowledge, I set up my firewall ages ago and haven't really touched it since :)
We decided to revert the responsible change because it caused problems in other areas as well. This patch should fix your problem.
Quoted 6 lines
Applied.
Now the question is what to do about the 2.6.12.x stable tree. I think we put the offending change there, now we need to revert it there too. Patrick, could you push this patch to [email protected] so we can resolve that too?
There's the first fix in the queue, I can either drop that one, or patch on top of it. Dropping what's in the queue[1] is fine for me. Below's the backport that Daniel sent over this morning (which applies if I drop what's in the queue). Patrick, does that look ok? I didn't queue this change yet, as I'd prefer it came either from you or with you Cc'd so you can ack it.
--
[NETFILTER]: Revert nf_reset change
Revert the nf_reset change that caused so much trouble, drop conntrack references manually before packets are queued to packet sockets.
Adapted for 2.6.12 by Daniel Drake <[email protected]>
net/ipv4/ip_output.c renamed from net/ipv4/ip_output.c_orig +0 -3
@@ -111,7 +111,6 @@ static int ip_dev_loopback_xmit(struct s#ifdef CONFIG_NETFILTER_DEBUG nf_debug_ip_loopback_xmit (newskb);#endif -nf_reset (newskb);netif_rx (newskb);return 0 ; }
@@ -196,8 +195,6 @@ static inline int ip_finish_output2(strunf_debug_ip_finish_output2 (skb);#endif /*CONFIG_NETFILTER_DEBUG*/ -nf_reset (skb); -if (hh) {int hh_alen;
net/ipv4/netfilter/ip_conntrack_standalone.c renamed from net/ipv4/netfilter/ip_conntrack_standalone.c_orig +7 -0
@@ -432,6 +432,13 @@ static unsigned int ip_conntrack_defrag( const struct net_device *out, int (*okfn)(struct sk_buff *)) { +#if !defined(CONFIG_IP_NF_NAT ) && !defined(CONFIG_IP_NF_NAT_MODULE ) +/* Previously seen (loopback)? Ignore. Do this before +fragment check. */ +if ((*pskb)->nfct) +return NF_ACCEPT ; +#endif +/* Gather fragments. */ if ((*pskb)->nh.iph->frag_off &htons (IP_MF |IP_OFFSET )) { *pskb = ip_ct_gather_frags(*pskb,
net/packet/af_packet.c renamed from net/packet/af_packet.c_orig +6 -0
@@ -274,6 +274,9 @@ static int packet_rcv_spkt(struct sk_bufdst_release (skb->dst); skb->dst =NULL ; +/* drop conntrack reference */ +nf_reset (skb); + spkt = (struct sockaddr_pkt *)skb->cb;skb_push (skb, skb->data-skb->mac.raw);
@@ -517,6 +520,9 @@ static int packet_rcv(struct sk_buff *skdst_release (skb->dst); skb->dst =NULL ; +/* drop conntrack reference */ +nf_reset (skb); +spin_lock (&sk->sk_receive_queue.lock); po->stats.tp_packets++;__skb_queue_tail (&sk->sk_receive_queue, skb);
Patrick McHardy
Acked-byQuoted 7 lines · Chris Wright
Quoted 4 lines · * David S. Miller ([email protected]) wrote:
Now the question is what to do about the 2.6.12.x stable tree. I think we put the offending change there, now we need to revert it there too. Patrick, could you push this patch to [email protected] so we can resolve that too?
Quoted 8 lines
There's the first fix in the queue, I can either drop that one, or patch on top of it. Dropping what's in the queue[1] is fine for me. Below's the backport that Daniel sent over this morning (which applies if I drop what's in the queue). Patrick, does that look ok? I didn't queue this change yet, as I'd prefer it came either from you or with you Cc'd so you can ack it.
Daniel's patch is fine, thanks.
Regards
Patrick
Quoted 15 lines · * Patrick McHardy ([email protected]) wrote:
Quoted 14 lines · Chris Wright
Quoted 4 lines · * David S. Miller ([email protected]) wrote:
Now the question is what to do about the 2.6.12.x stable tree. I think we put the offending change there, now we need to revert it there too. Patrick, could you push this patch to [email protected] so we can resolve that too?
There's the first fix in the queue, I can either drop that one, or patch on top of it. Dropping what's in the queue[1] is fine for me. Below's the backport that Daniel sent over this morning (which applies if I drop what's in the queue). Patrick, does that look ok? I didn't queue this change yet, as I'd prefer it came either from you or with you Cc'd so you can ack it.
Quoted 6 lines
Daniel's patch is fine, thanks.
Acked-by Patrick McHardy <[email protected]>
Great, thanks.
-chris