Issue with GRE tunnel, created with "local any" on kernel 3.14
Subsystems (2)
Hi. I have updated kernel on one of my servers to 3.14 and hit strange issue:
Simple setup:
serv1(172.30.0.100) <-> serv2(172.30.0.251)
Serv1 is on 3.11.7
Serv2 is on 3.13.6
Gre tunnel(serv1):
ip tunnel add tun_test mode gre remote 172.30.0.251 local any ttl 225 ifconfig tun_test 192.168.0.1/30
Gre tunnel(serv2):
ip tunnel add tun_test mode gre remote 172.30.0.100 local any ttl 225 ifconfig tun_test 192.168.0.2/30
Everything work as expected.
Now, with updated kernel on serv2 i have this:
serv2 ~ # ping 192.168.0.1 -c 10
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. 64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.339 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.303 ms
--- 192.168.0.1 ping statistics ---
10 packets transmitted, 2 received, 80% packet loss, time 9002ms rtt min/avg/max/mdev = 0.303/0.321/0.339/0.018 ms
Huh? tcpdump on serv1 show me really strange thing:
bgp ~ # tcpdump -i lan0 -nn ip proto gre
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0, link-type EN10MB (Ethernet), capture size 65535 bytes 16:49:33.338964 IP 172.30.0.251 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 1, length 64 16:49:33.339010 IP 172.30.0.100 > 172.30.0.251: GREv0, length 88: IP 192.168.0.1 > 192.168.0.2: ICMP echo reply, id 5259, seq 1, length 64 16:49:34.337945 IP 172.30.0.251 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 2, length 64 16:49:34.337988 IP 172.30.0.100 > 172.30.0.251: GREv0, length 88: IP 192.168.0.1 > 192.168.0.2: ICMP echo reply, id 5259, seq 2, length 64 16:49:35.336948 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 3, length 64 16:49:36.340999 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 4, length 64 16:49:37.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 5, length 64 16:49:38.340985 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 6, length 64 16:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64 16:49:40.340918 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 8, length 64 16:49:41.340947 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 9, length 64 16:49:42.340970 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 10, length 64 ^C
12 packets captured
14 packets received by filter
0 packets dropped by kernel
Huh? 0.0.0.0? o_O
I have tried to up and done tunnels few times. Sometimes 0.0.0.0 is replaced by correct address in the middle of icmp "session" for one or two packets, but usually only first one or two packets have correct src address.
I have tried some(not all of them) kernels from 3.14.2 up to 3.15.5 - all have the same issue.
If i set "local 172.30.0.251" on serv2 the issue is gone completely.
Looking on changelog i discovered some fixes to gre, but i could not understand if they can be culprit or not.
MTU on both real network ifaces are 1500, MTU on gre tunnels are 1476, so i suppose that's right(and not an issue, cause mentioned icmp packets are too small to hit MTU problem).
I am running Gentoo Linux, but issue is detected in vanilla kernel too, so it's not distro problem, i suppose.
Can anybody help me to track this issue down?
Signature · 4 lines
Best regards, Sergey Popov Gentoo developer Gentoo Desktop-effects project lead Gentoo Proxy maintainers project lead
Quoted 9 lines · On Fri, 18 Jul 2014 17:01:23 +0400 Sergey Popov <[email protected]> wrote:
Hi. I have updated kernel on one of my servers to 3.14 and hit strange issue:
serv1(172.30.0.100) <-> serv2(172.30.0.251)
Gre tunnel(serv1):
ip tunnel addtun_testmode gre remote 172.30.0.251 local any ttl 225 ifconfigtun_test192.168.0.1/30
Quoted 6 lines
Gre tunnel(serv2):
ip tunnel addtun_testmode gre remote 172.30.0.100 local any ttl 225 ifconfigtun_test192.168.0.2/3016:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64
Hi!
It seems we have this issue since 7d442fab0 (ipv4: Cache dst in tunnels). It is in ip_tunnel_xmit: fl4 is initialized with saddr = 0 in init_tunnel_flow(&fl4, protocol, dst, ...), net/ipv4/ip_tunnel.c:610, and isn't changed until iptunnel_xmit(skb->sk, ...), line 677, because connected = true and rt is almost always cached.
I have no patch to fix this issue, but as a workaround, you can use ip tunnel change tun_test tos inherit.
18.07.2014 22:32, Dmitry Popov пишет:
Quoted 21 lines
Quoted 9 lines · On Fri, 18 Jul 2014 17:01:23 +0400 Sergey Popov <[email protected]> wrote:
Hi. I have updated kernel on one of my servers to 3.14 and hit strange issue:
serv1(172.30.0.100) <-> serv2(172.30.0.251)
Gre tunnel(serv1):
ip tunnel addtun_testmode gre remote 172.30.0.251 local any ttl 225 ifconfigtun_test192.168.0.1/30Quoted 6 lines
Gre tunnel(serv2):
ip tunnel addtun_testmode gre remote 172.30.0.100 local any ttl 225 ifconfigtun_test192.168.0.2/3016:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64
Hi!
Quoted 8 lines
It seems we have this issue since
7d442fab0(ipv4: Cache dst in tunnels). It is inip_tunnel_xmit: fl4 is initialized with saddr = 0 ininit_tunnel_flow(&fl4, protocol, dst, ...),net/ipv4/ip_tunnel.c:610, and isn't changed untiliptunnel_xmit(skb->sk, ...), line 677, because connected = true and rt is almost always cached.I have no patch to fix this issue, but as a workaround, you can use ip tunnel change
tun_testtos inherit.
Thanks, your workaround works as expected if tos value defined explicitly. Though, I do not understand why: iproute2 documentation says that default tos value in 'ip tunnel' command is already 'inherit'.
Signature · 5 lines
Best regards, Sergey Popov Gentoo developer Gentoo Desktop Effects project lead Gentoo Qt project lead Gentoo Proxy maintainers project lead
Quoted 6 lines · On Thu, 31 Jul 2014 10:29:00 +0400 Sergey Popov <[email protected]> wrote:
Quoted 2 lines
I have no patch to fix this issue, but as a workaround, you can use ip tunnel change
tun_testtos inherit.Thanks, your workaround works as expected if tos value defined explicitly. Though, I do not understand why: iproute2 documentation says that default tos value in 'ip tunnel' command is already 'inherit'.
man ip-tunnel says default tos is 00, and that's true. If you're about *.tex files under iproute2.git/doc, they are more than 10 years old, I doubt if they are maintained. Otherwise I don't know where you found it.
This workaround works because setting tos=inherit disables tunnel route cache, which is buggy in your release (it doesn't cache source address).
Btw, thanks for the report, David has accepted the patch today, commit 95cb574 (ip_tunnel(ipv4): fix tunnels with "local any remote $remote_ip"), so I think next kernel releases will have it.