Route cache problem.
Subsystems (2)
Hi,
My workstation has an incorrect route cache entry:
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.34 192.168.235.251 0 0 3 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 4 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 2 eth0
(I don't know why there are three).
the correct routing cache entries would look something like this:
(this one works):
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.20 Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.20 192.168.235.4 0 0 1 eth0 192.168.234.20 192.168.235.8 192.168.235.8 l 0 0 0 lo 192.168.235.8 192.168.234.20 192.168.235.4 0 0 0 eth0
The routing table is:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.235.251 0.0.0.0 UG 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0 192.168.234.0 192.168.235.4 255.255.255.0 UG 0 0 0 eth0 192.168.235.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 192.168.235.2 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0
It's the third line that is supposed to steer packets for '234.34 to the proper router that knows how to reach the 234.0 network.
As a temporary workaround I've added the route to 192.168.235.2 which is that same host, but not in the nameserver, so it's annoying. (the other host that I can't reach due to this problem doesn't have a second IP address (yet)).
Oh... routing to 192.168.234.34 works on the router 192.168.235.4:
PING 192.168.234.34 (192.168.234.34) 56(84) bytes of data. 64 bytes from 192.168.234.34: icmp_req=1 ttl=64 time=41.5 ms
Anyway, what would you suggest for me to try to get that invalid route cache entry dropped?
Drop the default route? Ok. Done:
# route del default # ping 192.168.234.34 2 packets transmitted, 0 received, 100% packet loss, time 1007ms
I'm used to the default route being at the bottom, but deleting it should be enough to prevent it from being found first, right? :-)
Add a host route to this host explicitly naming the router?
assurancetourix:~# route add 192.168.234.34 gw driepoot assurancetourix:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0 192.168.234.0 192.168.235.4 255.255.255.0 UG 0 0 0 eth0 192.168.234.34 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0 192.168.235.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 192.168.235.2 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0 assurancetourix:~# ping 192.168.234.34 -c 2 .... 2 packets transmitted, 0 received, 100% packet loss, time 1008ms
Still the packets end up on the ethernet with the 192.168.235.251 router's Ethernet address.....
assurancetourix:~# route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.34 192.168.235.251 0 0 0 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 0 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 5 eth0
# ifconfig eth0 down # route -n <empty table> # ifconfig eth0 up <old routing table is restored automatically??? apparently with the routing cache entries as well....>
I initially thought that this was a problem with the routing cache entry being too persistent in the kernel. While documenting this while writing this email, I've found that I can flush the whole routing cache with "ip route flush cache" .
However the routing cache entry springs back to life when I first ping the 234.34 host. Even when the problem machine doesn't have a default route, so it shouldn't know about the 235.251 default router.
This is getting weirder and weirder.
During all this I have
# tcpdump -nei eth0 net 192.168.234.0/24 running. If my machine were to get an ICMP redirect from somewhere I'd see it, right?
It could be that the 192.168.235.251 router is proxy-arping (incorreclty) for the problem hosts. But then my workstation would have to be ARPing in the first place.
# route add 192.168.234.200 eth0 # ping 192.168.234.200 gives: 15:31:33.857343 00:23:54:15:1f:a9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.234.200 tell 192.168.235.8, length 28 in the TCPDUMP, so my machine is not arping for 192.168.234.34.
Any suggestions? Any at all?
Roger.
Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* The plan was simple, like my brother-in-law Phil. But unlike Phil, this plan just might work.
Quoted 4 lines · Rogier Wolff
Hi,
My workstation has an incorrect route cache entry:
What kernel version ?
Quoted 9 lines
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.34 192.168.235.251 0 0 3 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 4 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 2 eth0(I don't know why there are three).
Different routing keys, like TOS.
for tos in seq 2 2 34; do ping -c 1 -Q $tos 10.37.168.112; done
# route -nC|grep 10.37.168.112 10.37.168.112 192.168.20.108 192.168.20.108 l 0 0 479 lo 192.168.20.108 10.37.168.112 192.168.20.254 0 0 3 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 3 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 3 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 3 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 3 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 5 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 5 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 5 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 1 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 1 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 0 0 eth3 192.168.20.108 10.37.168.112 192.168.20.254 0 1 2 eth3
Quoted 60 lines
the correct routing cache entries would look something like this:
(this one works):
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.20 Kernel IP routing cacheSource Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.20 192.168.235.4 0 0 1 eth0 192.168.234.20 192.168.235.8 192.168.235.8 l 0 0 0 lo 192.168.235.8 192.168.234.20 192.168.235.4 0 0 0 eth0The routing table is:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.235.251 0.0.0.0 UG 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0 192.168.234.0 192.168.235.4 255.255.255.0 UG 0 0 0 eth0 192.168.235.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 192.168.235.2 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0It's the third line that is supposed to steer packets for '234.34 to the proper router that knows how to reach the 234.0 network.
As a temporary workaround I've added the route to 192.168.235.2 which is that same host, but not in the nameserver, so it's annoying. (the other host that I can't reach due to this problem doesn't have a second IP address (yet)).
Oh... routing to 192.168.234.34 works on the router 192.168.235.4:
PING 192.168.234.34 (192.168.234.34) 56(84) bytes of data. 64 bytes from 192.168.234.34: icmp_req=1 ttl=64 time=41.5 msAnyway, what would you suggest for me to try to get that invalid route cache entry dropped?
Drop the default route? Ok. Done:
# route del default # ping 192.168.234.34 2 packets transmitted, 0 received, 100% packet loss, time 1007msI'm used to the default route being at the bottom, but deleting it should be enough to prevent it from being found first, right? :-)
Add a host route to this host explicitly naming the router?
assurancetourix:~# route add 192.168.234.34 gw driepoot assurancetourix:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0 192.168.234.0 192.168.235.4 255.255.255.0 UG 0 0 0 eth0 192.168.234.34 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0 192.168.235.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 192.168.235.2 192.168.235.4 255.255.255.255 UGH 0 0 0 eth0 assurancetourix:~# ping 192.168.234.34 -c 2 .... 2 packets transmitted, 0 received, 100% packet loss, time 1008msStill the packets end up on the ethernet with the 192.168.235.251 router's Ethernet address.....
Quoted 7 lines
assurancetourix:~# route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.34 192.168.235.251 0 0 0 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 0 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 5 eth0
Better use "ip -s route list cache" to diagnose problems (more information)
Quoted 39 lines
# ifconfig eth0 down # route -n <empty table> # ifconfig eth0 up <old routing table is restored automatically??? apparently with the routing cache entries as well....>I initially thought that this was a problem with the routing cache entry being too persistent in the kernel. While documenting this while writing this email, I've found that I can flush the whole routing cache with "ip route flush cache" .
However the routing cache entry springs back to life when I first ping the 234.34 host. Even when the problem machine doesn't have a default route, so it shouldn't know about the 235.251 default router.
This is getting weirder and weirder.
During all this I have
# tcpdump -nei eth0 net 192.168.234.0/24 running. If my machine were to get an ICMP redirect from somewhere I'd see it, right?It could be that the 192.168.235.251 router is proxy-arping (incorreclty) for the problem hosts. But then my workstation would have to be ARPing in the first place.
# route add 192.168.234.200 eth0 # ping 192.168.234.200 gives: 15:31:33.857343 00:23:54:15:1f:a9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.234.200 tell 192.168.235.8, length 28 in the TCPDUMP, so my machine is not arping for 192.168.234.34.Any suggestions? Any at all?
Roger.
Quoted 7 lines · Eric Dumazet
Quoted 4 lines · Rogier Wolff
Hi,
My workstation has an incorrect route cache entry:
What kernel version ?
Linux version 3.0.0-12-generic (from Ubunutu oneiric.)
Quoted 8 lines
Quoted 8 lines
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface 192.168.235.8 192.168.234.34 192.168.235.251 0 0 3 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 4 eth0 192.168.235.8 192.168.234.34 192.168.235.251 0 0 2 eth0(I don't know why there are three).
Today there are four.
Quoted 1 line
192.168.20.108 10.37.168.112 192.168.20.254 0 1 2 eth3
That indeed got me a full complement of route cache entries.
Quoted 2 lines
Better use "ip -s route list cache" to diagnose problems (more information)
After doing the tos ping you suggested All TOS levels have a route cache entry.
192.168.234.34 from 192.168.235.8 tos 0x1c via 192.168.235.251 dev eth0
cache <redirected> age 77sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9
192.168.234.34 tos 0x1c via 192.168.235.251 dev eth0 src 192.168.235.8
cache <redirected> used 3 age 72sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9
192.168.234.34 from 192.168.235.8 tos 0x1c via 192.168.235.251 dev eth0
cache <redirected> age 72sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9Quoted 1 line
Quoted 1 line
Any suggestions? Any at all?
Last time, as well as this time, it is triggered by a network error that leads to the 192.168.235.4 router not being able to reach 192.168.234.34 or any other host on the 192.168.234.0/24 network.
During that time the VPN to 192.168.234.0/24 is down, so 192.168.235.4 doesn't have a route to 192.168.234.0/24 and it is logical that with that route gone, it sends packets for 192.168.234.0/24 to the default router 192.168.235.251. As it sees itself forwarding packets that come in on eth0 back to eth0, it will send a redirect. However that redirect should somehow expire, and not survive things like dropping the route to 192.168.234.0/24, dropping the default route, shutting down the interface or some time passing (that network problem was solved 20 hours ago)......
Roger.
Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* The plan was simple, like my brother-in-law Phil. But unlike Phil, this plan just might work.