Skip to contentSkip to conversation

Route cache problem.

DISCUSSION Rogier Wolff 2 participants 3 messages Nov 3, 2011 – Nov 18, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

My workstation has an incorrect route cache entry:

assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache

Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.34  192.168.235.251       0      0        3 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        4 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        2 eth0

(I don't know why there are three).

the correct routing cache entries would look something like this:
(this one works):
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.20 Kernel IP routing cache

Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.20  192.168.235.4         0      0        1 eth0
192.168.234.20  192.168.235.8   192.168.235.8   l     0      0        0 lo
192.168.235.8   192.168.234.20  192.168.235.4         0      0        0 eth0

The routing table is:

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.235.251 0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.234.0   192.168.235.4   255.255.255.0   UG    0      0        0 eth0
192.168.235.0   0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.235.2   192.168.235.4   255.255.255.255 UGH   0      0        0 eth0

It's the third line that is supposed to steer packets for '234.34 to the proper router that knows how to reach the 234.0 network.

As a temporary workaround I've added the route to 192.168.235.2 which is that same host, but not in the nameserver, so it's annoying. (the other host that I can't reach due to this problem doesn't have a second IP address (yet)).

Oh... routing to 192.168.234.34 works on the router 192.168.235.4:
PING 192.168.234.34 (192.168.234.34) 56(84) bytes of data. 64 bytes from 192.168.234.34: icmp_req=1 ttl=64 time=41.5 ms

Anyway, what would you suggest for me to try to get that invalid route cache entry dropped?

Drop the default route? Ok. Done:

# route del default
# ping 192.168.234.34
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

I'm used to the default route being at the bottom, but deleting it should be enough to prevent it from being found first, right? :-)

Add a host route to this host explicitly naming the router?

assurancetourix:~# route add 192.168.234.34 gw driepoot
assurancetourix:~# route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.234.0   192.168.235.4   255.255.255.0   UG    0      0        0 eth0
192.168.234.34  192.168.235.4   255.255.255.255 UGH   0      0        0 eth0
192.168.235.0   0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.235.2   192.168.235.4   255.255.255.255 UGH   0      0        0 eth0
assurancetourix:~# ping 192.168.234.34 -c 2
....
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

Still the packets end up on the ethernet with the 192.168.235.251 router's Ethernet address.....

assurancetourix:~# route -nC | head -2 ; route -nC | grep 234.34
Kernel IP routing cache
Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.34  192.168.235.251       0      0        0 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        0 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        5 eth0
# ifconfig eth0 down
# route -n 
<empty table> 
# ifconfig eth0 up
<old routing table is restored automatically??? apparently with the routing
cache entries as well....> 

I initially thought that this was a problem with the routing cache entry being too persistent in the kernel. While documenting this while writing this email, I've found that I can flush the whole routing cache with "ip route flush cache" .

However the routing cache entry springs back to life when I first ping the 234.34 host. Even when the problem machine doesn't have a default route, so it shouldn't know about the 235.251 default router.

This is getting weirder and weirder.

During all this I have

# tcpdump -nei eth0 net 192.168.234.0/24
running. If my machine were to get an ICMP redirect from somewhere
I'd see it, right? 

It could be that the 192.168.235.251 router is proxy-arping (incorreclty) for the problem hosts. But then my workstation would have to be ARPing in the first place.

# route add 192.168.234.200 eth0
# ping 192.168.234.200
gives: 
15:31:33.857343 00:23:54:15:1f:a9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.234.200 tell 192.168.235.8, length 28
in the TCPDUMP, so my machine is not arping for 192.168.234.34. 

Any suggestions? Any at all?

Roger.

Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
The plan was simple, like my brother-in-law Phil. But unlike
Phil, this plan just might work.
#1 lore raw
Quoted 4 lines · Rogier Wolff

Hi,

My workstation has an incorrect route cache entry:

What kernel version ?

Quoted 9 lines

assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache

Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.34  192.168.235.251       0      0        3 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        4 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        2 eth0

(I don't know why there are three).

Different routing keys, like TOS.

for tos in seq 2 2 34; do ping -c 1 -Q $tos 10.37.168.112; done

# route -nC|grep 10.37.168.112
10.37.168.112   192.168.20.108  192.168.20.108  l     0      0      479 lo
192.168.20.108  10.37.168.112   192.168.20.254        0      0        3 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        3 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        3 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        3 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        3 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        5 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        5 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        5 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        1 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        1 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      0        0 eth3
192.168.20.108  10.37.168.112   192.168.20.254        0      1        2 eth3
Quoted 60 lines

the correct routing cache entries would look something like this:
(this one works):
assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.20 Kernel IP routing cache

Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.20  192.168.235.4         0      0        1 eth0
192.168.234.20  192.168.235.8   192.168.235.8   l     0      0        0 lo
192.168.235.8   192.168.234.20  192.168.235.4         0      0        0 eth0

The routing table is:

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.235.251 0.0.0.0         UG    0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.234.0   192.168.235.4   255.255.255.0   UG    0      0        0 eth0
192.168.235.0   0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.235.2   192.168.235.4   255.255.255.255 UGH   0      0        0 eth0

It's the third line that is supposed to steer packets for '234.34 to the proper router that knows how to reach the 234.0 network.

As a temporary workaround I've added the route to 192.168.235.2 which is that same host, but not in the nameserver, so it's annoying. (the other host that I can't reach due to this problem doesn't have a second IP address (yet)).

Oh... routing to 192.168.234.34 works on the router 192.168.235.4:
PING 192.168.234.34 (192.168.234.34) 56(84) bytes of data. 64 bytes from 192.168.234.34: icmp_req=1 ttl=64 time=41.5 ms

Anyway, what would you suggest for me to try to get that invalid route cache entry dropped?

Drop the default route? Ok. Done:

# route del default
# ping 192.168.234.34
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

I'm used to the default route being at the bottom, but deleting it should be enough to prevent it from being found first, right? :-)

Add a host route to this host explicitly naming the router?

assurancetourix:~# route add 192.168.234.34 gw driepoot
assurancetourix:~# route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
192.168.234.0   192.168.235.4   255.255.255.0   UG    0      0        0 eth0
192.168.234.34  192.168.235.4   255.255.255.255 UGH   0      0        0 eth0
192.168.235.0   0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.235.2   192.168.235.4   255.255.255.255 UGH   0      0        0 eth0
assurancetourix:~# ping 192.168.234.34 -c 2
....
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

Still the packets end up on the ethernet with the 192.168.235.251 router's Ethernet address.....

Quoted 7 lines
assurancetourix:~# route -nC | head -2 ; route -nC | grep 234.34
Kernel IP routing cache
Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.34  192.168.235.251       0      0        0 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        0 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        5 eth0

Better use "ip -s route list cache" to diagnose problems (more information)

Quoted 39 lines
# ifconfig eth0 down
# route -n 
<empty table> 
# ifconfig eth0 up
<old routing table is restored automatically??? apparently with the routing
cache entries as well....> 

I initially thought that this was a problem with the routing cache entry being too persistent in the kernel. While documenting this while writing this email, I've found that I can flush the whole routing cache with "ip route flush cache" .

However the routing cache entry springs back to life when I first ping the 234.34 host. Even when the problem machine doesn't have a default route, so it shouldn't know about the 235.251 default router.

This is getting weirder and weirder.

During all this I have

# tcpdump -nei eth0 net 192.168.234.0/24
running. If my machine were to get an ICMP redirect from somewhere
I'd see it, right? 

It could be that the 192.168.235.251 router is proxy-arping (incorreclty) for the problem hosts. But then my workstation would have to be ARPing in the first place.

# route add 192.168.234.200 eth0
# ping 192.168.234.200
gives: 
15:31:33.857343 00:23:54:15:1f:a9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.234.200 tell 192.168.235.8, length 28
in the TCPDUMP, so my machine is not arping for 192.168.234.34. 

Any suggestions? Any at all?

Roger.

#2 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 4 lines · Rogier Wolff

Hi,

My workstation has an incorrect route cache entry:

What kernel version ?

Linux version 3.0.0-12-generic (from Ubunutu oneiric.)

Quoted 8 lines
Quoted 8 lines

assurancetourix:~> route -nC | head -2 ; route -nC | grep 234.34 Kernel IP routing cache

Source          Destination     Gateway         Flags Metric Ref    Use Iface
192.168.235.8   192.168.234.34  192.168.235.251       0      0        3 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        4 eth0
192.168.235.8   192.168.234.34  192.168.235.251       0      0        2 eth0

(I don't know why there are three).

Today there are four.

Quoted 1 line

192.168.20.108 10.37.168.112 192.168.20.254 0 1 2 eth3

That indeed got me a full complement of route cache entries.

Quoted 2 lines

Better use "ip -s route list cache" to diagnose problems (more information)

After doing the tos ping you suggested All TOS levels have a route cache entry.

192.168.234.34 from 192.168.235.8 tos 0x1c via 192.168.235.251 dev eth0

    cache <redirected>  age 77sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9
192.168.234.34 tos 0x1c via 192.168.235.251 dev eth0  src 192.168.235.8 
    cache <redirected>  used 3 age 72sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9
192.168.234.34 from 192.168.235.8 tos 0x1c via 192.168.235.251 dev eth0 
    cache <redirected>  age 72sec ipid 0xaa09 rtt 47ms rttvar 15ms ssthresh 7 cwnd 9
Quoted 1 line
Quoted 1 line

Any suggestions? Any at all?

Last time, as well as this time, it is triggered by a network error that leads to the 192.168.235.4 router not being able to reach 192.168.234.34 or any other host on the 192.168.234.0/24 network.

During that time the VPN to 192.168.234.0/24 is down, so 192.168.235.4 doesn't have a route to 192.168.234.0/24 and it is logical that with that route gone, it sends packets for 192.168.234.0/24 to the default router 192.168.235.251. As it sees itself forwarding packets that come in on eth0 back to eth0, it will send a redirect. However that redirect should somehow expire, and not survive things like dropping the route to 192.168.234.0/24, dropping the default route, shutting down the interface or some time passing (that network problem was solved 20 hours ago)......

Roger.

Signature · 5 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
The plan was simple, like my brother-in-law Phil. But unlike
Phil, this plan just might work.