route cache flush??

Hello,

I have been beating my head against the wall for 2 days trying to figure why when I change a route and do a "ip route flush cache" it still takes up to a minute for packets to start using the new route.

Is there a step I am missing?

kernel is 2.6.32

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)

Nothing comes to mind, please share more information ?

Sure

I have a box "A" with 2 interfaces ips 2.2.2.1 and 3.3.3.1 going to another box "B" that has 3 nics 2.2.2.254 3.3.3.254 and 1.1.1.254 there is a third box "C" with ip 1.1.1.1. Box B is to simulate the net.

I have 2 vpns on box A going to box C. The default rt on A is 2.2.2.254. If I don't do anything else routing wise esp packet originated on 1.1.1.1 come 3.3.3.1 on box A but the response packet from 3.3.3.1 goes out the default route. This works OK in the lab but in the field the isps generally drop packet that have a source address that doesn't match their subnet.

So I have found on the net how to set up a simple rule to route packets with src address 3.3.3.1 back out that interface.

$ ip r s
2.2.2.0/24 dev eth1  proto kernel  scope link  src 2.2.2.1
3.3.3.0/24 dev eth2  proto kernel  scope link  src 3.3.3.1
10.0.128.0/17 dev eth0  proto kernel  scope link  src 10.0.133.22
default via 2.2.2.254 dev eth1
L703103:~
$ ip r s table second
default via 3.3.3.254 dev eth2  src 3.3.3.1
L703103:~
$ ip rule list
0:      from all lookup local
200:    from 3.3.3.1 lookup second
32766:  from all lookup main
32767:  from all lookup default

So I run a script to disable the src route an flush the cache.$ date;sudo /usr/local/pgsql/storeproc/programs/src_rt_off.sh Fri Feb 3 10:20:16 EST 2012 --- notice the time.
++ id -un
+ '[' root '!=' root ']'
+ /sbin/ip route delete default table second
+ /sbin/ip rule delete table second
+ /sbin/ip route flush cache
+ /sbin/ip rule list

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

This is a tcpdump of the interface with 3.3.3.1 ip address - I ran the script at 10:20:16 Notice how long before response quit going out this interface and switch to the default route. I see similar behavior when I reinstall the src route and flush the cache.

10:20:19.102448 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x24e), length 116 10:20:19.103221 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x24e), length 116 10:20:19.498523 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x24f), length 116 10:20:19.498701 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x24f), length 116 10:20:30.704175 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x250), length 116 10:20:30.704357 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x250), length 116 10:20:34.217349 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x251), length 116 10:20:34.218150 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x251), length 116 10:20:34.333011 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x252), length 116 10:20:34.333795 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x252), length 116 10:20:40.826996 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x253), length 116 10:20:40.827775 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x253), length 116 10:20:41.412308 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x254), length 116 10:20:41.413081 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x254), length 116 10:20:41.910528 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x255), length 116 10:20:41.910711 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x255), length 116 10:20:42.413504 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x256), length 116 10:20:42.414260 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x256), length 116 10:20:43.413755 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x257), length 116 10:20:43.414510 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x257), length 116 10:20:44.413807 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x258), length 116 10:20:44.414560 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x258), length 116 10:20:49.468466 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x259), length 116 10:20:49.469267 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x259), length 116 10:20:49.576539 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x25a), length 116 10:20:49.577318 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25a), length 116 10:20:53.116021 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25b), length 116 10:20:53.116196 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x25b), length 116 10:21:04.320972 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25c), length 116<<<<<<< 10:21:04.720342 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25d), length 116 10:21:04.808653 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25e), length 116 10:21:08.422372 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25f), length 116 10:21:09.423730 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x260), length 116 10:21:10.424971 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x261), length 116 10:21:11.425469 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x262), length 116 10:21:15.525572 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x263), length 116

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)