Skip to contentSkip to conversation

Netfilter meets PBR - I'm starting to tear my hairs out...

DISCUSSION Henning Schmiedehausen 2 participants 2 messages Aug 12, 2004 – Aug 14, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

I have a question concerning the interaction between iptables and iproute2. I seem to miss some crucial point in the concepts... Let me explain:

I have a router with four interfaces:

ppp0  - 	DSL uplink with dynamic IP address
eth0  - 	regular uplink with fixed IP address
eth1  -		network with official ip addresses
eth2  -		network with private addresses and official

addreses

The idea is now that everything that has an official ip address is routed through the eth0 link (the addresses are from the range of this ISP) and the private addresses are NATed and routed through ppp0.

for illustration:

ppp0: dynamic
eth0: 100.100.100.1/30
eth1: 100.100.101.1/24
eth2: 100.100.102.1/24

	192.168.8.1/24

This is a regular Fedora Core 1 box with all updates installed. It reports itself as

Linux router 2.4.22-1.2199.nptl #1 Wed Aug 4 12:21:48 EDT 2004 i686 i686 i386 GNU/Linux

I use iptables-1.2.9-1.0 and iproute-2.4.7-13.2 as delivered by the Fedora Project.

I did the following thing with /sbin/iproute

# /sbin/ip route show
<DSL Provider Address> dev ppp0  proto kernel  scope link  src <dynamic DSL address>
100.100.100.1/30 dev eth0  scope link
100.100.101.1/24 dev eth1  scope link
100.100.102.1/24 dev eth2  scope link

192.168.8.0/24 dev eth2 proto kernel scope link src 192.168.8.1 127.0.0.0/8 dev lo scope link
default via 100.100.100.2 dev eth0

Now I added PBR:

echo "200 VLAN2" >> /etc/iproute2/rt_tables

/sbin/ip route add default dev ppp0 table VLAN2
/sbin/ip rule add from 192.168.8.0/24 table VLAN2

which seems to work:

/sbin/ip route list table VLAN2
default dev ppp0 scope link

/sbin/ip rule list

0:      from all lookup local
32765:  from 192.168.8.0/24 lookup VLAN2
32766:  from all lookup main
32767:  from all lookup default

When I ping from a host attached to eth2:

# tcpdump -i eth0
100.100.102.2 > internet_host: icmp: echo request
internet_host > 100.100.102.2: icmp: echo reply
100.100.102.2 > internet_host: icmp: echo request
internet_host > 100.100.102.2: icmp: echo reply
# tcpdump -i ppp0
192.168.8.2 > internet_host: icmp: echo request
192.168.8.2 > internet_host: icmp: echo request

(obviously no answer yet)

Now I add masquerading:

/sbin/iptables -t nat -A POSTROUTING -o ppp0 -s 192.168.8.0/24 -j MASQUERADE /sbin/iptables -t nat -A POSTROUTING -j ACCEPT

After this, there is silence on ppp0. No more packets pass through the link.

Instead, the kernel starts to spit out:

Aug 12 12:19:36 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:41 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:41 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:46 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:46 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:51 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:51 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:56 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:56 router kernel: MASQUERADE: Route sent us somewhere else.

The fun thing now is, that if I flush the chain again and do

/sbin/iptables -t nat -A POSTROUTING -o ppp0 -s 192.168.8.0/24 -j SNAT --to-source=<dynamic DSL address> /sbin/iptables -t nat -A POSTROUTING -j ACCEPT

then the PBR works. I see the packets going out the ppp0 link:

# tcpdump -i ppp0
<dynamic dsl address> > internet_host: icmp: echo request
internet_host > <dynamic dsl address>: icmp: echo reply
<dynamic dsl address> > internet_host: icmp: echo request
internet_host > <dynamic dsl address>: icmp: echo reply

Unfortunately I cannot use this in production, because the link will have to go up and down and I cannot rewrite the configuration scripts (which use masquerade).

So, what am I doing wrong? As far as I can understand, the MASQUERADE target is equal to the SNAT target, except that it takes the ip address for NATing from the interface and tears down the connections if the interface loses its link. Or not? Or have I simply stumbled over a well known bug in the RedHat kernel (which seems to be 2.4.22 + lots of patches from post-2.4.22)? Or am I just missing some concept that I need to add to my tables?

I'd appreciate Cc'ing me as I am not a regular subscriber to this list.

Regards
Henning

Signature · 13 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/
 
RedHat Certified Engineer -- Jakarta Turbine Development  -- hero for hire
   Linux, Java, perl, Solaris -- Consulting, Training, Development

"Fighting for one's political stand is an honorable action, but re-
 fusing to acknowledge that there might be weaknesses in one's
 position - in order to identify them so that they can be remedied -
 is a large enough problem with the Open Source movement that it
 deserves to be on this list of the top five problems."
                       --Michelle Levesque, "Fundamental Issues with
                                    Open Source Software Development"
#1 lore raw
Quoted 1 line · Henning Schmiedehausen

Hi,

<snip..>

Quoted 13 lines

Instead, the kernel starts to spit out:

Aug 12 12:19:36 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:41 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:41 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:46 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:46 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:51 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:51 router kernel: MASQUERADE: Route sent us somewhere else.
Aug 12 12:19:56 router kernel: NET: 4 messages suppressed.
Aug 12 12:19:56 router kernel: MASQUERADE: Route sent us somewhere else.

This had been fixed by Patrick McHardy :
http://lists.netfilter.org/pipermail/netfilter-devel/2004-January/013695.html

Quoted 18 lines

The fun thing now is, that if I flush the chain again and do

/sbin/iptables -t nat -A POSTROUTING -o ppp0 -s 192.168.8.0/24 -j SNAT --to-source=<dynamic DSL address>
/sbin/iptables -t nat -A POSTROUTING -j ACCEPT

then the PBR works. I see the packets going out the ppp0 link:

# tcpdump -i ppp0
<dynamic dsl address> > internet_host: icmp: echo request
internet_host > <dynamic dsl address>: icmp: echo reply
<dynamic dsl address> > internet_host: icmp: echo request
internet_host > <dynamic dsl address>: icmp: echo reply

Unfortunately I cannot use this in production, because the link will have to go up and down and I cannot rewrite the configuration scripts (which use masquerade).

Quoted 7 lines

So, what am I doing wrong? As far as I can understand, the MASQUERADE target is equal to the SNAT target, except that it takes the ip address for NATing from the interface and tears down the connections if the interface loses its link. Or not? Or have I simply stumbled over a well known bug in the RedHat kernel (which seems to be 2.4.22 + lots of patches from post-2.4.22)? Or am I just missing some concept that I need to add to my tables?

Initially, MASQUERADE was designed to work with basic & simple network setup. Funky routing such as using iproute2 or the ROUTE target isn't simple networking.

Both are overwriting the already-made routing decision. When it's time to MASQUERADE, that last one consults the routing table (again). Then, it compares both output interface decisions and NF_DROP in case they aren't equal.

For routing that iproute2 can't do, consider using the ROUTE target. It rocks.

Quoted 7 lines

I'd appreciate Cc'ing me as I am not a regular subscriber to this list.

Regards
Henning

--

HTH,

Samuel Jean
CookingLinux.org